On Data, Part Six: Data-Driven Disruption
Over the past six days, I've aired my thoughts on data use in public schools. I hope I've made it clear that I think quality data deserves strong consideration when schools make decisions about instruction. I hope I've also been clear, however, that data-informed decisions are far superior to data-driven decisions, particularly when we've clearly overshot creating quality data in the name of promoting its use no holds barred. Today, in my final post in this series, I will explore the reasons behind the cheating scandals we've recently seen in school districts across the country and draw two last comparisons between Wall Street and public education.
When you read about the pitfalls of standardized testing today, it's difficult to avoid coming across reference to Campbell's law.
The first time I read about Campbell's law was in Diane Ravitch's book, The Death and Life of the Great American School System. Anti-testing reformers allude to it often because it so aptly captures the problems inherent in NCLB's policy of using standardized test scores to evaluate school quality and punishing those schools that don't meet adequate yearly progress (AYP).
According to Wikipedia, Campbell's law says:
Earlier in this series on data I made a comparison between Wall Street and public education, but there are at least two other pertinent similarities that I think must be mentioned.
In the clip from Newshour, there is some discussion around who's to blame for the cheating we've seen afflict cities from Baltimore to Los Angeles to DC to Atlanta. Are the teachers and administrators the villains, or is this evidence of a more systematic problem? (Erich Martel, a DC social studies teacher, wrote convincingly on the topic here.)
In evaluating the causes of the global financial crisis of 2008, many observers appropriately note the disturbing decoupling of risk from reward for many of the world's biggest financiers (investors got all the risk and bankers got all the reward), resulting in what's known as moral hazard. One of the primary causes of this decoupling was investment banks' decision to transition from privately-held to publicly-owned companies. When banks sold their stock to investors, they effectively transferred the long-term risk of short-term decision-making from the banks' partners to the firm's investors. Knowing they would not share in the punishment if and when it came, traders' incentive to mitigate long-term risk dissipated, and before long, long-term risk was exacerbated in the name of gargantuan short-term gains.
I see a similar situation occurring in many of our high-profile urban school districts. Eschewing what should be long-term educational values (e.g. civic engagement, critical thinking, effective communication, a sense of shared responsibility), districts - and the politicians held responsible for them - see more incentive to maximize short-term testing gains in the interest of avoiding the punishments promised by NCLB for "failing" schools. (There are also the incentives of media prominence and near celebrity status afforded superintendents who appear to turn failing districts around.) Public education finds itself focusing on short-term gains because it's sold its stock (i.e. the right to make its decisions) not to the public, but to those who'd have you believe that standardized tests are appropriately used as the sole indicators of school quality.
On Wall Street and in public education, the new incentive structure is unquestionably biased toward serving the short-term. The reward for the Wall Street banker is vast sums of money. The reward for the urban teacher is the right to keep his or her job. (Read the fascinating story of a teacher who anonymously admits to cheating in the Philadelphia Public Schools Notebook.) Unless your view of human nature includes the widespread capacity for people to either deny $32 million bonuses in the name of fiscal responsibility or risk losing the ability to support one's family in the name of truth, we must accept that both of these massive problems are systematic. They're inherent to the incentive structures. While it may be tempting to vilify both bankers for ruining the lives of millions and teachers for failing their students, it should not surprise anyone if our myopic efforts to prosecute individuals while ignoring the systems that bore them don't produce results.
Where Wall Street and public education diverge, however, is in their respective systems' abilities to ultimately bring some sort of reckoning for abusing them. Money is a vastly better indicator of financial success than standardized test scores are of school quality. Additionally, when banks spend money they can't afford, especially when that money is borrowed, there sooner or later comes a reckoning that need not be initiated by an independent investigation. The reckoning is inherent to the system.
No such settling of accounts is inherent to the system of standardized testing. There will be no global knowledge meltdown if our standardized tests widely misrepresent school quality, at least not as far as we will be able to judge. There is no trigger that will, years from now, demand its points back when student learning is overvalued by cheaters. When students are deprived a quality education, the effects may often be just as severe (if not more) as an economic depression on those students' communities, but the media's privileged perception of those communities often focuses on other causes of their hazardous condition.
There will be no proper accounting for miseducation's future impact on society. We will not be able to place a neat number on it, like 12 trillion. But when we create a school system that sees its end goal in boosting annual numbers that indicate little about school quality, and then insists on disrupting education in schools that fail to meet them, it should give us cause to wonder what's really being driven by the data.
When you read about the pitfalls of standardized testing today, it's difficult to avoid coming across reference to Campbell's law.
The first time I read about Campbell's law was in Diane Ravitch's book, The Death and Life of the Great American School System. Anti-testing reformers allude to it often because it so aptly captures the problems inherent in NCLB's policy of using standardized test scores to evaluate school quality and punishing those schools that don't meet adequate yearly progress (AYP).
According to Wikipedia, Campbell's law says:
"The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor."Enter the Atlanta Public Schools cheating scandal, reported on yesterday by PBS's Newshour:
Watch the full episode. See more PBS NewsHour.
Earlier in this series on data I made a comparison between Wall Street and public education, but there are at least two other pertinent similarities that I think must be mentioned.
In the clip from Newshour, there is some discussion around who's to blame for the cheating we've seen afflict cities from Baltimore to Los Angeles to DC to Atlanta. Are the teachers and administrators the villains, or is this evidence of a more systematic problem? (Erich Martel, a DC social studies teacher, wrote convincingly on the topic here.)
In evaluating the causes of the global financial crisis of 2008, many observers appropriately note the disturbing decoupling of risk from reward for many of the world's biggest financiers (investors got all the risk and bankers got all the reward), resulting in what's known as moral hazard. One of the primary causes of this decoupling was investment banks' decision to transition from privately-held to publicly-owned companies. When banks sold their stock to investors, they effectively transferred the long-term risk of short-term decision-making from the banks' partners to the firm's investors. Knowing they would not share in the punishment if and when it came, traders' incentive to mitigate long-term risk dissipated, and before long, long-term risk was exacerbated in the name of gargantuan short-term gains.
I see a similar situation occurring in many of our high-profile urban school districts. Eschewing what should be long-term educational values (e.g. civic engagement, critical thinking, effective communication, a sense of shared responsibility), districts - and the politicians held responsible for them - see more incentive to maximize short-term testing gains in the interest of avoiding the punishments promised by NCLB for "failing" schools. (There are also the incentives of media prominence and near celebrity status afforded superintendents who appear to turn failing districts around.) Public education finds itself focusing on short-term gains because it's sold its stock (i.e. the right to make its decisions) not to the public, but to those who'd have you believe that standardized tests are appropriately used as the sole indicators of school quality.
On Wall Street and in public education, the new incentive structure is unquestionably biased toward serving the short-term. The reward for the Wall Street banker is vast sums of money. The reward for the urban teacher is the right to keep his or her job. (Read the fascinating story of a teacher who anonymously admits to cheating in the Philadelphia Public Schools Notebook.) Unless your view of human nature includes the widespread capacity for people to either deny $32 million bonuses in the name of fiscal responsibility or risk losing the ability to support one's family in the name of truth, we must accept that both of these massive problems are systematic. They're inherent to the incentive structures. While it may be tempting to vilify both bankers for ruining the lives of millions and teachers for failing their students, it should not surprise anyone if our myopic efforts to prosecute individuals while ignoring the systems that bore them don't produce results.
Where Wall Street and public education diverge, however, is in their respective systems' abilities to ultimately bring some sort of reckoning for abusing them. Money is a vastly better indicator of financial success than standardized test scores are of school quality. Additionally, when banks spend money they can't afford, especially when that money is borrowed, there sooner or later comes a reckoning that need not be initiated by an independent investigation. The reckoning is inherent to the system.
No such settling of accounts is inherent to the system of standardized testing. There will be no global knowledge meltdown if our standardized tests widely misrepresent school quality, at least not as far as we will be able to judge. There is no trigger that will, years from now, demand its points back when student learning is overvalued by cheaters. When students are deprived a quality education, the effects may often be just as severe (if not more) as an economic depression on those students' communities, but the media's privileged perception of those communities often focuses on other causes of their hazardous condition.
There will be no proper accounting for miseducation's future impact on society. We will not be able to place a neat number on it, like 12 trillion. But when we create a school system that sees its end goal in boosting annual numbers that indicate little about school quality, and then insists on disrupting education in schools that fail to meet them, it should give us cause to wonder what's really being driven by the data.
Great piece... but AYP is adequate yearly progress (although I admit that annual yearly progress is certainly redundant enough to come from DOE)
ReplyDeleteAck - duh! Whoops. Thanks for catching that mistake.
ReplyDeleteThank you for an excellent piece. It mirrors what our VIVA teachers said in their national report delivered last year to Arne Duncan. If everything rides on just one test, it's no wonder people try to game the system. But the cheating scandals are not a reason to abandon data. Rather, they are the reason to expand the sources of data to not only make it harder to cheat but also to give a truer picture of achievement. You can read the national and New York state reports at www.vivateachers.org.
ReplyDeleteNot abandon data, but certainly reconsider the way data is being used, as well as the quality of the data we're creating and the assumptions we hold about what it indicates.
ReplyDeleteSuperb series. As good and lucid an explanation as I've ever read on uses and misuses of data.
ReplyDeleteLooking for short-term gains at the expense of genuine investment in desired goals is a uniquely American trait. Call it the Wal-Martization of the culture.
Brilliant work, James. Thanks.