Monday, July 30, 2012

Loss Aversion: Paying Teachers More and Less in the Wrong Direction

You may have heard of a recent suggestion for enticing teachers to find ways to raise test scores: give teachers money at the beginning of the year, and then take it away if their scores don't improve appropriately. Its academic name is "Loss Aversion." The idea is that people may work harder when an undesirable behavior might cause them to lose something they already have than if a desirable behavior might earn them something in the future.

This scheme has come recently to the public debate over education following a recently published study conducted on Chicago teachers, which was subsequently reported on in The Atlantic and The Wall Street Journal.

Educators have understandably reacted negatively to the suggestion that districts might do well to consider having teachers give back money. Nobody wants to do that. But protecting teachers' financial well-being is hardly the most dire concern for the public to take issue with here.

It is not troublesome that researchers in the field of economics would construct such a study, nor should it worry us that the findings suggest that humans might be more strongly motivated when confronted with the loss of something they already have than when offered a reward for something later on. The research represents a valid question we'd naturally like to answer, and the findings may help us understand human behavior better.

What is extraordinarily worrisome, however, are the conclusions audiences are likely to draw given the media's unbalanced coverage in favor of, and omission of critical scrutiny regarding, efforts to tinker with teacher pay and motivation. I hope to make clear why I believe the media too often fails in its coverage of this issue by discussing three of those conclusions below.

Inaccurate Conclusion Number 1: Standardized tests so accurately represent student learning and teacher effectiveness that their pitfalls deserve to be discussed only rarely, if ever. 

The Atlantic, The Wall Street Journal, a host of corporate-reform-minded bloggers, and, disappointingly, the study's authors report the findings as if 1) standardized test scores were synonymous with student achievement, and 2) improvements in standardized test scores were synonymous with teacher quality. Neither suggestion is accurate. Even so, the study's abstract strongly implies that they are, despite that the authors stand on no ground to make such a claim (they're all economists, none of whom have expertise in the nuances of standardized testing) and that the pitfalls of standardized testing have been extraordinarily well-documented (recently and strikingly here).

A datum point becomes less and less useful the further from it inferences are made. To make assumptions about teacher effectiveness, or even more outlandish, school effectiveness, is academically dishonest given a metric designed only to diagnose student knowledge (many of which are so poorly constructed, we often can't honestly claim to infer much from them at all). When test scores increase, that is all that can be said - test scores increased. Unless you are intimately familiar with the test, curriculum, and instruction; it would be irresponsible to make any inferences beyond that. As a teacher, I find it depressing that stories like this are reported so frequently with no acknowledgement of the pitfalls of standardized testing. This misunderstanding is at the root of most of our failed education policies.

Inaccurate Conclusion Number 2: There are more benefits than costs to toying with teachers' compensation schemes.

By paying such attention to the Chicago study, the media implies that there are more benefits than costs to treating teachers like you train a dog, or a lazy teenager. Give them treats when they perform well and negative consequences when they don't do what you want.

Much of the dialogue around these policies assumes teachers have no intrinsic motivation to do well, or that extrinsic monetary motivation is the only kind that really makes a difference. Some research suggests that a person's efficacy at engaging in a cognitively complex task (like teaching) is not aided (and may be harmed) when rewards and punishments are added.

Teachers enter the profession to feel as if they have purpose, and they generally value respect and support over pay. Looking for ways to monetarily reward those who raise test scores abuses this reality in two ways: 1) it disrespects teachers (remember - the opposite of the external motivator they want) by implying they need monetary motivation for their work to improve (and suggests that, because they need that, their work is more akin to unskilled labor than a profession), and 2) it overlooks the external rewards good teachers actually value: professional collaboration and development, recognition, and appreciation.

Diane Ravitch suggests that if we're going to threaten to take away teachers money, why not go further? Why not threaten to cut off their fingers? And the blogger at Accountable Talk notes that thousands of teachers are already threatened with the loss of something much greater for failing to boost test scores: their jobs.

It seems to me that the notion that teachers ought to be treated like lazy teenagers comes from a good many people's previous experience with either a bad teacher or bad public employee. And, to be sure, those people exist in the teaching profession. I don't know how many of them there are, or if there are more now than ten years ago, but I do know that I've only worked alongside a handful in the five schools in which I've taught, and proceeding with efforts focused on toying with teachers' pay as a means of improving instruction will leave us with more of those types than less. Those teachers who are excellent, intelligent, and show great promise are among the many currently leaving in droves. They feel disrespected, hampered, and as if they have little control over a sketchy metric that will ultimately decide their pay, and possibly their career. If tying rewards (that teachers don't want) and punishments to indicators that force teachers to abandon creativity and autonomy relieve us of the absolute worst, they will also purge the good and the best.

Inaccurate Conclusion Number 3: Improving Teacher Quality is the Same as Improving Public Education.

When the media writes about the need to improve teacher quality as if it were synonymous with improving public education generally, it corrodes the public's understanding of the factors that go into creating an excellent educational system.

Even if we are to believe that toying with compensation schemes might improve instruction, our failure to provide educational equity (i.e. the existence of the "achievement gap") is dependent on far more than quality of instruction. The media's love affair with improving teacher quality as a means of improving public education amounts to focusing on a few hundred gallons in a problem the size of the Pacific - or obsessing over a few square inches of art on the mural in the Sistine Chapel.

Listening to the media and economists salivate over new ideas that might force teachers to work harder makes the word "curriculum" seem quaint and esoteric, as if it plays only an infinitesimally minor, and most certainly boring, role in improving education. It ignores (and contributes to) the reason the teaching profession earned a reputation for having fewer extremely bright and motivated professionals in the first place: our culture's general lack of respect for the job. It denies that class size, experienced staff, and access to up-to-date technology matter. And it allows us to remain blissfully unaware of (or at least inattentive toward) some of our society's most insidious and noxious troubles.

The ultimate problem with our educational system is not that we don't have great educators, or that we don't know how to "do education" well. It is that our system is catastrophically inequitable. Our richest students generally perform outstandingly in state and international tests, graduate high school and are accepted to college at high rates, and go on to earn lucrative jobs. The opposite is too often true for poor and minority students.

The United States bears the shame of having the highest rate of child poverty in the industrialized world and sending more black men to prison than to college. Our schools and communities are largely segregated by race and socio-economic standing. We have an increasing rate of income inequality. And we incarcerate a larger percentage of our population than any country in the world. These realities affect student achievement.

Despite our problems, I will most certainly grant that, save a complete cultural course alteration in the next five to ten years, we are still tasked with improving educational outcomes in the country we have, rather than the country we wish we had. But even that will remain a pipe dream if we're unable to move off our fixation for berating teachers until students give us the scores we want and engage in a dialogue that is indicative of a more nuanced and holistic understanding of what public education is, what it's for, and the factors that make it work.

I'm nearly done, but I don't want to leave you wanting for a counter solution. Were I asked to test my own aversion for loss, I'd suggest that instead of offering me the $4,000-$8,000 researchers gave teachers in the Chicago study, they use that money to buy my classroom the textbooks and basic materials we don't already have. Then, upon the study's completion, they could take the materials back, whether students scores improved or not! Better, they could feel free to do the same the following year, and the following, and the following...

11 comments:

  1. omg, that's such an awful idea. just when you think it can't get any worse . . .

    ReplyDelete
    Replies
    1. Exactly. It makes you wonder what's next.

      Delete
  2. One of the oddest things about this level of nastiness is that the student I feel that I made the least amount of progress with showed progress on her high stakes tests this year.

    How come increased test scores have not cured "Destiny?"

    “You never believe me when I am lying?” 4th grade Destiny complains through the thumb she got stuck in her mouth. I just look at her. “Sometimes I am not lying,” she declares. I resist asking when exactly she is not lying, and talk about the hard work of building trust. Destiny had over 90 incident reports in less than two years for disruptive, violent, or threatening behavior, but her mother only consented to have Destiny evaluated at the end of last year. One time she was suspended for listing the classmates that she wished to kill, and the ways she planned to kill them. She has vandalized the bathroom with horribly obscene drawing. Parents have threatened to pull their kids out of school because Destiny has caused so many disruptions to her class. This is the first year that Destiny has received the intense level of support she needs, and she has made some progress academically with unsustainable level of support. An instructional assistant or I have to work with her individually to do anything that remotely resembles her work. That has taken away my prep, his breaks, and most of my lunch, because there are other students who still need us. (Her mother refused consent for evaluation until the end of last year, and only decided to sign consent for services on the last day of school last year). Despite multiple interventions and counseling, she has made little progress in accepting responsibility for her behavior, and making better choices. She still can’t be trusted to be respectful to a substitute, attend a field trip, or participate in a group activity without trying to pick a fight or make another child cry, or stay in her regular classroom without making multiple disruptions. All of her antisocial behavior is always someone else’s fault, but hey, her MAP and state test scores went up.

    Sorrel

    ReplyDelete
    Replies
    1. Great point, Sorrel - just because test scores go up doesn't mean students are moving in the direction we want them to. At least it's a "success" you can try to build on with her, though.

      Delete
  3. Here's another story that I would like to share....

    The MAP scores indicate that Apollo (not his real name) has not met progress goals. A letter to his parents will tell them that Apollo is in the 1 percentile for his age group, and that he has not made progress. Apollo has autism and an IQ below 70. The MAP computer program said that Apollo has not made progress. The important people in his life are awestruck at the phenomenal progress he has made in the 3 years since kindergarten. Apollo was a runner. The only way to keep him safe was to sit behind him or hold his hand. He scream for 45 minutes straight. He threw counting bears, blocks, and crayons, and so forth instead of using them. He bit and spit. He spoke in echolia or two words phrases. He’d stand swinging his arm and make roaring sounds when not occupied. He stared at the ceiling instead of looking at his work. He had to be taught through hand-over-hand method. Beyond playing chase on the playground, he never played with toys or with other children. He needed the attention of an aide the entire time he was at school in order to be safe and productive. That was then.

    Now he will give you the brightest smile you have ever seen, and address you by name even if he only met you once before. He loves to run errands, and will do them exactly as directed. He has known the names of every adult in the school since kindergarten. He has learned to invite peers to play games, and has offered to teach them how if they don’t know the games. He speaks in complete sentences, and will ask how people are and where they have been, and listen to the answers. He will tell about events in his life, such as when his sister was in the hospital or his father was going on a trip without prompting. Honestly, his academic skills have remained low. He is still working below grade level, and struggles with retention and recall, but he loves reading Go Dog Go and Cat in the Hat. He’s beginning to answer questions about familiar text. He can be productive for increasingly longer stretches without the constant attention of the aide. I could go on and one. What does a computer program designed by strangers know about the progress Apollo has made? Nothing!


    A computer spews out numbers that indicate that this child has not made progress. Apollo has taken the MAP six times by now, and has clicked through it each and every time. The usual six minutes he takes to finish yield consistently low scores. He does not connect that he is rushing, and does not understand what the scores mean. I’ve tried having him practice with other computerized quizzes, such as Tumblebooks quizzes and Moby Math, but he gets bored and restless. He does not have the patience or the understanding to read through answer choices A through E, and frankly nor do I. He doesn’t know that his scores may one day affect my evaluations or pay.

    --Sorrel

    ReplyDelete
    Replies
    1. Thanks, again. This story really beautifully illustrates how little test scores often mean. We should flood people who remain unconvinced with them.

      Delete
  4. I have a bunch of stories, because I have been recording my memories over this past year.

    After huge amounts of interventions and lots of data collected to prove the interventions were not working for Destiny, we were finally able to get her into a program for students with emotional and behavior disorders. I think her new teacher will be the experience and expertise to help.

    ReplyDelete
  5. This is a fantastic piece. It is hard to even begin a discussion when the phrases "teacher quality" and "student achievement" are used as if they are synonyms for higher test scores.

    What most of us want is the conditions that allow us to do our jobs well: Necessary supplies, mold-free buildings, supportive, non-abusive administrators, and minimal time lost to test prep, testing, makeup testing, babysitting other teachers' classes while they are testing, etc.

    Instead we get, "our self-commissioned research tells us that teacher quality is the most important factor in student achievement." See above for the definitions of those two phrases. And look, we financed another movie about adorable kids cheated of their education by teachers who don't care as much about kids as profit-motivated non-teachers.

    ReplyDelete
    Replies
    1. Absolutely! Also see what Barnett Berry has to say about the study below:

      http://transformed.teachingquality.org/blogs/advancing-teaching-profession/07-2012/economists-return-your-salaries-producing-flawed-studies

      Delete
  6. Of all the blah-blah written on loss aversion (just typing that phrase makes my gorge rise)in what should be professional practice, this is by far the best and most comprehensive analysis of why letting economists tinker with educational "metrics" is flat-out wrong. Thanks.

    ReplyDelete
    Replies
    1. Thanks, Nancy. It's always nice to hear you approve ;)

      Delete