L.A. Times: Lies lies and more lies

On Sunday, the L.A. Times began publishing an important series of articles on teacher evaluations (Who’s Teaching L.A.’s Kids?). Important because the Los Angeles Unified School District is the nation’s second largest (Chicago is third); important because it appears in a major newspaper; important because they printed teachers names with the data, upping the ante in attacks on teachers (Duncan came out in support of publishing teacher evaluations); important because it previews what I suspect will be the same kind of arguments that will be used by Huberman’s administration against Chicago teachers.

I am late to the party on this -- I first saw a note of it on the District 299 blog (required reading to keep up with CPS news), then on the Answer Sheet blog, and the L.A. Times piece story showed up on NPR yesterday. I’m slow (actually, ironically I suppose, or sadly, I have been mushing and sorting students by their ISAT scores for an upcoming area observation).

The Who’s Teaching L.A.’s Kids? article looks at local student standardized test scores on a by-teacher basis, using a "value-added" statistical model, and based on that model, identifies teachers as "effective and "good" teachers versus "ineffective" and “bad" teachers.

On reading the article, I was reminded of Mary McCarthy’s famous quip about Lillian Hellman (and unfair I think), "every word she writes is a lie, including 'and' and 'the'." Every word in the L.A. Times article is a lie, including "and" and "the". In this case, not unfair.

The article doesn’t have to be "true" of course -- we are in a propaganda war, after all. The tactic of the Times authors is to dress up some statistics bullshit in a pretty hat, and parade it around as science, ergo truth. Everyone is so wowed by the hat, that they fail to recognize that underneath the hat, it’s just, well, just bullshit. But if enough scientistic magic power words are folded into the story, words like "Rand Corp.", "senior economist and researcher", "reliable data", "objective assessment", "effective", the narrative sweeps along and reaches it’s obvious, stinking conclusion.

Here is the essence of the Times’s method:

The Times used a statistical approach known as value-added analysis, which rates teachers based on their students' progress on standardized tests from year to year. Each student's performance is compared with his or her own in past years, which largely controls for outside influences often blamed for academic failure: poverty, prior learning and other factors. Though controversial among teachers and others, the method has been increasingly embraced by education leaders and policymakers across the country, including the Obama administration.
...
The approach, pioneered by economists in the 1970s, has only recently gained traction in education.
...
Value-added analysis offers a rigorous approach. In essence, a student's past performance on tests is used to project his or her future results. The difference between the prediction and the student's actual performance after a year is the "value" that the teacher added or subtracted.

There have been a number of challenges raised to value-added measures on methodological grounds (in particular, misidentification of “bad teachers -- see Study: Error rates high when student test scores used to evaluate teachers from the Answer Sheet blog. Also from the same blog (which is Really Good by the way), Willingham: Big questions about the LA Times teachers project. I have started a list of links here.)

But I think there is a more fundamental, worldview-type fault with the general approach demonstrated in the LA Times article, the Big Lie that makes everything in the article a lie, even "and" and "the". It’s not merely the concept of "value-added" as a metric, but the overall economic approach to education.

Through the lens of economics, teachers go to the education factory. They work on human widgets. At the end of the day, teachers have hopefully added value to the widgets. Value is added if the widgets score higher on multiple choice tests. The greater the change in test scores, the more value that has been added, and the more productive the teacher is.

Implicit in the economic argument is that the education factory must strive to be as productive as possible (i.e. raise test scores as much as possible). Teachers have a greater effect on students than any other single factor, so education reform should focus of identifying the most productive teachers. School districts must then devise incentives to keep the most productive teachers (hence merit pay). Or researchers need to determine what makes for a productive teacher (e.g., Building a Better Teacher, from the New York Times Magazine last March), and teach that in teacher education programs (hence let Teach for America certify its own teachers), and/or suss that out in teacher recruitment or on the firing line in the first couple of years of teaching (see Malcolm Gladwell’s piece "Most Likely to Succeed: How do we hire when we can’t tell who’s right for the job?").

Most all of the research on this approach -- what gives this approach its academic patina of respectability -- points back to the work of Eric Hanushek, an economist at the Stanford's Hoover Institution. He has been working on quantifying the effect of individual teachers, and trying to isolate the teacher effect in education since the early 1970s, and continues to work on it today. His work, and the work of people around him, is the academic foundation, the theory on which most of the official education rhetoric, from Obama to Duncan to Huberman (from what I can tell anyway), is based.

This economic model is taken a step further with the "value-added" notion. I’m not sure where the concept arose but it is an obvious extension of Hanushek’s work. CPS uses a version from the University of Wisconsin’s Value Added Research Center, as does New York City, Milwaukee and Dallas. The LA Times study referred to above was done by researchers at the Rand Corp. A company called Mathematica Policy Research, Inc. was noted in the Answer Blog as the contractor for the Washington, DC teacher evaluation system, which uses a value-added component (and used in the firing of teachers there recently.)

As noted above, "value-added" can be calculated in different ways, but all approaches are based on standardized test scores. As is the case for Hanushek’s work. Here is Hanushek's (and collaborator Steven Rifkin's) statistical justification for saying standardized test scores mean something:

One fundamental question––do these tests measure skills that are important or valuable? –– appears well answered, as research demonstrates that standardized test scores relate closely to school attainment, earnings, and aggregate economic outcomes (Murnane, Willett,and Levy 1995; Hanushek and Woessmann 2008). The one caveat is that this body of research is based on low-stakes tests that do not affect teachers or schools. The link between test scores and high-stakes tests might be weaker if such tests lead to more narrow teaching, more cheating, and so on. (from Hanushek and Rivkin’s Using Value-Added Measures of Teacher Quality, p. 2; emphasis added)

The economic view of education, as the above indicates, assumes the goal in life is earnings and/or academic attainment. If that assumption and mindset is rejected, then the rationale of standardized tests having any meaning evaporates, and the whole argument collapses.

jd

P.S. I skipped over their important caveat re: that the justifying research assumes that tests are low-stakes, which is not the case today with ISAT, ACT, Scantron, etc. The current cheating controversy in Atlanta speaks to the greater incentive to cheat as stakes get higher. In a more perfect world, tests would be part of a bigger assessment profile, and then they might mean something. In the words of Hanushek and Rivkin themselves, the standardized test score data is suspect.