Select Your Conclusions, Apply Data

The recent release of the National Assessment of Educational Progress (NAEP) and the companion Trial Urban District Assessment (TUDA) was predictably exploited by advocates to argue for their policy preferences. This is a blatant misuse of the data for many reasons that I have discussed here many times before, and I will not repeat them.

I do, however, want to very quickly illustrate the emptiness of this pseudo-empirical approach – finding cross-sectional cohort increases in states/districts that have recently acted policies you support, and then using the increases as evidence that the policies “work." For example, the recent TUDA results for the District of Columbia Public Schools (DCPS), where scores increased in all four grade/subject combinations, were immediately seized upon supporters of the reforms that have been enacted by DCPS as clear-cut evidence of the policy triumph. The celebrators included the usual advocates, but also DCPS Chancellor Kaya Henderson and the U.S. Secretary of Education Arne Duncan (there was even a brief mention by President Obama in his State of The Union speech).

My immediate reaction to this bad evidence was simple (though perhaps slightly juvenile) – find a district that had similar results under a different policy environment. It was, as usual, pretty easy: Los Angeles Unified School District (LAUSD).

Like DCPS, LAUSD also saw increases in all four grade/subject combinations of TUDA (though one of them was not statistically significant). Moreover, the increases were large – 4-5 scale score points.

So, by the flawed logic of those who trumpet TUDA increases as policy evidence, LAUSD is clearly doing something correctly. One could argue, therefore, that not implementing new teacher evaluations, while maintaining traditional salary schedules, tenure, and a strong local teachers’ union, are the keys to success.

(Or, alternatively, we might take the opposite approach: New York City has been very active on the reform front for several years, but there was no significant increase in any grade/subject combination.)

The problem with this kind of bad evidence is that you can use it to prove almost anything.

Now, don’t get me wrong: The increases in LAUSD and DCPS are good news. And it is very possibly the case that the increases reflect, at least to some degree, “real” progress, rather than differences, measurable and unmeasurable, in the sample of students taking the tests, and that the policies in these districts played a role in that. But they are not policy evidence. Period.

And I think that Chancellor Henderson and Secretary Duncan, both of whom are incredibly intelligent people who know far more about education than I ever will, are aware of this. I understand that their positions are hyper-political, that they are under enormous pressure to produce immediate results (or at least the appearance of immediate results), and that those who oppose their policies have shown themselves more than willing to do the same thing when they can.

But, as I’ve said before, there’s a very large group of us out here who are willing to applaud any high-level leader who refuses to misuse evidence, whether or not we happen to agree with their substantive policy positions. I’m sure there are leaders like that out there, and I wish they were more visible.

- Matt Di Carlo