Waiting For Methodology Man

I finally saw the entire “Waiting for Superman” movie last weekend, in a mostly-empty Georgetown theatre. I went with my mother, not just because she’s a great public school teacher, but also because I needed someone to comfort me while I watched.

We both had strong reactions to dozens of things about the film, and you almost have to admire the chutzpah. It is about education – with a primary focus on teachers – and includes sit-down interviews with superintendents, parents, students, businessmen, economists, and journalists, but not one teacher.

Given all the attention that has already been lavished on it, I’ll discuss just one other thing that struck me, one which I keep hearing elsewhere.  There is exactly one sentence in the whole film in which director Davis Guggenheim addresses the research on charter school effects beyond the anecdotal evidence that dominates his narrative.  He notes, “Only one in five charters is excellent," with the implication that these charters show that it can be done.

He is presumably referring to the CREDO study released last year, which is the largest (15 states plus D.C.) and arguably the most overplayed charter analysis in history (for other multi-state studies showing no charter effects, see here, here, here, and here).  The CREDO authors understandably framed their results in a “media-friendly” manner – by reporting the percentage of charters that did better than comparable regular public schools (17 percent), along with the proportion that did worse (37 percent).

My first point is that 17 percent is equivalent to one in six, not one in five. But beyond that, some charter advocates have taken the remarkable step of turning the finding that twice as many charters do worse than regular publics into “evidence” that the former should be expanded.  The rationale is, as Guggenheim puts it, that these “one in five” charters are “excellent," and if we can increase that proportion, we can fix our public education system. There is only one problem: That’s not what the study says. Guggenheim is either deliberately misleading his viewers or, more likely, just doesn’t know what he’s talking about.

Let’s look at the facts. The CREDO analysis compares a large sample of charter school students with hypothetical “virtual twins” (composites of similar students’ achievement) in the regular public schools classified as their “feeder schools” (schools which the charter students might have attended, sometimes just a few and other times schools throughout a whole district).  In simple terms, using regression models (with controls), the test score gains of each charter student are compared with those of their virtual twins in regular publics, and then these comparisons are “averaged” across students in each charter school. The result: only 17 percent of charter schools (actually, students in charters) produce average test score gains that are better than those of their regular public feeder schools by a statistically significant margin.

What does this finding mean? Statistical significance means only that the difference is non-zero, not that it is large. Although the CREDO study does not report a breakdown of effect sizes by "better/worse" classification, it is virtually certain that most of these 17 percent produce gains that are only slightly larger than those of comparison schools, while a small minority produce comparative gains that are much higher (like the schools featured in the movie).  Moreover, if the results are viewed the way they usually are in these analyses – from a pooled sample of all schools – the “charter effect” is, on the whole, negative and statistically significant. That is – charter schools actually decrease aggregate student performance.

Either way, “one in five” charters is not found to be “excellent." One in six is better at boosting scores than the schools to which they are compared. Most of these differences are probably very small. If I had to guess (and this is speculation), I would say that about 3-5 percent of charters are “excellent” by some reasonable standard (defined entirely in terms of score gains), while the remaining 12-14 percent are either moderately better or borderline identical (significant, but tiny, differences).

And the same interpretation goes for the regular public schools. Roughly one in three sets of feeders are found to be better (at boosting scores) than the charters to which they are compared. By Guggenheim’s logic, assuming that regular public schools follow a similar distribution, this means that one in three regular public schools is “excellent." Yet, by some incredible leap (of tall buildings in a single bound?), the interpretations of CREDO among some pro-charter folks seem to preclude the possibility that regular public schools can also be good. This is nonsense. 

Had Guggenheim (and others who spread this myth) been careful in the use of research, he might still have made a valid point that a small handful of both charters and regular public schools do appear to be very effective in raising test scores – and then carried out an honest investigation into why this is so and how we can expand upon it. But that particular exercise would have threatened the one-dimensional simplicity of Guggenheim’s central message.  It would have also given me and my mother less to talk about over dinner.