Remorse Code (Or Comments From A Crib Strangler)

Those who publicly advocate for the kind of education policies put forth in "Waiting for Superman" are now seeing the equivalent of a letter-high fastball down the middle. They can wait on it and crank it out, using the buzz created by a major motion picture to advance the movie’s/campaign’s arguments at face value. I'm a little late on this (at least by blog standards), but over at Fordham’s Flypaper blog, Mike Petrilli saw this fastball, yet instead of unloading, he sacrifice bunted the runner into scoring position for the good of the team.

Responding to an interview in which Davis Guggenheim, the film’s director, claims that charter schools have "cracked the code" on how to educate even the poorest kids, Petrilli warns against the hubris of thinking that we are anywhere beyond first steps when it comes to fixing urban schools. He points out that charters like KIPP benefit from selection effects (more motivated and informed parents seek out the schools), and that the degree to which these schools have actually "closed the gap" between poor and affluent schools has been somewhat oversold. Petrilli also notes that while some of these schools seem to have "cracked the code," there is still little idea of how to expand them to serve more than a tiny minority of poor kids.

Thoughtful comments like these should remind those of us who care about expanding quality education that, although we may have canyon-sized differences between us on what needs to be done (Petrilli claims that those who disagree with him are trying to "strangle" reforms "in their crib"), there may be a few important respects in which we are closer than we may appear. Still (in addition to the crib-strangling allegations), I would take issue with one of Petrilli’s central points – that charters like KIPP may have "cracked the code," and the main problem now is how to scale them up. From my perspective, the "code" is specific policies and practices that produce results. And on this front, we’re practically still using decoder rings from cereal boxes.

In large part, it is difficult to identify effective practices because it is difficult to isolate their effects. We can’t randomly assign policies to schools the way we sometimes can students (through lotteries), and separating out each policy’s effects from those of other school characteristics will always produce results that require cautious interpretations. For example, schools need more revenue to implement extra programs or pay employees to work more hours, and so revenue may appear to affect results when what really matters is how it is used. In addition, in many cases, school policies either defy easy measurement (e.g., rules surrounding student behavior), or don’t vary enough to analyze their effect (e.g., the number of days in the school year).

As a result, there are relatively few high-quality analyses of how specific charter school practices affect achievement or other outcomes (the recent report that convincingly showed that KIPP is effective also gave no real idea of why), and those studies that exist haven’t produced nearly enough in the way of concrete conclusions.

The examination of school characteristics in the Mathematica charter school study, for instance, produced only weak results in math (school size and achievement level tracking seemed to matter), and virtually none in reading (see this post for more discussion). Another analysis of New York City charters looked at the multivariate association between 30 charter school characteristics and school performance, and only three were found to be statistically significant at the conventional (95 percent confidence) level – more days in the year, a parent on the board, and a mission statement emphasizing achievement. (For my part, I have argued that among the most compelling reasons why high-profile charters like KIPP are effective is the fact that most of them have the students for up 60 percent more time.)

In other words, the important question is not which schools work, but why – finding out what works, and using it in all schools. That’s the code. And "cracking" it is a work in progress. People like Davis Guggenheim, who suggest that we know exactly what works and all that remains is the will to do it, are doing a great disservice. They are producing unrealistic expectations based on incredibly simplistic and misinformed premises.

So, when folks like Mike Petrilli, who probably (I don’t want to speak for him) agrees with much of Guggenheim’s prescription for fixing the public education system, nonetheless calls him out for vastly oversimplifying the issues and the proposed solutions, they deserve some extra credit.