Looking at Chubb and Moe's Numbers: "Slight of Hand Logic"

Political scientists John Chubb and Terry Moe, the authors of Politics, Markets and America's Schools (Washington DC: Brookings Institution, 1990) think that they know how to fix public education. They say there are two ways to organize schools: democratic control, which is how public schools are now run, and market control, which is the way private schools operate. They also tell us it's easier to have effective school organization with market control and that effective school organization is a leading cause of superior student achievement. So, it follows that abandoning democratically controlled schools and going to market control would lift student achievement out of the cellar. Chubb and Moe call their school plan "public school choice"; it is actually privatization at public expense.

Chubb and Moe complain that their critics are merely mouthing opinions, while they are talking facts; they claim they've proved their case in the statistical analysis that makes up half their book. They're right to complain, but not only about their critics. Few lay readers are equipped to negotiate their mountains of data or their fancy regression analysis. Now, however, John Witte, a professor of political science at the University of Wisconsin-Madison, and neither a friend nor a foe of market control, has given Chubb and Moe's analysis the kinds of close scrutiny that's been missing; his unpublished paper is called, "Understanding High School Achievement." And the questions he raises about Chubb and Moe's analysis ought to shake the confidence of even the book's most ardent supporters.

Chubb and Moe base their case on the High School and Beyond (HS&B) Survey of a random sample of American public and private high schools. In 1980, 36 randomly selected sophomores in these schools were interviewed and given a 63-minute, 116-question test. In 1982, in their senior year, these students were re-interviewed and retested using the same exam. Chubb and Moe try to account for the differences in the students' test score gains over these three years, and they conclude the differences are largely the result of the different ways the various schools were organized.

The first thing to notice is that Chubb and Moe are basing their radical recommendations on a highly questionable and biased set of data. One problem with HS&B, Witte points out, is that students self-reports on their parents' income and education were very unreliable. So we can't trust HS&B-based studies that find, as Chubb and Moe do, that family background doesn't matter much in student learning.

Another problem is HS&B test results. If the test really measures student learning, then there is almost no learning going on in public or private high schools. Students gained an average of only 6.6 more correct answers between their sophomore and senior year. It also turns out that despite the big differences in wealth and prior achievement of the students most private schools select, the differences in average achievement between public and private schools, Witte tells us, "are simply so small that we can draw almost no conclusion from them."

Chubb and Moe try nevertheless. One way they do this is to turn the small test gains in their data into years of learning. And that's how an 8-answer difference between their high- and low-performing schools becomes a chasm of 2.25 years of student learning. But they don't just inflate the difference in student achievement gains. They also give private schools a considerable edge by allowing "elite private" and "other private" schools to constitute 60 percent of their sample -- even though these schools represent only about one-third of private school students. That decision, says Witte, "will clearly increase the proportion of private schools in the upper quartile of high-performing schools .... "

But even after all this, Chubb and Moe's analysis still doesn't make the case that more effective school organization will produce lots more student achievement. In fact, all the reasons they try out to explain achievement differences in high school -- taken together -- don't explain more than 5 percent of the difference in student achievement! So when Chubb and Moe tell us that school organization has twice the impact on student achievement that peer influence has or the same impact as family background, they are telling us almost nothing; twice almost nothing is still almost nothing.

But Chubb and Moe still persist. Using a questionable maneuver, they jump an average student from a very poorly organized school to a very well-organized school -- the kind they imply market control produces almost automatically -- to see what would happen. And that's the half year of learning they tell us American students would be likely to gain if we embraced market control. What Chubb and Moe don't tell us, however, is that this "half-year" really means, at most, one more correct answer on the 116-item test -- and, says Witte, probably more like a quarter of an item more correct.

So is market control the "panacea" for this nations educational crisis? Not even according to Chubb and Moe's own analysis. As Witte observes, " ... in terms of student achievement, the house they build is very fragile." It certainly doesn't justify toppling democratic control of schools.