Chubb and More Revisited: Science or Ideology?

After decades of making speeches and writing articles, I've learned a few things. One of them is that the real tests for your arguments are not the people who are already with you or against you, but the ones who haven't made up their minds and those who might change their minds based on new evidence.

Judging from my travels and mail this past year, the idea of using public money to pay for the tuition of youngsters who choose (and are chosen by) private and religious schools has made some converts among this group of people who rely chiefly on evidence to make up their minds on important public policy issues. They don't have strong feelings one way or the other about ideological issues like our common school tradition or the separation of church and state. But they do care a lot about education and about preparing future citizens with the knowledge and skills America needs to compete in the global economy. They care about what works, and their bottom line is achievement. So, many of them tell me, often sadly, they must now support vouchers because the "evidence" shows that private schools outperform public schools, that market control of schools works better than democratic control.

The evidence, I ask? What about the results of the National Assessment of Educational Progress (NAEP) and other studies I've presented? And I show them that when you compare public and private school students who have taken the same courses and/or whose parents have similar education and income levels, there is no difference in their achievement. Neither public nor private schools are producing the levels of achievement we need.

This is persuasive for some people. Even Secretary of Education Lamar Alexander -- the President's top voucher salesman -- doesn't refute these NAEP results. But another response I often get is a diplomatic version of "We can't really trust an analysis of this issue by the president of a teachers' union. There's self-interest involved." And then I ask, "Whose analysis moved you to support vouchers?" The answer I usually get is John Chubb and Terry Moe's Politics, Markets and America's Schools.

The curious fact is that almost no one who cites Chubb and Moe has read their book, although they have all read favorable things about it in newspaper editorials or op-ed pages. The few non-technical people who have read it confess they could not follow the statistical analysis that makes up more than half the book and that is supposed to support the conclusions. This problem is shared by the majority of people who have written about the book in the popular press and by all the pro-voucher politicians who cite it.

So why has this book been so influential? The answer I most often hear is that the book must be true because it's science and its authors objective academics.

Not so, argue Professors Valerie Lee (University of Michigan) and Anthony Bryk (University of Chicago). In "Science of Policy Argument? A Review of the Quantitative Evidence in Chubb and Moe's Politics, Markets and America's Schools," a paper prepared for the Oct. 1, 1992, conference on school choice sponsored by the Economic Policy Institute in Wash., D.C., Lee and Bryk destroy Chubb and Moe's claim to have proven that private schools outperform public schools and that the reason is market control. They also demonstrate that Chubb and Moe made many technical decisions about how to conduct their analysis that tilted the results toward the policy conclusion they favored. "It is clear that Chubb and Moe have a personal preference for market control. It is not clear, however, that the empirical evidence favors the same argument."

Lee and Bryk are both sociologists and statisticians who are recognized by their peers as being experts on the literature, data and methods Chubb and Moe used. They are also academics who candidly admit their perspective. "We have published a considerable volume of research on Catholic schools, most of which reflects positively on these schools .... [W]e support at least some components of [Chubb and Moe's] argument." Yet Lee and Bryk are above all honest researchers whose extensive technical review of the Chubb and Moe research leads them to conclude "that the book falls considerably short of its stated aims. Having captured the public's attention, Chubb and Moe have artfully employed extensive, but not always solid, empirical evidence to support a policy argument to advance some preconceived notions about American schooling .... We contend that popular acceptance of this book as a source of scientific evidence to support its policy recommendations is unfortunate."

Lee and Bryk present important new evidence that tears apart the seemingly scientific foundation that made the Chubb and Moe book appear so credible. But equally important, they raise an important old worry about the role of science in a democratic society: Technical knowledge is essential to social improvement, but unless that knowledge is "broadly held, truly in the public domain," it might be used to promote anti-democratic and other harmful ends. As Lee and Bryk conclude, "Given the receptive ear for the message of Politics, Markets, and America's Schools at the highest levels of our government, and the lack of scrutiny its empirical base has received from the media or the public, this worry seems well founded.