How can we tell whether a researcher has reached sound conclusions?
We're in the midst of a national debate about school vouchers. Voucher supporters argue that all students, and particularly poor, minority students in inner-city schools, should be given a chance to go to private schools at public expense. Permitting students to do this, voucher supporters say, will greatly improve their academic performance.
In Milwaukee, Wisconsin, there has actually been a voucher experiment to see whether there is anything to voucher supporters' claims. Since 1990, a small number of poor youngsters have been given public money to go to private, non-religious schools. The Wisconsin Department of Public Instruction commissioned John Witte, a distinguished researcher and professor at the University of Wisconsin, to gather and interpret data, and for five years, Witte has been publishing reports about the experiment. He has looked at such questions as how many students elected to enter the voucher program, how many were selected by the private schools, how many subsequently dropped out, and, of course, how well the private school students have done in comparison with public school students from similar backgrounds. His conclusion? Even though many parents like the voucher schools, there is no difference, so far, between the math and reading scores of voucher students and comparable youngsters who have remained in public schools.
This would seem to provide us with an answer as to whether students who go to private schools do better than public school students. (Of course the experiment could be continued because it may take longer than four or five years to achieve an academic turnaround.) But several weeks ago, a group of researchers led by Harvard University professor Paul E. Peterson published a study of the Milwaukee voucher program that came up with totally different conclusions from Witte' s.
Peterson, assisted by Jay P. Greene of the University of Houston and Jiangtao Du of Harvard, took the data Witte used but employed different methods in analyzing them. According to their findings, by the time choice youngsters had been in their private schools for three and four years, their reading and math scores were markedly better than those of comparable students who had remained in public schools.
Of course, contradictory findings about student achievement are nothing new. After Education Alternatives, Inc. (EAi) had been running a group of Baltimore schools for two years, the media reported that student test scores were down. EAi then hired its own experts who said, "No, indeed! Test scores are up." In the end, pushed by the mayor, the school district asked a group of independent evaluators to assess EAi' s performance. The evaluators found that students in EAi schools were doing no better than comparable students in non-EAi schools. Parents and members of the public in situations like those in Baltimore and Milwaukee are always likely to face conflicting evaluations about the success of a program.
The question is, how can we tell whether a researcher has reached sound conclusions? The average person can't. The media can't either. Most reporters are not knowledgeable in the field of statistics. In writing about a technical study, they are likely to pick up what researchers claim to have proved and report it as fact -- which is like a jury's accepting a lawyer's closing argument as proof Perhaps the time will come when an impartial organization will evaluate reports of research the way Consumers Union evaluates products. In the meantime, there are certain clues that ordinary people can use to evaluate a piece of research. For example, what do other knowledgeable people think about it?
In education and other academic fields, researchers submit their work to professional journals and undergo a peer review process. Witte's work has been peer-reviewed, and interested experts have had five years to examine and critique his findings. Peterson et al, on the other hand, did not submit their research to peer review. They went directly to voucher advocacy groups and the media with their report. This is something that consumers should consider when they ask themselves whether Witte's findings of those by Peterson et al are more likely to hold water. And there are things in the report itself. For example,
• The table that contains Peterson et al' s main analysis - the one on which they base their conclusions - fails to take account of family background. This violates one of the basic rules of research on student achievement because accounting for family background is the only way researchers can distinguish between the advantages a student brings from home and the value added by the school. When researchers take account of family background, they can compare, for example, scores of students whose parents have had a high school education with those of students whose parents have had a college education. One of Peterson et al' slater tables does take account of family background. Their results are then identical to Witte's: There is no difference in achievement between voucher students and their public school peers. But Peterson et al slough off this table in their discussion, and no newspaper story that I've seen has picked up on it.
• Peterson et al also stack the deck by using a statistical standard that is much lower than researchers ordinarily use or accept. When researchers employ statistics to show that a program works, they look for results that are "statistically significant." This means there is little likelihood of error in their findings. Peterson et al do not talk much about "statistical significance" because their findings are not statistically significant. They talk, instead, about results that are "substantially significant" or "substantially important." These are made-up terms that have no meaning among researchers. But they do deceive the public into thinking that Peterson et al have proven their case.
The question of which schools do the best job of improving student achievement is not going to be easy. How can lay people decide when the bases for their decisions are highly technical reports that are likely to come to conflicting conclusions and when the media are not in a position to shed light on the question? If those who support vouchers expect us to make the kind of radical change that a voucher system will involve, they will have to come up with an answer to this problem.