How Can We Tell If Vouchers "Work"?

Brookings recently released an evaluation of New York City’s voucher program, called the School Choice Scholarship Foundation Program (SCSF), which was implemented in the late 1990s. Voucher offers were randomized, and the authors looked at the impact of being offered/accepting them on a very important medium-term outcome – college enrollment (they were also able to follow an unusually high proportion of the original voucher recipients to check this outcome).

The short version of the story is that, overall, the vouchers didn’t have any statistically discernible impact on college enrollment. But, as is often the case, there was some underlying variation in the results, including positive estimated impacts among African-American students, which certainly merit discussion.*

Unfortunately, such nuance was not always evident in the coverage of and reaction to the report, with some voucher supporters (strangely, given the results) exclaiming that the program was an unqualified success, and some opponents questioning the affiliations of the researchers. For my part, I’d like to make a quick, not-particularly-original point about voucher studies in general: Even the best of them don’t necessarily tell us much about whether “vouchers work."

In a typical voucher program, such as the SCSF in NYC, a relatively small group of recipients attend a participating private school; eligibility is often limited to certain school or subgroups, in SCSF's case students from low-income families. Analyses of how these students do might give some insight into whether participating private schools are any more effective than the public schools from which voucher recipients are drawn.**

(But, as Bruce Baker points out, even that is tricky, given the fact that being surrounded by higher-performing peers [e.g., those whose families can afford to pay full tuition] can by itself influence outcomes, regardless of the actual instructional effectiveness of the school. Such peer effects are a common issue, and very tough to address, random assignment or not.)

Regardless, placing a few poor kids into private schools every year isn’t really the purpose of vouchers. Rather, the endgame is a privatized school “marketplace,” in which students and parents can choose from a variety of private schools (and perhaps a mix of public schools as well). New schools open and close based on whether they attract students. This would, so the theory goes, generate competition and a kind of natural selection process in which "good schools" (or at least those well-suited to the students in the area) thrive and "bad schools" are forced to close.

This wouldn’t happen overnight, of course. Hypothetically, over time, more and more students would receive vouchers, and additional private schools would open up to meet the demand. It’s kind of like gradually opening up a whole new school system, except one that has minimal connection to government beyond its funding (via vouchers).

And it's exceedingly difficult to say whether the small group of private schools participating in recent voucher programs, such as NYC’s SCSF, would perform similarly to those comprising a vastly expanded marketplace of private schools (see a similar argument here).

Frankly, as mentioned above, it’s difficult to determine whether current private schools are “better” (or more cost effective), to say nothing of the quality of a sector that would eventually have to expand, in some places many times over, and would be operating in a very different environment (e.g., not competing with tuition-"free" alternatives). The schools that would arise under this system would almost certainly be different from current private schools, as would, of course, the students they serve.

In other words, the market-based scenario envisioned by voucher advocates cannot really be assessed without trying it (though there are alternative approaches, such as simulations and looking at other nations). At the very least, it cannot be assessed by allowing a tiny group of public school students to attend private schools for a few years.***

So, this Brookings evaluation is a useful addition to the evidence on voucher programs, and, given the design and size of the program, I hope to see future papers. But, when it comes to its implications for school privatization in general, this analysis, like almost all voucher studies, requires very cautious interpretation.

- Matt Di Carlo

*****

* As was roughly the case across all students, Hispanic students (47 percent of the sample) were only slightly more likely to enroll in college, and the difference was not statistically discernible at conventional levels. African-American students (41 percent), on the other hand, were significantly more likely to enroll, and the estimated difference was meaningful in size compared to the baseline enrollment rate.

** Voucher programs might also facilitate the examination of whether competition generates improvement among public schools (see here and here, for example).

*** There is a highly related, though somewhat different discussion to be had about charter schools, but that's a topic for a different post.