The Uncertain Future Of Charter School Proliferation

This is the third in a series of three posts about charter schools. Here are the first and second parts.

As discussed in prior posts, high-quality analyses of charter school effects show that there is wide variation in the test-based effects of these schools but that, overall, charter students do no better than their comparable regular public school counterparts. The existing evidence, though very tentative, suggests that the few schools achieving large gains tend to be well-funded, offer massive amounts of additional time, provide extensive tutoring services and maintain strict, often high-stakes discipline policies.

There will always be a few high-flying chains dispersed throughout the nation that get results, and we should learn from them. But there’s also the issue of whether a bunch of charters schools with different operators using diverse approaches can expand within a single location and produce consistent results.

Charter supporters typically argue that state and local policies can be leveraged to “close the bad charters and replicate the good ones." Opponents, on the other hand, contend that successful charters can’t expand beyond a certain point because they rely on selection bias of the best students into these schools (so-called “cream skimming”), as well as the exclusion of high-needs students.

Given the current push to increase the number of charter schools, these are critical issues, and there is, once again, some very tentative evidence that might provide insights.

One fairly simple way to take a look is to identify places where the charter sector as a whole has been shown effective. There are at least two major districts where this seems to be the case.

The first is New York City, as shown by a 2009 lottery study, which is discussed here (NYC charters were also shown effective in this paper). The other charter hotspot is Boston. A 2008 evaluation, part of which also used random assignment, found that, among Boston middle and high schools, charter students outperformed their regular public school counterparts by substantial margins (also see this analysis).*

Charter supporters frequently point to these analyses as evidence that it’s not just a few chains scattered throughout the nation doing well – that NYC and Boston are among the largest districts in the U.S., both serve large proportions of disadvantaged kids, and that the charters in these cities are “showing it can be done."

It’s fair to say that most of these schools are probably very well-run and doing great things. But, in the question of whether these efforts represent proof that charter sectors as a whole can produce results, it’s also important to note that, in both cities, charter “market share” is exceedingly small. In other words, in these two large, urban districts, there are very few charter schools. The Boston study included just 16 "oversubscribed" schools - those with more applicants than spaces, which usually requires a lottery - within the city (about 60 statewide), whereas the NYC analysis included data for around 75 oversubscribed charters (fewer during the earlier years included in the data). In both cases, charters as a whole (over- and undersubscribed) served only a miniscule slice of each city’s total enrollment.

This matters because, put simply, charters compete for several types of finite resources, and if there are fewer schools, each will get a larger share than if there are many schools. For instance, virtually all high-profile charters receive private funding, often a great deal. This is definitely the case in NYC. Though evidence on how these well-heeled charters spend their private money is limited, the additional funds might matter for providing the kind of services, such as tutoring and massively extended time, that appear to be associated with higher relative performance (NYC’s oversubscribed charters provide, on average, 30 percent more time).

On a similar note, many charters “save money” by maintaining a higher-turnover teacher force. They tend to hire young teachers, who are less expensive, and many leave within a few years. More schools would make it harder to maintain this approach without suffering a decrease in the “quality” of applicants. The same might be said about administrators, consultants and other staff.

Finally, one might also argue that market share has implications for the students charters serve – i.e., that charters in districts with lower market share are more likely to benefit from processes like selection bias and selective attrition. This is certainly plausible.**

In any case, it’s tough to argue that the analyses of charter effects in NYC and Boston, though both are randomized controlled trials, represent evidence that charters can successfully expand within a single location. In fact, in some respects, they suggest the opposite – that the few places where an entire charter sector does well, at least those for which good evaluations are available, tend to be those where the sector as a whole serves only a small fraction of the students.

There is also some evidence (though it is very weak) for the importance of market share at the state level. In the national CREDO study, there is a moderately strong association at the state-level between the percent of all students who attend charters and estimated effects, especially on math scores. There are exceptions, but states with the smallest charter populations (as a percentage of total enrollment), such as Indiana and Missouri, tended to show positive effects (relative to comparable regular public schools), whereas those with the largest shares, including Ohio and Arizona, tended to get worse results.

It is certainly possible, as is frequently argued, that those states (and localities) with more “quality-focused” laws will produce better results, and that states need to take an active role in “shutting down bad charters and replicating good ones." But empirical evidence for these claims – whether such policies actually work in practice - is lacking, and serious questions remain.

Closing schools is exceedingly difficult, both logistically and politically. And, even if states and localities implement a closure regime, the real question is whether they can open superior alternatives. There is some brutal logic here. There is only handful of charters that consistently do very well (e.g., KIPP, Achievement First). They rely heavily on private donations, as does their capacity to expand. They also tend to employ practices, such as 8-9 hour days and unusually strict discipline policies (including, in some cases, frequent suspensions), that are not easily replicated in regular public schools, and are unlikely to produce the same results when/if they are.

Even the most well-established and heavily funded charter operators have trouble succeeding and expanding, as evident in a very recent Mathematica/CRPE study of charter management organizations (CMOs), which found that these organizations vary widely in their results (also see here). Many saw a decline in results as they opened more schools (also see this 2009 report).

In addition, the national CREDO study is among the only ones that includes enough states to examine the associations between state policies and outcomes, and it found a small negative effect of state policies allowing charters to choose from multiple authorizers. This represents tentative support for the idea that these policies matter, but not necessarily the claim that they are a decisive factor.***

So, states and localities can potentially play a role in creating an environment for charters that is conducive to “quality control," but the real factors determining success seem to be money, time and attention. As a result, very few of these schools produce meaningfully superior results, and charter sectors as a whole tend to succeed where their numbers are limited.

In general, after more than 20 years of fairly rapid expansion, charter schools haven’t yet taught us much we didn’t already know, at least in terms of what influences test-based outcomes (which are of course far from the only relevant kind). Certainly, a small group have accomplished a great deal, but they are outliers, and they are dispersed throughout the nation. The available evidence is nowhere near comprehensive, but it strongly suggests that even these successes cannot be scaled up without, at the very least, large private investments. As a result, their implications for regular public schools remain limited.

What’s truly needed is for charter schools to aggressively pursue their role as “educational laboratories” – accepting high-needs students, and trying new and innovative ways to help them succeed (as well as, perhaps, offering specialized programs such as language immersion). These schools should, like KIPP, open themselves up to high-quality research programs testing the effects of these practices, in an effort to inform and improve all schools. Their test scores and other results might serve as valuable policy evidence instead of judge and jury, and the schools themselves should be seen as partners, not adversaries. None of this is likely to occur so long as both “sides” of this debate – both supporters and opponents – continue to wage test-based trench warfare that is so obviously locked in stalemate.

- Matt Di Carlo

*****

* An additional lottery study used data from Chicago, but it included only a tiny handful of schools run by the same network. One other location that gets a lot of attention is New Orleans, much of it focused on changes in cross-sectional proficiency rates that are not valid gauges of progress. Based on news reports, however, CREDO has done an analysis of that city’s charters, but the full report (including overall effect sizes) has not been released to the public.

** It is certainly fair to say that processes like selection and attrition play a role in determining the "raw" outcomes of charter schools. The question is the degree to which rigorous evaluations of charter effects can account for these factors. The best studies, especially those that use random assignment (e.g., the lottery studies discussed above), can do so to a substantial degree, but even they cannot adequately address the peer effects stemming from attrition without replacement, nor can they address the possibility that results may differ between applicants to oversubscribed charters and non-applicants (external validity).

*** CREDO also found a small negative association of states approaching their charter caps (i.e., states in which the number of charters is over 90 percent of the cap). There is also evidence that certain types of charter schools and authorizers get better results, on average, and this is information that authorization laws might exploit. For instance, this analysis of authorizers in Ohio (where there are many) found that non-profit organizations got better results, while other studies conclude worse results among schools with local district authorizers and virtual charters.