The Equity Projection

A new Mathematica report examines the test-based impact of The Equity Project (TEP), a New York City charter school serving grades 5-8. TEP opened up for the 2009-10 school year, receiving national attention mostly due to one unusual policy: They paid teachers $125,000 per year, regardless of experience and education, in addition to annual bonuses (up to $25,000) for returning teachers. TEP largely makes up for these unusually high salary costs by minimizing the number of administrators and maintaining larger class sizes.

As is typical of Mathematica, the TEP analysis is thorough and well-done. The school's students' performance is compared to that of similar peers with a comparable probability of enrolling in TEP, as identified with propensity scores. In general, the study’s results were quite positive. Although there were statistically discernible negative impacts of attendance for TEP’s first cohort of students during their first two years, the cumulative estimated test-based impact was significant, positive and educationally meaningful after three and four years of attendance. As always, the estimated effect was stronger in math than in reading (estimated effect sizes for the former were very large in magnitude). The Mathematica researchers also present analyses on student attrition, which did not appear to bias the estimates substantially, and they also show that their primary results are robust when using alternative specifications (e.g., different matching techniques, score transformations, etc.).

Now we get to the tricky questions about these results: What caused them and what can be learned as a result? That’s the big issue with charter analyses in general (and with research on many other interventions): One can almost never separate the “why” from the “what” with any degree of confidence. And TEP, with its "flagship policy" of high teacher salaries, which might appeal to all "sides" in the education policy debate, provides an interesting example in this respect.

The initial reaction to the Mathematica findings, which was predictable (and understandable), was to interpret TEP’s positive results as a direct result of their teacher compensation policy. That is, the school offered salaries far beyond those that most NYC teachers (especially newer teachers) could earn in the district or even outside the city. This was designed to attract a consistent supply of strong candidates to the positions, thus permitting selectivity in hiring and renewing contracts, as well as, perhaps, minimizing any instances of voluntary attrition due to inadequate compensation.

This is clearly plausible, and almost certainly played at least some role. But school performance, at least to the degree testing gains capture it, is complex and multifaceted, and a single policy cannot explain it. It is important to examine other possibilities that might help explain TEP's apparent early success (some of which may be directly or indirectly related to the compensation policy), as there is always a danger entailed in drawing conclusions about a single policy without understanding the full picture.

For one thing, TEP, like virtually all charter schools that have been shown to increase test scores, offers a large extension of the school day. TEP’s day is eight hours long, compared with the roughly six hours offered by the city’s regular public middle schools. (TEP's days were actually even longer in their first year of operation, but were shortened because the principal thought them too long for the teachers and students.)

Increasing time by 25 percent is the regular public school equivalent of two extra months of learning. Surely, time must be used wisely, and TEP seems to offer a diverse set of programs beyond the primary tested subjects of math and reading, but it’s difficult to believe that most schools would not fare at least somewhat better in test-based evaluations were they afforded an extra couple of months of instruction (though attrition due to workload may be less of an issue when salaries are high). And, on the flip side, it’s virtually impossible to find a high-performing charter school chain that does not extend the day substantially.

Another interesting descriptive finding in the report is the fact that TEP teachers are not necessarily the fresh-out-of-college type that some people might think they are. In fact, only two of the school’s 42 teachers had fewer than three years of experience, and the median was six years. Perhaps the school’s ability to continually hire experienced, accomplished educators is a factor in their (apparent) test-based success. This was no doubt at least somewhat aided by the high salaries, but it could also certainly serve as an example of a goal that might be accomplished in other ways. (Also note that the experience distribution implies that TEP seems to be largely hiring teachers who were already in the classroom, rather than attracting new candidates to the profession.)

TEP also places a great deal of emphasis on screening applicants. Teachers who apply endure a lengthy process in which they must submit evidence of effectiveness and sample lesson plans and agree to teach several demonstration classes, among other requirements. The benefits of such a process (if any), could, once again, potentially be facilitated by a large, strong applicant pool due to high salary offers, but may be improved in other ways as well. Further complicating the situation, the report also notes that around half of TEP’s teachers in any given year are not teaching there the next year, and that most of this (about 35 of the approximately 45 percent attrition rate) is due to their not being rehired. Such a high deselection rate could be attributable, at least in part, to high standards for contract renewal, but in any case it suggests that identifying strong educators during the hiring process is a tricky business, and that even massive increases in starting salaries does not necessarily solve the problem.

And there are additional possibilities as well, some of which seem largely independent of the high salaries. For example, each incoming TEP cohort has assigned to them a social worker, who sticks with those students throughout their tenure at the school. Having a social worker assigned to such a small group might confer advantages not shared by the schools to which TEP is compared in this report, in which social workers are less common and are responsible for much larger groups of students (this might, for example, help explain why the school did not use out-of-school suspensions or expulsions, and instead suspended students in-school, under the supervision of their social worker).

Compared with their regular public school counterparts, TEP teachers also reported considerably more control over their school’s policies, as well as a high degree of collaboration between them and their colleagues. There is some evidence that these conditions are associated with school performance.

The point here is not to discount the possibility that TEP’s high salaries contribute to its apparent success in raising test scores. Indeed, it is more than likely that this played at least some role. In addition, to reiterate, the salaries are not independent of many of the other possible factors discussed above, such as TEP's accomplished, experienced teachers (perhaps lured to the school by the high salaries). It's also fair to say that TEP provides pretty good tentative evidence that this kind of financial model may be feasible, at least under certain circumstances.

The point, rather, is that even the best charter school analyses are typically only designed to tell us (with a reasonable degree of confidence) whether students’ test results improved—but not why they may have done so, or how other schools might benefit from this knowledge. What’s more, this is particularly important to bear in mind when only one school is being examined after just a few short years of operation, especially one which employs a highly unusual policy (high teacher salaries) that might enjoy widespread support from all "sides" in the education policy debate. The prudent conclusion at this point is that TEP provides some promising evidence of a model that can be replicated elsewhere and examined more thoroughly as to the underlying factors, including but not nearly limited to the high salaries, that may be responsible for its results.

- Matt Di Carlo