The IMPACT Of Teacher Turnover In DCPS

Teacher turnover has long been a flashpoint in education policy, yet these debates are rife with complications. For example, it is often implied that turnover is a “bad thing,” even though some turnover, as when low-performing teachers leave, can be beneficial, whereas some retention, as when low-performing teachers stay, can be harmful. The impact of turnover also depends heavily on other factors, such as the pool of candidates available to serve as replacements, and how disruptive turnover is to the teachers who are retained.

The recent widespread reform of teacher evaluation systems has made the turnover issue, never far below the surface, even more salient in recent years. Critics contend that the new evaluations, particularly their use of test-based productivity measures, will cause teachers to flee the profession. Supporters, on the other hand, are in a sense hoping for this outcome, as they anticipate that, under the new systems, voluntary and involuntary separations will serve to improve the quality of the teacher workforce.

A new working paper takes a close look the impact of teacher turnover under what is perhaps the most controversial teacher evaluation system in the nation – that used in the District of Columbia Public Schools (DCPS). It's a very strong analysis that speaks directly to policy in a manner that does not fit well into the tribal structure of education debates today.

The paper’s authors, Melinda Adnot, Thomas Dee, Veronica Katz, and James Wyckoff, analyze DCPS data between 2009-10 and 2012-13, specifically how the evaluation scores of teachers leaving their schools compared with those of their replacements, and how and whether this eventually influenced student achievement (the evaluation system is called IMPACT). During this time period, overall attrition in DCPS (teachers leaving the district entirely) was around 18 percent. There was, however, a great deal of variation in this rate by IMPACT rating. For example, among teachers rated “effective” or “highly effective,” attrition was roughly 13 percent, compared with over 45 percent among teachers rated “minimally effective” or “ineffective.” Some of the teachers in the latter group left involuntarily (i.e., they were fired for low ratings), whereas others left on their own.

The impact of this turnover on overall teacher effectiveness depends on the difference in effectiveness (measured here with IMPACT scores) between exiting teachers and their replacements, while the effect on student achievement is further dependent upon the relationship between IMPACT scores and test-based effectiveness. These are the primary questions of interest for Adnot et al., who, put very simply, compare grade/school units that experienced turnover to those that did not.

They find, in brief terms, that teachers who left were replaced by teachers with higher IMPACT scores the following year. The difference is large in magnitude (about one third of one standard deviation), and corresponds to an educationally meaningful increase in achievement scores – about eight percent of a standard deviation in math and five percent in reading, though the latter estimate is only significant at the 90 percent confidence level. (Note that this does not mean that districtwide achievement scores improved by these amounts, as these estimates represent the year-to-year change in achievement in school/grade units that experienced turnover, compared to those that did not.)

There is, however, substantial variation by measured teacher effectiveness, as well as by school poverty. The estimated impact on both evaluation scores and student achievement is positive, large in magnitude, and statistically discernible among low-rated teachers who were replaced, but negative (though not statistically significant) when higher-rated teachers are replaced. This means, as would be expected, that exits of low-rated teachers may be beneficial, whereas exits of high-rated teachers are, on balance, harmful, or at least not helpful.

In addition, there is a rather large difference in the impact of turnover by student poverty (i.e., subsidized lunch eligibility rates). In high poverty schools (which means most DCPS schools), the estimated impact was large and statistically discernible. In low poverty schools, though, the effect was basically zero.

(Before discussing these results, I feel obliged to mention that DCPS has been very accommodating about providing their data to some researchers, but we at the Shanker Institute, despite extensive efforts over the course of almost two years, were unable to obtain data on teacher race and ethnicity for our recent teacher diversity study, despite the fact that the data we requested were far less sensitive and complicated than those used for this and other DCPS studies.)

Having said that, there are a few implications of these important results that bear articulating.

Teacher turnover/attrition is neither good nor bad by itself. This first and most obvious point is often ignored in our education policy discourse, but when a teacher leaves, the impact of this departure depends primarily on how good she is vis-à-vis the performance of her replacement. There is no reason to believe that aggregate turnover would necessarily have a positive or negative impact on meaningful outcomes – it all depends on who leaves, who replaces them, and how the churn affects those who remain. Prior research on this question, though not abundant, finds that the test-based impact of turnover may be negative (e.g., Ronfeldt et al. 2013).

The results of this paper, in contrast, suggest that, in DCPS, at least during this time period and measuring outcomes in terms of IMPACT scores and student achievement tests, the effect of teachers leaving, while heterogeneous, may have been positive overall. This certainly is not to say that schools and districts should ignore or encourage turnover, but it does illustrate empirically the underappreciated idea that the impact of teachers leaving can cut both ways. The degree to which, in this particular study, the estimated impact is due to the IMPACT system itself is an open question, one discussed a bit further below.

The (prospective) teacher labor pool in DC appears quite deep, at least for now. This is a very interesting and important finding. Now, in theory, dismissing and paying teachers based on IMPACT scores, which include test-based effectiveness measures, should produce higher IMPACT scores and, ultimately, higher achievement as well. The reality, of course, is that any number of factors can intervene in this process and create problems. In addition to and the effectiveness of teachers who choose voluntarily to leave, one huge (and unpredictable) factor is the pool of available replacements.

In DCPS, even though almost one in five teachers must be replaced every year, it seems that the labor market was sufficient not only to maintain teacher quality (at least as measured by these outcomes), but perhaps even to increase it. And this is even more significant considering the fact that, during the period of time in this study, the average measured performance of exiting teachers improved. This might have attenuated the postive impact of replacing them, but for the fact that the performance of their replacements also improved. Whether this is sustainable, and whether the same situation applies elsewhere, remain open questions to say the least.

The impact of turnover, and not just the level, may vary by school context. It is well established that teacher turnover is higher in schools serving larger proportions of disadvantaged students (e.g., Ingersoll 2001), but the results from Adnot et al. indicate that the impact of that turnover on meaningful outcomes also may vary by contextual factors, in this case student characteristics. Specifically, as mentioned above, replacements in low poverty schools were, on the whole, no different from their predecessors, whereas there was a rather large discrepancy in high poverty schools. This is important to keep in mind when, for example, looking at aggregate statistics on teacher turnover at the district level, and especially at the state or national levels.

Is this paper an evaluation of IMPACT itself? No, but that's not a black and white answer. The authors are very careful to note that the paper should not be interpreted that way (although that didn’t stop several commentators and reporters from doing so). This is an example of great researchers taking steps to ensure that their work is interpreted correctly.

The primary problem with seeing this analysis as an evaluation of IMPACT per se is that most attrition is voluntary, and the level and nature of teacher leaving during this time period cannot be attributed to IMPACT directly (nor, of course, can the quality of replacements). It may have occurred under a different system, or may even have been occurring to an extent before IMPACT was implemented. This may seem like a trivial issue, but it is not - this is an important distinction between, on the one hand, evidence on the effect of turnover under IMPACT and, on the other hand, the effect of turnover because of IMPACT (see Dee and Wyckoff 2015 for better evidence).

That said, IMPACT was directly intended to influence the quality composition of the DCPS teacher workforce, and the results from Adnot et al., combined with those from previous studies, certainly suggest that turnover under IMPACT may be having a positive impact. In that sense, these results can hardly be viewed as agnostic on the effect of the system.