New Evidence On Teaching Quality And The Achievement Gap

It is an extensively documented fact that low-income students score more poorly on standardized tests than do their higher income peers. This so-called “achievement gap” has persisted for generations and is still one of the most significant challenges confronting the American educational system.

Some people tend to overstate -- while others tend to understate -- the degree to which this gap is attributable to differences in teacher (and school) effectiveness between lower and higher income students (with income usually defined in terms of students’ eligibility for subsidized lunch assistance). As discussed below, the evidence thus far suggests that lower income students are a more likely than higher income students to have less “effective” teachers -- with effectiveness defined in terms of the ability to help raise student test scores, or value-added, although the magnitude of these discrepancies varies by study. There are also some compelling theories as to the possible mechanisms behind these (often modest) discrepancies, most notably the fact that schools in low-income neighborhoods tend to have fewer resources, as well as more trouble recruiting and retaining highly qualified, experienced teachers.

The Mathematica Policy Research organization recently released a very large, very important study that addresses these issues directly. It focuses on shedding additional light on the magnitude of any measurable differences in access to effective teaching among students of different incomes (the “Effective Teaching Gap”), as well as the way in which hiring, mobility, and retention might contribute to these gaps. The analysis uses data on teachers in grades 4-8 or 6-8 (depending on data availability) over five years (2008-09 to 2012-13) in 26 districts across the nation.

In short, the authors, Eric Isenberg, Jeffrey Max, Philip Gleason, Matthew Johnson, and Jonah Deutsch from Mathematica, along with Michael Hansen from AIR, find that, across all 26 districts, there exists only a very small Effective Teaching Gap. That is, the value-added of teachers of lower income students is, on average, only very slightly lower than that of teachers of higher income students, and both groups of students are equally likely to have access to the teachers with the highest and lowest value added scores. It follows, then, that persistent test-based achievement gaps among students may not in fact be due to the distribution of teachers by student income (eligibility for federal lunch assistance). Moreover, the Mathematica researchers find that hiring, transfer, and leaving patterns are either basically consistent with the small overall gap, or do not contribute either way.

There are a number of important aspects implications of this report's findings, but we will focus here on just a few of them.

This report's findings do not quite square with the consensus opinion in the public discourse. The consensus among many education commentators, which, as mentioned above, is intuitively compelling, is that access to effective teaching varies substantially by student income, and that reducing this discrepancy would make a big dent in the income-based achievement gap. The findings reported by the Mathematica/AIR researchers do not support this view.

To be clear, the results do show a gap in access between higher and lower income students, but, across all of the 26 districts included in this report, the gap is not large -- about one percentile point, on average. And lower and higher income students have roughly the same chance of having a teacher who is highly effective in boosting scores, as well as the same chance of having a teacher who is not.

This does not, of course, suggest that teaching quality is unimportant, only that, overall, if we could magically equalize access to effective teaching between higher and lower income students, this would not have not much impact on the achievement gap, at least in most (but not all) of these 26 districts.

(Note that, theoretically, an incredibly effective teaching force could still contribute to achievement gaps, while an ineffective teaching force could mitigate such gaps. This report is not about the overall effectiveness of teachers, but rather about how they are distributed.)

But these results are not entirely inconsistent with the previous research on these issues. One need not look far to find an argument that achievement gaps are significantly (or even largely) attributable to differences between impoverished and more affluent students in access to effective teaching. Yet the existing body of evidence on students’ income-based gaps in access to effective teaching is not particularly large, and while the results generally indicate that lower income students (and schools) have less access to teachers who are the most effective in boosting test scores, the magnitude of these differences varies quite a bit by study, as well as by the methods employed, particularly, it seems, model specification.

The Mathematica report provides an excellent summary of this evidence (Appendix A), which readers should review for themselves. But it bears mentioning, once again, the obvious point that many of the “facts” taken for granted in our discourse, even when they have a strong theoretical or logical basis, don't always square with empirical evidence. This report and its predecessors (e.g., Glazerman and Max 2011), in testing one such proposition directly, are examples of good social science.

The overall “no gap” finding may get most of the attention, but the underlying results pertaining to hiring, mobility, and retention are also very important (and interesting). The overall “Effective Teaching Gap” discussed above might mask differences at different points in the “pipeline.” For example, higher poverty schools may hire teachers who are very effective in boosting test scores, but this gap may be erased if these new teachers are more likely to leave or transfer than their colleagues in lower poverty schools. The Mathematica researchers do find some such differences, but some are either “cancelled out” by other factors or are not large enough to make a difference overall. For example:

• Hiring: Even though the new hires in higher and lower poverty schools are equally effective in raising test scores (which is itself quite surprising), higher poverty schools, of course, tend to hire more of these teachers each year. This does contribute to the overall Effective Teaching Gap, but its impact is attenuated because the difference in hiring rates is not huge (5 and 11 percent in lower and higher poverty schools, respectively), new hires only represent a small proportion of all teachers, and also because new teachers in both groups of schools tend to improve very quickly in their first year;
• Tranfers: Similarly, teachers who transfer to higher poverty schools tend to be less effective than those who transfer to lower poverty schools, which contributes to the Effective Teaching Gap, but once again the contribution is partially muffled by the fact that relatively few teachers transfer from higher- to lower poverty schools, and vice-versa;
• Attrition: Higher poverty schools do lose more teachers each year than do lower poverty schools, but the fact that stayers tend to have higher value-added scores than do leavers virtually nullifies the degree to which attrition contributes to the Effective Teaching Gap.

All of these findings, put simply, illustrate how teacher labor markets have a lot of working parts. For instance, the impact of high attrition rates depends on who leaves and who stays, while transfers affect the gap only insofar as teachers move to schools with different student populations. This report serves to “connect these dots,” and the results discussed above carry clear policy relevance.

The results of this Mathematica study, as a whole, are (mostly) encouraging. Putting aside the obvious and important limitation of this study, which is that it is based entirely on test scores, it might be tempting for some to profess disbelief at this analysis’ apparent implication that students are not being underserved by the schools in low income neighborhoods. There are, however, a couple of reasons why this is an incomplete interpretation.

Most obviously, while the overall Effective Teaching Gap was found to be very small (especially relative to the size of test-score gaps), there were districts in which the gap was substantial, suggesting, as always, that districts vary in the distribution of teachers by value added scores. It seems fair to say that the Effective Teaching Gap may not be as large as previously assumed, but there’s no basis for any grand conclusions beyond that. Bear in mind, however, that even a large Effective Teaching Gap would be small compared to the achievement gap to which it may contribute, and seemingly minor differences can make big “real world” differences for students.

Second, it is plausible that schools serving large proportions of low-income students are less effective than those serving more affluent populations, even if part of this gap does not manifest itself entirely (or even mostly) in terms of differences in teachers’ test-based effectiveness. Other school-based factors, such as gaps in principal effectiveness, curriculum, resources, facilities, etc., may also contribute to achievement gaps between student subgroups.

So, to reiterate, this study does not support the idea that simply equalizing test-based effectiveness between teachers of higher and lower income students, within or between these 26 districts, would have a large aggregate impact insofar as access is already pretty much equal, on the whole. This also means that moving “effective” teachers to higher poverty schools could actually create “reverse” Effective Teaching Gaps.

But teaching quality is a not a zero sum game. If lower and higher income students indeed have roughly the same access to effective teaching, or at least if the gap is far smaller than previously assumed, then current efforts to improve teaching quality across the board, if they succeed uniformly, could benefit all students equally, without perpetuating unequal access. In addition, if effectiveness could be improved more quickly among teachers in higher poverty schools through targeted training, support and incentives, it could actually help make a real dent in the achievement gap, rather than just staunch the bleeding.