The Increasing Academic Ability Of New York Teachers

For many years now, a common talking point in education circles has been that U.S. public school teachers are disproportionately drawn from the “bottom third” of college graduates, and that we have to “attract better candidates” in order to improve the distribution of teacher quality. We discussed the basis for this “bottom third” claim in this post, and I will not repeat the points here, except to summarize that “bottom third” teachers (based on SAT/ACT scores) were indeed somewhat overrepresented nationally, although the magnitudes of such differences vary by cohort and other characteristics.

A very recent article in the journal Educational Researcher addresses this issue head-on (a full working version of the article is available here). It is written by Hamilton Lankford, Susanna Loeb, Andrew McEachin, Luke Miller and James Wyckoff. The authors analyze SAT scores of New York State teachers over a 25 year period (between 1985 and 2009). Their main finding is that these SAT scores, after a long term decline, improved between 2000 and 2009 among all certified teachers, with the increases being especially large among incoming (new) teachers, and among teachers in high-poverty schools. For example, the proportion of incoming New York teachers whose SAT scores were in the top third has increased over 10 percentage points, while the proportion with scores in the bottom third has decreased by a similar amount (these figures define “top third” and “bottom third” in terms of New York State public school students who took the SAT between 1979 and 2008).

This is an important study that bears heavily on the current debate over improving the teacher labor supply, and there are few important points about it worth discussing briefly.

First, this is good news. Granted, SAT scores are a highly imperfect proxy for an individual’s knowledge and skills, but any evidence that the paper qualifications of new teachers are improving can only be considered encouraging. At the very least, it suggests, albeit tentatively, that individuals with stronger measurable qualifications (and, perhaps, higher labor market value) are pursuing teaching as a career, which in turn may indicate that the status of the profession is getting better (also see Goldhaber and Walch 2013). (Note that the authors of this paper also find that teachers are increasingly coming from selective undergraduate institutions.)

In short, then, as the title of the paper suggests, this is an encouraging finding.

Second, the improvement is difficult to explain, but policy does seem to matter. Upon hearing that the SAT scores of new teachers have been improving recently, one’s initial reaction may be to suspect that economic conditions drive these trends. For example, the improvement in teacher qualifications coincides with two recent recessions – the 2001 recession and the “great recession” that began in 2007. The story here would be that economic downturns may have forced districts to cut back on hiring, thus allowing them to be more selective. In addition, college graduates may have been choosing to teach during the recessions, particularly the more recent one, because the labor market was poor, and their options were more limited. The authors attempt to test these explanations. Their analyses are not the final word, of course (they employ proxies for labor market conditions, such as enrollment and hiring), but they find that these factors explain very little of the trend. This does not mean that labor market conditions play no role in in explaining the trend (they most certainly do), but they may not be the primary factor.

The authors identify some alternative factors that may have contributed to the improvement, including national policies (e.g., the highly qualified teacher requirements of NCLB), statewide policies (New York’s increasing standards for teacher licensure), and district-specific practices (NYC’s hiring patterns by certification). They make a strong, albeit inevitably inconclusive attempt to test these factors empirically based on the timing of policies/practices and trends in teacher qualifications, and they do find some evidence supporting them. This suggests that policies to improve teacher qualifications may work.

Third, incoming New York State teachers have never been disproportionately drawn from the “bottom third” of (statewide public school) SAT test takers. As is so often the case, the “U.S. teachers are drawn from the bottom third” talking point ignores a great deal of underlying variation. New York provides a good example of this. At no point during the time period of this analysis were more than 30 percent of incoming teachers drawn from the bottom third (as a rough guide, anything above 33 percent would be overrepresentation). In fact, in 1986, only about 22 percent of incoming teachers had bottom third SAT scores. Moreover, conversely, at no point were less than 30 percent of incoming teachers drawn from the top third.

This is, needless to say, not to imply that we should settle for proportional representation (and remember that these are New York-only distributions), but it bears noting that one should be careful when interpreting national trends.

Fourth, on a related note, the situation in New York City is quite different from that in New York State as a whole. The “bottom third” talking point didn’t really apply to New York State (at least based on statewide SAT distributions), but it did apply to New York City. Between 1985 and 2000, roughly 20-25 percent of incoming NYC teachers had top third SAT scores, while 35-45 percent scored in the bottom third on the SAT. In other words, new teachers in NYC had lower SAT scores than their colleagues outside of the city.

Similarly, the paper also finds that, prior to 2000, teachers whose first placements were in the poorest schools (lowest quintile) in New York State (most of them in the city) had SAT scores between one and four standard deviations below the statewide mean. The situation improved a great deal between 2000 and 2009, to the point where NYC teachers are now disproportionately drawn from the top third statewide (also see Boyd et al. 2008). Still, the fact that their average scores remain lower than the scores outside of the city is indicative, perhaps, of the recruitment problems faced by schools (and districts) serving larger populations of disadvantaged students. That is, to whatever degree there is an overall teacher recruitment problem, it is distributed unevenly, and the variation seems heavily influenced by student and school characteristics (which some teachers may view as a proxy for working conditions).

Fifth and most importantly, pre-service characteristics, including SAT scores, are limited as predictors of new teacher quality. In general, it is extremely difficult to predict who will become a skilled teacher based on pre-service characteristics, including cognitive tests, undergraduate GPA, selectivity of college/university attended, etc. (see this post for a review of this literature, including research on other pre-service measures). Accordingly, the relationship between SAT/ACT scores and (test-based) effectiveness is not especially strong (e.g., Harris and Sass 2011; Rockoff et al. 2011).

Granted, much of the empirical evidence addressing this issue defines teacher quality in terms of test-based productivity measures (e.g., math and reading value-added scores). In addition, SAT scores and other tests of cognitive skills are among the few characteristics that actually do provide at least some predictive power, and one must always evaluate measures against their alternatives. From this perspective, the use of pre-service characteristics, including SAT/ACT scores, is certainly worthwhile, both for gauging the quality of the labor supply and for hiring decisions (Clotfelter et al. 2007).

That said, in reality, the “best candidates” are frequently those who don’t attend the best universities, or receive the best grades, or score most highly on college entrance exams. This caveat seems often to get lost in the ubiquitous, high-profile use of the “bottom third” and similar talking points, and, sometimes, in the attempts to overhaul long-standing personnel policies solely to attract a particular conceptualization of the “best candidates.”

So, the finding that the SAT scores of New York teachers are improving most likely reflects improvement in the skills and abilities of the typical incoming teacher, and this is encouraging, but when it comes to predicting who will be a good teacher, humility is still the best mindset.