Paper Summary

Ready, Aim, Hire: Predicting The Future Performance Of Teacher Candidates

In a previous post, I discussed the idea of “attracting the best candidates” to teaching by reviewing the research on the association between pre-service characteristics and future performance (usually defined in terms of teachers’ estimated effect on test scores once they get into the classroom). In general, this body of work indicates that, while far from futile, it’s extremely difficult to predict who will be an “effective” teacher based on their paper traits, including those that are typically used to define “top candidates," such as the selectivity of the undergraduate institutions they attend, certification test scores and GPA (see here, here, here and here, for examples).

There is some very limited evidence that other, “non-traditional” measures might help. For example, a working paper, released last year, found a statistically discernible, fairly strong association between first-year math value-added and an index constructed from surveys administered to Teach for America candidates. There was, however, no association in reading (note that the sample was small), and no relationships in either subject found during these teachers’ second years.*

A recently-published paper – which appears in the peer-reviewed journal Education Finance and Policy, originally released as working paper in 2008 –  represents another step forward in this area. The analysis, presented by the respected quartet of Jonah Rockoff, Brian Jacob, Thomas Kane, and Douglas Staiger (RJKS), attempts to look beyond the set of characteristics that researchers are typically constrained (by data availability) to examine.

In short, the results do reveal some meaningful, potentially policy-relevant associations between pre-service characteristics and future outcomes. From a more general perspective, however, they are also a testament to the difficulties inherent in predicting who will be a good teacher based on observable traits.

A Look Inside Principals' Decisions To Dismiss Teachers

Despite all the heated talk about how to identify and dismiss low-performing teachers, there’s relatively little research on how administrators choose whom to dismiss, whether various dismissal options might actually serve to improve performance, and other aspects in this area. A paper by economist Brian Jacob, released as working paper in 2010 and published late last year in the journal Education Evaluation and Policy Analysis, helps address at least one of these voids, by providing one of the few recent glimpses into administrators’ actual dismissal decisions.

Jacob exploits a change in Chicago Public Schools (CPS) personnel policy that took effect for the 2004-05 school year, one which strengthened principals’ ability to dismiss probationary teachers, allowing non-renewal for any reason, with minimal documentation. He was able to link these personnel records to student test scores, teacher and school characteristics and other variables, in order to examine the characteristics that principals might be considering, directly or indirectly, in deciding who would and would not be dismissed.

Jacob’s findings are intriguing, suggesting a more complicated situation than is sometimes acknowledged in the ongoing debate over teacher dismissal policy.

Fundamental Flaws In The IFF Report On D.C. Schools

A new report, commissioned by the District of Columbia Mayor Vincent Gray and conducted by the Chicago-based consulting organization IFF, was supposed to provide guidance on how the District might act and invest strategically in school improvement, including optimizing the distribution of students across schools, many of which are either over- or under-enrolled.

Needless to say, this is a monumental task. Not only does it entail the identification of high- and low-performing schools, but plans for improving them as well. Even the most rigorous efforts to achieve these goals, especially in a large city like D.C., would be to some degree speculative and error-prone.

This is not a rigorous effort. IFF’s final report is polished and attractive, with lovely maps and color-coded tables presenting a lot of summary statistics. But there’s no emperor underneath those clothes. The report's data and analysis are so deeply flawed that its (rather non-specific) recommendations should not be taken seriously.

The Persistence Of Both Teacher Effects And Misinterpretations Of Research About Them

In a new National Bureau of Economic Research working paper on teacher value-added, researchers Raj Chetty, John Friedman and Jonah Rockoff present results from their analysis of an incredibly detailed dataset linking teachers and students in one large urban school district. The data include students’ testing results between 1991 and 2009, as well as proxies for future student outcomes, mostly from tax records, including college attendance (whether they were reported to have paid tuition or received scholarships), childbearing (whether they claimed dependents) and eventual earnings (as reported on the returns). Needless to say, the actual analysis includes only those students for whom testing data were available, and who could be successfully linked with teachers (with the latter group of course limited to those teaching math or reading in grades 4-8).

The paper caused a remarkable stir last week, and for good reason: It’s one of the most dense, important and interesting analyses on this topic in a very long time. Much of the reaction, however, was less than cautious, specifically the manner in which the research findings were interpreted to support actual policy implications (also see Bruce Baker’s excellent post).

What this paper shows – using an extremely detailed dataset and sophisticated, thoroughly-documented methods – is that teachers matter, perhaps in ways that some didn’t realize. What it does not show is how to measure and improve teacher quality, which are still open questions. This is a crucial distinction, one which has been discussed on this blog numerous times (also here and here), as it is frequently obscured or outright ignored in discussions of how research findings should inform concrete education policy.

What Are "Middle Class Schools"?

An organization called “The Third Way” released a report last week, in which they present descriptive data on what they call “middle class schools." The primary conclusion of their analysis is that “middle class schools” aren’t “making the grade," and that they are “falling short on their most basic 21st century mission: To prepare kids to get a college degree." They also argue that “middle class schools” are largely ignored in our debate and policymaking, and we need a “second phase of school reform” in order to address this deficit.

The Wall Street Journal swallowed the report whole, running a story presenting Third Way’s findings under the headline “Middle class schools fail to make the grade."

To be clear, I think that our education policy debates do focus on lower-income schools to a degree that sometimes ignores those closer to the middle of the distribution. So, it’s definitely worthwhile to take a look at “middle class schools’” performance and how it can be improved. In other words, I’m very receptive to the underlying purpose of the report.

That said, this analysis consists mostly of arbitrary measurement and flawed, vague interpretations. As a result, it actually offers little meaningful insight.

Investment Counselors

NOTE: With this post, we are starting a new “feature” here at Shanker Blog – periodically summarizing research papers that carry interesting and/or important implications for the education policy debates. We intend to focus on papers that are either published in peer-reviewed journals or are still in working paper form, and are unlikely to get significant notice. Here is the first:

Are School Counselors a Cost-Effective Education Input?

Scott E. Carrell and Mark Hoekstra, Working paper (link to PDF), September 2010

Most teachers and principals will tell you that non-instructional school staff can make a big difference in school performance. Although we may all know this, it’s always useful to have empirical research to confirm it, and to examine the size and nature of the effects. In this paper, economists Scott Carrell and Mark Hoekstra put forth one of the first rigorous tests of how one particular group of employees – school counselors – affect both discipline and achievement outcomes. The authors use a unique administrative dataset of third, fourth, and fifth graders in Alachua County, Florida, a diverse district that serves over 30,000 students.  Their approach exploits year-to-year variation in the number of counselors in each school – i.e., whether the outcomes of a given school change from the previous year when a counselor is added to the staff.

Their results are pretty striking: The addition of a single full-time counselor is associated with a 0.04 standard deviation increase in boys’ achievement (about 1.2 percentile points). These effects are robust across different specifications (including sibling and student fixed effects). The disciplinary effects are, as expected, even more impressive. A single additional counselor helps to decrease boys’ disciplinary infractions between 15 to 26 percent.