Students First, Facts Later

On Wednesday, Michelle Rhee’s new organization, Students First, rolled out its first big policy campaign: It’s called “Save Great Teachers," and it is focused on ending so-called “seniority-based layoffs."

Rhee made several assertions at the initial press conference and in an accompanying op-ed in the Atlanta Constitution Journal (and one on CNN.com). At least three of these claims address the empirical research on teacher layoffs and quality. Two are false; the other is misleading. If history is any guide, she is certain to repeat these “findings” many times in the coming months.

As discussed in a previous post, I actually support the development of a better alternative to seniority-based layoffs, but I am concerned that the debate is proceeding as if we already have one (most places don't), and that there's quite a bit of outrage-inspiring misinformation flying around on this topic. So, in the interest of keeping the discussion honest, as well as highlighting a few issues that bear on the layoff debate generally, I do want to try and correct Rhee preemptively.

First, she claims that seniority-based layoffs hurt the least advantaged students most (she made this assertion at the press conference, in the op-ed, and in a short video on her website). This is based on one very rough previous calculation (perhaps this one too), as well as on inference from the fact that high-poverty schools (or those with more minority students) tend to have less experienced teachers. But, although layoff policies vary from district to district, none really proceeds based on seniority alone. Most take many factors into account, including need. As a result, the two best layoff analyses available - one using data from New York City and the other from Washington State, both sophisticated and thorough - find that layoffs are dispersed relatively evenly between higher- and lower-poverty/minority schools (see here for more Washington results). There is little disparate impact. This is not the final word, of course, but according to the best available evidence, Rhee's claim is not correct: Layoffs hurt all students.

Second, probably referring to a study by the same author as the one she uses for her disparate impact claim, Rhee goes on to assert that, compared with layoffs based on "performance," seniority-based layoffs would require the dismissal of 33 percent more teachers to achieve the same budgetary savings. (she only used the 33 percent figure at the press conference, while making the claim generically [no numbers] in her op-ed and video.)

This, too, is incorrect. It’s true that a layoff that is based solely or mostly on seniority is likely to require more layoffs, to make up the same budget shortfall, than one using some other criteria. But the analysis Rhee uses to quantify the extent of those “savings” is completely inappropriate for this purpose, and by design overstates the difference. It compares a seniority-based layoff with a hypothetical scenario in which teachers at all levels of experience are equally likely to be fired (i.e., a “seniority-neutral layoff”). This is, essentially, a random layoff, which means that Rhee’s “evidence” of how many jobs would be saved by a performance-based layoff does not actually simulate a performance-based layoff of any kind.

There is, in reality, no way to know how a “quality-based” layoff would actually play out under a realistic alternative system in which both tested and non-tested teachers were fired based on multiple measures of performance and need. It would almost certainly result in fewer layoffs than a largely seniority-based system, but the size of this difference would, in large part, depend on how one measured performance.

Still, even by the narrow measure of test score growth, there is every reason to believe that the least senior teachers would be, on average, the least effective (i.e., more senior teachers would be retained than in a random layoff), and, as mentioned above, actual layoffs are almost never based solely on seniority (i.e., fewer brand new teachers would be fired than in a "purely" seniority-based layoff). This means that this particular analysis overstates the “savings," probably by a very wide margin. The last thing we need in a budget crisis is this type of exaggeration, and the "33 percent" claim is a careless use of evidence.

Finally, Rhee’s third empirical assertion, although not incorrect, is rather misleading. Here it is, as stated in the op-ed:

Rules that mandate layoffs by seniority instead of quality do incredible damage to children and schools. An ineffective teacher generates only half the learning that an effective teacher does. Conversely, a highly effective teacher generates 50 percent more learning than an average teacher. This means that kids learn three times more in a highly effective teacher’s classroom than in an ineffective teacher’s.

Basically, all the "50 percent" finding shows is that teacher quality - as measured by test score gains - varies. Well, yes. But it doesn't follow that seniority-based layoffs would “do incredible damage” by retaining all the “ineffective” teachers, As discussed above, seniority bears a strong relationship to test-based effectiveness during a teacher's first few years.

Perhaps more importantly, the statement implies that effective/average teachers would be retained (since that's how they're selected for retention). This not only assumes that value-added is accurate in any given year (it is not), and is a sufficient measure of “quality," but it also ignores the fact that the “benefits” of layoffs based on value-added decline rather rapidly over time, because the estimates are unstable (as the NYC analysis found). A teacher that produces 50 percent more “learning” than average in one year might only produce 20 percent more the next; she might very well be below average, especially if she’s a newer teacher with a small sample of students (which would make her first-year estimate more error-prone). Overall, without context or elaboration, Rhee's statement is a non-sequitur of sorts, an inappropriate juxtaposition of findings and a policy position.

For the record, I don’t think that Michelle Rhee is “lying” by her misrepresentation of these findings - I just don’t think she’s very careful about education research. In general, we should all support the use of empirical evidence to guide policy decisions, so long as those who use it are diligent about keeping up with the work and presenting it appropriately. Michelle Rhee is running a national campaign, using other people’s money. To kick off that campaign with inaccuracies and exaggerations does a disservice to her to supporters and opponents alike, and I hope she corrects these errors going forward.