Data-Driven Decisions, No Data

According to an article in yesterday’s Washington Post, the outcome of the upcoming D.C. mayoral primary may depend in large part on gains in students’ “test scores” since Mayor Adrian Fenty appointed Michelle Rhee to serve as chancellor of the D.C. Public Schools (DCPS).

That struck me as particularly interesting because, as far as I can tell, Michelle Rhee has never released any test scores to the public. Not an average test score for any grade level or for any of the district’s schools or any subgroup of its students. None.

DCPS only announces the percentage of students (by school and subgroups) whose scores fall within the four categories of below basic, basic, proficient, and advanced. As I discussed in a previous post, these numbers are not test scores – they are one big step removed – and year-to-year changes in these rates don’t necessarily tell us very much about the typical DCPS student. For one thing, none of these changes, whether scores or rates, are actually gains per se, they are cohort-to-cohort changes (e.g., the rates of third graders one year are compared with rate of a different group of third graders the next year). Moreover, as that post explained, it’s possible for a school’s proficiency rate to increase even when there is a decline in average test scores across grades (and vice-versa). Consequently, changes in proficiency rates are a very poor measure of student progress.

The week after that previous post went up, New York State illustrated the point in an unusual, rather dramatic fashion. The state revised the "cut point" scores above which students are called basic, proficient, and advanced. Literally overnight, proficiency rates dropped dramatically. In New York City, we found out that the “proficiency gap” between white and minority students hadn't narrowed anywhere near as much as we had previously thought – actually, not at all. The proficiency rates for poor and special education students saw particularly steep declines, as did the city's charter schools. And all of this without a single score moving an inch.

New York, however, was not left in the dark. Like virtually all states, it releases average test scores along with proficiency rates. This means that parents, researchers, and the public can make a more accurate assessment of cohort-to-cohort progress by checking both the changes in rates as well as the changes in scores, which are a better (though still limited) measure of progress.

As the New York situation illustrates, changes in proficiency or other benchmark rates are, by themselves, not only a poor way to gauge performance over time, they are potentially very misleading.

Yet, in DC, the rates are all we are given. For all we know, average test scores in the district have been flat for years, even though proficiency rates were rising until this year. It’s also entirely possible that the proficiency rate increases in D.C. over the past few years mask far more modest growth in test scores – or that this year’s proficiency decreases mask rising average scores. The rates can change a great deal even if scores are only marginally different between two cohorts of students.

The significant increases in DC’s NAEP scores between 2003 and 2009 (in fourth grade math and reading and eighth grade math) only provide information about a tiny slice of DCPS students (two grades) every other year – that’s why DCPS’ own results are so important. Even if they don’t track individual students over time (NAEP doesn’t either), they at least tell us – every year – how the typical student in each grade (or the typical student at each school) tested, and how this compares with previous cohorts over time.

The point is that we don’t really have that much reliable information about how well D.C. students are doing, because average test score data aren’t publicly available (at least not that I could find, at either DCPS or the Office of the State Superintendent of Education).

So, while education certainly appears to be one of the most important issues in this year’s Democratic Party primary, any vote based on actual test scores will have to be cast by absentee ballot.