Shanker Blog

Creating A Valid Process For Using Teacher Value-Added Measures

** Reprinted here in the Washington Post

Our guest author today is Douglas N. Harris, associate professor of economics and University Endowed Chair in Public Education at Tulane University in New Orleans. His latest book, Value-Added Measures in Education, provides an excellent, accessible review of the technical and practical issues surrounding these models. 

In particular, how should “value-added” measures be used in teacher evaluation? President Obama’s Race to the Top initiative expanded the use of these measures, which attempt to identify how much each teacher contributes to student test scores. In doing so, the initiative embraced and expanded the controversial reliance on standardized tests that started under President Bush’s No Child Left Behind.

In many respects, The Race was well designed. It addresses an important problem - the vast majority of teachers report receiving limited quality feedback on instruction. As a competitive grants program, it was voluntary for states to participate (though involuntary for many districts within those states). The Administration also smartly embraced the idea of multiple measures of teacher performance.

But they also made one decision that I think was a mistake.  They encouraged—or required, depending on your vantage point—states to lump value-added or other growth model estimates together with other measures. The raging debate since then has been over what percentage of teachers’ final ratings should be given to value-added versus the other measures. I believe there is a better way to approach this issue, one that focuses on teacher evaluations not as a measure, but rather as a process.

A Case Against Assigning Single Ratings To Schools

The new breed of school rating systems, some of which are still getting off the ground, will co-exist with federal proficiency targets in many states, and they are (or will be) used for a variety of purposes, including closure, resource allocation and informing parents and the public (see our posts on the systems in IN, FL, OH, CO, NYC).*

The approach that most states are using, in part due to the "ESEA flexibility" guidelines set by the U.S. Department of Education, is to combine different types of measures, often very crudely, into a single grade or categorical rating for each school. Administrators and media coverage usually characterize these ratings as measures of school performance - low-rated schools are called "low performing," while those receiving top ratings are characterized as "high performing." That's not accurate - or, at best, it's only partially true.

Some of the indicators that comprise the ratings, such as proficiency rates, are best interpreted as (imperfectly) describing student performance on tests, whereas other measures, such as growth model estimates, make some attempt to isolate schools’ contribution to that performance. Both might have a role to play in accountability systems, but they're more or less appropriate depending on how you’re trying to use them.

So, here’s my question: Why do we insist on throwing them all together into a single rating for each school? To illustrate why I think this question needs to be addressed, let’s take a quick look at four highly-simplified situations in which one might use ratings.

When You Hear Claims That Policies Are Working, Read The Fine Print

When I point out that raw changes in state proficiency rates or NAEP scores are not valid evidence that a policy or set of policies is “working," I often get the following response: “Oh Matt, we can’t have a randomized trial or peer-reviewed article for everything. We have to make decisions and conclusions based on imperfect information sometimes."

This statement is obviously true. In this case, however, it's also a straw man. There’s a huge middle ground between the highest-quality research and the kind of speculation that often drives our education debate. I’m not saying we always need experiments or highly complex analyses to guide policy decisions (though, in general, these are always preferred and sometimes required). The point, rather, is that we shouldn’t draw conclusions based on evidence that doesn't support those conclusions.

This, unfortunately, happens all the time. In fact, many of the more prominent advocates in education today make their cases based largely on raw changes in outcomes immediately after (or sometimes even before) their preferred policies were implemented (also see here, here, here, here, here, and here). In order to illustrate the monumental assumptions upon which these and similar claims ride, I thought it might be fun to break them down quickly, in a highly simplified fashion. So, here are the four “requirements” that must be met in order to attribute raw test score changes to a specific policy (note that most of this can be applied not only to claims that policies are working, but also to claims that they're not working because scores or rates are flat):

Are Stereotypes About Career And Technical Education Crumbling?

The stereotypes, bias, and misunderstanding that have for many decades surrounded and isolated Career and Technical Education (CTE) may slowly be crumbling.  A recent report by the National Research Center for Career and Technical Education (NRCCTE) argues that traditional CTE typology -- the way in which CTE students are identified and classified -- is obsolete.  The distinctions between “CTE” students and “academic” students are no longer useful. Today, nearly all high school students, including the highest achieving academic-track students, enroll in some CTE courses.

Moreover, a significant number of students complete “high intensity” CTE courses as well as academic courses, in patterns that cross SES lines. In order to understand the contemporary high school experience, these researchers argue, we need a new typology based on the reality of today’s classroom, students, and curricula.

The October 2012 study, “A Typology for Understanding the Career and Technical Education Credit-taking Experience of High School Students," proposes a new, more nuanced classification system --  one the authors believe would more accurately capture the high school experience and needs of today’s students. The researchers argue that these long-overdue changes could alter experts’ views of what students actually study in high school, break down the obsolete conceptual barriers that currently divide CTE and academic curricula, and help educators work with students to devise the most appropriate pathways to academic and career success.

Value-Added, For The Record

People often ask me for my “bottom line” on using value-added (or other growth model) estimates in teacher evaluations. I’ve written on this topic many times, and while I have in fact given my overall opinion a couple of times, I have avoided expressing it in a strong “yes or no” format. There's a reason for this, and I thought maybe I would write a short piece and explain myself.

My first reaction to the queries about where I stand on value-added is a shot of appreciation that people are interested in my views, followed quickly by an acute rush of humility and reticence. I know think tank people aren’t supposed to say things like this, but when it comes to sweeping, big picture conclusions about the design of new evaluations, I’m not sure my personal opinion is particularly important.

Frankly, given the importance of how people on the ground respond to these types of policies, as well as, of course, their knowledge of how schools operate, I would be more interested in the views of experienced, well-informed teachers and administrators than my own. And I am frequently taken aback by the unadulterated certainty I hear coming from advocates and others about this completely untested policy. That’s why I tend to focus on aspects such as design details and explaining the research – these are things I feel qualified to discuss.  (I also, by the way, acknowledge that it’s very easy for me to play armchair policy general when it's not my job or working conditions that might be on the line.)

That said, here’s my general viewpoint, in two parts. First, my sense, based on the available evidence, is that value-added should be given a try in new teacher evaluations.

Describing, Explaining And Affecting Teacher Retention In D.C.

The New Teacher Project (TNTP) has released a new report on teacher retention in D.C. Public Schools (DCPS). It is a spinoff of their “The Irreplaceables” report, which was released a few months ago, and which is discussed in this post. The four (unnamed) districts from that report are also used in this one, and their results are compared with those from DCPS.

I want to look quickly at this new supplemental analysis, not to rehash the issues I raised about“The Irreplaceables," but rather because of DCPS’s potential importance as a field test site for a host of policy reform ideas – indeed, the majority of core market-based reform policies have been in place in D.C. for several years, including teacher evaluations in which test-based measures are the dominant component, automatic dismissals based on those ratings, large performance bonuses, mutual consent for excessed teachers and a huge charter sector. There are many people itching to render a sweeping verdict, positive or negative, on these reforms, most often based on pre-existing beliefs, rather than solid evidence.

Although I will take issue with a couple of the conclusions offered in this report, I'm not going to review it systematically. I think research on retention is important, and it’s difficult to produce reports with original analysis, while very easy to pick them apart. Instead, I’m going to list a couple of findings in the report that I think are worth examining, mostly because they speak to larger issues.

The Educational Attainment Of Girls And Boys: Two Sides of the Same Coin

Last month, Malala Yousafzai, a 14-year-old Pakistani girl, was shot in the head, in an attempted assassination by Taliban militants. Her “crime” was daring to advocate for girls’ education. In a New York Times column, Nicholas Kristof observes that we in the West find it “easy to dismiss such incidents as distant barbarities," and uses the example of sex trafficking to illustrate that we “have a blind spot for our own injustices." I agree. However, I am not sure we need to go so far to find domestic injustices.

How about a close look within this very area: The education of girls (and boys) in the U.S.? Stories about how girls have surpassed boys in educational attainment have become common, and are often linked to statements about how boys are forgotten and/or lost. This rhetoric is troubling for several reasons. First, it can be read to imply a zero-sum equation; that is, that the educational advancement of girls is the cause of boys’ educational neglect. Second, stories about girls’ “successes” and boys’ “failures” may obscure more than they reveal.

There are the "lost boys" of higher education and the "missing girls" of STEM. We worry about boys and reading and girls and math. Recurring questions include where are the women in technology? Or, are there enough novels that cater to boys? Women have sailed past men in obtaining college degrees but, importantly, continue to concentrate in different fields and need Ph.D.s to match men with bachelor’s in the workplace.

When issues are addressed in this fragmented manner, it’s hard to tell if it’s girls or boys that we should be worrying about. Well, both and neither. What all these pieces of the puzzle really say is that – at least in this day, age, and nation – gender still matters.

New Books For The Youngest Victims Of Hurricane Sandy

Citizens across the United States have been deeply moved by the scenes of devastation wrought by Hurricane Sandy. At the Albert Shanker Institute, we offer our condolences to those who have lost loved ones, and to the thousands upon thousands who have lost their homes.

Educators and those who care about our public schools can make a special contribution to the recovery efforts. One of the great losses in the flooding that came with Hurricane Sandy was books. The Albert Shanker Institute is partnering with the organization First Book in a drive to replenish school, classroom, and home libraries that were destroyed. We are asking our friends and fellow educators to join in this campaign: your help will ensure that children in need will have new books -- stories at bedtime, the chance to be transported to another world, and the opportunity to return to normalcy.

As a result of contributions from the Albert Shanker Institute, the AFT, and other First Book partners, your contribution to this drive will be matched, dollar for dollar.

Annual Measurable Objections

As states’ continue to finalize their applications for ESEA/NCLB “flexibility” (or “waivers”), controversy has arisen in some places over how these plans set proficiency goals, both overall and for demographic subgroups (see our previous post about the situation in Virginia).

One of the underlying rationales for allowing states to establish new targets (called “annual measurable objectives," or AMOs) is that the “100 percent” proficiency goals of NCLB were unrealistic. Accordingly, some (but not all) of the new plans have set 2017-18 absolute proficiency goals that are considerably below 100 percent, and/or lower for some subgroups relative to others. This shift has generated pushback from advocates, most recently in Florida, who believe that lowering state targets is tantamount to encouraging or accepting failure.

I acknowledge the central role of goals in any accountability system, but I would like to humbly suggest that this controversy, over where and how states set proficiency targets for 2017-18, may be misguided. There are four reasons why I think this is the case (and one silver lining if it is).

The Structural Curve In Indiana's New School Grading System

The State of Indiana has received a great deal of attention for its education reform efforts, and they recently announced the details, as well as the first round of results, of their new "A-F" school grading system. As in many other states, for elementary and middle schools, the grades are based entirely on math and reading test scores.

It is probably the most rudimentary scoring system I've seen yet - almost painfully so. Such simplicity carries both potential advantages (easier for stakeholders to understand) and disadvantages (school performance is complex and not always amenable to rudimentary calculation).

In addition, unlike the other systems that I have reviewed here, this one does not rely on explicit “weights," (i.e., specific percentages are not assigned to each component). Rather, there’s a rubric that combines absolute performance (passage rates) and proportions drawn from growth models (a few other states use similar schemes, but I haven't reviewed any of them).

On the whole, though, it's a somewhat simplistic variation on the general approach most other states are taking -- but with a few twists.