Education

When The Legend Becomes Fact, Print The Fact Sheet

The New Teacher Project (TNTP) just released a "fact sheet" on value-added (VA) analysis. I’m all for efforts to clarify complex topics such as VA, and, without question, there is a great deal of misinformation floating around on this subject, both "pro-" and "anti-."

The fact sheet presents five sets of “myths and facts." Three of the “myths” seem somewhat unnecessary: that there’s no research behind VA; that teachers will be evaluated based solely on test scores; and that VA is useless because it’s not perfect. Almost nobody believes or makes these arguments (at least in my experience). But I guess it never hurts to clarify.

In contrast, the other two are very common arguments, but they are not myths. They are serious issues with concrete policy implications. If there are any myths, they're in the "facts" column.

The False Conflict Between Unionism and Professionalism

Some people have the unfortunate idea that unionism is somehow antithetical to or incompatible with being a professional. This notion is particularly salient within education circles, where phrases like “treat teachers like professionals” are often used as implicit arguments against policies associated with unions, such as salary schedules and tenure (examples here, here, here and here).

Let’s take a quick look at this "conflict," first by examining union membership rates among professionals versus workers in other types of occupations. As shown in the graph below, if union membership and professionalism don’t mix, we have a little problem: Almost one in five professionals is a union member. Actually, union membership is higher among professionals than among any other major occupational category except construction workers.

Schedule Conflicts

As most people know, the majority of public school teachers are paid based on salary schedules. Most (but not all) contain a number of “steps” (years of experience) and “lanes” (education levels). Teachers are placed in one lane (based on their degree) and proceed up the steps as they accrue years on the job. Within most districts, these two factors determine the raises that teachers receive.

Salary schedules receive a great deal of attention in our education debates. One argument that has been making the rounds for some time is that we should attract and retain "talent" in the teaching profession by increasing starting salaries and/or the size of raises teachers receive during their first few years (when test-based productivity gains are largest). One common proposal (see here and here) for doing so is reallocating salary from the “top” of salary schedules (the salaries paid to more experienced teachers) down to the “bottom” (novice teachers’ salaries). As a highly simplified example, instead of paying starting teachers $40,000 and teachers with 15 years of experience $80,000, we could pay first-year teachers $50,000 and their experienced counterparts $70,000. This general idea is sometimes called “frontloading," as it concentrates salary expenditures at the “front” of schedules.

Now, there is a case for changes to salary schedules in many places – bargained and approved by teachers – including, perhaps, some degree of gradual frontloading (though the research in this area is underdeveloped at best). But there is a vocal group of advocates who assume an all-too-casual attitude about these changes. They seem to be operating on the mistaken assumption that salary schedules can be easily overhauled – just like that. We can drastically restructure them or just “move the money around” without problem or risk, if only unions and "bureaucrats" would get out of the way.**

The Impact Of The Principal In The Classroom

Direct observation is way of gathering data by watching behavior or events as they occur; for example, a teacher teaching a lesson. This methodology is important to teacher induction and professional development, as well as teacher evaluation. Yet, direct observation has a major shortcoming: it is a rather obtrusive data gathering technique. In other words, we know the observer can influence the situation and the behavior of those being observed. We also know people do not behave the same way when they know they are being watched. In psychology, these forms of reactivity are known as the Hawthorne effect, and the observer- or experimenter- expectancy effect (also here).

Social scientists and medical researchers are well aware of these issues and the fact that research findings don’t mean a whole lot when the researcher and/or the study participants know the purpose of the research and/or are aware that they are being observed or tested. To circumvent these obstacles, techniques like “mild deception” and “covert observation” are frequently used in social science research.

For example, experimenters often take advantage of “cover stories” which give subjects a sensible rationale for the research while preventing them from knowing (or guessing) the true goals of the study, which would threaten the experiment’s internal validity – see here. Also, researchers use double-blind designs, which, in the medical field, mean that neither the research participant nor the researcher know when the treatment or the placebo are being administered.

In Research, What Does A "Significant Effect" Mean?

If you follow education research – or quantitative work in any field – you’ll often hear the term “significant effect." For example, you will frequently read research papers saying that a given intervention, such as charter school attendance or participation in a tutoring program, had “significant effects," positive or negative, on achievement outcomes.

This term by itself is usually sufficient to get people who support the policy in question extremely excited, and to compel them to announce boldly that their policy “works." They’re often overinterpreting the results, but there’s a good reason for this. The problem is that “significant effect” is a statistical term, and it doesn’t always mean what it appears to mean. As most people understand the words, “significant effects” are often neither significant nor necessarily effects.

Let’s very quickly clear this up, one word at a time, working backwards.

NAEP Shifting

** Also posted here on “Valerie Strauss’ Answer Sheet” in the Washington Post

Tomorrow, the education world will get the results of the 2011 National Assessment of Educational Progress (NAEP), often referred to as the “nation’s report card." The findings – reading and math scores among a representative sample of fourth and eighth graders - will drive at least part of the debate for the next two years, when the next round comes out.

I’m going to make a prediction, one that is definitely a generalization, but is hardly uncommon in policy debates: People on all “sides” will interpret the results favorably no matter how they turn out.

If NAEP scores are positive – i.e., overall scores rise by a statistically significant margin, and/or there are encouraging increases among key subgroups such as low performers or low-income students – supporters of market-based reform will say that their preferred policies are working. They’ll claim that the era of test-based accountability, which began with the enactment of No Child Left Behind ten years ago, have produced real results. Market reform skeptics, on the other hand, will say that virtually none of the policies, such as test-based teacher evaluations and merit pay, for which reformers are pushing were in force in more than a handful of locations between 2009 and 2011. Therefore, they’ll claim, the NAEP progress shows that the system is working without these changes.

If the NAEP results are not encouraging – i.e., overall progress is flat (or negative), and there are no strong gains among key subgroups – the market-based crowd will use the occasion to argue that the “status quo” isn’t producing results, and they will strengthen their call for policies like new evaluations and merit pay. Skeptics, in contrast, will claim that NCLB and standardized test-based accountability were failures from the get-go. Some will even use the NAEP results to advocate for the wholesale elimination of standardized testing.

Similar Problems, Different Response: “We Are Public Education”

Thousands of people from all over Spain demonstrated Saturday October 22nd in Madrid against severe austerity measures affecting public education in several Spanish regions. The march on Madrid, which attracted more than 100,000 protesters – huge by Spanish standards – was jointly organized by national education unions and the national parents’ association, CEAPA. Taking part in the protest, a somewhat unprecedented coalition: educators, parents, and students.

The economy in Spain is in terrible shape. Parents and teachers don’t always have an ideal relationship, yet  Spaniards seem to have avoided the divisive and unproductive quarrels we often read about in the US education debate – e.g., adults versus children or teachers versus parents – in an attempt to prioritize long-term educational investment over short-term, budget-driven savings. This broad alliance is building consensus around the notion of “the education community." As the protest’s manifesto notes, such community is “society as a whole," which must unite to oppose drastic budget cuts in public education and attacks by political leaders on public school teachers.

The nationwide protest was triggered by a recent government decision that bans the temporary hiring of teachers as part of a plan to reduce government spending. In various parts of the country, teachers have already been laid off, class sizes and teaching hours have increased significantly, and teachers will have to teach subjects they are not specialized in. Many schools will have to reduce extra-curricular activities, remedial classes for struggling students and integration classes for the children of immigrants. This situation triggered a series of regional demonstrations across Spain throughout the months of September and October – including student demonstrations in defense of public education – with protesters arguing that education quality has been put at risk. National in scale, the march on Madrid sends a broader message, with the potential of immediate political impact.

The Ratings Game: New York City Edition

Gotham Schools reports that the New York City Department of Education rolled out this year’s school report card grades by highlighting the grades’ stability between this year and last. That is, they argued that schools’ grades were roughly the same between years, which is supposed to serve as evidence of the system’s quality.

The city’s logic here is generally sound. As I’ve noted before, most schools don’t undergo drastic changes in their operations over the course of a year, and so fluctuations in grades among a large number of schools might serve as a warning sign that there’s something wrong with the measures being used. Conversely, it’s not unreasonable to expect from a high-quality rating system that, over a two-year period, some schools would get higher grades and some lower, but that most would stay put. That was the city’s argument this year.

The only problem is that this wasn’t really the case.

Higher Education: The Great Equalizer Or Business As Usual?

** Also posted here on "Valerie Strauss' Answer Sheet" in the Washington Post

Several weeks ago, a survey of college admission directors and enrollment managers conducted by Inside Higher Education sparked considerable media coverage about an issue that is not entirely new: Money, not only merit, matters in college admissions.

According to the survey of 462 directors and managers, in the face of generalized budget cuts, universities are favoring applicants who don't need financial assistance to pay their tuition. About  22 percent agreed that “the financial downturn [had] forced them to pay more attention to an applicants’ ability to pay when [making] admissions decisions." Directors acknowledged seeking more candidates who would not need financial aid, including out-of-state and international students. Furthermore, 10 percent of four-year colleges reported that the admitted students who could pay in full had lower grades than their peers who couldn’t.

These findings resulted in headlines like “College Admission Directors Finally Admit They Want Rich Students More Than Smart Students” or “Universities Seeking Out Students of Means." Journalists wrote that higher education institutions are no longer in the business of recruiting “the best and brightest […] but the richest” and noted that “money is talking a bit louder in college admissions these days.” David A. Hawkins, director at the National Association for College Admission Counseling, told the New York Times: “As institutional pressures mount, between the decreased state funding, the pressure to raise a college’s profile, and the pressure to admit certain students, we’re seeing a fundamental change in the admissions process. Where many of the older admissions professionals came in through the institution and saw it as an ethically centered counseling role, there’s now a different dynamic that places a lot more emphasis on marketing."

The actual finding that money matters in admissions did not strike me as a particularly surprising. After all, social scientists have long been skeptical of meritocracy’s role in higher education – see here and here. What surprised me was that survey respondents were comfortable giving a “socially inappropriate” answer. Let me explain.

The Teachers' Union Hypothesis

For the past couple of months, Steve Brill's new book has served to step up the eternally-beneath-the-surface hypothesis that teachers’ unions are the primary obstacle to improving educational outcomes in the U.S. The general idea is that unions block “needed reforms," such as merit pay and other forms of test-based accountability for teachers, and that they “protect bad teachers” from being fired.

Teachers’ unions are a convenient target. For one thing, a significant proportion of Americans aren’t crazy about unions of any type. Moreover, portraying unions as the villain in the education reform drama facilitates the (mostly false) policy-based distinction between teachers and the organizations that represent them – put simply, “love teachers, hate their unions." Under the auspices of this dichotomy, people can advocate for changes , such as teacher-level personnel policies based partially on testing results, without having to address why most teachers oppose them (a badly needed conversation).

No, teachers’ unions aren’t perfect, because the teachers to whom they give voice aren’t perfect. There are literally thousands of unions, and, just like districts, legislatures and all other institutions, they make mistakes. But I believe strongly in separating opinion and anecdote from actual evidence, and the simple fact is that the pervasive argument that unions are a substantial cause of low student performance has a weak empirical basis, while the evidence that unions are a primary cause of low performance does not exist.