• The Education Reporter's Dilemma

    I’ve written so many posts about the misinterpretation of testing data in news stories that I’m starting to annoy myself. For example, I’ve shown that year-to-year changes in testing results might be attributable to the fact that, each year, a different set of students takes the test. I’ve discussed the fact that proficiency rates are not test scores – they only tell you the proportion of students above a given line – and that the rates and actual scores can move in opposite directions (see this simple illustration). And I’ve pleaded with journalists, most of whom I like and respect, to write with care about these issues (and, I should note, many of them do so).

    Yet here I am, back on my soapbox again. This time the culprit is the recent release of SAT testing data, generating dozens of error-plagued stories from newspapers and organizations. Like virtually all public testing data, the SAT results are cross-sectional – each year, the test is taken by a different group of students. This means that demographic changes in the sample of test takers influence the results. This problem is even more acute in the case of the SAT, since it is voluntary. Despite the best efforts of the College Board (see their press release), a slew of stories improperly equated the decline in average SAT scores since the previous year with an overall decline in student performance – a confirmation of educational malaise (in fairness, there were many exceptions).

    I’ve come to think that there’s a fundamental problem here: When you interpret testing data properly, you don’t have much of a story.

  • Public Servants Versus Private Contractors ... Again

    Has the battle over public sector compensation turned a decisive corner? Have much-maligned government workers won an evidence-based victory?

    Reasonable people might think so, thanks in part to a study by the Project on Government Oversight (POGO), a nonpartisan group that keeps close tabs on government operations. According to the findings of the POGO report – findings that they call "shocking" – the "federal government approves service contract billing rates … that pay contractors 1.83 times more than the government pays federal employees in total compensation, and more than 2 times the total compensation paid in the private sector for comparable services."

    More specifically, federal government employees cost less than private contractors in 33 of the 35 occupational classifications reviewed – and non-federal private sector worker compensation was lower than contractor billing rates in all of the reviewed classifications. In one case, contractor bill rates were nearly "5 times more" than the full compensation rates paid to comparable federal workers.

  • What Are "Middle Class Schools"?

    An organization called “The Third Way” released a report last week, in which they present descriptive data on what they call “middle class schools." The primary conclusion of their analysis is that “middle class schools” aren’t “making the grade," and that they are “falling short on their most basic 21st century mission: To prepare kids to get a college degree." They also argue that “middle class schools” are largely ignored in our debate and policymaking, and we need a “second phase of school reform” in order to address this deficit.

    The Wall Street Journal swallowed the report whole, running a story presenting Third Way’s findings under the headline “Middle class schools fail to make the grade."

    To be clear, I think that our education policy debates do focus on lower-income schools to a degree that sometimes ignores those closer to the middle of the distribution. So, it’s definitely worthwhile to take a look at “middle class schools’” performance and how it can be improved. In other words, I’m very receptive to the underlying purpose of the report.

    That said, this analysis consists mostly of arbitrary measurement and flawed, vague interpretations. As a result, it actually offers little meaningful insight.

  • Be Careful What You Think The Public Thinks About Tenure

    Monmouth University polling director Patrick Murray offered this response to my criticism of how he described tenure in a recent poll of New Jersey public opinion (see my brief reply and Bruce Baker's as well).

    I’m not particularly keen on arguing about the wording of poll questions. As I stated in my original post, wordings are never perfect, and one must always take this into account when trying to interpret polling results. I took issue with Monmouth’s phrasing because it is demonstrably inaccurate, nothing more.

    But I do want to quickly add that, as is often the case, responses to poll questions about tenure are very sensitive to the nature of the description offered. A 2009 PDK/Gallup poll provides an illustration.

  • Collective Bargaining Teaches Democratic Values, Activism

    Some people must have been startled by President Obama’s decision to draw a line in the sand on collective bargaining in his jobs speech to the Congress last week. Specifically, the President said: “I reject the idea that we have to strip away collective bargaining rights to compete in a global economy."

    Given the current anti-union tenor of many prominent Republicans, started by Wisconsin Governor Scott Walker, it seems pretty clear that worker rights is shaping up to be a hot-button issue in the 2012 campaign. Collective bargaining rights as presidential campaign plank? It wasn’t that long ago that anything to do with unions was considered to be an historic anachronism – hardly worth a major Republican presidential candidate’s trouble to bash. Times have changed.

  • Serious Misconduct

    A recent Monmouth University poll of New Jersey residents is being widely touted by Governor Chris Christie and his supporters as evidence that people support his education reform plans. It’s hardly unusual for politicians to ignore the limitations of polling, but I’d urge caution in interpreting these results as a mandate.

    Others have commented on how some of the questions are worded in a manner that could skew responses. These wording issues are inherent to polling, and that’s one of the major reasons why they must be interpreted carefully. But one of the questions caught my eye – the question about teacher tenure – and it’s worth quickly discussing.

  • Revisiting The Merits Of Merit Pay

    Al Shanker was very concerned about the need to identify and replace incompetent teachers. The first time he wrote a column about it, his wife was one of the many people who warned him that the union’s teachers would be up in arms (see here). Shanker wasn’t worried, replying that "All of my members will read that, and they’ll all agree, because not one of them will think that they are one of the bad teachers that I’m talking about."

    He was right. Most of the members were very supportive, probably for a variety of reasons. First, most teachers take their responsibilities as teachers very seriously, thus favoring the establishment and enforcement of high standards of professional practice. Second, teachers who don’t believe themselves to be effective are more likely to leave the profession – see here. And third, we know from research that most of us just believe that we are simply better than most other people. Psychologists describe this "illusory superiority" or "above average" effect as the tendency to make self-serving comparisons between oneself and others, with the consequence that an overwhelming majority of people judge themselves to be "better than average" on a variety of traits, skills, and socially desirable dimensions ( here and here).

    Nevertheless, there are many teachers who support the idea of performance pay, even if they're wary of the details of how "merit" is defined (specifically, whether or not it includes test scores).

    Now, it’s no secret that I think merit pay for teachers is of limited practical utility. Nevertheless, it’s important to understand why, evidence aside, some people (including teachers) might find the policy to be attractive. These are my thoughts on the issue:

  • The Real Charter School Experiment

    The New York Times reports that there is a pilot program in Houston, called the "Apollo 20 Program" in which some of the district’s regular public schools are "mimicking" the practices of high-performing charter schools. According to the Times article, the group of pilot schools seek to replicate five of the practices commonly used by high-flying charters: extended school time; extensive tutoring; more selective hiring of principals and teachers; “data-driven” instruction, including frequent diagnostic quizzing; and a “no excuses” culture of high expectations.

    In theory, this pilot program is a good idea, since a primary mission of charter schools should be as a testing ground for new policies and practices that could help to improve all schools. More than a decade of evidence has made it very clear that there’s nothing about "charterness" that makes a school successful – and indeed, only a handful get excellent results. So instead of arguing along the tired old pro-/anti-charter lines, we should, like Houston, be asking why these schools excel and working to see if we can use this information productively.

    I’ll be watching to see how the pilot schools end up doing. I’m also hoping that the analysis (the program is being overseen by Harvard’s EdLabs) includes some effort to separate out the effects of each of the five replicated practices. If so, I’m guessing that we will find that the difference between high- and low-performing urban schools depends more than anything else on two factors: time and money.

  • In Ohio, Charter School Expansion By Income, Not Performance

    For over a decade, Ohio law has dictated where charter schools can open. Expansion was unlimited in Lucas County (the “pilot district” for charters) and in the “Ohio 8” urban districts (Akron, Canton, Cincinnati, Cleveland, Columbus, Dayton, Toledo, and Youngstown). But, in any given year, charters could open up in any other district that was classified as a “challenged district," as measured by whether the district received a state “report card” rating of “academic watch” or “academic emergency." This is a performance-based standard.

    Under this system, there was of course very rapid charter proliferation in Lucas County and the “Ohio 8” districts. Only a small number of other districts (around 20-30 per year) “met” the  performance-based standard. As a whole, the state’s current charter law was supposed to “open up” districts for charter schools when the districts are not doing well.

    Starting next year, the state is adding a fourth criterion: Any district with a “performance index” in the bottom five percent for the state will also be open for charter expansion. Although this may seem like a logical addition, in reality, the change offends basic principles of both fairness and educational measurement.

  • How Cross-Sectional Are Cross-Sectional Testing Data?

    In several posts, I’ve complained about how, in our public discourse, we misinterpret changes in proficiency rates (or actual test scores) as “gains” or “progress," when they actually represent cohort changes—that is, they are performance snapshots for different groups of students who are potentially quite dissimilar.

    For example, the most common way testing results are presented in news coverage and press releases is to present year-to-year testing results across entire schools or districts – e.g., the overall proficiency rate across all grades in one year compared with the next. One reason why the two groups of students being compared (the first versus the second year) are different is obvious. In most districts, tests are only administered to students in grades 3-8. As a result, the eighth graders who take the test in Year 1 will not take it in Year 2, as they will have moved on to the ninth grade (unless they are retained). At the same time, a new cohort of third graders will take the test in Year 2 despite not having been tested in Year 1 (because they were in second grade). That’s a large amount of inherent “turnover” between years (this same situation applies when results are averaged for elementary and secondary grades). Variations in cohort performance can generate the illusion of "real" change in performance, positive or negative.

    But there’s another big cause of incomparability between years: Student mobility. Students move in and out of districts every year. In urban areas, mobility is particularly high. And, in many places, this mobility includes students who move to charter schools, which are often run as separate school districts.

    I think we all know intuitively about these issues, but I’m not sure many people realize just how different the group of tested students across an entire district can be in one year compared with the next. In order to give an idea of this magnitude, we might do a rough calculation for the District of Columbia Public Schools (DCPS).