Education Research

New Policy Brief: The Evidence On The Florida Education Reform Formula

The State of Florida is well known in the U.S. as a hotbed of education reform. The package of policies spearheaded by then Governor Jeb Bush during the late 1990s and early 2000s focused, in general, on test-based accountability, competition, and choice. As a whole, they have come to be known as the “Florida Formula for education success,” or simply the “Florida Formula.”

The Formula has received a great deal of attention, including a coordinated campaign to advocate (in some cases, successfully) for its export to other states. The campaign and its supporters tend to employ as their evidence changes in aggregate testing results, most notably unadjusted increases in proficiency rates on Florida’s state assessment and/or cohort changes on the National Assessment of Educational Progress. This approach, for reasons discussed in the policy brief, violates basic principles of causal inference and policy evaluation. Using this method, one could provide evidence that virtually any policy or set of policies “worked” or “didn’t work,” often in the same place and time period.

Fortunately, we needn’t rely on these crude methods, as there is quite a bit of high quality evidence pertaining to several key components of the Formula, and it provides a basis for tentative conclusions regarding their short- and medium term (mostly test-based impact. Today we published a policy brief, the purpose of which is to summarize this research in a manner that is fair and accessible to policymakers and the public.

How Effective Are Online Credit Recovery Programs?

Credit recovery programs in the U.S. have proliferated rapidly since the enactment of No Child Left Behind (NCLB), particularly in states that are home to a large number of urban schools with high dropout rates (Balfanz and Legters 2004).

Although definitions vary somewhat, credit recovery is any method by which students can earn missed credits in order to graduate on time (Watson and Gemin 2008). Online credit recovery is a common form of these programs, but others include mixed online/in-person instruction, and in-person instruction (McCabe and Andrie 2012). At least three major school districts – Boston, Chicago, and New York City – offer credit recovery programs, as do several states, including Missouri and Wisconsin. Private companies such as Plato, Pearson, Apex, and Kaplan have also tried to fill this niche by offering to charge between $175 and $1,200 per student per credit. Online credit recovery represents approximately half of all instruction in the $2 billion online education industry.

Yet, despite the rising presence of online credit recovery programs, there exists scant evidence as to their effectiveness in increasing high school graduation rates, or their impact on other outcomes of interest.

Developing Workplaces Where Teachers Stay, Improve, And Succeed

** Republished here in the Washington Post

Our guest authors today are Matthew A. Kraft and John P. Papay. Kraft is an Assistant Professor of Education at Brown University. Papay is an Assistant Professor of Education and Economics at Brown University. In 2015, they received the American Educational Research Association Palmer O. Johnson Memorial Award for the research discussed in this essay. 

When you study education policy, the inevitable question about what you do for a living always gets the conversation going. Controversies over teachers unions, charter schools, and standardized testing provide plenty of fodder for lively debates. People often are eager to share their own experiences about individual teachers who profoundly shaped their lives or were less than inspiring.

A large body of research confirms this common experience – teachers have large effects on students’ learning, and some teachers are far more effective than others. What is largely absent in these conversations, and in the scholarly literature, is a recognition of how these teachers are also supported or constrained by the organizational contexts in which they teach.

The absence of an organizational perspective on teacher effectiveness leads to narrow dinner conversations and misinformed policy. We tend to ascribe teachers’ career decisions to the students they teach rather than the conditions in which they work. We treat teachers as if their effectiveness is mostly fixed, always portable, and independent of school context. As a result, we rarely complement personnel reforms with organizational reforms that could benefit both teachers and students.

New Teacher Attrition And The Recession

The Beginning Teacher Longitudinal Study (BTLS), a terrific project by the National Center for Education Statistics, tracks a nationally representative cohort of beginning teachers (those who started in 2007 or 2008) through their first five years and documents their turnover outcomes. Results from this survey have been trickling out every year (see our post here), but the most recent report presents outcomes from these teachers’ first four years.

The headline story, as reported in the Washington Post, was that roughly 17 percent of these new teachers had left the profession entirely within their first four years. A number of commenters, including the Post article, hastened to point out that the BTLS estimates are far lower than the “conventional wisdom” statistic that 40-50 percent leave the profession within the first five years (see here for more on this figure; also see Perda 2013 for a similar five-year estimate using longitudinal data). These findings are released within a political context where teacher attrition (somewhat strangely) has become a contentious political issue, one which advocates tend to interpret in a manner that supports their pre-existing beliefs about education policy.

Putting this source of contention aside, the BTLS results clearly show that new teacher attrition during these years was far lower than is often assumed, and certainly that teachers are not fleeing the classroom at a greater clip than in previous years. This is important, and cannot be "explained away" by any one factor, but we should still be careful about generalizing too strongly these findings beyond this particular time period, given that the BTLS cohort of teachers entered the classroom almost precisely at the time that the "great recession" began. This is, of course, not a new or original point – it was, for instance, mentioned briefly in the Post article. And it's hardly groundbreaking to note that labor market behavior does not occur in a vacuum. Still, given all the commentary about the BTLS results, it may be worth reviewing briefly.

Is The Social Side Of Education Touchy Feely?

That's right, measuring social and organizational aspects of schools is just... well, "touchy feely." We all intuitively grasp that social relations are important in our work environments, that having mentors on the job can make a world of difference, that knowing how to work with colleagues matters to the quality of the end product, that innovation and improvement relies on the sharing of ideas, that having a good relationship with supervisors influences both engagement and performance, and so on.

I could go on, but I don't have to; we all just know these things. But is there hard evidence, other than common sense and our personal experiences? Behaviors such as collaboration and interaction or qualities like trust are difficult to quantify. In the end, is it possible that they are just 'soft' and that, even if they’re important (and they are), they just don't belong in policy conversations?

Wrong.

In this post, I review three distinct methodological approaches that researchers have used to understand social-organizational aspects of schools. Specifically, I selected studies that examine the relationship between aspects of teachers' social-organizational environments and their students' achievement growth. I focus both on the methods and on the substantive findings. This is because I think some basic sense of how researchers look at complex constructs like trust or collegiality can deepen our understanding of this work and lead us to embrace its implications for policy and practice more fully.

Lessons And Directions From The CREDO Urban Charter School Study

Last week, CREDO, a Stanford University research organization that focuses mostly on charter schools, released an analysis of the test-based effectiveness of charter schools in “urban areas” – that is, charters located in cities located within in 42 urban areas throughout 22 states. The math and reading testing data used in the analysis are from the 2006-07 to 2010-11 school years.

In short, the researchers find that, across all areas included, charters’ estimated impacts on test scores, vis-à-vis the regular public schools to which they are compared, are positive and statistically discernible. The magnitude of the overall estimated effect is somewhat modest in reading, and larger in math. In both cases, as always, results vary substantially by location, with very large and positive impacts in some places and negative impacts in others.

These “horse race” charter school studies are certainly worthwhile, and their findings have useful policy implications. In another sense, however, the public’s relentless focus on the “bottom line” of these analyses is tantamount to asking continually a question ("do charter schools boost test scores?") to which we already know the answer (some do, some do not). This approach is somewhat inconsistent with the whole idea of charter schools, and with harvesting what is their largest potential contribution to U.S. public education. But there are also a few more specific issues and findings in this report that merit a little bit of further discussion, and we’ll start with those.

Turning Conflict Into Trust Improves Schools And Student Learning

Our guest author today is Greg Anrig, vice president of policy and programs at The Century Foundation and author of Beyond the Education Wars: Evidence That Collaboration Builds Effective Schools.

In recent years, a number of studies (discussed below; also see here and here) have shown that effective public schools are built on strong collaborative relationships, including those between administrators and teachers. These findings have helped to accelerate a movement toward constructing such partnerships in public schools across the U.S. However, the growing research and expanding innovations aimed at nurturing collaboration have largely been neglected by both mainstream media and the policy community.

Studies that explore the question of what makes successful schools work never find a silver bullet, but they do consistently pinpoint commonalities in how those schools operate. The University of Chicago's Consortium on Chicago School Research produced the most compelling research of this type, published in a book called Organizing Schools for Improvement. The consortium gathered demographic and test data, and conducted extensive surveys of stakeholders, in more than 400 Chicago elementary schools from 1990 to 2005. That treasure trove of information enabled the consortium to identify with a high degree of confidence the organizational characteristics and practices associated with schools that produced above-average improvement in student outcomes.

The most crucial finding was that the most effective schools, based on test score improvement over time after controlling for demographic factors, had developed an unusually high degree of "relational trust" among their administrators, teachers, and parents.

The Debate And Evidence On The Impact Of NCLB

There is currently a flurry of debate focused on the question of whether “NCLB worked.” This question, which surfaces regularly in the education field, is particularly salient in recent weeks, as Congress holds hearings on reauthorizing the law.

Any time there is a spell of “did NCLB work?” activity, one can hear and read numerous attempts to use simple NAEP changes in order to assess its impact. Individuals and organizations, including both supporters and detractors of the law, attempt to make their cases by presenting trends in scores, parsing subgroups estimates, and so on. These efforts, though typically well-intentioned, do not, of course, tell us much of anything about the law’s impact. One can use simple, unadjusted NAEP changes to prove or disprove any policy argument. And the reason is that they are not valid evidence of an intervention's effects. There’s more to policy analysis than subtraction.

But it’s not just the inappropriate use of evidence that makes these “did NCLB work?” debates frustrating and, often, unproductive. It is also the fact that NCLB really cannot be judged in simple, binary terms. It is a complex, national policy with considerable inter-state variation in design/implementation and various types of effects, intended and unintended. This is not a situation that lends itself to clear cut yes/no answers to the “did it work?” question.