Peer Effects And Attrition In High-Profile Charter Schools

An article in last week’s New York Times tells the story of child who was accepted (via lottery) into the highly-acclaimed Harlem Success Academy (HSA), a charter school in New York City. The boy’s mother was thrilled, saying she felt like she had just gotten her son a tuition-free spot in an elite private school. From the very first day of kindergarten, however, her child was in trouble. Sometimes he was sent home early; other times he was forced to stay late and “practice walking the hallways” as punishment for acting out. During his third week, he was suspended for three days.

Shortly thereafter, the mother, who had been corresponding with the principal and others about these incidents, received an e-mail message from HSA founder Eva Moskowitz. Moskowitz told her that, at this school, it is “extremely important that children feel successful," and that HSA, with its nine-hour days, during which children are “being constantly asked to focus and concentrate," can sometimes “overwhelm children and be a bad environment." The mother understood this to be a veiled threat of sorts, but was not upset at the time. Indeed, she credits HSA staff with helping her to find a regular public school for her child to attend. Happily, her son eventually ended up doing very well at his new school.

It’s very important to remember that this is really only one side of the story. It’s also an anecdote, and there is no way to tell how widespread this practice might be at HSA, or at charter schools in general. I retell it here because it helps to illustrate a difficult-to-measure “advantage” that some charter schools have when compared with regular neighborhood schools – the peer effects of attrition without replacement.

Charter critics often contend that many charters have high attrition, and that lower-performing students leave – whether due to “counseling out” (as may have been the case in the NYT story) or on their own volition – which artificially boosts test scores. The standard reply to this argument from charter supporters is to point to studies (such as this paper on New York City charters and this one on KIPP schools) showing that charter school attrition is similar to that of regular public schools. In addition, supporters point out that these studies that include high-performing charters, though limited in scope and number, use techniques to ensure that attrition does not directly affect their results (for example, put simply, “following” students who leave charters into their new schools).

These arguments seem to be supported by available evidence (by the way, KIPP in particular deserves credit for “subjecting itself” to high-quality research that provides this evidence). But, as others have pointed out, they do not rule out attrition effects. Some analyses can compare attrition between charters and nearby regular publics, and sometimes even account for the performance of students who leave the charter schools. But they cannot address the effect of that attrition on the students who stay behind.

Many (but not all) high-profile charter schools only accept students at one “intake grade," usually the lowest grade that the school serves. Each year, they get a new cohort of students who enter at that intake grade (sometimes via random lottery). That cohort stays together throughout their time at the school. As is frequently the case, student attrition can be high.

And it is not random. On the whole, students who leave a school tend to be those who are not doing well (there are, of course, exceptions). In some cases, as with the family in the article, they are gently encouraged to find another placement (remember, again, that the extent of this practice is not well-documented). Some schools, most notably KIPP, require parents to sign contracts stating that their kids will meet certain expectations, such as being on time and doing their homework. If students fail to meet these requirements, they are subject to a variety of disciplinary actions, including suspension and expulsion (for example, according to the KIPP DC Handbook, 20 unexcused absences results in automatic unenrollment, while three suspensions results in expulsion [see pages 27-30 of the handbook for a list of suspension-inducing infractions]). In many instances, parents will pull their children out voluntarily, because they’re just not succeeding. And, finally, for secondary schools, students drop out. Again, in none of these cases is attrition random.

Yes, there is plenty of mobility in regular public schools as well, but there’s a big difference: Often, when students leave charter schools, they are not replaced. That is why, at high-profile charter schools, it is not unusual to find that the number of students in each cohort declines rapidly over time.

Regular publics, on the other hand, must typically accept all students, both at the beginning and in the middle of schools years.  This includes, in many cases, those who leave the charter schools.

So, for many charters the end result is that each cohort of incoming students is gradually whittled down, eventually losing many of the students with the most difficult academic and/or behavioral problems. Since they’re often not replaced, who’s left behind?

The students who remain tend to be the students who fit in well with the school’s culture, meeting its expectations, and not disrupting class. These remaining students interact with each other socially and academically, feeding off each other’s successes and abilities. And, from decades of research, we know that these peer effects can have a substantial influence on student achievement.

Even the best studies cannot, unfortunately, account for peer effects (there are techniques, but they’re limited). It remains one of the big, unanswered questions in the research on high-profile charter schools like KIPP and HSA – the degree to which their successes are at least partially attributable to the fact that they shed struggling students and need not take in new ones (also see this discussion of extended school time). In other words, the problem isn’t as much selection bias as it’s deselection bias (with non-replacement).

Note that I am not saying that we should disregard the high performance of those few charters, such as the KIPP schools, that seem to do well. For one thing, some of these schools do replace leaving students. In addition, among those that don’t, I cannot say how much of their success is due to the peer effects. My guess is that it’s important, but explains only part of the student gains – i.e., the schools would still be getting good results even without the non-replacement issue. What I am saying is that this means that even the best studies of high-profile charters should be interpreted carefully, with this limitation in mind.

- Matt Di Carlo