Student Attrition Is A Core Feature Of School Choice, Not A Bug
The issue of student attrition at KIPP and charter schools is never far beneath the surface of our education debates. KIPP’s critics claim that these schools exclude or “counsel out” students who aren’t doing well, thus inflating student test results. Supporters contend that KIPP schools are open admission with enrollment typically determined by lottery, and they usually cite a 2010 Mathematica report finding strong results among students in most (but not all) of 22 KIPP middle schools, as well as attrition rates that were no higher, on average, than at the regular public schools to which they are compared.*
As I have written elsewhere, I am persuaded that student attrition cannot explain away the gains that Mathematica found in the schools they examined (though I do think peer effects of attrition without replacement may play some role, which is a very common issue in research of this type).
But, beyond this back-and-forth over the churn in these schools and whether it affected the results of this analysis, there’s also a confusion of sorts when it comes to discussions of student attrition in charters, whether KIPP or in general. Supporters of school choice often respond to “attrition accusations” by trying to deny or downplay its importance or frequency. This, it seems to me, ignores an obvious point: Within-district attrition - students changing schools, often based on “fit” or performance - is a defining feature of school choice, not an aberration.
The idea of school choice is that there should be an educational marketplace, if you will. Schools run by a variety of different operators, using a variety of different models, would all compete for enrollment, allowing parents to choose what’s best for their kids. From this perspective, student attrition (and mobility in general) based on how students are doing or "fitting in" is to be expected, as it signals the exercise of choice in a world of imperfect information (e.g., parents might not choose the best option on the first or second try). In other words, the proliferation of charter schools in any given area will almost invariably increase the non-random churn of students in and out of schools.**
So, instead of downplaying the existence and importance of student attrition/selection, charter supporters might clarify this debate by stating clearly that it’s part of the deal in a choice-based school system.
Moreover, from this perspective, the best case scenario is that a robust enough set of choices might provide something for virtually everyone, but the results of any one school or model cannot easily be generalized for all. Arguably, this is particularly true in the case of approaches such as KIPP’s, which entail intensive interventions, such as extremely rigid discipline and 40-50 percent more school time, that might not be right for many students, regardless of background.
What is clear about KIPP is that they are, from all indications, well-run schools. They serve mostly disadvantaged students, and those who do well really do well. This shouldn’t be dismissed or diminished, but it also shouldn’t be used to draw strong conclusions about the success or potential of charter schools in general, especially given the fact that so few models have produced consistent results.
Finally, I can’t speak to whether KIPP “counsels out” students (though I of course think it’s wrong for schools to do so). The brutal truth, however, is that there’s a somewhat thin line between “counseling out," pushing out, and helping parents find the best fitting school for their kids. This is a line that any school choice system must necessarily straddle, and we can’t really have a productive conversation about it without first acknowledging its existence.
- Matt Di Carlo
*****
* As usual, the average attrition rates mask a great deal of underlying variation among these 22 schools (see, for example, the figure on page 10 of this report).
** It bears mentioning that, in theory, this selection need not necessarily “benefit” one type of school or the other. For example, parents may choose to switch their kids out of regular public schools when they aren’t doing well.
Matt,
What do you think of this argument: that the other problem with studies around attrition is that the analyses are not sensitive to pick up the real issue of attrition--the tiny handful of challenging students who actually account for a disproportionate amount of administrative time.
Lots of kids move around urban districts-- hence attrition generally looks the same in KIPP and local publics. But that doesn't necessarily address the key argument about attrition, raised in the HVA article you cite: how do we know that KIPP and other charter schools are not systematically counseling out the handful of children who are the most difficult?
Let's assume a kind of pareto principle of discipline, most kids are relatively easy to teach, about 20% are responsible for 80% of the challenges, and a tiny handful are disproportionately responsible for problems in discipline, administrative support, etc.
The analysis done by Mathematica assumes that if the demographics of the kids leaving are the same, then attrition is not to explain differences in achievement. But that argument doesn't work if the problem isn't just having more kids from certain categories, but that public schools end up as the catchment for the handful of the most challenging kids (Dan Koretz has a name for this effect in Measuring Up, named after a particular kid, I can't remember the name right now).
That's always been my main concern with the inferences drawn from the Mathematica study. It's important that KIPP seems to be teaching the same demographic categories of kids as everyone else-- but it doesn't demonstrate that KIPP isn't counseling out (or that lotteries are a barrier to entry for) the handful of kids who are not just difficult to teach, but can disrupt classrooms and schools.
Justin,
My research in Texas looking at 9 charter operations (one of which was KIPP) found that KIPP does NOT enroll students from the same demographics categories. KIPP enrolls higher achieving students, fewer special needs students, and fewer ELL students. They enroll about the same percentage of poor students, but the KIPP students in poverty have achievement around .25 to 0.3 standard deviations greater than students not enrolling in KIPP. My study found few differences in attrition between KIPP and comparison schools, but you are correct that these labels of groups of kids do not identify the kids that are tough to teach. In fact, a high performing kid may make it harder for kids around him to learn if the HP kid is a distraction in class. Matt is correct as well that merging the effects of attrition with replacement provides different results than just looking at attrition. That is something that I plan to do with the Texas data if I can find the time.
See my blog for the details:
http://fullerlook.wordpress.com/2012/08/23/tx_ms_charter_study/
Very thoughtful blog and 2 good comments.
1. Justin, good point.
2. Ed I often find myself disagreeing with your comments, but your data lines up with what we have here in Boston.
The Kane/Angrist study here found that those attending charters were, on average, about .2 SDs above the district as a whole.
But the researchers also found:
-In-district "pilot schools" had the exact same .2 SD differential. Ie, it certainly wasn't a "district versus charter" issue.
-From the .2 SD starting point, the charter lottery winners made huge gains, and the lottery losers actually regressed (the full 0.2 SDs in middle school math, for example).
3. Matt, I'd ask you to consider another aspect of the attrition and choice discussion.
Different standards for promotion.
For example, imagine a student fails his courses at XYZ State College. He then transfers to ABC State. Does ABC give him credit? Of course not.
But that's the single most typical experience of, I'd argue, a KIPP student.
a. Didn't pass, say, 9th grade in one of their 20 or so high schools.
b. Teachers, principal implore him to stay.
c. Nearby traditional school offers kid automatic promotion to 10th grade.
In that case, it's NOT an "inherent feature" of choice. Districts could do what colleges do. Simply honor the report cards of the teachers at the charter, offer the kid enrollment but not social promotion, and you'd see a reduction in charter attrition.
In other words, schools should behave like colleges. Since they honor one another's transcripts and professor decisions about grades, it allows for choice without an incentive for enrolled students to shop for a new school if they're failing, as a way to bypass accountability.
MG--the increase and decrease of lottery winners and losers (as well as the increase in magnet, pilot, early college, etc) schools could largely be due to peer effects. If so, then how scalable is the reform? Its not. Current reform strategies are simply segregating kids into winner and loser schools and then providing fewer resources to the loser schools. That does not sound like the meritocracy or equality that we speak of in our political rhetoric.
In talking with teachers in schools receiving students from charters, most of the students said that they were forced to leave because they did not follow rules or meet expectations. For example, one KIPP school in Houston required the parents of a student to follow the student around school for one day because the student had been found wandering the hall without a pass. The parents could not afford to miss work, so the students was forced to withdraw.
Ed,
Sorry for the delay.
Let me get your view straight:
a. If a charter lottery study shows charter kids do WELL, then it's not the charter which caused those gains. It's magical "peer effects."
b. If a charter lottery study shows charter kids do BADLY, then it IS the charter. Bad charter.
I mean, c'mon.