A Quick Look At The DC Charter School Rating System

Having taken a look at several states’ school rating systems  (see our posts on the systems in IN, OH, FL and CO), I thought it might be interesting to examine a system used by a group of charter schools – starting with the system used by charters in the District of Columbia. This is the third year the DC charter school board has released the ratings.

For elementary and middle schools (upon which I will focus in this post*), the DC Performance Management Framework (PMF) is a weighted index composed of: 40 percent absolute performance; 40 percent growth; and 20 percent what they call “leading indicators” (a more detailed description of this formula can be found in the second footnote).** The index scores are then sorted into one of three tiers, with Tier 1 being the highest, and Tier 3 the lowest.

So, these particular ratings weight absolute performance – i.e., how highly students score on tests – a bit less heavily than do most states that have devised their own systems, and they grant slightly more importance to growth and alternative measures. We might therefore expect to find a somewhat weaker relationship between PMF scores and student characteristics such as free/reduced price lunch eligibility (FRL), as these charters are judged less predominantly on the students they serve. Let’s take a quick look.

The most simple descriptive statistic is to simply calculate the average FRL and special education rates of the schools within each tier (again, charters are sorted into Tiers 1-3, with Tier 1 being the highest performing, at least according to this system). These averages are presented in the table below for all 51 elementary/middle charter schools that received a 2012 rating (keep in mind that this is a small group of schools - 51 in total, and that most are concentrated in Tier 2, which means that the averages can be influenced substantially by a small set of schools).

There is no apparent relationship between tier and special education (the rightmost column) – schools in Tier 1 actually have the highest rates.

The FRL averages, in contrast, do exhibit a pattern – schools in the lower tiers have a higher FRL rate, on average, than schools in the higher tiers. The difference is most stark when comparing Tier 1 (66.4 percent) to Tier 2 (77.7 percent). This is to no small extent a result of the measures employed by the system - not only of the proficient/advanced rates, which are heavily associated with characteristics such as FRL, but also the growth measure (median growth percentiles), which usually are modestly correlated with these traits.

Averages, of course, sometimes mask what's happening underneath, so it might be useful to take a look at the scatterplot of FRL by the actual index scores upon which the tier ratings are based.

You can see a relationship, but it is quite messy. Although virtually all of the schools in Tier 3 (those with index scores below 35 percent) are high poverty, the schools with the four highest index scores have FRL rates over 80 percent, while several schools with rates below 50 percent receive relatively low index scores.***

It is therefore fair to characterize this relationship as discernible (statistically and otherwise) and modest-to-low. It is also noticeably less strong than that in most of the other states I have reviewed.

The reason for this is very simple: The PMFs rely less heavily on absolute performance measures, which are strongly correlated with student characteristics such as FRL. This does not necessarily mean these DC charter ratings are “better” or more fair than their counterparts elsewhere, but it does show how the choice of constituent measures has a predictable and often substantial impact on the association between the results and the students schools serve.

- Matt Di Carlo

***** 

* The DC charter board does not make the entire PMF dataset available in a single spreadsheet, which is why I limited this simple analysis to elementary/middle schools (and it's also why I concentrated solely on one outcome - schools' final score/tier).

** For elementary and middle schools, growth measures (median growth percentiles) are 40 percent (20 percent reading, 20 percent math). Math and reading proficiency rates contribute another 20 percent, while an additional five percent is based on advanced rates in both subjects. 15 percent is based on what the board calls gateway indicators, which are proficient/advanced rates in reading among third graders for elementary schools, and proficient/advanced rates in math for eighth graders among middle schools. The final 20 percent of schools’ scores – called “leading indicators” – are attendance and re-enrollment rates (10 percent each) for both elementary and middle schools. The formula is slightly different for combined elementary/middle schools.

*** Two quick side notes: First, as suggested by the table above, which looks at tier categories, there is essentially no correlation between index scores and the proportion of schools’ students that are classified as special education; second, as in most accountability systems that sort scores into categories, there are a number of schools right at the Tier 1/2 and Tier 2/3 cutoff points.