For Florida's School Grading System, A Smart Change With Unexpected Effects

Last year, we discussed a small but potentially meaningful change that Florida made to its school grading system, one that might have attenuated a long-standing tendency of its student “gains” measures, by design, to favor schools that serve more advantaged students. Unfortunately, this result doesn’t seem to have been achieved.

Prior to 2014-15, one of the criteria by which Florida students could be counted as having “made gains” was scoring as proficient or better in two consecutive years, without having dropped a level (e.g., from advanced to proficient). Put simply, this meant that students scoring above the proficiency threshold would be counted as making “gains,” even if they in fact made only average or even below average progress, so long as they stayed above the line. As a result of this somewhat crude “growth” measure, schools serving large proportions of students scoring above the proficiency line (i.e., schools in affluent neighborhoods) were virtually guaranteed to receive strong “gains” scores. Such “double counting” in the “gains” measures likely contributed to a very strong relationship between schools’ grades and their students’ socio-economic status (as gauged, albeit roughly, by subsidized lunch eligibility rates).

Florida, to its credit, changed this “double counting” rule effective in 2014-15. Students who score as proficient in two consecutive years are no longer automatically counted as making “gains.” They must also exhibit some score growth in order to receive the designation.

This seemingly trivial change had the potential to attenuate the degree to which Florida’s “gains” measures “level the playing field” by focusing on progress rather than how highly students score (see here for further discussion of how and why this matters). To be clear, Florida, like virtually all states, still relies heavily on simple proficiency rates to grade schools, and so the overall grades still strongly “favor” schools with more advantaged students, regardless of the growth measures used. But the change still represents the state’s recognition of the problem, as well as acknowledgement of the need to make progress toward better measurement in accountability systems.

The results for the latest year of school grades, however, indicate that this change has had little or no apparent effect. The table below presents the correlations between the “gains” components of Florida’s system for grading elementary and middle schools and each schools’ free and reduced price lunch eligibility (FRL) rates, as well as the correlations between “gains” and schools’ proficiency rates in the same subject. The correlations are given for 2014 (before the change to the “gains” measure) and 2017 (after the change).

Let’s begin with the correlations between the “gains” measures and FRL rates. Given that proficient students are no longer automatically scored as making “gains,” higher income (lower FRL) schools, which tend to serve larger proportions of proficient students, no longer have the “double counting” advantage. We might thus expect to find a weaker relationship between schools’ “gains” scores and their FRL rates in 2017 versus 2014.

Instead, we find the opposite. The correlation coefficients are actually higher after the change.

Another way to examine the impact of the “double counting” rule is simply to calculate correlations between “gains” scores and proficiency rates. In other words, if proficient students are no longer automatically counted as making “gains,” then we should expect a weaker correlation between “gains” and proficiency in 2017 than we find in 2014.

Once again, this is not the case. The correlations are stronger in 2017, by considerable margins. Schools with higher proficiency rates now tend to receive even higher “gains” scores than they did in 2014. Partially as a result, the final row of the table shows that the association between FRL and schools’ overall scores also increased between 2014 and 2017. Moreover, given that the correlations between FRL and proficiency rates (not shown) did not increase, it seems that the “gains” measures are the primary reason why this relationship has strengthened.

There are a few possible explanations for these counterintuitive results (and they are not mutually exclusive). One is that the relationship between these variables in Florida’s grading system was substantially altered by the introduction of new tests. Put differently, it is possible that the correlations might have decreased had Florida continued to use its previous state tests.

(This also suggests that the new tests may have rather serious implications for Florida’s school grades, including their relationship with student characteristics, and one definitely hopes that state officials are on top of this.)

Another possible explanation is that the majority of proficient students are still being “double counted” – i.e., most of them achieved the small increase in score required to qualify as having made “gains.” This wouldn’t explain the increase in the correlations in the table above, but it would at least account for the fact that they did not decrease. Unfortunately (and understandably), no data are available on how many students met each of the criteria required to be coded as making "gains."

In any case, Florida still deserves credit for attempting to improve its system. It shows that they are paying attention to the details of their accountability system and adjusting course based on sound measurement principles. On the other hand, the simple results presented above also suggest that the relationship between school grades and student income in Florida may have changed, and that the “gains” measures seem to be the cause.  Clearly, a lot more room for improvement still remains, most notably the option of choosing a different growth model.