Promoting Productive Collaboration Through Inquiry: The Limits Of Policy Mandates
Our guest author today is Robert Shand, the Novice G. Fawcett Postdoctoral Researcher in Educational Studies at The Ohio State University. His research focuses on the economics of education, teacher collaboration and professional development, and how teachers and school leaders make decisions based on data and research to improve student outcomes.
In some ways, it is hard to dispute the traditional view that K-12 teaching is a professionally solitary activity. At the end of the day, most instruction still occurs with a single teacher standing in front of a classroom. When I tell folks that I study teacher collaboration for a living, some are puzzled – other than team teaching, what would teachers even collaborate about? Some former colleagues from my time as a middle and high school teacher even bristle at the growing demands by administrators that they collaborate. These former colleagues no doubt envision pointless meetings, contrived team-based scenarios, and freeloading colleagues trying to offload their work onto others.
Despite these negative preconceptions, there is growing evidence that meaningful work with colleagues can enhance teacher productivity, effectiveness, and professional growth, and even increase job satisfaction. Teachers can share ideas and instructional strategies, divide the work of developing curriculum, learn from colleagues, and analyze data and evidence to solve instructional problems and help meet diverse student needs. The evidence for the potential benefits of collaboration is so compelling, and collaborative work in education is becoming so pervasive, that the Every Student Succeeds Act legally redefines professional development to include “collaborative” as part of the definition.
Many studies have found links between teacher collaboration and positive outcomes for teachers and students. Historically, much of this evidence has been observational (e.g., from rich qualitative work on Professional Learning Communities) or correlational, which provides some evidence in support of the value of collaboration but leaves open the question of whether collaboration makes teachers better, or simply that better teachers collaborate more (or sort into schools with more collaborative cultures). Recent research has made use of randomized experiments (Gallimore, Ermeling, Saunders & Goldenberg, 2009; Papay, Taylor, Tyler & Laski, 2016), changes over time within the same teacher or school (Jackson & Bruegmann, 2009; Ronfeldt, Farmer, McQueen & Grissom, 2015), and other strategies (e.g., Sun, Loeb & Grissom, 2016) to estimate a causal link between teacher collaboration and effectiveness.
In spite of these recent advances in collaboration research, important questions remain. Even with robust evidence that collaboration can cause improvements in teaching and learning, my former colleagues may have reason to be skeptical of policy initiatives that impose additional collaboration requirements on them. First, it is clear that not all collaboration is equally productive. While some research exists on the conditions that make collaboration more effective – time, sustained focus, and meaningful participation and support by leaders are commonly cited conditions – the evidence on the features of productive collaboration is less robust than the evidence on the benefits of collaboration overall. Secondly, most existing research, even the causal work, captures naturalistic variation in collaboration. We are left with the question of whether effective collaboration – and the conditions that support it – can be induced by policy.
A collaboration initiative as part of a larger reform in a large urban school district provides an opportunity to answer that question. Traditional policies to enhance teacher knowledge, productivity, and effectiveness revolve around professional development. More recently, particularly catalyzed by Race to the Top and No Child Left Behind waivers, policy has shifted toward a performance management approach, whereby schools and teachers are held accountable for how much they contribute to growth in student learning. Both of these approaches have advantages but also important limitations – the empirical evidence on the effectiveness of most traditional PD is weak, and there are concerns about validity, fairness, bias, and gaming with value-added and teacher incentive systems. Depending upon their implementation as part of a larger accountability and policy framework, they can either contribute to or undermine the conditions that lead to productive collaborative cultures in schools.
The school district mentioned above aimed to “square the circle,” so to speak, by holding schools accountable for student learning outcomes while also providing them with greater autonomy over staffing, budget, and curriculum decisions, and investing in supports to build school and teacher capacity to use assessments, data, and other tools to solve instructional problems. A key aspect of these reforms was reorganizing traditional professional development around team-based problem-solving, with the goals of making PD better tailored to the needs of teachers and students, leveraging the knowledge-sharing and positive teacher peer effects that could arise from teamwork, and encouraging a continuous improvement mindset within schools. This approach, known as collaborative inquiry or inquiry teams, was inspired by the Japanese kaizen model of ongoing improvements through team-based problem-solving in business, as well as by theories of customization of provision of social services, including education, by autonomous professionals while preserving democratic oversight of the process.
The basic team process proceeded as follows. Teams of teachers, administrators, and other professionals with some shared goal or challenge would assemble – most often, the teams would map on to some existing collaborative structure within a school, such as a team of teachers within the same grade level, teachers of the same subject, or teachers who worked with one specific population of high-needs students. The team would then identify a small target group of students who were struggling with a particular skill and use refined assessment instruments and data analysis protocols to identify a very specific sub-skill to serve as the team’s initial area of focus. With a sub-group and sub-skill in hand to give focus and discipline to the team’s work, the team would enter into a series of iterative cycles by which they would locate or develop instructional strategies to help improve student performance in the sub-skill, systematically monitor improvement and revise or test new strategies, and then, as they found strategies that were effective, share them with colleagues and move on to new or more carefully refined sub-skills. This cycle-based approach had three objectives: to improve performance for the targeted sub-group of students so as to close achievement gaps within the school; to build teacher capacity in data analysis and instructional problem-solving via the team process; and to identify and share best instructional practices identified through action research.
The initiative builds upon what we know from prior research about the potential power of collaboration. In particular, inquiry teams relate to the crucial interaction between human capital and social capital. Human capital development is inhibited when it occurs in isolation, as teachers facing instructional challenges with their students often do not know where to go to seek new solutions and strategies. Inquiry teams help bridge that gap by connecting teacher professional development with collaborative work with their colleagues – teachers can learn new strategies from one another, and also work together to problem-solve and consult academic research to address novel challenges.
The phase-in of the inquiry team initiative over time provides an ideal setting for research. In the pilot year of the initiative, 2007-2008, each school in the district was required to assemble one team, selected by the principal among teacher volunteers. In 2008-2009, each school was to have multiple teams, with the ultimate aim of at least 90 percent of teachers participating on a team by 2009-2010 and the culture of inquiry suffusing all professional development and collaboration throughout the school. Since teachers volunteered to participate on teams and were selected by their principals, a simple comparison of the effectiveness of teachers on teams to those not on teams would likely be biased – we wouldn’t be able to tell if inquiry helped teachers become more effective, or if more effective teachers chose to participate in inquiry.
Instead, I use two natural experiments and one more descriptive or correlational approach to analyze the effects of inquiry: I compare first-year teachers who happen to be placed in a grade with an inquiry team to those placed in grades without teams, and teachers who switch grades from a grade without a team to a grade with a team to those who were at grades that either always or never had a team. The basic underlying assumption behind these experiments is that if a teacher or principal thought that teamwork would be especially useful to a teacher, it would be easier to form a team at the grade a teacher was teaching rather than to switch grades; therefore, the effectiveness of teamwork is unrelated to grade switching.
The descriptive approach analyzes differences within the same school in test score growth for grade-subject-student subgroup (e.g., English language learners) combinations that are identified as the target of team work versus those that are not. This is less causally robust than the natural experiments but provides some evidence on the proximal effects of teamwork on the students it directly targeted.
Using these methods, I found extremely modest effects of collaborative inquiry overall. There are some small (on the order of 5 percent of a standard deviation) effects on test scores for the students directly targeted by inquiry teams, and some very small effects on retention and value-added for first-year teachers. However, there are no effects on value-added of teachers in general, or via the grade-switching experiment.
This pattern of results, along with analysis of heterogeneity and intensity of implementation by teams, suggests a few hypotheses. First, inquiry teams seem modestly effective at improving outcomes for targeted students, but not teacher capacity overall, suggesting that targeting instruction helps, but if not done appropriately, could come at the expense of other students (e.g., redirecting teacher time to providing tutoring and extra support to the targeted sub-group). The effects on first-year teachers suggest that inquiry could serve as a kind of on-the-job training or induction and mentoring program, and help new teachers adapt to the unique needs of a particular school environment. Finally, textual analysis of team narrative entries into a centralized database on team process suggests that implementation of the initiative was weak overall, potentially hobbled by other demands on teacher time or lack of time and support. By the third year of the initiative, only 6 percent of teams engaged in more than one “cycle” of inquiry activity, and almost 30 percent of teams didn’t even set a goal for their work.
While the results are somewhat disappointing, they do not suggest that inquiry teams specifically, and teacher collaboration more generally, are not beneficial. Rather, they point to the challenge of implementing effective collaboration through policy mandates, as well as the careful balance required in what to mandate.
This policy mandated that all teachers participate, but deliberately made the process itself very flexible. Perhaps this was not quite the right balance – unlike the game show Jeopardy!, inquiry is not the best response to every question, and teachers likely needed more support and resources, in addition to better targeting, to make the policy work. Finally, my research speaks to the critical importance of ongoing work in how best to measure teacher collaboration. On the one hand, this study represents an important contribution in that it’s the first I’m aware of that uses administrative, rather than survey, data to assess teacher collaboration, and is therefore less subject to bias due to teachers saying what they think researchers want to hear, as opposed to what they actually did, on surveys. On the other hand, it’s clear that subtle differences in quality and intensity of implementation matter a great deal, and those nuances are very difficult to perceive in administrative data. Future work will focus on what we can glean from the narrative responses to refine survey instruments to better capture the “secret sauce” that makes some collaborations more fruitful than others.