Is The Social Side Of Education Touchy Feely?
That's right, measuring social and organizational aspects of schools is just... well, "touchy feely." We all intuitively grasp that social relations are important in our work environments, that having mentors on the job can make a world of difference, that knowing how to work with colleagues matters to the quality of the end product, that innovation and improvement relies on the sharing of ideas, that having a good relationship with supervisors influences both engagement and performance, and so on.
I could go on, but I don't have to; we all just know these things. But is there hard evidence, other than common sense and our personal experiences? Behaviors such as collaboration and interaction or qualities like trust are difficult to quantify. In the end, is it possible that they are just 'soft' and that, even if they’re important (and they are), they just don't belong in policy conversations?
Wrong.
In this post, I review three distinct methodological approaches that researchers have used to understand social-organizational aspects of schools. Specifically, I selected studies that examine the relationship between aspects of teachers' social-organizational environments and their students' achievement growth. I focus both on the methods and on the substantive findings. This is because I think some basic sense of how researchers look at complex constructs like trust or collegiality can deepen our understanding of this work and lead us to embrace its implications for policy and practice more fully.
***
1. Known Measures, New Models
Kirabo Jackson and Elias Bruegman (2009) used value added scores and other student and teacher level data such to demonstrate that a teacher’s own test-based effectiveness is influenced by the quality of her peers. They found that teachers, particularly new teachers, improved their ability to raise student tests scores when they worked in schools with more effective colleagues. Using complex statistical models, the researchers were able to demonstrate that the mechanism through which colleagues influence teachers’ effectiveness is peer learning (direct or induced).
In a separate paper, Jackson (2013) used student level data to examine the importance of the match between teachers and schools for student achievement, and demonstrated that match quality can “explain” a quarter of the variation in teacher quality.
The results indicate that the inclusion of match effect reduces the explanatory power of teacher effects by between 10 and 50 percent ─ indicating that part of what we typically interpret as a teacher quality effect may in fact be a match quality effect that is not portable across schooling environments.
Thus, Jackson explains, "policy simulations based on teacher quality estimates that do not account for match quality could be quite inaccurate." He also notes that it is necessary "to consider the working environment in which teachers operate" when making hiring and firing decisions. Furthermore,
The fact that a teacher who performs well in one school may not be as effective in another means that policy-makers should be cautious about identifying strong teachers in one school and moving them to another.
2. New Measures, Known Methods
Although the use of social network analysis (SNA) is relatively new in education, it's been a key method in sociology for many decades. SNA examines the ties or patterns of relations among individuals within and/or across organizations and systems. It is relevant here because all collaboration occurs primarily through social interactions.
The general goal of SNA is to gather information about who individuals interact with and for what purpose -- e.g., asking teachers how often they seek (and/or give) advice or support from colleagues. This kind of information provides a very rich picture of the social context of schools – for example, the extent to which teachers work together, trust each other, etc. Network (or sociometric) measures have been used to understand whether and how social-relational aspects influence teacher effectiveness and student learning. In fact, many of the studies that we have discussed in the social side of education series on this blog draw on SNA -- see here and here.
For example, professors Frits Pil and Carrie Leana (2009) followed over one thousand 4th and 5th grade teachers in a representative sample of 130 urban elementary schools. They looked at how much each student’s math knowledge advanced in the year they spent with a particular teacher. What's particularly original about Pil and Leana's work is that, in addition to using measures of student and teacher characteristics, they collected network data by asking teachers to provide information about who they spoke with when they had questions or needed advice, and the extent to which they trusted the source of the advice that they received. As Leana and Pil explained in this post in our series:
Teachers sought advice from one another and students showed higher gains in math achievement when their teachers reported frequent conversations with their peers that centered on math, and when there was a feeling of trust and closeness between the teachers.
Social capital (i.e., these patterns of interactions among teachers) "was the most powerful predictor of student achievement gains above and beyond teacher experience or ability in the classroom." In fact, "even low-ability teachers could perform as well as teachers of average ability if they had strong social capital." Finally, "students whose teachers were both more capable (higher human capital) and had stronger ties with their peers (higher social capital) showed the highest gains in math achievement."
A second, similar, network study by Alan Daly and colleagues - see Daly, Moolenaar, Der-Martirosian & Liou 2014 - looked the relationship between teachers' social capital (as measured by their interactions and social networks) and student achievement. The researchers found that the depth and density of teachers' relations were associated with increased student achievement.
Finally, Nienke Moolenaar, Peter Sleegers, and Alan Daly (2012), who collected and used teachers' social network data from 53 Dutch elementary schools, found that "well-connected teacher networks were associated with strong teacher collective efficacy, which in turn supported student achievement."
3. New Measures, New Models
The third set of studies draws on new measures of school context constructed from student, teacher, and principal responses to district and statewide surveys (e.g., Teaching, Empowering, Leading and Learning (TELL) Survey.) These surveys, which are administered annually, contain Likert-scale questions designed to capture detailed information about how educators view teaching and learning conditions in their schools.
In a study using data from Massachusetts -- a statewide survey of school working conditions with demographic and student achievement data -- Susan M. Johnson, Matthew Kraft and John Papay (2012) found that stronger principal leadership, relationships among colleagues, and positive school culture predicted higher median student achievement growth among schools.
Combining similar data sources from large urban district in North Carolina, Kraft and Papay (2014) examined how measures of the school context influenced the degree to which teachers become more effective over time. The authors find that there is substantial variation by school in terms of teacher effectiveness growth, and that such variation is explained, in part, by differences in teachers' professional environments:
Teachers who work in more supportive environments become more effective at raising student achievement on standardized tests over time than do teachers who work in less supportive environments.
***
Measuring trust or collegiality isn't easy or straightforward but it is possible, it is being done, and it is essential work to do if we hope to understand educational progress. The research cited above represents only some recent examples of the exciting, rich work currently underway.
In his book Linked: How Everything Is Connected To Everything Else, Albert-Lázló Barabási says:
We have taken the universe apart and have no idea how to put it back together. After spending trillions of research dollars to disassemble nature in the last century, we are just now acknowledging that we have no idea how to continue - except to take it further apart.
It seems to me, though, that something quite different is taking place in education, particularly among the growing and diverse group of scholars who care about and want to understand the social dimension of educational improvement. These scholars are looking at the same complex issues from different prisms, using different tools but with one aim: To put things back together and stop taking them further apart.