Shanker Blog

Research On Teacher Evaluation Metrics: The Weaponization Of Correlations

Our guest author today is Cara Jackson, Assistant Director of Research and Evaluation at the Urban Teacher Center.

In recent years, many districts have implemented multiple-measure teacher evaluation systems, partly in response to federal pressure from No Child Left Behind waivers and incentives from the Race to the Top grant program. These systems have not been without controversy, largely owing to the perception – not entirely unfounded - that such systems might be used to penalize teachers.  One ongoing controversy in the field of teacher evaluation is whether these measures are sufficiently reliable and valid to be used for high-stakes decisions, such as dismissal or tenure.  That is a topic that deserves considerably more attention than a single post; here, I discuss just one of the issues that arises when investigating validity.

 The diagram below is a visualization of a multiple-measure evaluation system, one that combines information on teaching practice (e.g. ratings from a classroom observation rubric) with student achievement-based measures (e.g. value-added or student growth percentiles) and student surveys.  The system need not be limited to three components; the point is simply that classroom observations are not the sole means of evaluating teachers.   

In validating the various components of an evaluation system, researchers often examine their correlation with other components.  To the extent that each component is an attempt to capture something about the teacher’s underlying effectiveness, it’s reasonable to expect that different measurements taken of the same teacher will be positively related.  For example, we might examine whether ratings from a classroom observation rubric are positively correlated with value-added.

Fighting For Fairness For U.S. Domestic Workers

On September 17, 2013, the U.S. Department of Labor (DOL) announced the Home Care Final Rule, which extends the Fair Labor Standards Act’s (FLSA) minimum wage and overtime protections to domestic workers who provide home care assistance to the elderly, the infirm, and the disabled. The Home Care Final Rule is essential to improving the lives of two million domestic workers who, unlike other U.S. workers, are in many states not protected by the FLSA regarding minimum wage, overtime, sick leave, and vacation. Domestic work differs from other jobs in that the work takes place inside other people’s homes, which often puts domestic workers’ wellbeing at the mercy of their employers.

The exclusion of domestic workers from the FLSA was a concession to Southern politicians in the early 1900’s. It had left many homecare aides vulnerable to abuse and mistreatment by their employers. The rule was scheduled to go into effect on January 1, 2015. However, lawsuits filed by homecare corporations have hindered the change and served as an excuse for states to postpone implementation. For example, in Home Care Association of America v. Weil, U.S. District Court Judge Richard Leon vacated the portion of the Rule that prevents third-party home care providers from using the companionship services exemption, and later vacated the revised definition of companionship services.

As of July 2015, only five states have passed the Domestic Workers Bill of Rights: New York; Hawaii; California; Massachusetts; and Oregon. New York was the first state to pass the law (in July 2010) after six years of efforts by domestic workers, unions, employers, clergy and community organizations. The bill was introduced in two other states, Connecticut and Illinois, but has yet to be passed.

Recruiting And Retaining Educators Of Color

Our guest authors today are Audra Watson, Travis Bristol, Terrenda White and Jose Vilson. Watson is Program Officer and Director of Mentoring and Induction Strategy at the Woodrow Wilson National Fellowship Foundation. Bristol is a Research and Policy Fellow at Stanford Center for Opportunity Policy in Education. White is Assistant Professor at the University of Colorado-Boulder. Vilson is a math educator, blogger, speaker, and activist in New York City, NY

On Thursday, May 7, the White House Initiative on Educational Excellence for African Americans and the Stanford Center for Opportunity Policy in Education (SCOPE) co-sponsored an hour-long webinar, in which researchers, policy makers, and practitioners shared best practices and strategies for increasing the racial/ethnic diversity of the country’s teaching force.

As discussed during the webinar, a racially and ethnically diverse teaching force is important for several reasons. First, in this flat, or interconnected, world, our children need a diverse teaching force to prepare them to be global citizens. Second, teachers of color are positioned to serve as role models and cultural brokers for children of color, who account for 50.2 percent of all U.S. public school students (NCES, 2015). Despite this diverse student population, Latino, Black, Asian, and Native American teachers comprise only 17.3% of all teachers (Ingersoll, Merrill & Stuckey, 2014). Third, several large-scale studies point to increased learning -- as measured by a standardized exam -- for students when they have a teacher of the same race (Dee, 2001; Egalite, Kisida,& Winters, 2015); Not discussed at the time, but equally important, is the fact that a diverse teaching force challenges the assumption that some of the qualities needed most by high-quality, effective teachers -- intelligence, intellectual curiosity, and deep content knowledge -- are difficult to find in large supply amongst individuals of color seeking to enter the teaching profession.

Empower Teachers To Lead, Encourage Students To Be Curious

Our guest author today is Ashim Shanker, a former English Language Arts teacher in public schools in Tokyo, Japan. Ashim has a Master’s Degree in International Education Policy from Harvard University and is the author of three books, including Don’t Forget to Breathe. Follow him on Twitter at @ashimshanker.

In the 11 years that I was a public school teacher in Japan, I came to view education as a holistic enterprise. Schools in Japan not only imbued students with relevant skills, but also nurtured within them the wherewithal to experience a sense of connection with the larger world, and the exploratory capacity to discover their place within it.

In my language arts classes, I encouraged students to read about current events and human rights issues around the world. I asked them to make lists of the electronics they used, the garments they wore, and the food products they consumed on a daily basis. I then had them research where these products were made and under what labor conditions.

The students gave presentations on child laborers and about modern-day slavery. They debated about government secrecy laws in Japan and cover-ups in the aftermath of the Fukushima nuclear disaster. They read an essay on self-reliance by Emerson and excerpts on civil disobedience by Thoreau, and I asked them how these two activists might have felt about the actions of groups like Anonymous, or about whistleblowers like Edward Snowden. We discussed the Milgram Experiment and the Stanford Prison experiment, exploring how obedience and situational role conformity might tip even those with the best of intentions toward acts of cruelty. We talked about bullying, and shared anecdotes of instances in which we might unintentionally have hurt others. There were opportunities for self-reflection, engagement, and character building—attributes that I would like to think foster the empathic foundations for better civic engagement and global citizenship.

Measuring The Success Of Informal STEM Programs For Girls

Despite substantial improvement over the past half decade, the gender pay gap still persists in the U.S. When the Equal Pay Act was passed in 1963, women made 59 cents for every dollar earned by men. Fifty years later, in 2014, women earned 81 cents on the male dollar. One area on which educators and policymakers have focused to rectify the gender pay differential is encouraging young girls to go into the fields of science, technology, engineering and math, or STEM.

According to the U.S. Department of Commerce, women account for half of the workforce in the U.S., but they are employed in less than 25 percent of STEM positions. Women in STEM fields earn 33 percent more than comparable women in non-STEM occupations, but women hold far fewer STEM undergraduate degrees than men, especially in engineering. And, perplexingly, even women with a STEM degree seem less inclined to work in STEM jobs; they are more likely than STEM-educated men to choose education or healthcare. The main explanations for these discrepancies include lack of female role models, discrimination against females at school and in the work place, and gender stereotypes (Hill et al. 2010).

Informal STEM programs offer one potential means of improving female participation in these fields, with the “informal” meaning that the learning occurs in an out-of-school environment (Krishnamurthi and Rennie 2013). A program called Girls Inc. Operation SMART, for example, runs an initiative called Eureka!, which seeks to provide eighth grade girls with internship opportunities in math, science, and technology. The organization Techbridge offers Girls Go Techbridge, which holds learning activities, and a role model training program, which connects girls with volunteers who are passionate about motivating them to get involved in science, technology and engineering. Other initiatives include the National Science Partnership for Girl Scouts and Science Museums (NSP), Women in Natural Sciences (WINS), and Rural Girls in Science.

New School Climate Tool Facilitates Early Intervention On Social-Emotional Issues: Bullying And Suicide Prevention

Our guest author today is Dr. Alvin Larson, director of research and evaluation at Meriden Public Schools, a district that serves about 8,900 students in Meriden, CT. Dr. Larson holds a B.A. in Sociology, M. Ed., M.S. in Educational Research and a Ph.D. in Educational Psychology. The intervention described below was made possible with support from Meriden's community, leadership and education professionals.

For the most part, students' social-emotional concerns start small; if left untreated, though, they can become severe and difficult to manage. Inappropriate behaviors are not only harmful to the student who exhibits them; they can also serve to increase the social bruising of his/her peers and can be detrimental to the climate of the entire school. The problem is that many of these bruises are not directly observable – or not until they become scars. School psychologists and counselors are familiar with bruised students who act out overtly, but some research suggests that 4.3% of our students carry social-emotional scars of which counselors are unaware (Larson, AERA 2014). To develop a more preventative approach, foster pro-social attitudes and a positive school climate, we need to be able to identify and support the students with hidden bruises as well as intervene with pre-bullies early in their school careers.

Since 2011, Connecticut’s Local Education Agencies (LEAs) have been required to purchase or develop a student school climate survey. The rationale for this is that anti-social attitudes and a negative school climate are associated with lower academic achievement, current behavior problems, as well as future criminal behaviors (DeLisi et al 2013; Hawkins et al 2000) and suicide ideation (King et al 2001). There are hundreds of anonymous school climate surveys, but none of them was designed to provide the kind of information that we need to help individual students.

Do We Know How To Hold Teacher Preparation Programs Accountable?

This piece is co-authored by Cory Koedel and Matthew Di Carlo. Koedel is an Associate Professor of Economics and Public Policy at the University of Missouri, Columbia.

The United States Department of Education (USED) has proposed regulations requiring states to hold teacher preparation programs accountable for the performance of their graduates. According to the proposal, states must begin assigning ratings to each program within the next 2-3 years, based on outcomes such as graduates’ “value-added” to student test scores, their classroom observation scores, how long they stay in teaching, whether they teach in high-needs schools, and surveys of their principals’ satisfaction.

In the long term, we are very receptive to, and indeed optimistic about, the idea of outcomes-based accountability for teacher preparation programs (TPPs). In the short to medium term, however, we contend that the evidence base underlying the USED regulations is nowhere near sufficient to guide a national effort toward high-stakes TPP accountability.

This is a situation in which the familiar refrain of “it’s imperfect but better than nothing” is false, and rushing into nationwide design and implementation could be quite harmful.

Will Value-Added Reinforce The Walls Of The Egg-Crate School?

Our guest author today is Susan Moore Johnson, Jerome T. Murphy Research Professor in Education at Harvard Graduate School of Education. Johnson directs the Project on the Next Generation of Teachers, which examines how best to recruit, develop, and retain a strong teaching force.

Academic scholars are often dismayed when policymakers pass laws that disregard or misinterpret their research findings. The use of value-added methods (VAMS) in education policy is a case in point.

About a decade ago, researchers reported that teachers are the most important school-level factor in students’ learning, and that that their effectiveness varies widely within schools (McCaffrey, Koretz, Lockwood, & Hamilton 2004; Rivkin, Hanushek, & Kain 2005; Rockoff 2004). Many policymakers interpreted these findings to mean that teacher quality rests with the individual rather than the school and that, because some teachers are more effective than others, schools should concentrate on increasing their number of effective teachers.

Based on these assumptions, proponents of VAMS began to argue that schools could be improved substantially if they would only dismiss teachers with low VAMS ratings and replace them with teachers who have average or higher ratings (Hanushek 2009). Although panels of scholars warned against using VAMS to make high-stakes decisions because of their statistical limitations (American Statistical Association, 2014; National Research Council & National Academy of Education, 2010), policymakers in many states and districts moved quickly to do just that, requiring that VAMS scores be used as a substantial component in teacher evaluation.

New Policy Brief: The Evidence On The Florida Education Reform Formula

The State of Florida is well known in the U.S. as a hotbed of education reform. The package of policies spearheaded by then Governor Jeb Bush during the late 1990s and early 2000s focused, in general, on test-based accountability, competition, and choice. As a whole, they have come to be known as the “Florida Formula for education success,” or simply the “Florida Formula.”

The Formula has received a great deal of attention, including a coordinated campaign to advocate (in some cases, successfully) for its export to other states. The campaign and its supporters tend to employ as their evidence changes in aggregate testing results, most notably unadjusted increases in proficiency rates on Florida’s state assessment and/or cohort changes on the National Assessment of Educational Progress. This approach, for reasons discussed in the policy brief, violates basic principles of causal inference and policy evaluation. Using this method, one could provide evidence that virtually any policy or set of policies “worked” or “didn’t work,” often in the same place and time period.

Fortunately, we needn’t rely on these crude methods, as there is quite a bit of high quality evidence pertaining to several key components of the Formula, and it provides a basis for tentative conclusions regarding their short- and medium term (mostly test-based impact. Today we published a policy brief, the purpose of which is to summarize this research in a manner that is fair and accessible to policymakers and the public.

How Effective Are Online Credit Recovery Programs?

Credit recovery programs in the U.S. have proliferated rapidly since the enactment of No Child Left Behind (NCLB), particularly in states that are home to a large number of urban schools with high dropout rates (Balfanz and Legters 2004).

Although definitions vary somewhat, credit recovery is any method by which students can earn missed credits in order to graduate on time (Watson and Gemin 2008). Online credit recovery is a common form of these programs, but others include mixed online/in-person instruction, and in-person instruction (McCabe and Andrie 2012). At least three major school districts – Boston, Chicago, and New York City – offer credit recovery programs, as do several states, including Missouri and Wisconsin. Private companies such as Plato, Pearson, Apex, and Kaplan have also tried to fill this niche by offering to charge between $175 and $1,200 per student per credit. Online credit recovery represents approximately half of all instruction in the $2 billion online education industry.

Yet, despite the rising presence of online credit recovery programs, there exists scant evidence as to their effectiveness in increasing high school graduation rates, or their impact on other outcomes of interest.