How many times have you sat through a professional development that promises to be the panacea to all of education’s woes, only for it to be tossed aside a couple of years later when it can’t deliver on its promises? Meanwhile, the frustration at the waste of our time, effort, and money leads to a drop in morale for teachers, students, and parents. So, how do we avoid falling into this trap?
Well, in Part One of this series, we looked at the importance of ensuring that any programs or strategies we choose use strong sampling methods and therefore increase the likelihood that their results will translate to our students. In Part Two, we’ll look at another common reason that “research-based” programs can fall short of delivering their promises: the difference between correlation and causation.
So, what does this mean? Let’s look at an example. It shouldn’t come as a great surprise that ice cream sales go up in the summer. Did you also know that crime goes up in the summer? Therefore, it’s pretty obvious that one of the biggest causes of crime is…eating ice cream.
If that paragraph seems a bit nonsensical to you, that’s because it is. Of course ice cream isn’t the cause of crime; what’s actually happening is that both ice cream consumption and crime go up in warmer weather (people tend to commit less crime when the weather is poor). It’s actually the weather that’s the cause of the crime increase, and the ice cream is just an extraneous variable.
Frozen dairy treats aside, you’ll often see studies in education where this happens. For instance, a researcher might poll students about whether they have regular access to personal computers or other devices at home. The researcher then looks at how the students score on their end-of-year assessments, and realizes that those who said they have regular access to personal computers or other devices score better, on average, than their peers who don’t have access. Therefore, the researcher concludes that having regular access to personal computer or other devices causes student achievement scores to go up, and therefore recommends that district leaders push for a program that will provide devices to all students in their homes.
Do you see what went wrong here?
Students who have access to personal devices at home are often in a higher socioeconomic status. You have probably know from other research that higher socio-economic status tends to be correlated with higher student scores. So, it might not be the devices that are making a difference; it could be that these students have parents with more education who help their children do homework, or these kids have a quiet place to study, or that their parents get them tutors when they don’t do well in a subject… and so on.
Meanwhile, a school or division has spent a lot of money on a program that might only be correlated with student success, rather than the cause. This actually gets to the heart of why educational research can be so tricky: there are just so many variables, and it can be hard to control for them all in an experiment that proves, definitively, that one variable is actually the cause of a rise in student achievement.
So how do we, as school leaders, keep ourselves from the accidentally falling into the correlation = causation trap?
Well, whenever you are presented with educational research, ask to read the actual study. Determine who was in the study, and whether the researchers accounted for things like socioeconomic status, previous achievement, etc., when interpreting their results and drawing conclusions. Here’s an example article on homework effectiveness from Edutopia. Notice how at the end, all the references are listed, including links to several actual studies? That’s great! It equips you as the educational leader with the tools needed to independently apply your knowledge of research “gotchas” when presented with summary claims.
Also, check out more than one study on a particular program or strategy. In fact, the more studies, the better. For instance, on some forms in the eObservations database, cooperating learning is recommended strategy with a .59 effect size, or a 22 percentile point average gain for students who engage in that strategy versus those who do not. Like many of our strategies, this is based on the work of John Hattie and his meta-analyses in his 2009 book Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. Because Hattie synthesizes the results of multiple studies that use very diverse populations, the likelihood that this strategy actually causes gains—rather than is simply correlated with them due to some extraneous factor—is greater.
Of course, you might be thinking that teasing apart correlation versus causation is just so tricky, so why don’t we just rely on the fast-growing sector of brain research for hard data on what works and what doesn’t. Well… we’ll get to that in Part Three.
Interested in this topic? Read more here:
Cook, T.D. & Campbell, D.T. (1979). Quasi-Experimentation: Design & Analysis Issues for Field Settings. Chicago: Rand McNally College Publishing Company.
“Extraneous and confounding variables and systematic vs. non-systematic error” (n.d.) Psychology World. Retrieved from http://web.mst.edu/~psyworld/extraneous.htm.
Hattie, J. (2009). Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement. New York: Routledge.