Another must read essay by Bob Tinker was published in the latest newsletter from the Concord Consortium (which he runs). [The other one is an earlier post by him about funding for educational research & development that I discussed earlier.]
This essay is about bad educational research and bad conclusions drawn from that research. He brings up two specific examples which actually I have discussed before also , although he presents them in a much more entertaining way 🙂 One is animations vs. diagrams research. Researchers are finding there is no significant difference between animations and diagrams when you control exactly for the content and interactions (or lack thereof). Tinker refers to this as the “hobbled horse race.” Researchers discard studies showing advantages of animations when the animations provide more content than the diagrams, or when the participants interact with or control the animation (which shows much bigger benefits). As I said in my earlier post on this, the fact that an animation provides more information than a diagram is not a confound but an advantage when you are a designer trying to create the best learning environment possible.
The second example is what Tinker calls the “trivial treatment.” The government spent millions of dollars in a large scale study that showed that kids using software did just as well as kids who did not, but, the catch is that students only used the software on average a few minutes a week. The intervention was too weak to show anything, plus they only assessed learning effects with multiple choice questions, no open ended questions or concept maps or other techniques to assess higher-order and critical thinking skills.
I disagree with Bob that we should simply ignore all of this research though, flawed or not, because as I said in my earlier post about the latter study, there is some validity and there are some lessons to be learned. All too often we see the effectiveness of some research-based educational technology or curricula fall off dramatically the more widely it is disseminated. The exceptions to that rule which I have seen are interventions that tightly control every aspect and virtually every minute of the teaching & learning experience with the software, such as the Read 180 program by Ted Hasselbring, but that isn’t so feasible or desirable for other learning scenarios. And the research on animations is showing that interactivity, giving students control over the pacing of animations rather than merely presenting them, is very important.