A recent study found that female students with more feminine names (such as Isabella, Anna, as rated by others) were less likely to pursue careers in math and science than female students with less feminine names (Abigail, Ashley). It reminds me of another study (can’t remember reference) that found that the simple act of asking students to state their gender and race on a test affects their test scores (negatively, for women and minorities). You see these kind of “devil in the details” examples all the time in educational research and educational design:
A tiny change in wording of a test question, even if it doesn’t change the “meaning” of a question, can significantly influence students’ answers, and sometimes devalue the usefulness of the question. We all know from experience that when you see words like “never” or “always” in a test question, that’s a clue to what answer to choose. That’s why there are many books on test development and design, and you usually have to come up with many more questions than you will end up using, and test your questions out with hundreds of students before the test is finally ready and validated.
A small change in a visual representation of some concept, such as electrical current, can fundamentally alter students’ conceptions. Do you show current as particles flowing through a pipe? That might lead students to an “empty pipe” misconception about current, thinking current fades or dies out as it travels through a wire, in violation of Kirchoff’s Laws.
If you want to add a helpful 3D avatar to your software application, what should it look like? How should it behave? If it’s gender and racial makeup doesn’t match that of the learner, that may impact its effectiveness. If it looks realistic vs. cartoony, that has an impact. If the avatar looks at you when you are speaking or typing vs. looking around randomly, that can make a difference, etc. See for example this paper:
The visual appearance of the avatars was either basic and genderless (like a “match-stick” figure), or more photorealistic and gender-specific. Similarly, eye gaze behavior was either random or inferred from voice, to reflect different levels of behavioral realism.Our comparative analysis…inferred gaze significantly outperformed random gaze. However responses to the lower-realism avatar are adversely affected by inferred gaze, revealing a significant interaction effect between appearance and behavior.
Our intuitions about how students learn best and what choices to make in designing a learning environment are often wrong. Teachers and even experts have “blind spots.” We forget what it was like to learn a topic after we have mastered it, or we have mistaken conceptions about the nature of learning.
We all know how tough math word problems are, for example. It seems like it would be better for students to just present the math problem straight-forward. Well it turns out this isn’t always the case. From the abstract of “Expert Blind Spot Among Pre-Service Mathematics and Science Teachers“:
In this work we investigate “Expert blind
spot” (EBS). Participants (n=16) were pre-service high school teachers majoring in mathematics or science. All had high levels of mathematics content knowledge. As predicted, participants’
rankings of mathematics problems favored a “symbol precedence view” of algebraic development (r=.94), and over 80% agreed that students’ symbolic reasoning precedes verbal reasoning and story problem solving. These findings are contrary to high school students’ actual performance patterns. Participants’ views also appear to conflict with others views they hold of
learning and pedagogy, which follow reform-based views. These data, along with prior findings in language arts and medical education suggests that EBS does exist and can adversely affect educators’ beliefs and practices.
Say you need to give instructions to someone, basic instructions such as how to get somewhere or how to put something together. Telling them the instructions is best, of course? This study by Tversky & Lozano found surprisingly that using gestures alone can sometimes be better (like the game of charades):
Communicators explained how to assemble an object or how to get from A to B to recipients, using speech and gesture freely, or restricted to gesture alone. Explanations were examined for uses of gesture and speech in conveying semantic content as well as narrative structure. For both tasks, communicators who explained using only gestures performed better than those using gestures and speech. Recipients also learned both tasks better after watching explanations that only used gestures.
The importance of details is also important in game design. You can have the best and most realistic graphics of any game, but not good gameplay. The developers of the Nintendo Wii spent an inordinate amount of time finessing the dynamics of the motion-sensitive controller, since it is the key to gameplay. So educational researchers and designers can learn from that field as well.
An important lesson is that the environment you designed will not be perceived by users the same way as you:
When a designer constructs a computer microworld to represent some subdomain of mathematical or scientific knowledge, there is no guarantee that the user will see what the designer intended” (p. 73).
Edwards, L.D. (1998). Embodying mathematics and science: Microworlds as representations. Journal of Mathematical Behavior, 17(1),53-78.
And sum it all up, I would probably agree that educational research and design/development is “the hardest science of all”, which is the title of a 2002 paper by David Berliner. See also the other articles in this special issue of Educational Researcher, and an excerpt:
Doing science and implementing scientific findings are so difficult in education because humans in schools are embedded in complex and changing networks of social interaction. The participants in those networks have variable power to affect each other from day to day, and the ordinary events of life (a sick child, a messy divorce, a passionate love affair, migraine headaches, hot flashes, a birthday party, alcohol abuse, a new principal, a new child in the classroom, rain that keeps the children from a recess outside the school building) all affect doing science in school settings by limiting the generalizability of educational research findings. Compared to designing bridges and circuits or splitting either atoms or genes, the science to help change schools and classrooms is harder to do because context cannot be controlled.