Here’s a paper we wrote that was recently accepted to the 2008 ASEE Conference (American Society for Engineering Education). In the paper we discuss the development of a concept inventory multiple choice test on AC and DC circuits for high school and undergraduate students. We discuss revisions we made to the test questions in order to improve the test’s reliability and reduce susceptibility to test-wiseness. We found a significant proportion of undergraduate engineering majors still have misconceptions about certain concepts and behaviors relating to electrical circuits even after instruction. We have already begun exploring ways to address this with for example an animated circuit simulation, an online dynamic assessment environment that facilitates qualitative reflection on circuit concepts, and challenge-based (problem-based) labs and course activities.
This apparently isn’t so well known in the literature, so I’d thought I’d share it here. If you are designing a multiple-choice or true-false quiz to assess students’ knowledge and mastery of some material, there is a good chance that part of what you are assessing is not knowledge of the content but rather how good students are at taking tests (test-wiseness). Basically, students can guess the correct answer or narrow down the options based on things such as wording and grammar (secondary cues) rather than knowledge of the topic being assessed.
If you are a student, this is a good thing to know about. If you are teacher/researcher trying to assess student knowledge, you need to closely look at and revise each of your questions based on criteria such as those listed below (derived from references such as these: 1, 2, and 3):
- Categorical Exclusive: Distracters contain words such as “all”, “always”, or “every.” If a choice contains the word “all” or or “every”, a student can usually rule it out. Don’t use those terms in the question or in the choices.
- Phrase-Repeat: Correct answer contains a key sound, word, or phrase that is contained in the question’s stem.
- Absurd/Implausible Answer: Distracters are unrelated to the stem. Some incorrect choices may be more obviously wrong and easier to eliminate. Sometimes a choice sticks out like a sore thumb and is very easy to eliminate.
- Precision/Specificity: Correct answer is more precise, clear, or qualified than the distracters. It is very detailed and provides no room for ambiguity. That suggests it is the correct answer.
- Length: Correct answer is longer than the distracters. Try to make each of the choices relatively equal in length.
- Grammar: Distracters do not match the verb tense of the stem, or there is not a match between articles (“a”, “an”, “the”).
- Give-Away: Correct answer is given away by another item in the test. Sometimes the answer to question 2 can be found in the stem of question 1, for example.
- Order of Answer: Don’t use a predicatible pattern for the answers (for example, the correct answer is usually C or D). In fact, apparently the correct answer IS usually C.
- Number of Options and Guessing: Out of a hundred multiple-choice items with three answer options each, a student with no knowledge of the content will get 33 correct simply by guessing blindly. A five-answer-option question cuts the chances down to 20%.
- Odd Man Out: If you make two or more choices difficult to distinguish from one another conceptually, that usually helps you rule them out. If three answers are very similar and one is very different, the different one may likely be the correct answer.
- Spelling: Another one mentioned is that test authors are more likely to make spelling errors on incorrect choices, since they mainly review the correct answers.
Here’s an example quiz that you can answer correctly even though it doesn’t make sense, due to test-wiseness (see the above link for the answers and explanations):
- The answer to this one refers to an a) overture b) mountain c) building d) misnomer
- In which pifflerock did the zorkrans inkle? a) gi hien b) gis inkle c) gish frankel d) gishen fronks
- If there were a question here, which would be the correct answer? a) vanquished b) victorious c) conquered d) defeated
- Hixek norfolken piffle? a) Hiku nippon ibitus b) Efil Yadlan Ruoj c) Yokon Gnithol d) Ikkek zippo unkerzotz notiaplan hipposlump enslife yolent
- Dome ukerland dimmel? a) Nepal b) Canada c) Britain d) United States
- Zarfarkle, en Ko day? a) Henkledorf b) Ricktoffen c) Ifetain d) Krator
- The correct answer here is a) soveriegn b) glucose c) maveric d) masculinne