Linda Suskie

  A Common Sense Approach to Assessment & Accreditation

Blog

How hard should a multiple choice test be?

Posted on March 18, 2017 at 8:25 AM

My last blog post on analyzing multiple choice test results generated a good bit of feedback, mostly on the ASSESS listserv. Joan Hawthorne and a couple of other colleagues thoughtfully challenged my “50% rule”—that any questions that more than 50% of your students get wrong may suggest something wrong and should be reviewed carefully.

 

Joan pointed out that my 50% rule shouldn’t be used with tests that are so important that students should earn close to 100%. She’s absolutely right. Some things we teach—healthcare, safety—are so important that if students don’t learn them well, people could die. If you’re teaching and assessing must-know skills and concepts, you might want to look twice at any test items that more than 10% or 15% of students got wrong.

 

With other tests, how hard the test should be depends on its purpose. I was taught in grad school that the purpose of some tests is to separate the top students from the bottom—distinguish which students should earn an A, B, C, D, or F. If you want to maximize the spread of test scores, an average item difficulty of 50% is your best bet—in theory, you should get test scores ranging all the way from 0 to 100%. If you want each test item to do the best possible job discriminating between top and bottom students, again you’d want to aim for a 50% difficulty.

 

But in the real world I’ve never seen a good test with an overall 50% difficulty for several good reasons.

 

1. Difficult test questions are incredibly hard to write. Most college students want to get a good grade and will at least try to study for your test. It’s very hard to come up with a test question that assesses an important objective but that half of them will get wrong. Most difficult items I’ve seen are either on minutiae, “trick” questions on some nuanced point, or questions that are more tests of logical reasoning skill than course learning objectives. In my whole life I’ve written maybe two or three difficult multiple choice questions that I’ve been proud of: that truly focused on important learning outcomes and didn’t require a careful nuanced reading or logical reasoning skills. In my consulting work, I’ve seen no more than half a dozen difficult but effective items written by others. This experience has led me to suggest that “50% rule.”

 

2. Difficult tests are demoralizing to students, even if you “curve” the scores and even if they know in advance that the test will be difficult.

 

3. Difficult tests are rarely appropriate, because it’s rare for the sole or major purpose of a test to be to maximize the spread of scores. Many tests have dual purposes. There are certain fundamental learning objectives we want to make sure (almost) every student has learned, or they’re going to run into problems later on. Then there are some learning objectives that are more challenging—that only the A or maybe B students will achieve—and those test items will separate the A from B students and so on.

 

So, while I have great respect for those who disagree with me, I stand by my suggestion in my last blog post. Compare each item’s actual difficulty (the percent of students who answered incorrectly) against how difficult you wanted that item to be, and carefully evaluate any items that more than 50% of your students got wrong.

Categories: Multiple choice, How to assess

Post a Comment

Oops!

Oops, you forgot something.

Oops!

The words you entered did not match the given text. Please try again.

11 Comments

Reply Nursing Project Help
8:00 AM on March 20, 2018 
My friend recommended this blog and he was totally right keep up the fantastic work! https://nursingassignment.com/
Reply Do My Finance Project
7:58 AM on March 20, 2018 
Good way of telling, good post to take facts regarding my presentation subject matter, which i am going to deliver in my college https://financeprojectshelp.xyz/
Reply Harvard case solution excel file
7:57 AM on March 20, 2018 
Such a nice post, keep providing good resources. https://www.case-study-solutions.com/
Reply Engineering Assignment Help Service
7:55 AM on March 20, 2018 
This site and the resources you provide is really nice keep it up. https://engineeringhelponline.xyz
Reply Harvard Business Case Study Solution
7:53 AM on March 20, 2018 
Such a nice post, keep up the fantastic work https://case-study-analysis.com
Reply Buy Harvard Business Case Studies Solutions
4:03 AM on March 20, 2018 
Science Channel?s are giving a complete knowledge to its viewers about every thing students write done dissertation on this subjects and show its importance. https://casesolutions.xyz/
Reply Business Ethics Homework Help
3:59 AM on March 20, 2018 
only professional writers can make this kind of material, cheers http://assignmentsmba.com/business-ethics-assignment-help-12435
Reply Levi Strauss & Co. (B) Case Study Analysis
3:42 AM on March 20, 2018 
Really i appreciate the effort you made to share the knowledge. This is really a great stuff for sharing. Keep it up . Thanks for sharing. https://caseism.com/levi-strauss-co-b-case-solution-98391
Reply Note on Financial Analysis Solutions to Problems Case Study Analysis
3:25 AM on March 20, 2018 
Science Channel?s are giving a complete knowledge to its viewers about every thing students write done dissertation on this subjects and show its importance. https://casegurus.com/note-on-financial-analysis-solutions-to-pro
blems-21271
Reply Spss Programming Assignment Help
3:10 AM on March 20, 2018 
Amazing article thanks or sharing.. https://www.assignmentsprogramming.xyz/
Reply lilaluna
6:24 AM on December 11, 2017 
I truly value this post. I've been searching all over for this! Thank heavens I discovered it on this blog . I truly respect your work and i seek in future i will return after more data. like this one.You have filled my heart with joy! I have some good work experience with best essay writing service ( https://buyessays.us )and my words are clearly based on what I felt through such processes in the past.