Linda Suskie

  A Common Sense Approach to Assessment in Higher Education

Blog

How hard should a multiple choice test be?

Posted on March 18, 2017 at 8:25 AM

My last blog post on analyzing multiple choice test results generated a good bit of feedback, mostly on the ASSESS listserv. Joan Hawthorne and a couple of other colleagues thoughtfully challenged my “50% rule”—that any questions that more than 50% of your students get wrong may suggest something wrong and should be reviewed carefully.

 

Joan pointed out that my 50% rule shouldn’t be used with tests that are so important that students should earn close to 100%. She’s absolutely right. Some things we teach—healthcare, safety—are so important that if students don’t learn them well, people could die. If you’re teaching and assessing must-know skills and concepts, you might want to look twice at any test items that more than 10% or 15% of students got wrong.

 

With other tests, how hard the test should be depends on its purpose. I was taught in grad school that the purpose of some tests is to separate the top students from the bottom—distinguish which students should earn an A, B, C, D, or F. If you want to maximize the spread of test scores, an average item difficulty of 50% is your best bet—in theory, you should get test scores ranging all the way from 0 to 100%. If you want each test item to do the best possible job discriminating between top and bottom students, again you’d want to aim for a 50% difficulty.

 

But in the real world I’ve never seen a good test with an overall 50% difficulty for several good reasons.

 

1. Difficult test questions are incredibly hard to write. Most college students want to get a good grade and will at least try to study for your test. It’s very hard to come up with a test question that assesses an important objective but that half of them will get wrong. Most difficult items I’ve seen are either on minutiae, “trick” questions on some nuanced point, or questions that are more tests of logical reasoning skill than course learning objectives. In my whole life I’ve written maybe two or three difficult multiple choice questions that I’ve been proud of: that truly focused on important learning outcomes and didn’t require a careful nuanced reading or logical reasoning skills. In my consulting work, I’ve seen no more than half a dozen difficult but effective items written by others. This experience has led me to suggest that “50% rule.”

 

2. Difficult tests are demoralizing to students, even if you “curve” the scores and even if they know in advance that the test will be difficult.

 

3. Difficult tests are rarely appropriate, because it’s rare for the sole or major purpose of a test to be to maximize the spread of scores. Many tests have dual purposes. There are certain fundamental learning objectives we want to make sure (almost) every student has learned, or they’re going to run into problems later on. Then there are some learning objectives that are more challenging—that only the A or maybe B students will achieve—and those test items will separate the A from B students and so on.

 

So, while I have great respect for those who disagree with me, I stand by my suggestion in my last blog post. Compare each item’s actual difficulty (the percent of students who answered incorrectly) against how difficult you wanted that item to be, and carefully evaluate any items that more than 50% of your students got wrong.

Categories: Multiple choice, How to assess

Post a Comment

Oops!

Oops, you forgot something.

Oops!

The words you entered did not match the given text. Please try again.

29 Comments

Reply 카지노사이트
7:17 PM on October 13, 2019 
Thank you for your information and good article.
https://star77.app 바카라사이트
https://star77.app/contact/ 온라인바카라
https://star77.app 카지노사이트
https://star77.app 온라인카지노
https://star77.app/services/ 실시간바카라
https://star77.app 인터넷카지노
https://star77.app 슬롯사이트
https://star77.app/contact/ 카지노주소
https://star77.app/about/ 밀리언클럽
https://star77.app/blog/ 로얄카지노
https://star77.app/services/ 카심바코리아
https://star77.app 맥스카지노
https://star77.app 슬롯머신게임
https://star77.app 모바일카지노
Reply Activate espn
3:29 PM on September 28, 2019 
Wow, amazing blog layout! How long have you been blogging for?
you make blogging look easy. The overall look of your site
is fantastic, as well as the content!
Reply 온카지노
11:10 PM on September 4, 2019 
온카지노 http://www.88enk.com/ 온카지노
온카지노 http://www.88enk.com/ 온카지노카톡
온카지노 http://www.88meh.com/ 온카지노직영
온카지노 http://www.88meh.com/ 온카지노본사
온카지노 http://www.wjj77.com/ 온카지노콤프
온카지노 http://www.wjj77.com/ 온카지노루징
붐카지노 http://xn--o80b92upyjuve.com/ 붐카지노VR
붐카지노 http://xn--o80b92upyjuve.com/ 붐카지노아이폰
더드림카지노 https://xn--o80b78av7awzjz9wnog.com/ 카지노영상
더드림카지노 https://xn--o80b78av7awzjz9wnog.com/ 라이브카지노

온카지노 http://www.88enk.com/ 온카지노카톡
온카지노 http://www.88enk.com/ 온카지노추천
온카지노 http://www.88enk.com/ 온카지노바로가기
온카지노 http://www.88enk.com/ OnCasino
온카지노 http://www.88enk.com/ 온카지노합법
Reply casino
11:29 PM on August 25, 2019 
https://www.rnm77.com/ 카지노
https://www.rnm77.com/casinosa/ 카지노사이트
https://www.rnm77.com/yescasino/ 예스카지노
https://www.rnm77.com/firstcasino/ 퍼스트카지노
https://www.rnm77.com/thekingcasino/ 더킹카지노
https://www.rnm77.com/hgame/ 호게임
https://www.rnm77.com/blog-1/ Blog
https://yes1477.com/ 예스카지노
https://time14.com/ 더킹카지노
https://first147.com/ 퍼스트카지노
https://www.dbk222.com/ 온라인카지노
https://www.dbk222.com/yescasino11/ 예스카지노
https://www.dbk222.com/thekingcasino11/ 더킹카지노
https://www.dbk222.com/firstcasino11/ 퍼스트카지노
https://www.dbk222.com/dkswjs11/ 안전놀이터
https://www.dbk222.com/casinosa11/ 카지노사이트
https://www.dbk222.com/blog/ 블로그
https://www.caba-agency.com/ CABA카지노
https://www.caba-agency.com/yescasino/ 예스카지노
https://www.caba-agency.com/thekingcasino/ 더킹카지노
https://www.caba-agency.com/firstcasino/ 퍼스트카지노
https://www.caba-agency.com/blog/ blog
Although these theologians showed her great consideration, their eternal questions wearied her; their slowness and heaviness exasperated her.
Reply 바카라카지노PPAP3.COM
8:52 AM on August 15, 2019 
Reply online casino
3:35 AM on July 24, 2019 
If you're looking for a proven and reliable casino company, this is it. You can enjoy various games such as baccarat, blackjack, roulette, mezabo, and big wheel safely.
https://www.zzy29.com/trump
Reply 카지노사이트
3:34 AM on July 24, 2019 
Baccarat site - the top in the industry. Staying 1 in the casino industry for 10 consecutive years! Proven baccarat site Enjoy a variety of games including baccarat, blackjack, roulette, and slot machines on the Baccarat site.
https://www.zzy29.com
Reply occ369
7:04 AM on July 10, 2019 
Great Article it its really informative and innovative keep us posted with new updates. its was really valuable. thanks a lot https://www.occ369.com
Reply agency
7:01 AM on June 25, 2019 
When you come to the CN Agency, you can enjoy a variety of end-of-line casino games.
CN-agency homepage is : https://www.cn-agency.com
Reply mvpgame
7:00 AM on June 25, 2019 
Woori casino series or our affiliate casino means the lower affiliates that originated from our casino. From now until 11 years ago, our casino has been operating independently under the name of our casino, but since 2007 we have been operating under the name of our casino operating subsidiary operating system online The casino site system has changed the way we operate. We operate the lower affiliates in our casinos as our casinos or our casinos.
My web site is : https://www.mvpgame-win.com
Reply bme787
6:59 AM on June 25, 2019 
If you have ever been to a casino site or have been around for a long time, you might have heard it before. It is the word "our casino or our casino".
Our casino, which can be called the history of Korea online casino site in 2007, will stop its own operation and will be divided into shares and operated by several subsidiaries.
Each year we change our name, and we call this lineage our casino series or our series to renew the renewal.
The wooricasino homepage is : https://www.bme787.com
Reply Kamila (Seven luck cas, Kor)
8:21 AM on May 29, 2019 
Reply Freya (Seven luck cas, Kor)
8:20 AM on May 29, 2019 
Reply Ruth (Seven luck cas, Kor)
8:20 AM on May 29, 2019 
https://dancesweb.com/woori/ - 우리카지노
https://dancesweb.com/ - 슈퍼카지노
https://dancesweb.com/theking/ - 더킹카지노
https://dancesweb.com/first/ - 퍼스트카지노

https://dancesweb.com/trump/ - 트럼프카지노
https://dancesweb.com/33casino/ - 33카지노
https://dancesweb.com/gatsby/ - 개츠비카지노

https://dancesweb.com/bacara/ - 바카라사이트
https://dancesweb.com/casino/ - 카지노사이트
https://dancesweb.com/yesca/ - 예스카지노
https://dancesweb.com/yesno/ - YES카지노
https://dancesweb.com/obama/ - 오바마카지노

https://dancesweb.com/coincasino/ - COIN카지노
https://dancesweb.com/coinkore/ - 코인카지노
https://dancesweb.com/theninecasino/ - 더나인카지노
Reply Angela (Seven luck cas, Kor)
8:19 AM on May 29, 2019 
https://custory.com/super/ - 슈퍼카지노
https://custory.com/theking/ - 더킹카지노
https://custory.com/woori/ - 우리카지노
https://custory.com/33casino/ - 33카지노
https://custory.com/gatsby/ - 개츠비카지노

https://custory.com/first/ - 퍼스트카지노
https://custory.com/bacara/ - 바카라사이트
https://custory.com/cass/ - 카지노사이트
https://custory.com/trump/ - 트럼프카지노
https://custory.com/foryou/ - 포유카지노

https://custory.com/mcasino/ - M카지노
https://custory.com/ - 월드카지노
https://custory.com/f1casino/ - F1카지노
https://custory.com/ebiang/ - 에비앙카지노
https://custory.com/4ucasino/ - 4U카지노

https://custory.com/davinchi/ - 다빈치카지노
https://custory.com/live/ - 라이브카지노
https://custory.com/yes/ - 예스카지노
https://custory.com/yescasino/ - YES카지노
https://custory.com/obama/ - 오바마카지노

https://custory.com/coincasino/ - COIN카지노
https://custory.com/coinkor/ - 코인카지노
https://custory.com/theninecasino/ - 더나인카지노
Reply Michelle (Seven luck cas, Kor)
8:19 AM on May 29, 2019 
https://como79.com/ - 우리카지노
https://como79.com/theking/ - 더킹카지노
https://como79.com/yescasino/ - 예스카지노
https://como79.com/yes/ - YES카지노
https://como79.com/mcasino/ - M카지노

https://como79.com/obama/ - 오바마카지노
https://como79.com/first/ - 퍼스트카지노
https://como79.com/super/ - 슈퍼카지노
https://como79.com/trump/ - 트럼프카지노
https://como79.com/gatsby/ - 개츠비카지노

https://como79.com/coincasino/ - COIN카지노
https://como79.com/coinkorea/ - 코인카지노
https://como79.com/theninecasino/ - 더나인카지노
Reply rental mobil medan
4:30 AM on September 18, 2018 
I am in love with this web site. I have visited this site so frequently. I discovered this web site on the internet. I have gotten a good stuff of information. Many thanks. keep updating me
https://www.aretarentcarmedan.com
Reply rental mobil medan
4:29 AM on September 18, 2018 
Reply Nursing Project Help
8:00 AM on March 20, 2018 
My friend recommended this blog and he was totally right keep up the fantastic work! https://nursingassignment.com/
Reply Do My Finance Project
7:58 AM on March 20, 2018 
Good way of telling, good post to take facts regarding my presentation subject matter, which i am going to deliver in my college https://financeprojectshelp.xyz/