Multiple choice tests

Using Multiple Choice Tests in higher education can be controversial. It can seem to students as if multiple choice questions:

  • test only their recall of facts and not their understanding of content.
  • award a lot of marks on the basis of pure guesswork.
  • are only introduced to make grading easier for teachers.

At the same time, it is worth noting that well designed multiple choice questions:

  • can test the ability to derive, develop or apply ideas as well as the ability to recall facts.
  • can be constructed so guess work is not rewarded.
  • can allow for more questions in the same time period, when compared to traditional exams; this can allow the test to assess a wider range of topics.
  • can allow more consistent grading than in the case of some traditional exams.

It is also worth noting that there is substantial research evidence that students who do well on traditional exams typically also do well on well-constructed multiple choice tests. This has been found to be true across a wide range of disciplines including science and engineering subjects. In fact, for numerical-type questions, both multiple choice and traditional type questions do seem to measure the same underlying student abilities or characteristics.

Given the sometimes negative student reaction to multiple choice tests, if you would like to use a multiple choice test in your course, you should first discuss it with the section director.

Automated correction and scanning of multiple choice tests 

Multiple choice tests typically take a lot longer to develop than traditional exams. However, with the use of optical mark reader software, the process of grading can be quicker. The trade-off between time spent developing the exam and time spent grading it means multiple choice questions are often not a good investment of time for smaller classes but can be worthwhile for larger classes.

Learn more about the available services and resourcees for Automated correction and scanning of multiple choice tests

It should be noted that the regulations applying to written exams apply in the case of multiple choice tests. See here for a summary of relevant regulations.

One challenge with multiple choice questions is that students who do not know the answers can guess them: given a test with 4 possible answers to each question, anyone with a pencil can get 25% correct on average.  Worse, this seems to reward weaker students over stronger ones (since, if you know nothing you can get 25% by guessing, but if you know half the answers you can only get an additional 12.5% by guessing).

Indeed, if students use a bit of logic, they can actually reduce the chance of a wrong answer even further:

  • If some of the proposed answers are logically implausible or obviously wrong, then students may only have to guess between two or three possibilities.
  • Typically, the different possible wrong answers are chosen to reflect student errors. In calculation questions, some errors will cause things to be overestimated and some will cause them to be underestimated. Therefore the middle answer is often the student’s best guess.
  • If there are not enough wrong answers (for example, in simple true/false questions) the reward for guessing is quite high.

One way of addressing this problem is through the grading system. There are two options you might consider:

  • Raising the bar to reflect the fact that some of the marks can be obtained by guessing. For example, if you have 4 options per question, you might rescale your marking scheme based on a 25% to 100% scale (since students can get, on average, 25% of the marks ‘for free’).  In this case a mark of 62.5% on a multiple choice test would be thought of as being comparable to a mark of 50% on a traditional exam, and a mark of 85% would be thought of as being comparable to a mark of 80% on a traditional exam.
  • Penalising guesses through negative marking. You can discourage students from guessing by subtracting marks for an incorrect answer, but awarding no marks for a blank answer.  For example, if I have an exam with 4 possible correct answers and 5 marks per correct answer, I could also have -1.66 marks for an incorrect answer (this would give a score of zero, on average, for a student who responded with nothing but guesses).

Which option you take is up to you (there is some research evidence that in fact both systems give rise to similar outcomes).

Good multiple choice questions are first and foremost, good questions. The following are some suggestions for writing good multiple choice questions.

  • The question should be important. Don’t focus on trivial issues or ‘tricky’ issues.
  • Ideally, some items will assess application of knowledge and thinking rather than just recall. For example if students have been given a formula for calculating the force of gravity exerted between two masses at a given distance, you could ask them for the formula (a recall question), or you could ask them to derive a formula relating gravity to density and volume (an application of knowledge question).
  • Ideally the question should be self-contained and it should be possible to answer the question without seeing the options.

A good starting point is to make a list of questions that seem to you to be relatively important in your course. Try then to phrase them in such a way that they have a single correct answer. This then becomes your question set.

In multiple choice questions the wrong answers are called ‘distractors’.

Writing distractors is often the hardest part of writing multiple choice exams.

A good distractor is a plausible distractor and a plausible distractor is usually one that is based upon common student errors. This means that writing good wrong answers will depend on knowing your students well. You might:

  • ask your assistants to take a note each week of common errors in exercise sessions and use this to design distractors.
  • take a note of student questions or points that you are asked to explain a second time.
  • build a short exercise into your lecture every now and again and collect the (anonymous) answers.
  • look at exercises in textbooks (in good textbooks exercises are often designed to help students recognize common errors).

Here are a few further tips for writing good distractors

  • Present all the answers in a logical order (smallest to largest, for example), so that the order of answer does not in itself give the reader a clue. An alternative is to present all answers in completely random order (this can be done, for example, with the AMC software).
  • Make sure, if possible, that excluding one answer does not automatically exclude another (for example, if one answer is “warmer than zero degrees Centigrade” and a second answer is “above boiling point for water”, then excluding the first automatically excludes the second).
  • Try to have distractors of similar length (when faced with one answer longer than another, students generally find that the longer answer is more often than not the right one).
  • For numerical questions, try to vary the position of the correct answer rather than having it consistently in the middle.
  • Try to make sure that all distractors are grammatically consistent with the question (students often find that the grammatically consistent answer is more likely to be the correct one!)
  • If you need a lot of detail, put it in the question and not in the answer. Long questions with short possible answers are better than short questions with long possible answers.
  • Have enough good distractors. Some authors recommend providing at least five options (i.e. four distractors and a correct answer).
  • Try to avoid using “none of the above” as an answer.

Timing is important in multiple choice exams (just as in other exams). There are a number of specific issues to take into account with multiple choice exams:

  • you want your students to think about and figure out the answer; however, if students feel under time pressure, they may be more likely to simply guess answers.
  • choosing the right answer can often involve reading and understanding both the questions and the answers very precisely. This can be an issue, for example, where distractors appear quite similar to each other but have a subtle, but crucial, difference. For students taking a course through a second-language, it may take them longer to read and understand questions with this level of precision, when compared to native speakers. If the timing of a test is tight, therefore, the test may end up being a test of language fluency rather than of content knowledge.

Just as with any other type of exam, some multiple choice questions will take longer than others. For example, questions that require sketching, graphing, deriving or calculating may well take longer than definition-based questions. It is therefore impossible to set a rule of thumb as to how many questions can be asked in a given time period. One approach that is sometimes suggested, however, is to have an assistant complete the test and give the students four times as long to complete the exam (this may need to be even longer for students learning through a second-language).

As with any other assessment it can be a good idea to run some statistical tests on your exam. This can give you feedback that will help you to improve the design of the assessment for the following year. These include:

  • Testing the difficulty of each question.
  • Testing how well the question discriminates between better and weaker students.

These tests are described in the “assess your assessment” section of the advice on grading page.

For multiple choice tests, it can also be useful to look at which distractors were most effective and which were not at all effective. You may want to consider revising ineffective distractors if a similar question is to be used in the future.

These tests can be easily run with spreadsheet package, while optical reader software used for multiple choice exams can sometimes calculate these kind of statistics automatically.