Articles

Multiple-choice questions are in use for a variety of tests ranging from driving tests to standard attainment tests for school children (not to mention quiz programmes).

Yes, they are easy to mark, but it is sometimes believed that you can't test more than surface learning of facts with multiple-choice questions; an averagely lucky monkey could expect to get reasonable marks in an multiple-choice questions test. Can we use multiple-choice questions to assess effectively the skills and understanding which we wanted to test?

Investigating the myths

Multiple-choice questions are credited with a number of advantages. They are objective, so variations in marking due to subjective factors are eliminated (although the questions themselves still have to be scrutinised to ensure that they are not biased) and this also makes them easy to mark (and they do not necessarily require an experienced tutor to mark them). This is likely to be a significant advantage for over-burdened staff. They are efficient because questions take less time to complete, and therefore it is possible to test a greater range of the syllabus through setting compulsory questions (there seems to be an inherent assumption that multiple-choice questions tests will feature compulsory rather than optional questions). Nevertheless, there are a number of potential issues associated with multiple-choice questions, and these are considered here.

1. Are multiple-choice questions assessments easy?

   This seems to be the major concern raised about multiple-choice questions. They have a reputation as being 'easy' and there is certainly a danger that they may be, but any form of assessment can be designed for a particular level of difficulty. With multiple-choice questions, the difficulty level depends upon the nature of the questions asked and also how the multiple-choice questions fit into the overall scheme of assessment. Careful attention needs to be paid, as with any assessment mechanism, to the ability of the questions to test the intended learning outcomes set for the candidates. In a subject discipline where application and writing skills are required, replacing the whole of an assessment strategy with multiple-choice tests comprising simple knowledge-based questions would not allow an adequate assessment of the required achievement, which would rightly be regarded as dumbing-down - but the same questions might be extremely useful for formative assessment throughout a unit, or as part of the summative assessment. Have you ever despaired, like us, at the desire of candidates to 'question spot by 'learning' only a limited portion of the syllabus? A traditional end-of-year examination where candidates answer a small number of questions out of a range covering the whole syllabus (a very popular model in legal teaching, and one often regarded as the 'proper' way to assess candidate achievement in law) arguably encourages them to do so. Having a compulsory multiple-choice questions section allows efficient testing across a greater range of the syllabus - if that is what you want to do - which is probably a more acceptable alternative to setting a large number of compulsory essays or problems.

   Additionally, multiple-choice questions need not be limited merely to testing knowledge; tests can include more challenging comprehension/application-based questions, and these might well allow more of the intended learning outcomes for a unit to be assessed efficiently and thus be valid for summative assessment. In other words, multiple-choice questions are versatile, and it is only if they are inappropriately used or poorly designed that there is a risk of 'dumbing-down'. Some issues of question design are considered later (4).

2. Are multiple-choice questions scores unrealistically high?

   Also contributing to the reputation of multiple-choice questions as 'easy'; is that the scores tend to be higher than those attained in non-objective assessments in our experience. This exposes the discrepancy between assessments which use the full marking range (0 - 100 per cent) and more traditionally-marked assessments (in undergraduate law at least) where there may be a 'glass ceiling' around the 80 per cent mark. This may be problematic where an multiple-choice questions assessment is used within a diet of conventional assessments. Questions must be carefully designed to ensure the same level of achievement is required for a pass in an multiple-choice questions test as for other comparable assessments (although it is really the subjective 'glass ceiling' effect which is at fault, rather than the multiple-choice questions which are, after all, objective).

   High scores pose less of a problem where the multiple-choice questions are used as part of formative assessment, but candidates may have optimistic expectations of their abilities if they have scored highly in multiple-choice questions throughout the unit but are then faced with a final formal assessment asking them to display different skills. The purposes of any formative assessment (and the summative assessment, if different) need to be made explicit to candidates.

3. Can candidates guess their way to success in an multiple-choice questions test?

   A second scoring issue is that a candidate has a theoretical chance of 'guessing'the correct answer in an multiple-choice questions the 'lucky monkey' problem. Arguably this is no worse than a candidate who adopts the write-all-you-know approach to a question for which s/he can generally expect to pick up marks for the correct points the marker has laboriously identified within the largely irrelevant answer. Nevertheless this does present a problem to which there is no definitive solution, although there are various possibilities which can be adopted6, including:

   • use of negative marking to discourage candidates from equating multiple choice with multiple guess;
   • adopting mathematical strategies to 'normalise marks achieved;
   • ensuring there are sufficient options (answer choices) for each question and/or raising the overall pass mark for the test to reduce the likelihood of a candidate passing through chance.
   
   

   Even if these solutions don't sound satisfactory to make multiple-choice questions a valid option for complete summative assessment, the problems are less significant for formative assessment (although, as already mentioned, care does need to be taken about false candidate expectations where the strategies for formative assessment differ from the summative assessment). However, the advantage of objectivity in multiple-choice questions summative assessment should be stressed. Traditional forms of assessment, for example essays, raise considerable problems in terms of subjective marking, even where assessment criteria and moderation strategies are adopted, and the phrase 'it's an art not a science' defensively employed. One of the lessons we learned was that an assessment method shouldn't be rejected just because it is new or different in a particular subject discipline, any more than another is automatically valid simply because it is the one that has been used for years. In many cases carefully designed multiple-choice questions that are challenging and require satisfaction of (all or part of) the intended learning outcomes may be sufficient to reduce the effect of the scoring problems outlined here.

4. Does using multiple-choice questions assessment encourage rote or surface learning?

   Some candidates might 'rote-learn' whatever method of assessment is adopted. 'Problem' questions allow examiners to distinguish between surface learners and those who have shown understanding through a higher level of application and analysis, and hence avoid rewarding (and by implication encouraging) surface learning. Appropriate multiple-choice questions can be used in the same way to distinguish the surface learner from the deeper learner, where this is the purpose of the assessment, by setting questions which require comprehension and application skills, although testing requires a sufficient number of questions to eliminate those lucky monkeys. We have to remember however that any form of valid assessment is likely to require the candidate to display some form of knowledge acquired by rote learning. Knowledge-based questions are also a useful formative assessment tool to check whether candidates have grasped the basics. Possibly giving candidates a sample of the type of questions they could expect was the best way of encouraging them to adopt appropriate learning methods.

5. Can multiple-choice questions test oral and written skills?

   This may well be a drawback for institutions considering incorporating multiple-choice questioning in their teaching. Multiple-choice questions can test much more than knowledge but they cannot test oral or written skills. They cannot test that candidates can form appropriate arguments and discuss the subtleties of a topic. Where the intended learning outcomes of a unit require oral or written skills a multiple-choice test cannot form the entire assessment. However, multiple-choice questions offer ample scope to test much of the knowledge and skills required by the legal curriculum and can therefore be extremely useful for formative assessment purposes and, where appropriate, as part of summative assessment. We decided to concentrate on what we could test rather than on what we couldn't!

Features of good question design

Many of the potential problems with multiple-choice questions identified above can be 'designed out' with well-written questions. Poorly designed questions may:

• Give away clues to the answer
• Fail to test the skills required by the intended learning outcomes
• Contain implausible distracters - (obviously wrong answers) which can be eliminated by candidates with only limited knowledge
• Encourage rote learning
• Confuse or frustrate candidates with sound understanding

All of the above problems could result in scores which do not reflect candidate ability.

There are many factors but one crucial factor will be the 'distracters' which are written. A popular TV game show provides us with an illustration. The following question came up: Which US state was the third state to accede to the Union in 1787? This could have been a comparatively difficult question, but the level of difficulty became laughable when the options were revealed, which were (a) New Cardy (b) New Woolly (c) New Jumper and what you have probably already realised is the correct answer (d) New Jersey. We use this to emphasise that the art of providing likely wrong answers (or plausible distracters to give them their correct term) is a key factor in writing good multiple-choice questions. The same question would have been harder if the options had been, say, New Hampshire, New Jersey, New Mexico or New York because all four are actually states. Here the difficulty lies not with the question stem itself but with the plausibility of the distracters.

It should be emphasised that tutor perception of the plausibility of a distracter may differ widely from that of candidates taking the test. We once set a question about the burden of evidential proof in a criminal case which we intended to be a fairly simple knowledge-based 'opener' to a test. The scenario involved a man who had been arrested for an offence involving a musical monkey, and we were dismayed at the number of candidates who felt that the burden of proof in this case would be on the monkey (What this says about our teaching is quite possibly more depressing.) One of the best ways of writing plausible distracters is to utilise common candidate mistakes. This forms a 'trap' for weaker candidates but allows candidates with better understanding to demonstrate it. The use of distracters based on common misconceptions in web-based formative assessment situations is explored below.

A further trap which needs to be avoided is the temptation to make a factual multiple-choice questions 'harder' by simply testing more obscure knowledge. Factual questions need to be related to the intended learning outcomes for the course. Best practice suggests that designing harder questions should involve moving through questions which demand more comprehension or application.

Conclusions

The intention is to evaluate the potential multiple-choice questions in areas where they have traditionally been rejected because of the myths we identified earlier. multiple-choice questions are not, ultimately, a mechanism for reducing the marking workload; there are skills they simply cannot test and care needs to be taken with the scoring scheme. So why bother to consider multiple-choice questions? Exploring the use of objective questions really opens up the potential for web-based formative assessment so candidates can get instant feedback and 'individual' (albeit automated) feedback on their understanding. Having the staff resources to provide this individual support face-to-face is an increasingly rare luxury as candidate numbers grow. Of course, feedback from a PC screen isn't the same as from a nice human tutor, but our candidate evaluation forms show most of the candidates find it helpful. Some even prefer it, because it has the advantage of flexibility; candidates can interact at the times that suit them. Online exercises are a valuable way to transform web resources to something more engaging.

Sources

Edwina Higgins and Laura Tatham - Manchester Metropolitan University (http://www.ltu.mmu.ac.uk/ltia/issue4/higginstatham.shtml)