Theme: Assessment: Written and Feedback
  • Currently nan/5
  • 1
  • 1
  • 2
  • 2
  • 3
  • 3
  • 4
  • 4
  • 5
  • 5

Rating: nan/5 (0 votes cast)

Logo
Correlation between scores on multiple choice and short essay questionnaires
Authors: Rune Standal
Marite Rygg
Hilde Grimstad
Torstein Vik
Institutions:
 
Background

Multiple choice questions (MCQ) are considered the most reliable and valid way of assessing understanding and application of knowledge. Although less reliable and valid, essay questionnaires (EQ) are often used to assess reasoning. We wanted to study how these two examination forms correlate and how they agree in identifying students who fail the examination.

Summary of Work

Results of the end-of- the fourth year examinations during 2009-12 at the Faculty of Medicine, Norwegian University of Science and Technology were analyzed. At this examination both MCQs and EQs are applied and a sum score is calculated whereby MCQs count 60% and EQs count 40% of the total score (100%). In order to pass the examination a total score of 65% is required.

Summary of Results

A total of 459 students completed the written part of the included examinations. The correlation between MCQ and EQ scores was high (Pearson correlation coefficient: 0.62; p<0.01). A total of 38 (8.3%) students failed the examination, and 22 (4.8%) failed on both the MCQ- and the EQ-part. Moreover, 40 (8.7%) students failed the MCQ part, while 64 (13.9%) failed the EQ part. The agreement between MCQ and EQ on which students should have failed the examination was only fair (Cohen’s kappa = 0.35).

All MCQ items at the faculty are categorized as either K1 items that only require recall or K2 items that require interpretation and problem solving. MCQ scores for  359 students that graduated in the years 2010 to 2012 where split into a K1 and a K2 score to see whether there was a greater correlation between essay scores and the score obtained on the K2 questions, see figure 3. We found that essay scores are more correlated with the scores obtained on the K2 items than scores obtained on K1 items, (Z = -2.68, p = 0.007, Meng Rosenthal, & Rubin Test).

Conclusion
 
Our results may be consistent with the notion that EQs are less reliable and valid than MCQs, and that more topics may be tested by MCQs. However, the results may also reflect the possibility that different abilities are tested.
Scores obtained on essay questions correlate better with scores obtained on MCQ items that requires interpretation and problem solving than items that only requires recall.
Take-home Messages
A variety of examination forms should be applied in high-stake written examinations.
Background
Summary of Work
Summary of Results

Figure 1: Scatter plot showing MCQ scores versus EQ scores. The crosshair shows the pass/fail score of 65%.

Table 1: Cross tabulation of candidates that failed the MCQ and essay parts respectively.

 

Essay Total
  Failed Passed
MCQ Failed 22 18 40
Passed 42 377 419
Total 64 395 459

 

Figure 2: Scatter plot of candidates that failed the examination based on the total score.

 


Figure 3. Scatter plots of Essay scores versus MCQ scores split in two fractions based on the cognintion levels needed to answer them. Recall items are classified as K1 while items that need interpretation and problem solving are classified as K2.

Conclusion
Take-home Messages
Send ePoster Link