Multiple choice questions (MCQ) are considered the most reliable and valid way of assessing understanding and application of knowledge. Although less reliable and valid, essay questionnaires (EQ) are often used to assess reasoning. We wanted to study how these two examination forms correlate and how they agree in identifying students who fail the examination.

Results of the end-of- the fourth year examinations during 2009-12 at the Faculty of Medicine, Norwegian University of Science and Technology were analyzed. At this examination both MCQs and EQs are applied and a sum score is calculated whereby MCQs count 60% and EQs count 40% of the total score (100%). In order to pass the examination a total score of 65% is required.

A total of 459 students completed the written part of the included examinations. The correlation between MCQ and EQ scores was high (Pearson correlation coefficient: 0.62; p<0.01). A total of 38 (8.3%) students failed the examination, and 22 (4.8%) failed on both the MCQ- and the EQ-part. Moreover, 40 (8.7%) students failed the MCQ part, while 64 (13.9%) failed the EQ part. The agreement between MCQ and EQ on which students should have failed the examination was only fair (Cohen’s kappa = 0.35).

All MCQ items at the faculty are categorized as either K1 items that only require recall or K2 items that require interpretation and problem solving. MCQ scores for  359 students that graduated in the years 2010 to 2012 where split into a K1 and a K2 score to see whether there was a greater correlation between essay scores and the score obtained on the K2 questions, see figure 3. We found that essay scores are more correlated with the scores obtained on the K2 items than scores obtained on K1 items, (Z = -2.68, p = 0.007, Meng Rosenthal, & Rubin Test).