OSCE has been designated as one of the three examinations needed to acquire a Medical Practice License in Thailand for the past 4 years. In order to prepare medical students for the national exams, the Faculty of Medicine, Chiang Mai University, has run its own mock-up OSCE exams for final year (6^th year) students every year. The topics tested are based on the same scheme as the National Tests. 20 topics (20 stations), comprising History Taking (4 stations), Physical Examinations (4 stations), Manual Skill (4 stations), Communication Skill (3 stations and Interpretation Skill (5 stations).

This study involves the evaluation of 14 checklist exams undertaken by 185 students. In each checklist a weighed score is assigned to each item according to its difficulty and importance, the total score in each checklist being 100.

A group of 6-8 experts comprising teaching staff from OB-GYN, Surgery, Medicine and Pediatric Department was assigned to review all OSCE questions from the teaching staff from various departments. They then collectively evaluated and weighed each item and determined whether it should be included in the MPL.

In a 14-station OSCE administered to 185 6th-year medical students, each examiner, in addition to evaluating the students' performance on the checklist, also provided the overall rating of students as fail, borderline, pass or excellent. The students' scores were analyzed to obtain a passing score using MPL assigned by experts, adjusted experts' MPL, Borderline Group Method and Borderline Regression Method.

It was found that if the experts' MPL was used, only 46 students (24.86 %) passed the exams. The number passing increased if the MPL from the Adjusted experts' MPL, the Borderline Group Method and the Borderline Regression Method, being 66 (35.67%), 144 (77.83%) and 145 (78.37%), respectively.

Table 1. Comparison of Four Standard Settings

Table 2. Borderline Regression Metrics

The judgmental methods of standard setting give vastly different results from the empirical methods, the former resulting in much higher failure rates. Checklists with the coefficient of determination on R² less than 0.5 need to reviewed. This study has led to an understanding of standard setting for OSCE and eventually to improve of checklists and assessor training.

References :

1. Kaufman DM, Mann KV, Miujtjens AMM, Van der Vleuten CPM. A Comparison of Standard setting procedures for an OSCE in undergraduate medical education. Acad Med 75: 267-71

2. Pell G, Fuller R, Homer M, Roberts TE. How to measure the quality of the OSCE : A review of metrics – AMEE guide no.49. Medical Teacher 32: 802-81

3. Wood T, Humphrey-Murto S and Norman G. Standard Setting in a Small Scale OSCE : A comparison of the Modified Borderline-Group Method and the Borderline Regression Method. Advance in Health Sciences Education. 11:115-122