Theme
Clinical Assessment and the OSCE
INSTITUTION
Aga Khan University, Karachi Pakistan
Foundation for Advancement of International Medical Education & Research (FAIMER)
Effective communication and interpersonal skills are key components for the optimal performance of any health care professional. Developing these skills is an integral part of residency training. To assess the interpersonal and communication skills of trainees, and to identify potential areas needing improvement, we conducted 360 degree evaluations of residents. The measurement properties of the multi-source ratings were investigated.
A cross-sectional survey of a cohort of internal medicine residents (PGY-1 – PGY-4) was conducted. Using a 360 degree evaluation technique, every resident was evaluated by eight other co-workers. A self-evaluation was also completed. The mean scores obtained by each resident for each domain were calculated and compared. To gather evidence to support validity, the residents were grouped together based on their year of training, and their mean scores compared. Analysis of Variance (ANOVA) was employed to test for differences in mean scores, both for rater type and residency year. Generalizability theory was employed to estimate the reliability of the ratings.
EVALUATORS |
NO. |
Head Nurse |
1 |
Nurse |
1 |
Unit Receptionist (UR) |
1 |
Service coordinator |
1 |
Peers (fellow residents) |
2 |
Faculty |
2 |
Self |
1 |
Total |
9 |
The 360 degree evaluation technique is effective for measuring the communication skills of trainees. Individuals who interact with trainees on regular basis can, on average, provide meaningful judgments of their abilities provided there are enough of them. Resident self-evaluations may, however, differ from those of other evaluators.
The use of 360-degree evaluation is feasible and can provide meaningful data to guide resident feedback. Three sixty degree evaluations can be used to assess professionalism and interpersonal & communication skills. Gathering non-physician ratings is an integral part of 360 degree evaluation process
I would like to acknowledge internal medicine residents, the nurses and the faculty of the Department of Medicine
I would also like to mention FAIMER Institute for all the help and guidance, which I received from its Faculty and staff
We received a total of 367 completed forms for the 360 degree evaluations (response rate of 83.2%). There was a significant effect attributable to rater type (F=5.2, p<0.01). There were significant differences in mean ratings (p<.05) between the unit staff (M=6.2, SD=1.3) and self evaluations (M=5.4, SD=1.0), and unit staff and nurses (M=5.4, SD=1.3). There were no significant differences in mean scores by level of training. The mean resident self-assessment scores were significantly lower than those provided by faculty (p<0.01). Based on 8 raters, the generalizability of the ratings was 0.39.
Table: Inter-rater differences across various domains
Variables |
Mean Score by Faculty
N= 62 |
Mean Score by Nurses
N= 94 |
Mean Score by Residents
N= 95 |
Mean Score by Unit Staff
N= 84 |
Mean Self Evaluation
N= 32 |
Communication skills attitude with staff /nurses |
5.62
SD = 1.10 |
5.47
SD =1.35 |
5.63
SD = 1.18 |
6.14
SD = 1.39 |
5.18
SD = 1.06 |
Teamwork Skills |
5.60
SD = 1.18 |
5.44
SD = 1.46 |
5.74
SD = 1.25 |
6.05
SD = 1.29 |
5.18
SD = 1.20 |
Compassionate & Respectful |
5.86
SD = 0.99 |
5.56
SD = 1.44 |
5.90
SD = 1.35 |
6.20
SD = 1.39 |
5.62
SD = 1.12 |
Communication skills attitude with patients & families |
5.67
SD =1.26 |
5.48
SD = 1.50 |
5.88
SD = 1.14 |
6.21
SD = 1.27 |
5.59
SD = 0.97 |
Educating & Counseling Patients |
5.53
SD = 1.21 |
5.31
SD = 1.42 |
5.65
SD = 1.18 |
6.20
SD = 1.30 |
5.31
SD = 1.22 |
Punctuality & Accessibility |
5.95
SD = 1.16 |
5.61
SD = 1.31 |
5.81
SD = 1.33 |
6.07
SD = 1.42 |
5.00
SD = 1.52 |
Reliability (Dependability) |
5.83
SD = 1.30 |
5.36
SD = 1.33 |
5.83
SD = 1.26 |
6.26
SD = 1.51 |
5.87
SD = 1.12 |
Overall Professional Competence |
5.76
SD = 1.23 |
5.48
SD = 1.33 |
5.83
SD = 1.16 |
6.25
SD = 1.47 |
5.32
SD = 1.01 |
tot_scr1
|
5.73
SD = 1.03 |
5.43
SD = 1.26 |
5.77
SD = 1.02 |
6.16
SD = 1.28 |
5.39
SD = 0.95 |
tot_scr2
|
5.73
SD = 1.04 |
5.43
SD = 1.26 |
5.78
SD = 1.02 |
6.17
SD = 1.28 |
5.38
SD = 0.93 |
Fig 1: Inter-rater differences
Fig 2: Ratings across domains by level of training