The 360 degree evaluation is an effective and promising method of assessment for measuring communication, interpersonal skills and professionalism of the trainees in the workplace.  A 360 degree evaluation involves the collection of data (often peer ratings) from multiple sources.  Rodgers et al. describe it as multisource feedback, multi-rater assessment, full-circle appraisal, and peer evaluation (Rodgers & Manifold, 2002).  While 360 degree evaluation can be used to assess all six ACGME core competencies, it is a particularly useful tool for the evaluation of professionalism and interpersonal & communication skills (Lynch et al., 2004). Joshi et al (Joshi, Ling, & Jaeger, 2004), through a ten item questionnaire filled in by the residents and evaluators, found that a 360-degree instrument yielded reliable evaluations of residents’ competency in interpersonal and communication skills, and could effectively be used to guide formative feedback.

A diverse group of practitioners and staff who interact with residents, each with their own perspective, can provide the individual evaluations.  The evaluators may include faculty, fellow residents, medical students, nurses, ancillary staff, patients, families, and the resident self-assessment (Rodgers & Manifold, 2002).  Ramsey and colleagues (1996) in their analysis of Multisource ratings of 187 physicians through 3005 questionnaires found that 10 to 11 responses per assessee were needed to achieve a generalizability coefficient of 0.7. Wood & Campbell (2006) demonstrated the need for eight raters to give an intra-class coefficient of 0.8.

The purpose of this investigation was to assess, via 360 degree evaluations, residents’ communication and interpersonal skills, and to identify their potential weaknesses.  As part of this investigation, we gathered preliminary data on which to judge the efficacy of the evaluation system and the reliability and validity of the multisource ratings.

This was a cross sectional study conducted at the Aga Khan University Hospital between November 2009 and March 2010. This study was a component of a larger project designed to look at the impact of structured verbal feedback on residents’ performance.  Initially, we designed a 360 degree evaluation form based on input of the authors as well as using extensive literature search and expert opinion. The evaluation form included a list of questions pertaining to various attributes of the residents including their communication skills, as well as questions related to the residents’ teamwork skills, punctuality, reliability (dependability), and overall professional competence. Each item (dimension) on the communication instrument was rated on a global rating scale ranging from poor to excellent. A series of discussion meetings were subsequently held in which the evaluation form was further modified and refined based on input from various faculty members. We decided to add an ‘unable to comment’ option if the evaluator did not feel he or she was qualified to make a judgment concerning skill level for a particular attribute. The 360 evaluation form was pilot tested with a total of 15 evaluators including residents, fellows and faculty.  The opinions of the evaluators regarding the form were sought; they generally seemed satisfied with the form and did not suggest any major revisions. Therefore, no major changes in the evaluation form were required, only minor modifications and clarifications in the explanations of what the individual items were measuring.

The final 360 degree evaluation form was then administered to each of the 49 total residents enrolled in our Internal Medicine residency training program, ranging from year 1 (R1) to year 4 (R4). Every resident was evaluated by 8 different evaluators, each of whom had interacted with the resident in the preceding 6 months.  A ninth self-evaluation was also completed by the residents using the same form.  Based on a list of individuals with whom, the resident had interacted with over the past 6 months (as part of their clinical rotations); evaluators were randomly selected by the Research Associate with the help of the Chief Residents. The 8 different evaluators included 2 nurses (including 1 head nurse of a particular ward), 2 faculty members, 1 UR (Unit Receptionist), 1 ward service coordinator (ward manager), and 2 peers (fellow residents).

Informed consent was sought from all residents prior to beginning the study and from each evaluator before they filled out the evaluation forms. Prior to implementation, meetings were also held with all the stakeholders who were part of the study, including head nurses, the faculty, the unit staff, as well as all the residents in order to ensure their full understanding and cooperation, and to facilitate a smooth implementation of the study. Prior to beginning the study, ethics approval was obtained from the Ethical Review Committee (ERC) at the Aga Khan University Hospital (AKUH), Karachi, Pakistan. 

The data was entered and analyzed using SAS version 9.1. The mean scores obtained by each resident for each domain, as well as the mean of all domains for each resident, were calculated and compared. The different categories of evaluators were also grouped together and compared in their mean ratings for all the residents. To gather evidence to support validity, the residents were grouped together based on their year of training, and their mean scores compared. The results were expressed as means (± standard deviation) and percentages. Analysis of Variance (ANOVA) was employed to test for differences in mean scores, both for rater type and residency year.  Based on the total score (average of dimension ratings), Generalizability theory was employed to estimate the reliability of evaluation ratings.(Boulet, 2005; Shavelson, Webb, & Rowley, 1989)

For purposes of analysis and comparison, the raters were divided into 5 broad categories; faculty, nurses, residents, unit staff (unit receptionist/service coordinators) and self evaluators. We received a total of 367 evaluations, but 3 were omitted due to incomplete data. We received 62 evaluations from faculty, 94 from nurses, 95 from the residents, 84 from URs and 32 (of 49) self evaluations.

Batalden, P., Leach, D., Swing, S., Dreyfus, H., & Dreyfus, S. (2002). General competencies and accreditation in graduate medical education. Health Affairs, 21(5), 103-111.

Boulet, J. R. (2005). Generalizability Theory: Basics Encyclopedia of statistics in behavioral science: John Wiley & Sons, Ltd.

Butterfield, P. S., & Mazzaferri, E. L. (1991). A new rating form for use by nurses in assessing residents humanistic behavior. Journal of general internal medicine, 6(2), 155-161.

Davis, D., Mazmanian, P., Fordis, M., Van Harrison, R., Thorpe, K., & Perrier, L. (2006). Accuracy of physician self-assessment compared with observed measures of competence: a systematic review. JAMA: the journal of the American Medical Association, 296(9), 1094.

Epstein, R. M. (2007). Assessment in Medical Education. N Engl J Med, 356, 387-396.

Eva, K., & Regehr, G. (2005). Self-assessment in the health professions: a reformulation and research agenda. Academic medicine: journal of the Association of American Medical Colleges, 80(10 Suppl), S46.

Fletcher, C. (1999). The implications of research on gender differences in self?assessment and 360 degree appraisal. Human Resource Management Journal, 9(1), 39-46.

Hodges, B. D., Ginsburg, S., Cruess, R., Cruess, S., Delport, R., Hafferty, F., . . . Ohbu, S. (2011). Assessment of professionalism: Recommendations from the Ottawa 2010 Conference. Medical Teacher, 33(5), 354-363.

Joshi, R., Ling, F. W., & Jaeger, J. (2004). Assessment of a 360-degree instrument to evaluate residents' competency in interpersonal and communication skills. Academic Medicine, 79(5), 458-463.

Kaplan, C. B., & Centor, R. M. (1990). The use of nurses to evaluate houseofficers humanistic behavior. Journal of general internal medicine, 5(5), 410-414.

Lurie, S. J., Mooney, C. J., & Lyness, J. M. (2009). Measurement of the General Competencies of the Accreditation Council for Graduate Medical Education: A Systematic Review. Academic Medicine, 84(3), 301-309 310.1097/ACM.1090b1013e3181971f3181908.

Lynch, D. C., Surdyk, P. M., & Eiser, A. R. (2004). Assessing professionalism: a review of the literature. Medical Teacher, 26(4), 366-373.

Mabe, P. A., & West, S. G. (1982). Validity of self-evaluation of ability: A review and meta-analysis. Journal of Applied Psychology, 67(3), 280.

Massagli, T. L., & Carline, J. D. (2007). Reliability of a 360-degree evaluation to assess resident competence. American Journal of Physical Medicine & Rehabilitation, 86(10), 845-852.

McLeod, P. J., Tamblyn, R., Benaroya, S., & Snellmd, L. (1994). Faculty ratings of resident humanism predict patient satisfaction ratings in ambulatory medical clinics. Journal of general internal medicine, 9(6), 321-326.

Norcini, J. J. (2005). The Mini Clinical Evaluation Exercise (mini-CEX). The Clinical Teacher, 2(1), 25-30. doi: 10.1111/j.1743-498X.2005.00060.x

Ogunyemi, D., Gonzalez, G., Fong, A., Alexander, C., Finke, D., Donnon, T., & Azziz, R. (2009). From the eye of the nurses: 360-degree evaluation of residents. Journal of Continuing Education in the Health Professions, 29(2), 105-110.

Overeem, K., Wollersheim, H., Driessen, E., Lombarts, K., Van De Ven, G., Grol, R., & Arah, O. (2009). Doctors perceptions of why 360-degree feedback does (not) work: a qualitative study. Medical education, 43(9), 874-882.

Ramsey, P. G., Carline, J. D., Blank, L. L., & Wenrich, M. D. (1996). Feasibility of hospital-based use of peer ratings to evaluate the performances of practicing physicians. Academic medicine: journal of the Association of American Medical Colleges, 71(4), 364.

Ramsey, P. G., Wenrich, M. D., Carline, J. D., Inui, T. S., Larson, E. B., & LoGerfo, J. P. (1993). Use of peer ratings to evaluate physician performance. Journal American Medical Association, 269, 1655-1655.

Rodgers, K. G., & Manifold, C. (2002). 360-degree Feedback: Possibilities for Assessment of the ACGME Core Competencies for Emergency Medicine Residents. Academic Emergency Medicine, 9(11), 1300-1304.

Sargeant, J., Mann, K., Sinclair, D., Van der Vleuten, C., & Metsemakers, J. (2007). Challenges in multisource feedback: intended and unintended outcomes. Medical education, 41(6), 583-591.

Shavelson, R. J., Webb, N. M., & Rowley, G. L. (1989). Generalizability theory. American Psychologist, 44(6), 922.

Swing, S. (2007). The ACGME outcome project: retrospective and prospective. Medical Teacher, 29(7), 648.

Swing, S., & Bashook, P. (2000). Toolbox of assessment methods. version 1.1. Evanston, USA: Accreditation Council for Graduate Medical Education/American Board of Medical Specialties, 15.

Van der Heijden, B. I., & Nijhof, A. H. (2004). The value of subjectivity: problems and prospects for 360-degree appraisal systems. The International Journal of Human Resource Management, 15(3), 493-511.

Van der Vleuten, C. P. M. (1996). The assessment of professional competence: Developments, research and practical implications. Advances in Health Sciences Education, 1(1), 41-67. doi: 10.1007/bf00596229

Wass, V., Van der Vleuten, C., Shatzer, J., & Jones, R. (2001). Assessment of clinical competence. The Lancet, 357(9260), 945-949.

Wenrich, M. D., Carline, J., Giles, L., & Ramsey, P. (1993). Ratings of the performances of practicing internists by hospital-based registered nurses. Academic medicine: journal of the Association of American Medical Colleges, 68(9), 680.

Wilkinson, T. J., Wade, W. B., & Knock, L. D. (2009). A blueprint to assess professionalism: results of a systematic review. Academic Medicine, 84(5), 551-558.

Wood, J., Collins, J., Burnside, E. S., Albanese, M. A., Propeck, P. A., Kelcz, F., Schmaltz, L. M. (2004). Patient, faculty, and self-assessment of radiology resident performance. Academic radiology, 11(8), 931-939.

Wood, L., Hassell, A., Whitehouse, A., Bullock, A., & Wall, D. (2006). A literature review of multi-source feedback systems within and without health services, leading to 10 tips for their successful design. Medical Teacher, 28(7), e185.

Woolliscroft, J. O., Howell, J. D., Patel, B. P., & Swanson, D. B. (1994). Resident-patient interactions: the humanistic qualities of internal medicine residents assessed by patients, attending physicians, program supervisors, and nurses. Academic Medicine.