ePoster
Abstract Title | Are learning portfolios useful in assessing paediatric competencies? A resident perspective

Authors

  1. Dr. Meera Umamaheswaran & Dr. Moyez Ladhani

Theme

Portfolios and ePortfolios

Category

Portfolios

INSTITUTION

McMaster University

Conclusion

Overall paediatric residents at McMaster perceive portfolios as useful tools for tracking professional development which helps them to identify and achieve their career goals.  The major drawback highlighted was that of time.  Where, similar to the literature, paediatric residents felt portfolios were too time consuming to maintain.  Requests to make the portfolio electronic may help to improve the time demands and efficiency of cultivating one throughout a busy residency. 

 

Despite the literature’s suggestion that reflective practice improves physician self awareness, paediatric residents at McMaster perceived reflections less favourably.  More promising, however, is the increasing appreciation for meta-cognition in developing advocacy skills and in resolving difficult situations and ethical dilemmas with progression from junior to senior residency. This would suggest that through training thinking evolves and matures allowing residents to become more reflective in their practice.  In addition, the personal nature of reflections should be treated with sensitivity and confidentiality if we are to expect residents to divulge their perceived weaknesses and frustrations genuinely without fear of repercussion.

Background

The role of a physician has evolved from a ‘medical expert’ to encompass those of a communicator, collaborator, manager, health advocate, scholar, and professional, as outlined in CanMEDS by the Royal College of Physicians and Surgeons of Canada.  The assessment of knowledge as a ‘medical expert’ is less equivocal than assessing proficiency at the other expected competencies.  The learning portfolio is one method that is becoming popular worldwide to more efficiently track and assess intrinsic competencies.  However, there are few reports on resident perceptions of this process, and almost none specific to paediatric residents.

Summary of Work

The Paediatric Residency Program at McMaster University implemented learning portfolios in 2008, with the purpose of guiding a resident’s progress through reflective learning and by providing them a means to record their evaluations, learning contracts, activities and accomplishments by CanMEDS roles, thereby helping to demonstrate the competencies achieved.  To determine the usefulness of the learning portfolio as well as resident perception of and attitude towards it, a survey was sent to paediatric residents at McMaster.

Take-home Messages

Acknowledgement

Special thanks to Ms. Sharyn Kreuger for survey design and administration and Dr.Kelly Dore for statistical analysis.

Summary of Results
References

 

References

1.       van Tartwijk, J, and E Driessen. “Portfolios for assessment and learning: AMEE guide no.45.” Medical Teacher 45, no. 31 (2009): 790-801.

2.       Driessen, E, J van Tartwijk, CPM van der Vleuten, and V Wass. “Portfolios in medical education: Why do they meet with mixed success? A systematic review.” Medical Education, 2007b: 1224-1233.

3.       Epstein, R, and E Hundert. “Defining and assessing professional competence.” JAMA 287, no. 2 (2002): 226-235.

4.       Royal College of Physicians andSurgeons of Canada. “CanMEDS: better standards, better physicians, better care.” Royal College of Physicians and Surgeons of Canada. 2012. http://www.royalcollege.ca/portal/page/portal/rc/canmeds (accessed 2012  June).

5.       Royal College of Physicians and Surgeons of Canada. “CanMEDS 2005 Framework.” Royal College of Physicians and Surgeons of Canada. 2005. http://www.royalcollege.ca/portal/page/portal/rc/common/documents/canmeds/framework/the_7_canmeds_roles_e.pdf (accessed 2012 June).

6.       Epstein, R. “Assessment in medical Education.” N Engl J Med 356 (2007): 387-396.

7.       Carraccio, C, and R Englander. “ANALYSES/LITERATURE REVIEWS: Evaluating Competence Using a Portfolio: A Literature Review and Web-Based Application to the ACGME Competencies.” Teaching and Learning in Medicine: An International Journal 16, no. 4 (2004): 381-387.

8.       Nagler, A, K Andolsek, and J Padmore. “The unintended consequences of portfolios in graduate medical education.” Academic Medicine 84, no. 11 (2009): 1522-1526.

9.       Royal College of General Practitioners. “Portfolio-based learning in general practice: report of a working group on higher professional education.” Occaisional paper 63, 1993.

10.   van Tartwijk, J, E Driessen, K Stokking, and C van der Vleuten. “Factors influencing the successful introduction of portfolios.” Qual Higher Educ 13, no. 1 (2007): 69-79.

11.   Ladhani, M. “Resources for Residents.” McMaster Pediatrics Residency Program. 2008. http://www.macpeds.com/documents/MacPedsPortfolio.pdf (accessed 2012  January).

12.   Kalet, A, et al. “Promoting professionalism through an online professional development portfolio: successes, joys and frustrations.”Acad Med. 82 (2007): 1065-1072.

13.   Davis, M, G Ponnamperuma, and J Ker. “Student perceptions of a porfolio assessment process.” Medical Education 43 (2009): 89-98.

Conclusion
Background

The evolutionary landscape of medical curricula over the last 20 years has shown a shift from knowledge based to competency based education in an effort to develop well rounded health professionals and ultimately improve health outcomes.(1)

 

Professional competence has been defined as “the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individuals and communities being served.”(2) (3)  Thus, it requires the application of theoretical knowledge in relevant real life tasks rather than just the acquisition of knowledge.

 

This trend has been widely supported by many professional organizations such as the Royal College of Physicians and Surgeons of Canada (RCPSC), the Accreditation Council of Graduate Medical Education (ACGME) and the General Medical Council (GMC) to name a few.  In 1996 the RCPSC formally adopted an educational competency-based framework that it developed known as the CanMEDS Physician Competency Framework” to describe the knowledge, skills and abilities expected of physicians in Canada that would lead to improved patient care.  Central to the seven roles is that of medical expert.  In addition, the physician is also expected to embody other “soft skills” and be a communicator, collaborator, manager, health advocate, scholar and professional. (1) (4) (5)


This shift in paradigm has been gaining significant momentum over recent years due to challenges in finding an appropriate evaluative tool that demonstrates the achievement of competence.  Since competence tends to be more of an inferred quality and is less directly measurable, traditional methods for knowledge based learning are not as reliable.(2) (3) (6)  The end result of appraisement should be to foster learning, inspire confidence, and enhance one’s ability to self assessment.(1) (3) (7)


The literature would suggest that the portfolio shows the greatest promise as either a formative and/or summative evaluative instrument and has been highly recommended by the ACGME as one of the most desirable evaluative tools for competency assessment.(2) (8) (9)  Due to this, portfolios have been increasingly introduced into all stages of medical education over the last 10-15 years.(2)


However, significant variations in portfolio definition and composition exist and therefore the most effective design remains elusive.  Despite these variations, a compiled definition would suggest that a portfolio is a collection of evidence, either electronic or paper based, that reports on learning experiences, achievements, feedback and assessments received, as well as plans for improvement which can show progression and form the foundation for a process of lifelong learning.  Through its compilation, self reflection is also stimulated as one looks back and appraises their accomplishments.(1) (2) (6) (7) (8) (10)


In 2008 the Paediatric Residency Program at McMaster University designed and implemented mandatory learning portfolios for formative assessment, organized according to the competencies defined by the RCPSC CanMeds Roles.  Its creation was guided by the RCPSC “Portfolios and CanMEDS Competencies” workshop of the 2006 International Conference on Residency Education.  For purposes of our program the portfolio is defined as a flexible multifaceted means of collecting evidence of achievements of competence over time and represents a collection of materials that demonstrates the breadth and quality of a resident’s work as well as reflecting upon key events and planning their further progress in their professional career.(11)


At the start of training, each resident is given an in depth document as to what is expected in each section of the portfolio, where each section represents one of the CanMeds roles.  For example, under the section of “Resident as a Medical Expert” one would be asked to include learning objectives and strategies while reflecting on objectives achieved, electives taken and why, scores in practice examinations, as well as various evaluations.  More details on required portfolio content can be seen in Appendix A.(11)  The portfolio is then reviewed biannually with the program director to ensure adequate progression through the program is being achieved.


While there is a growing body of literature on portfolio use and implementation in health sciences, the majority is directed towards undergraduate training programs.  Those that do focus on post graduate medical training are not specific to paediatrics.  In order to determine the perceived usefulness of the learning portfolio for paediatric residents at McMaster University, a study was conducted to seek resident perception of and attitude towards learning portfolios after 4 years of use by way of a survey.

Summary of Work

A 40 item online satisfaction survey was designed based on attitudes towards portfolio content and design.  The items were organized according to CanMeds roles and a 5-point Likert scale (strongly disagree to strongly agree) was used for 38 of the 40 items.  The other 2 questions documented level of training and additional comments to allow for open feedback.


This voluntary survey was sent to all 45 paediatric residents at McMaster and responses were collected anonymously.  The survey was open for 1 month from March 1st to the 31st of 2012.  In order to help increase response rates 3 reminders were also sent by electronic mail further requesting completion of surveys once per week throughout the collection period.  At the start of the survey, residents were reminded of the definition and purpose of the portfolio as seen in Table 1.  In addition, respondents had to complete the previous section before proceeding to the next section.  Please see Table 1 for the survey items.


Table 1. McMaster Paediatric Resident Portfolio Post Satisfaction Survey. *Adapted questions.(12) (13).

Survey Introduction

 

Objective: To determine resident satisfaction with, and perception of, learning portfolios after four years of implementation in a paediatric residency program and to assess possible areas for improvement in portfolio categories.


        A Portfolio is:

  • A collection of materials to demonstrate the breadth and quality of a student's work and reflect upon and plan their further progress
  • A collection of materials that records and reflects on key events and processes in a professional's career
  • A flexible and multifaceted means of collecting evidence of the achievement of competence over time.

 

       A Reflection:  

  • Is a form of meta-cognition - thinking about thinking
  • Allows a candidate to critically assess his or her own beliefs and actions while considering salient issues around the CanMEDS competencies to further improve his/her skills  

All residents in the McMaster Paediatrics Program will maintain a portfolio. The portfolio is a mandatory component of the program. The purpose of the portfolio is to record activities and accomplishments in order to guide a resident's progress. It will also help demonstrate the seven CanMEDS competencies achieved. It will help the resident reflect on his/her achievements and help guide their career as a resident. It will help the program director determine if a resident is progressing appropriately through the program.  Keeping in mind the purpose of the portfolio, please answer the following questions to the best of your ability. There are 9 short sections. 

Survey Items

What level of training are you


The Resident as a Professional

  • Overall the portfolio is a very useful tool for tracking my professional development e.g. keeping records of professional memberships, achievements, etc.*
  • The portfolio helped improved my self confidence by allowing me to summarize my accomplishments
  • Assessing my own professionalism has raised my awareness of my professional responsibilities*
  • Overall, I found reflections in the portfolio helpful for improving self awareness*
  • Reflecting helps me cope with difficult situations and uncertainty
  • Ethical reflections have helped me to express my views on difficult situations
  • Ethical reflections have helped in conflict resolution
  • Writing reflections take more time than I feel they are worth

The Resident as a Medical Expert
  • Writing and reflecting on learning strategies and goals for the upcoming year is useful
  • The portfolio helps me to organize and keep my academic records in one place
  • The portfolio helped me improve my critical thinking skills
  • The portfolio allows me to integrate theory and practice

The Resident as a Communicator

  • Reflecting on communication has allowed me to improve my skills
  • Self evaluation of dictations have helped me to improve on my written communication skills
  • The scoring sheet used for self evaluation of dictations is useful
  • It would be more helpful to submit dictations to a third party e.g. colleague, staff, mentor etc. to evaluate my dictations with the scoring tool and provide feedback

The Resident as a  Collaborator

  • I find the CAPE tool useful in evaluating my collaborative experiences and identifying potential areas for improvement
  • Reflecting on collaborative experiences is helpful to my professional development

The Resident as a Manager

  • Career reflection has helped me to set realistic goals and develop a means in achieving them
  • The portfolio helped me take responsibility for my career
  • Administrative roles helped in developing my leadership skills
  • The portfolio helped me improve my organizational skills
  • The portfolio helped me improve patient management skills

The Resident as an Advocate

  • Reflecting on health advocacy has helped me determine ways to become actively involved members of the community
  • The portfolio helped me improve my empathy

The Resident as a Scholar

  •  I find it helpful to keep a record of academic activities given and/or attended e.g. Conferences/CME events, teachings given etc.
  • The portfolio helped me assess my procedural skills
  • The portfolio improved my ability to learn independently

General Questions

  • I feel meetings with the PD are better than it would have been without the portfolio*
  • I felt as if the PD had clearly read my portfolio
  • It would be more helpful to my learning to review content and get feedback on my portfolio more frequently e.g. 4 times/year in order to facilitate improvement
  • I have reviewed my portfolio with my advisor in addition to the PD
  • My advisor was able to provide useful feedback after reviewing my portfolio
  • Creating the portfolio was easy*
  • Overall I am satisfied with the portfolio
  • I would prefer if the portfolio were electronic/online
  • I find keeping a portfolio too time consuming
  • The portfolio would be better without the self-reflections

If you have other comments or concerns about the portfolio process, please let us know.

Take-home Messages
Acknowledgement
Summary of Results

References

 


 

Appendix A:  McMaster Pediatric Residency Program Portfolio Requirements by Section. (11)

 


Resident as a Medical Expert

  • A copy of your rotational schedule for the year.
  • A copy of each of your rotational evaluation forms.
  • Copies of your learning contracts.
  • A document outlining your learning strategies and learning goals for the year and how you have achieved these.
  • A document outlining the electives you have taken detailing: when and where they have occurred and the specific goals and objectives and how these were achieved.
  • Performance documents regarding your medical expertise and proficiency from patients/allied healthcare professionals during your rotations
  • Your OSCE/MCQ/SAQ performances
  • Your in-training results from the ABP(American Board of Pediatrics)
  • Your Practice Long Cases
  • All of your end of year summaries by your advisor

 

Resident as a Communicator

  • A reflection on how you have improved on your communication skills both written and verbal. Please provide a reflection every 6 months providing examples or teaching sessions that have helped you make a change, may be of assistance.
  • Self evaluate your dictations. Use the attached scoring sheet to self evaluate your dictations, 2/6 months, 4/year.

 

Resident as a Collaborator

  • A reflection on how you have improved on your collaboration skills. Please provide a reflection every 6 months; providing examples or teaching sessions that have helped you make a change, will be of assistance.
  • Reflection on team meetings you may have run (1/yr)
  • Reflection on conflict resolution if there were any
  • Any off service evaluations of collaboration from allied health professionals/parents/family meetings/multidisciplinary rounds etc.
  • Complete the reflective CAPE tool yearly (found at the end of this document)
  • Have completed two evaluations/yr from the mock codes(PCCU/NICU/Trauma)

 

Resident as a Manager

  • A reflection on career planning i.e. thoughts of future career goals and what you are doing to explore and achieve these goals. This should be done every 6 months
  • All residents should have an administrative role , please list committee involvement and/or leadership roles and your achievements in these roles
  • Responsible organizer for specific resident activities within the program
  • Role as chief resident in the organization and structure of the residency program
  • Resident as a Health Advocate
  • Please describe your involvement with health advocacy, this can be hospital based and/or community based advocacy.
  • Telephone consultative advice for the care of subspecialty patients
  • Promoting health within the community through participation in specific clinics (e.g. SISO), camps and follow-up home care
  • Member of community, school and regional health programs advocating for child health

 

Resident as a Scholar

  • Maintain MGLA
  • Maintain a procedure log
  • List all conferences and courses attended (including NRP, PALS, ATLS etc.)
  • Planned learning activities through the use of audio/video tapes, computer/internet, CME
  • List all teaching you have done (undergrad, postgrad, can include rounds, sit down sessions etc)
  • Senior residents, have completed 2/year, an evaluation form from a junior resident (form attached)
  • List preparation of presentations and examinations (undergraduate/postgraduate OSCE’s, MCQ’S, SAQ’S)
  • List your Research/Scholarly project (completed and in progress)
    • Participation in the application for research grants/studies
    • Participation in research studies and clinical trials
    • List all publications, posters, abstracts and presentations under local/national/international
  • List any exams taken with qualifications

 

Resident as a Professional

  • List all professional memberships (CMPA, CPSO, OMA, AAP, PAIRO etc.)
  • List any nominations, awards and achievements
  • List patient surveys, institution audit activities and development of standards for professional/clinical practice, policies and procedures
  • Personal letters supporting your commitment to patient care and the profile of McMaster Children’s Hospital
  • Participation in fund raising activities for McMaster Children’s Hospital and community health agencies
  • Reflect on an ethical issue encountered in a clinical setting. Resident may use information discussed during the ethics curriculum
Send ePoster Link