1	A Program Director’s Guide to Teaching and Assessing the Seven Competencies July 16, 2004 Victoria Village, Auditorium, Columbus, Ohio
2	Overview of AOA Core Competency Integration Plan By Steve Davis, AOA Competency Overview
3
4	Documents:
5	Core Competency Task Force Report July 2003 Executive Summary + 9 Sections: Definition of Professional Competence History of Competency-Based Education Background and Statement of the Problem Approach to the Problem Chronicle of Committee Activity Seven Core Competencies of the Osteopathic Profession (Definition, Elements, Suggested Methods for Evaluation) Questions for Further Consideration Integration of All Competencies Recommendations to the Board of Trustees
6	#1. Definition of Professional Competence “Professional competence is the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection in daily practice for the benefit of the individual and community being served.”
7	#8. Integration of All Competencies Year 1 – July 2004: Osteopathic Philosophy and Osteopathic Manipulative Medicine & Medial Knowledge Year 2 – July 2005: Patient Care & Professionalism Year 3 – July 2006: Practice-Based Learning and Improvement & Systems-Based Practice
8	#9. Recommendations to the Board of Trustees Incorporated into all AOA-approved Basic Standards for internship and residency training. All on-site inspections will evaluate the core competency requirements & specialty affiliates survey workbook will incorporate the core competency criteria starting January 2005. All SOS specialty certification and re-certification board examinations incorporate core competency testing beginning July 2007. AOA Council on CME incorporate core competency requirements in life-long learning and the CME process, offering 1-A credit. Intern and resident institution training programs and all specialty college Program Director and Resident Annual Reports incorporate the core competencies into the evaluation process. All OPTIs are required to participate with partner training institutions and programs. AOA and specialty affiliates are required to incorporate core competency education, training, methodology and evaluation into conferences, conventions, and program director’s seminars.
9	PART I Core Competency Compliance Program (CCCP)
10	PART II Core Competency Compliance Program (CCCP) AOA Core Competency MAP Outlines 8 evaluation tools by Use(s), Advantage(s), & Disadvantage(s) 360-Degree Evaluation Checklist OSCE Monthly Service Rotation Evaluation Procedure/Case Logs Portfolios Written Examination Chart Stimulated Oral Recall Examination Pulled from ACGME Toolbox – just suggestions
11	PART III Core Competency Compliance Program (CCCP) Program Director’s Annual Evaluation Report Instrument to assist DME & Program Director report competency integration Timeline Forms for each competency and required elements Method of Evaluation Suggested Evaluation Tool Evaluation Rating and Comments
12	Summary of AOA Documents 55 pages outlined in ~10 slides 4 KEY documents with Parts I-III drawn from the main (1)Task Force Report. (2)Part I = ICCP (3)Part II = Competency Map (Eval tools) (4)Part III = Sample Annual Report Instrument Questions and/or comments?
13
14	Institutional Core Competency Plan Presented by: Olivia Ojano Sheehan, Ph.D. July 16, 2004
15
16	What is the ICCP? Dynamic document that serves as an institutional road map for continuous improvement in teaching and in evaluating competency-based education. The written plan must address content issues that define responsibilities, goals, methodologies, and evaluation activities.
17	Why is the ICCP Needed? In order to assess effectiveness in the integration of core competencies, the AOA requires its development and implementation. To create an internal process that outlines the methods chosen by the institution to achieve compliance with implementation deadlines and evaluation activities.
18	Who is responsible for the ICCP? The DME (the institutional official recognized by the AOA) is charged to draft the plan. Medical Education Committee approves the plan. Medical Education Committee will monitor the plan’s implementation and performance. Internship, residency directors, and faculty are responsible for implementation of the plan.
19	How will the ICCP be monitored? Annual report - a written progress report of the ICCP must be reviewed and updated annually by the DME, with approval required of the Medical Education Committee. Copy of the annual report should be sent to Osteopathic Graduate Medical Committee of the OPTI.
20	What specifically should be in the ICCP? Part 1 of the AOA Competency Map Document enumerates the important items that need to be incorporated in the ICCP.
21	What guidelines will help in the design of the ICCP? Use the Core Competency Map and the Program Director’s Annual Evaluation Report to assist in the design. Choose one recommended methodology and one recommended evaluation tool. Starting June 2006, all residents are to be assessed by at least two evaluation tools for each Core Competency to qualify for program complete status. Institutions have flexibility in choosing the methods that work best for their postdoctoral programs.
22
23
24	American Osteopathic Association Program Director’s Annual Evaluation Report and CORE Evaluation of Intern Performance Form Presented by: Angela Degnan, B.S. July 16, 2004
25
26	AOA Requirements The American Osteopathic Association requires DMEs and Program Directors to implement training, and Program Evaluators to assess, the AOA Core Competencies in all AOA training programs. The Annual Report instrument was designed to assist you in this process; it was developed based on the AOA core competencies map and its associated references. Specialty specific yearly reports and documents should be included with this instrument upon submission.
27	Purpose & Process of the Annual Report Purpose: To show completion of a residency program; an annual report from the resident and the program director for each year of training is required. Process: Report observations of the resident in both in-hospital and ambulatory care settings should be included. Complete Annual Report within thirty (30) days of the completion of the training year. Submitted Annual Report directly to the appropriate specialty college. Annual Reports will be reviewed by the specialty colleges and become part of the resident’s permanent file
28	Elements of the Annual Report Evaluate the resident on each required element of each of the seven core competencies, specifically, you are asked to: select the methods, outcomes, or demonstrations of compliance that were utilized, select the evaluation tool utilized to document the methods, outcomes, or demonstrations of compliance, rate the resident as Deficient, Usually meets Competencies, Consistently meets Competencies, or Exceptional comment on each required element of each competency Complete the Trainee Assessment Element (applicable for interns) Complete the Program Complete Summary – Final Resident Assessment Element (if appropriate; residents only)
29	Sample of Competency Assessment Element
30	Trainee Assessment Element (used for both interns & residents) Specify that you have reviewed and approved the trainee’s research assignment (e.g., scientific paper, etc.) Specify the trainee’s participation in the annual resident in-service examination as required by the specialty college Specify that you have reviewed the results of the COMLEX III or the resident’s in-service examination with the trainee Specify that the trainee met the requirement for the management of a panel of patients followed throughout the year in an ambulatory continuity setting, and Specify that the trainee completed all other specialty specific requirements for the year Specify trainee’s progress in the program, promise as a physician, and in other areas not specifically mentioned above. State that trainee has made satisfactory progress in this training program and is capable to proceed to the next year. If not, you must attach the trainees quarterly evaluations upon submission of the form.
31	Program Complete Summary – Final Resident Assessment Element State that the resident has been assessed with at least two evaluation tools for each required element of each of the seven competencies. Attach a document portfolio of the resident’s “best performance” evaluations for each competency Attest that the graduating resident has successfully completed all the requirements of the training program, and is recommended for program complete status. This element of the Annual Report is not completed for Interns
32
33	CORE Evaluation of Intern Performance Developed by the Doctors Hospital Medical Education Department and adopted by the CORE DME’s to be used statewide This form is to be completed at the end of each rotation and can be used to help develop your year-end trainee summary. This form was created before the final versions of the core competency documents were completed by the AOA, so it may be updated to better match the wording of the required elements the AOA developed This form will be uploaded into your New Innovations database by the end of the month. Doctors Hospital is working on creating a Intern Evaluation of Rotation and Preceptor form that will be based on the core competencies; They will share this form with the CORE when it is completed this fall.
34	Elements of the Evaluation of Intern Performance The form has you rate the trainee based on three to four specific aspects of each of the seven competencies. You will rate the trainee as unsatisfactory/needs attention, competent/meets expectations, skilled, exceeds expectations, or you may specify if a certain aspect of the competency does not apply to the particular trainee. For all unsatisfactory ratings received by a trainee, you must specify an action plan or recommendation for remediation. The form also requests that you specify an overall evaluation for the trainee - advancement or remediation. The form must be signed by the preceptor, the DME, and the intern. The intern is also required to initial the form to specify that the rotation did not violate the 80 hour work policy
35	Sample of Assessment of a Competency
36
37	AOA Residency Program Planning and Assessment Tools Presented by: Elaine Soper, Ph.D. July 16, 2004
38
39
40
41	Match the tool with the type of assessment
42	Evaluation Tools 360 Evaluation Checklist OSCE Monthly Service Rotation Evaluation Procedure/Case Logs Portfolios Written Exams Chart Stim. Oral Exam Simulations & Models Oral Exam CEX / Mini-CEX Patient Survey A/V Reviews Med. Records Review Global Ratings
43