Evaluation: Making It Work

Acknowledgements

This monograph was developed by the MAHEC Office of Regional Primary Care Education, Asheville, North Carolina. It was developed with support from a HRSA Family Medicine Training Grant. The monograph was provided to our Office of Faculty Development with permission to modify and use in our faculty development program.

Introduction

Evaluation is a valuable process that begins before the rotation starts and integrates fully with the entire learning experience. The result of a well-integrated evaluation process is an enhanced learning experience for the student and increased satisfaction for the preceptor. Integrating evaluation into the entire educational experience and linking it with other important teaching skills, such as setting expectations and providing effective feedback, can actually make a difficult task easier and more effective.

Many preceptors find evaluation to be an unpleasant task associated with their community-based teaching. They enjoy teaching, but dislike having to “grade” their learners. The occasional difficult situation resulting from conflict over a grade can leave an unpleasant aftertaste. If evaluation was just “giving a grade” it would hardly be worth it for the preceptor.This monograph provides additional discussion on the importance and benefits of evaluation and a practical roadmap to the process of evaluating learners, starting before the rotation begins and continuing throughout the entire clinical experience.

The goals for this monograph are to:

1. Review characteristics of evaluation and discuss why it is important.

2. Discuss pitfalls in the evaluation process.

3. Outline a practical system for effective evaluation.

Evaluation: A Challenging Component of Teaching

One of the reasons that preceptors may dislike evaluating learners is that things can go wrong. Read the scenario below and make a mental list of items that may not have gone perfectly in this situation.

It is the next to last day of the rotation and the learner, with whom you have been working for a month, hands you a form that you have never seen before. She states, “I need you do a final evaluation with me for the rotation before I leave at noon tomorrow.” You look at your packed schedule and decide that if you are lucky you can fit in a 30-minute session between AM rounds and the start of office hours. You recall that the learner arrived the Monday after you were on vacation for a week and you didn't get to do your usual orientation.

That night, when you get home from admitting a patient at 10:00 PM, you reflect on the student's performance. She was very pleasant and friendly and was really well liked by your office staff. Her clinical performance was adequate but not as good as the last student you had. There were times when you recommended that she do some reading and report back to you and it was not done. A couple of patients said they had felt the learner was a little “pushy” at times. You look at the form and see sections where covering the learner's history taking skills and physical exam skills and you realize that you never directly observed those skills. You decide she will probably be happy with a High Pass.

After a very hectic morning arriving late to the office from rounds, you finally meet at 1:00PM and review the form, the learner is irate saying “I need Honors.... My career depends on it!” The learner points to several areas on the form stating, “You never told me I needed to do better on that. How can I improve if you never told me!”

The evaluation process is a key component of community-based teaching. I'm sure you were able (with the aid of hindsight) to identify several areas where the preceptor in the example may have avoided some of the problems leading to the uncomfortable result. By consciously and conscientiously integrating the evaluation process into the entire learning experience, you can simultaneously enhance the quality of the educational experience and avoid or minimize the potential discomfort of the process. Evaluation: Why is it Important?

There are three key reasons for effectively evaluating those that you teach. Firstly, evaluating learners is a basic expectation of teaching health professions students in the community. Almost every school or training program expects some type of report, form or grade for learners that they send to you for teaching. Although the level of detail and complexity may vary depending on the rotation, some expectation for evaluation is implied when you agree to work with a learner. Evaluation is a critical function of the role of “teacher”. Just as your chemistry professor was obliged to give you a grade, you are expected to evaluate the learners that you teach.

Many preceptors volunteer their time and energy to teach in the community because of a sense of wanting to return something to the profession and to play a role in the future of health care. Accurate and meaningful evaluation is a key component of that role. By evaluating learners, you are helping to judge future members of your profession to see of they possess the appropriate knowledge attitudes and skills for their level of training. It is not only important to the school or program which needs to make decisions regarding advancement, promotion and licensing based (in part) on your evaluation. You are also modeling for the learner how to assess their current strengths and weaknesses in order to continue their professional growth and development.

Finally, effective and ongoing evaluation can enhance the quality of the learning experience, as well as setting the stage for future professional growth. When integrated into the entire learning experience, the evaluation process can enhance the educational value of the rotation for the individual. Developing a strategy and a system for accomplishing this is a primary goal of this monograph.

Evaluation: What is it?

Many persons think of evaluation as the brief meeting at the end of a learning experience, but this is only a small part of the picture. This evaluation session is a scheduled, formal session between the learner and preceptor. The content should be based on objective data, and from personal observation. One end product is a summary of performance that contributes to a grading process established by the school. The evaluation session takes only small proportion of the time relative to the length of the learning experience, but requires a significant amount of background to be valid and effective.

The evaluation process is an ongoing series of steps and interactions that form the foundation for the information that you share in your evaluation sessions. This process should be fully integrated into the entire rotation.

The entire educational experience -- from setting expectations on the first day, to ongoing observation and behavior-specific feedback -- all these components are part of the evaluation process. An awareness of evaluation should be maintained through out the rotation.

One component of evaluation is assessment. Although often used interchangeably with evaluation, the two words have different meanings when applied in education. Assessment is the on-going process of collecting information about the learner's current level of knowledge, skills and attitudes. In your daily work with the learner, you are continually becoming more aware of what they know and don't know. In this way, you identify areas where the student needs to improve. Think of assessment as “an educational diagnosis” for the learner. Based on this educational diagnosis you are able to provide ongoing feedback to the learner. When summarized over a period of time and analyzed in terms of improvements made, your assessment is the basis for your evaluation. An accurate ongoing assessment is the basis for effective feedback and a functional evaluation.

Feedback is the act of providing the information for your ongoing observations and assessment to the learner. It is best if it is very specific to the behavior observed and given as close in time to the event as possible. Feedback is an important educational tool, but feedback and evaluation sessions are sometimes confused. They differ in several important areas. (Table 1) Feedback is best if it is “Timely”-- given as close to the learning situation as possible. Evaluation sessions are usually a scheduled event at a specific time. Feedback is often informal--fit in at appropriate, brief sessions, into a busy workday. Evaluation sessions are usually performed in a more formal setting by sitting down for a more “official review”.

Table 1. Comparing Feedback and Evaluation Sessions
FEEDBACK EVALUATION
Timing Timely

Scheduled

Setting Informal Formal
Basis Observation Observation
Content Objective Objective
Scope Specific Actions Global Performance
Purpose Improvement “Grading” & Improvement

There are some important similarities. The basis for both feedback and evaluation should be objective data -- specific behaviors that you observed. In addition the content of both the feedback and evaluation sessions should be as objective as possible. The scope or focus of feedback is on specific events or actions, whereas evaluation should encompass a broader review of performance and skills.

The underlying purposes of feedback and evaluation may seem to be very different IF the emphasis of evaluation is solely to generate a grade. As you progress through this monograph you will see that an integrated approach to evaluation makes it an important tool for enhancing learning and professional growth.

Evaluation: What Can Go Wrong

As you observed in the opening scenario. One of the challenges of evaluation is that, without careful attention, a well-meaning preceptor can find himself or herself in an uncomfortable and unpleasant situation. There are a number of things that can go wrong. Some potential problems and pitfalls are listed in Table 2.

Table 2: Potential Problems with Evaluation
“Halo Effect”

“Oops”-Insufficient Evidence

“You never told me that!”

“But I NEED Honors!”

“Uh-Oh”-Should they pass?

“Lake Wobegon” Effect

The “Halo Effect” can make giving a realistic and practical evaluation difficult. The term refers to the situation where one unrelated but outstanding characteristic has an effect on other aspects of evaluation. For example, a learner who is very nice, friendly, out going and well liked by staff but clinically mediocre could get a very high grade. Likewise, quiet, reserved, introspective but clinically excellent learner could receive a mediocre grade. It is important to look beyond personality traits and consider the entire package of underlying knowledge, attitudes, skills and performance when evaluating the learner.

At times a preceptor will arrive at the end of the rotation with a sense that a student's performance is inadequate in some areas, but is not able to recall the details of specific instances where this was demonstrated. This makes it challenging to explain to the learner why a certain area was evaluated as it was and what specifically could be done to improve. Paying close attention to specific instances of clinical performance, and having a system for recording them, will greatly assist the preceptor in doing and accurate evaluation and explaining it to the learner.

The preceptor often has certain standards of performance in mind against which each learner is compared. If the final evaluation is the first time these are discussed with the learner, the preceptor is likely to hear, “But you never told me that!” It is important to discuss your criteria for evaluation as early in the rotation as possible and to give ongoing feedback on areas in need of improvement. For example, you may expect a learner to read about common clinical problems encountered during the rotation. You should mention this very early in the rotation and state that you will periodically ask the learner what they have learned from their reading. If you consider that they are not meeting your expectations in this area, specific and clear feedback should be given with examples of what was not done. When you arrive at the final evaluation session, your assessment of how they have met your goal will be well supported.

Most learners are primarily focused on their professional growth and development, but some bring expectations for a particular grade or evaluation to the rotation. The final day of the rotation is not the best time to become aware of these goals and perceived needs. An early discussion of the learners goals and expectations for the rotation will allow a process of helping the student to meet those goals or facilitate their understanding of why they will not be able to meet those goals in your setting.

Occasionally the preceptor will arrive at the end of the rotation with the realization that, despite significant and sometimes heroic effort on their part, the student's performance remains substandard. Now, at the very end of the rotation the question is, “Should this learner pass this rotation?” The final evaluation is not the best time to begin contemplating this issue. If a learner appears marginal or problematic it is crucial to get help early. Contact the school or training program. Significant help and guidance is available and should be sought early. (See PDP Monograph: Dealing with the Difficult Learning Situation to learn more.)

A strategy that is sometimes used to make evaluations “painless” may recall memories of the Prairie Home Companion radio show and the mythical town of Lake Wobegon: “Where all our students are above average.” Some preceptors consider that if they give all learners a high grade in spite of their performance then everyone should be happy...Right? Not exactly. Both the school and the learner are mislead and are unable to benefit from the opportunities for growth and improvement that an accurate evaluation can provide. The evaluator has abdicated his responsibility to the learner, to the school and to the health profession. Future patients and the profession, as well as the learner may suffer as a result.

These are a few of many potential pitfalls. The good news is that most of these situations can be consistently avoided, making evaluation a useful educational tool rather than an uncomfortable chore. Incorporating evaluation into the learning environment from the very beginning of the experience is essential. What follows is a strategy to help you to accomplish it.

Evaluation: Making it Work

The key to avoiding evaluation pitfalls is to recognize that Evaluation is not just something that you do for an hour at the end of the experience. It is vitally linked to the entire experience. When this connection is lost or ignored problems are more likely to occur. Integrating evaluation throughout the rotation will make evaluation easier, more productive and will help it to have a positive educational effect long after the learner has moved on. Our G-R-A-D-E strategy can help you accomplish this. (See Table 3.) We will review each component in detail.

G... Get Ready

One of the most difficult, but perhaps one of the most important parts of the evaluation process is “getting ready.” The challenge comes from the fact that this is best done before the learner even arrives. Fortunately it can be integrated into another important component of community-based teaching: Setting Expectations.

Table 3: The G-R-A-D-E Strategy for Evaluation
G ... Get Ready
  • Review course expectations
  • Review Evaluation Form
  • Consider unique Opportunities/Challenges of your site
  • What are your Expectations for the learner?

R ... Review Expectations with Learner

  • Meet very early in the experience.
  • Determine knowledge/skill level.
  • Review: Program Goals, Your Goals, Learner's Goals
  • Describe the Evaluation Process

A ... Assess

  • Observe
  • Record
  • Provide Feedback Regularly
  • Have Learner Self-Assess

D ... Discuss Assessment at Mid-Point

  • Formal Meeting
  • Learner and evaluator fill out form in advance
  • Compare evaluations together.
  • Discuss differences and how expectations are being met.

E ... End with a “Grade”

  • Complete evaluation in advance
  • Schedule sufficient time.
  • Support your evaluation with examples
  • Highlight items that can be worked on in the future

Before you first meet the learner, you should review a copy of the “course expectations” and the evaluation form. If this is a new rotation for you or if it has been a while since you have worked with this type of learner, it is best to ask the school or program for these at the time you schedule the rotation. If you are unable to get a copy in advance, ask the student at the very beginning of the experience to give you a copy-- ideally before you sit down together for an orientation meeting. Even if you have had learners for the same rotation before, a brief review of the expectations of the school or program and a review of the criteria that you will be evaluating the learner on can help you focus from the start. It is useful to reflect in advance on your schedule for the period of the rotation and the status of your practice for that time period. If there is time when you or your partners will be away or if the rotation is during your busy season you may need to be more creative in scheduling time for feedback and evaluation, not to mention patient care and teaching.

Decide in advance what YOU want from the learner. What are your goals? What do you expect from the learner? When do they need to be in the office or hospital? What are your “On Call” expectations? What format do you want their notes written in? What style of case presentation do you prefer? Above and beyond the expectations of the school, knowing what your expectations are in advance will set the stage for a successful rotation and evaluation process. (For more information in this area, see the PDP monograph on “Setting Expectations.”)

R...Review Expectations with the Learner

Now that you have determined the expectations of the school or program and your expectations for the learner, it is vital to review these with the learner. This should occur very early in the experience as part of an orientation meeting. You should also review the goals and expectations of the school or program with the learner. You may wish to highlight specific methods by which these expectations can be met in your setting or indicate some goals that may be challenging (e.g. male physical exam in an obstetrician's office). Don't forget to review your expectations for the learner. The more specific you can be and the earlier you tell the learner, the more likely the learner will be able to implement your suggestions and meet your requirements.

Lastly the learner's expectations should be reviewed. What do they hope to get from the experience? Are there specific clinical entities that they wish to see or specific procedures that they expect to learn or perform? Do they expect or wish to achieve a specific grade? Not all learners are up front with (or have even thought about) their own expectations, but this kind of questioning promotes an adult learning style. Knowing their expectations up front can help you to avoid potential problems and conflicts that can come from unrealistic or unmet goals.

The training level of the learner should be determined at this time. This can vary significantly for students at the same stage of training. For example, two medical students in the middle of their third year may have had completely different clinical rotations – one may have had OB/GYN and Surgery another may have had Medicine and Pediatrics. As a result their knowledge base and skills may be quite different and your expectations and your criteria for evaluation will need to be different.

At the very beginning of the rotation, you should discuss how you plan to evaluate the learner. How and when do you plan to give ongoing feed back? How they can give you feedback on how the rotation is meeting their needs? When will the evaluation sessions occur? Review with them the actual form that they will be evaluated on and the criteria you will use.

As you will see this early meeting and discussion with the learner sets the stage for a productive learning experience that is likely to be free of unpleasant surprises for anyone.

A...Assess

By reviewing the expectations and goals of the school or program, the learner and the preceptor, as well as the mechanism and criteria for ongoing assessment and evaluation, you can now readily assess the learner's performance in a functional and educationally productive manner. Assessment should be an ongoing process throughout the rotation. The preceptor should compare the learner's performance to the expectations that were discussed at the beginning.

Observation is a key component of Assessment. Observing student/patient encounters (history and physical exam), presentations, and reviewing notes or write-ups provides the basis for your evaluation.

Important data from observation can sometimes be lost. Time passes quickly and useful details from the beginning of a rotation may be lost to memory by the end. A method for briefly recording observations may be helpful in remembering useful details. A note card in a pocket or quick notes slipped into a file folder can give the preceptor a quick way to jog the memory. A quick phrase, a diagnosis or a patient name may be enough to recall the desired details. This card or note can be used to provide more behavior specific feedback at the end of the day and can be saved to help you remember at evaluation time.

Regular feedback is a key component of the evaluation process. Issues, concerns, or problems should not be saved up for the end of the rotation. Timely, directed feedback will give the learner the opportunity to try to improve and will give you a chance to observe their efforts and successes. By the same token, positive comments on a good performance, quality interactions with patients, etc. should not wait for an arbitrary scheduled time. Few things reinforce good behavior and growth like timely, positive feedback.

As a practicing professional, much of the “feedback” and “evaluation” of your day to day performance must come from self-assessment of your day to day performance. Encouraging learners to assess themselves promotes this important behavior and gives you valuable data on how they view their own performance. For example, asking the learner, “How do you think your encounter with that patient went?” before giving your own feedback, can give a new perspective to both the learner and preceptor.

D...Discuss Evaluation at the Mid-Point

After you have charted a clear course for the learning experience by discussing expectations early and are tracking the learner's progress with an ongoing assessment, the next step is to do a formal evaluation at the midpoint of the learning experience. Time passes very quickly, but taking the time to do a mid-rotation evaluation creates a valuable opportunity to review the experience to date, detect any unexpected issues, and plan for the remainder of the educational experience.

It is important to schedule a specific time for this meeting. This separates this session from your ongoing feedback, and defines it as a summary of performance to date and an opportunity to plan for the remainder of the experience. A minimum of 30 minutes is needed to ensure adequate time for discussion.

Both the preceptor and the learner should prepare for the mid-rotation evaluation. If an evaluation form will be used for the final evaluation, make two extra copies. You should fill out one in advance of your meeting and the student should complete their copy before you meet. By filling them out separately you avoid the chance of influencing each other's honest assessment. The learner should be asked to consider how the rotation has met their needs and expectations so far, and to be prepared to discuss any suggestions for improvement at the meeting.

At the meeting, compare forms and review the learner's evaluation to date. Significant discrepancies between the learner's assessment and the preceptors should be discussed in detail and expectations revisited. Areas of good performance and areas for improvement should be highlighted and specific recommendations for improvement given. Areas where either the preceptor's or the learner's expectations are not being met should be discussed and a plan for improvement agreed upon.

This mid-point evaluation provides an excellent opportunity to shape the learner's growth and development for the remainder of the experience. It can also point out important issues such as misinterpreted expectations, or discrepancies in preceptor and student evaluations that are best detected before the final evaluation. A relatively small amount of time and effort during the middle of the experience can dramatically improve the ultimate satisfaction of the learner and the preceptor.

E...End with a Grade

Well, after a great deal of discussion, we've come to that which we usually consider when we think about evaluation – the final evaluation. This time, by beginning the process before the rotation began and keeping evaluation in mind throughout the experience, we are prepared to provide a high-quality evaluation with less effort.

It is important to schedule sufficient time for a formal, private meeting. Consider setting aside an hour. A late start, and an interruption or two can whittle away a half-hour down to nothing, and near the end of the rotation options for rescheduling are limited.

Take the opportunity to complete the evaluation in advance. This allows you time to carefully reflect on the learner's performance -- their knowledge, skills and attitudes, how they have improved during the rotation and where there is room for further growth. It is difficult to do this effectively with the learner looking over your shoulder. Support your evaluation with specific examples (this is where your notes that you have kept can be very helpful). Rather than just saying, “You have a good rapport with patients”, describe a specific instance where this was apparent. Consider including examples on the written evaluation. This can be very helpful to the school or program in giving them a clear picture of the learner's performance.

Some learning experiences require you to assign a specific grade for the rotation. This can be one of the most difficult parts of evaluation. Some schools or programs give specific descriptions of the level of performance expected for a particular grade, which may be helpful. If you have had several learners for this rotation in the past, you have a useful standard for comparison. You may wish to discuss the learner's performance with partners or colleagues who have had experience with this type of learner or the contact person at the school or program who is overseeing this learning experience. They usually have experience with scores of learners and can give you valuable perspective, advice and guidance, particularly if this is a new teaching experience for you. It is most important that you be able to justify your final decision based on the actual performance of that learner.

Your evaluation should be future-oriented. Although this particular learning experience is ending, the learner's education and professional career will continue long after. Apply lessons learned, and areas for improvement to future goals and directions so that the benefits of this experience can continue as long as possible.

The Paperwork

Like many other things in life and medicine, “It's not over until the paperwork is done.” In that spirit, we should spend a moment discussing this aspect of evaluation. There are so many different evaluation forms it is not possible to get specific here, but there are some important general principles.

The preceptor should become familiar with the paperwork required as early as possible in the rotation. Review the forms before or at the beginning of the rotation, if possible. If you have any concern or confusion regarding the materials or your role in evaluation, get in touch with the school's or program's contact person to discuss it. When filling out the form, you should be prepared to discuss how you arrived at your evaluation with the student and/or the school or program. Most of the time this level of discussion does not occur, but if you are prepared, it means you have done a thoughtful and accurate evaluation. Specific examples of performance should be considered whenever possible.

Many forms require or request written comments from the preceptor. These can be extremely useful to the school or program. They can help describe the details of a less than ideal performance and why a particular grade or assessment is warranted. They are often incorporated into Dean's Letters or other summary documents that are used to communicate the learner's performance to others outside the school. Your comments should be as specific as possible -- describing positive attributes and strengths as well as reviewing areas for improvement. Both the comments and the other aspects of the form should reflect the learners overall level of performance. For example, an “Honors” level performance should be reflected by the use of superlatives in describing the learner and a description of the areas in which expectations were significantly exceeded.

Complete any necessary paperwork as promptly as possible. It is surprising how quickly the memory of a learner and the details of their performance fade as the parade of life and practice moves quickly forward. Completing the paperwork in advance of your final evaluation meeting gets most of the work out of the way before the learner has left and while the experience is still fresh in your mind. You can also reserve the last 10 minutes of your final evaluation meeting to wrap up any paperwork, after the learner has left the room. Prompt completion of the required forms makes the process much easier, and prevents follow-up phone calls from schools, programs, or students asking you to dig out those papers and send them in.

Evaluation -- Making it Work

Evaluation is not just something that you do at the end of the rotation. It is not just assigning a grade or getting some ink down on paper. Evaluation is an ongoing process that should begin before the learning experience starts and should continue as a part of the entire rotation. Other important teaching skills and tools are vital components of effective evaluation: setting expectations, assessing the learner, and providing effective feedback. When integrated into the rotation, the evaluation process improves the quality of the educational experience and contributes to the satisfaction of the learner and the preceptor. The G-R-A-D-E approach can help accomplish that vital integration.

Evaluation is not just a grade, but a process for guiding and contributing to the growth and development of our future colleagues in the health professions.

References/Resources

DeWitt, T. G., Goldberg, R. L., & Roberts, K. B. (1993). Developing community faculty. Principles, practice, and evaluation. American Journal of Diseases of Children, 147, 49-53.

Rubenstein, W. & Talbot, Y. (1992). Medical Teaching in Ambulatory Care. New York: Springer Publishing Company.

Stritter, F. T. & Flair, M. D. Effective Clinical Teaching. Bethesda, MD: US Dept. of Health Education and Welfare.

Westberg, J. & Jason, H. (1993). Collaborative Clinical Education. New York: Springer Publishing Company.


  Hit Counter