The OSPE: Objective Structured Pathology Examination

Abstract

Objectives

Resident assessment tends to consist of multiple-choice examinations, even in nuanced areas, such as quality assurance. Internal medicine and many other specialties use objective structured clinical examinations, or OSCEs, to evaluate residents. We adapted the OSCE for pathology, termed the Objective Structured Pathology Examination (OSPE).

Methods

The OSPE was used to evaluate first- and second-year residents over 2 years. The simulation included an anatomic pathology sign-out session, where the resident could be evaluated on diagnostic skills and knowledge of key information for cancer staging reports, as well as simulated frozen-section analysis, where the resident could be evaluated on communication skills with a “surgeon.” The OSPE also included smaller cases with challenging quality issues, such as mismatched slides or gross description irregularities. All cases were scored based on the Pathology Milestones created by the Accreditation Council for Graduate Medical Education.

Results

Using this OSPE, we were able to demonstrate that simulated experiences can be an appropriate tool for standardized evaluation of pathology residents.

Conclusions

Yearly evaluation using the OSPE could be used to track the progress of both individual residents and the residency program as a whole, identifying problem areas for which further educational content can be developed.

The approach to assessment of residents and programmatic evaluation of pathology training programs has dramatically changed over the past decade. Competency-based education and evaluation were formally introduced into graduate medical education in 1998 via the Outcome project by the Accreditation Council for Graduate Medical Education (ACGME). In 2013, the ACGME began implementation of the Next Accreditation System (NAS).¹ While the emphasis on the six core competencies^2,3 remains, the system also aims to institute metrics in graduate training programs based on resident achievement of specialty-specific milestones. The NAS also gives programs a larger role in continuous self-assessment, which puts a greater emphasis on assessment and evaluation of residents. In July 2015, the ACGME, in collaboration with the American Board of Pathology, released the Pathology Milestone Project. This collection of 27 categories, ranked on a scale from 0 to 5, is designed to evaluate the six core competencies as they relate to pathology,⁴ with a recommendation that they be reviewed during residents’ semiannual evaluations. These milestones have recently been updated and simplified, with the 2019 Pathology Milestones rolled out in the 2019-2020 academic year. Individual institutions are tasked with developing their own methods of evaluating residents using these milestones. The ACGME further introduced the Clinical Learning Environment Review (CLER) program in 2012 to provide feedback to graduate medical education (GME) programs related to six areas of patient care, including aspects such as professionalism and patient safety.⁵ At our institution, we have piloted the implementation of objective structured clinical examinations (OSCEs) as a method for resident evaluation as relating to many aspects of the ACGME milestones, and thus core competencies and the CLER focus on areas that are relevant to pathology training.

Performance-based assessments or OSCEs have been a mainstay in undergraduate medical education, particularly in the evaluation and assessment of clinical skills as they relate to the continuum of training.^6-8 In addition, residencies in areas such as internal medicine have adapted these evaluations to their programs as a way of formally tracking resident progress.^9,10 While milestones have been clearly delineated by the ACGME, pathology residency programs are also in need of tools that can be used to objectively evaluate resident performance. Most OSCEs have been developed to assess patient interactions and clinical decision making, and to our knowledge OSCEs have not been implemented in pathology residency programs as a means of assessment. Instead, some programs predominately focus on direct observation during normal residency workflow, which will fall short of truly evaluating a resident’s competence in practice.¹¹ Simulation is ideally used to demonstrate the developmental progression for common situations and offer opportunities to experience uncommon situations not readily encountered. These objective assessments using behavioral anchored checklists can also identify areas of weakness hidden from routine rote subjective evaluations, particularly those that deal with communication, interprofessional interactions, and systems practice.

With that in mind, we set out to develop a proof of concept for an Objective Structured Pathology Examination (OSPE), with one portion focused on quality assurance. We hoped to identify those residents who had not yet developed a practical understanding of the types of quality issues faced in the surgical pathology department, as well as those with specific areas for improvement with regard to communication skills and professionalism. The open-endedness of this experience helped to facilitate resident assessment in these areas.

Materials and Methods

Participants

The OSPE was given to all postgraduate year (PGY) 1 and PGY-2 anatomic pathology (AP) or AP/clinical pathology (CP) residents within a university residency program. The OSPE was given in 2 consecutive years. The purpose and a general overview of the OSPE were explained to the residents but not the specific topics that would be covered. The residents were told that the OSPE would be considered a part of their mandatory biannual evaluation process, but they could opt out of the OSPE and instead discuss with the residency director. During this time, 21 total residents participated; eight of the residents participated as PGY-1 level in the first year and as PGY-2 the following year, thus taking the OSPE twice. As our program is more AP focused during the first 2 years of residency education, we hoped that the OSPE would allow us to assess if by PGY-2 residents had a good understanding of the AP-focused topics in the OSPE. The residents were all enrolled in the GME research registry that was approved by the institutional review board at the institution.

OSPE Content and Format

A team consisting of four attending surgical pathologists developed a Simulation Examination Blueprint, which was carried out over 2 years. The OSPE consisted of both simulated interactions as well as the evaluation of surgical pathology cases. The simulated interactions were developed with the idea of being authentic to daily clinical practice within an AP service. Table 1 contains the instructions and materials provided to the residents prior to the simulated encounters for reference; the overview and script provided to the evaluator are shown in Table 2.

The first simulated scenario consisted of a general surgical pathology sign-out with an emphasis on how effectively the resident could describe microscopic findings and communicate their importance in their “report,” specifically related to cancer diagnosis and the information required in a complete report.

The second simulated case consisted of a frozen-section encounter that contained an unexpected malignancy and centered on the communication and interaction with the attending surgeon regarding the pathologic diagnosis and their request for additional information.

The third simulated scenario consisted of a frozen-section encounter that contained an ovarian tumor and centered on communication with an attending surgeon regarding the risks of the lesion and potential limitations of frozen-section diagnosis.

The resident was given the clinical history, instructions, and a general list of tasks prior to entering the room (Table 1). The only room requirements were a microscope with working pointer, a notepad with pen, two chairs, and a telephone.

The second component of the session consisted of four short surgical pathology cases that tested both the resident’s diagnostic skill as well as the ability to identify problems focused on quality assurance. This section was developed with the group of pathologists and a quality specialist. Each of the four cases consisted of a set of microscopic slides, requisition paperwork, gross description, and/or pictures. The residents were not told these cases contained preanalytic issues. Case A was labeled a “mass.” The material on the slide provided did not explain or feasibly correlate with a clinical mass, requiring the resident to recognize the need for further gross examination, review of the patient’s clinical history, and review of prior pathology materials. Case B presented the residents with slides in which the tissue had been poorly processed, with a large hole in the center, requiring the resident to recognize the need to review the gross description and request a recut of the tissue after examining the block. Case C presented the resident with two slides showing squamous carcinoma from a specimen labeled “left leg, possible recurrence” and a gross description that mentioned submitting only one cassette, requiring the resident to recognize the need to discuss the case with the “tech” and request that more tissue be submitted on the case. Case D consisted of a biopsy specimen of an invasive adenocarcinoma of the colon, whereas the clinical information stated the patient was 14 years old, requiring the resident to recognize the need to double check the requisition, bottles, and clinical history for possible specimen mislabeling. The information, including the diagnosis, quality assurance (QA) issue, and possible corrective actions provided to the attending evaluators, is shown in Table 3.

Scoring

Each OSPE scenario was scored based on the Pathology Milestones deemed relevant to that specific encounter, using the 0 to 4 scoring guidelines provided by the ACGME. The ACGME Milestones were the basis of the rubric used by the attending evaluator for formative assessment for the simulated interactions. The case and slides were chosen by a task force group of pathologists interested in resident education. Due to time constraints of the available attendings, the attending evaluator also played the standardized role. The attending evaluator spent 1 hour learning how to play a standardized role by reviewing the script. A detailed sample of the scripts for the simulated cases is in Table 2. In the first simulated case, the evaluator was also instructed to answer questions as if they were the person who grossed the specimen. In all, 13 of the milestones were evaluated.

The QA section, consisting of the four case-based sections, were scored based on if the resident was able to correctly describe and identify the preanalytic problem and solution(s) available. This section was scored on a 0 to 2 scale with three components: diagnosis, QA issue, and follow-up action. For each component, a score of 0 was given if nothing was recognized, a score of 1 if there was partial recognition, and a score of 2 if completely identified. The highest total score was 6 for each case and 24 for all four cases.

Following the simulated experience, the residents received immediate verbal feedback on their performance and had an opportunity to discuss the encounter with the evaluator. Later, the resident received the written narrative feedback via the electronic evaluation system and their ACGME Milestone level based on the rubric. The resident received the correct preanalytic issues and written feedback on their performance on the QA section 1 week later, and a debrief session was held to go over the preanalytic issues presented in the cases.

Results

The average milestone score was 1.4 out of 5 for first-year residents and 2.3 out of 5 for second-year residents. Individual residents’ average scores ranged from 1 to 3.5, as shown in Figure 1. This was slightly lower than the historic average milestones for PGY-1 level residents obtained from more traditional means and around the same for PGY-2 level residents. Table 4 contains the individual data for milestone scores, by resident year. There was an overall improvement in the milestone level of the residents who were assessed over both of the years the OSPE was given, as shown in Figure 2.

For the four surgical pathology cases with QA issues, the average score was 15 of 24 for PGY-1 residents and 11.5 of 24 for PGY-2 residents. Within PGY-1, the average scores for the diagnosis, problem identification, and potential solutions section were 1.5, 1.2, and 1.1, respectively, on the 0 to 2 score scale described above. For PGY-2, the average scores for the diagnosis, problem identification, and potential solutions section were 2, 0.4, and 0.5, respectively.

Discussion

This pathology simulation pilot, OSPE, was a proof-of-concept study demonstrating that the assessment of pathology residents in a simulated environment is possible and may be further developed as a method to assess resident performance, give feedback, and observe overall trends in milestone and other objective fulfillment. The OSPE provides pathology residency programs with a standardized method of evaluation of residents, particularly around topics such as communication skills that do not lend themselves easily to multiple-choice examinations. The OSPE can also identify deficiencies early on in a resident’s training, as well as allowing the program to effectively provide solutions for typical quality and safety issues that occur in the AP laboratory. In our institution, for example, we were able to identify that several of our residents had difficulty identifying the practical preanalytic issues in our QA cases, and even when the error was identified, some could not effectively explain how they would deal with this error. This led us to create additional teaching sessions focused on these issues within our quality and laboratory administration teaching. In addition, the evaluators noted that two residents had difficulty explaining the importance and limitations of frozen section diagnoses to the “surgeons.” Using the OSPE, a program can thus focus resident education by identifying resident weakness and curriculum gaps in training. We posit that further implementation and serial follow-up will improve competency and can be extended even to junior attendings who are new to practice. This is an improvement on the current common practices for pathology education, which involve a generalized curriculum identical for all residents. If time were to be built into the schedule for “competency” rotations, when problem areas are identified for residents, as in our case with quality issues or frozen section communication, this time could be devoted specifically to improvement in these areas. For example, an additional rotation as dealing with frozen sections could be assigned. A long-term goal would be the development of greater flexibility in the pathology resident’s schedule, with OSPE-like assessments helping to determine the rotations needed.

As this method becomes more standard across programs and the pathology milestones better developed, rubrics for readiness related to graduated responsibility can be standardized and globally applied across programs.

With careful planning and vetting, the OSPE has the potential for use in resident assessment in other areas of AP and CP. Rare reports, such as the one written by Olson and George,¹² describe the use of simulation in pathology education and training. Focused more on on-call responsibilities, the authors describe a workshop to prepare residents for commonly encountered scenarios. While this workshop did not lay out a method for evaluation of residents, this type of activity could be transformed so that competency was assessed using the OSPE model. Simulation for tumor board presentation or other forms of pathology consultation in AP could be delivered via an OSPE.

The evaluation and assessment of knowledge, skills, and attitudes can be difficult in pathology residency training. In the AP portion of training, many times this evaluation comprises direct observation over the course of a rotation that is linked to a milestone-based electronic evaluation. Some programs give homegrown assessments based on unknown slides or other types of examinations. Other sources of information on resident knowledge for practice are the resident in-service examination and programmatically the board pass rate. While all of these assessments provide crucial information on resident medical knowledge and some information on a resident’s performance, they may miss the opportunity to address a resident’s weakness in translation of these skills to practice. Pathology simulation with feedback provides an objective delineation of behaviors and evaluation of those behaviors for the purposes of informing the residents’ practice pattern. Because of the link to rubrics or scoring, the OSPE can provide metrics for evaluation of resident performance in a variety of specific topics from cancer staging to frozen section evaluation. Over time, pathology programs can follow resident performance and use this to document progression of clinical skills and levels of achievement and areas in need of improvement. The benefits could also extend to attending pathologists in the form of faculty development such as giving feedback.

While simulation-based assessment is not without drawbacks, most issues encountered in this study were logistical. The OSPE required a time commitment by both residents and supervising pathologists. Planning of an OSPE requires specific behavioral anchored metrics to examine and to gather pertinent materials such as slides, gross images, and cases. In addition, the actual simulation requires time where both residents taking the assessment and pathologists administering the assessment are free from clinical duties.

Some of the limitations of this study include the lack of a control group and the use of the attending as an evaluator and standardized role player. As this was a feasibility study, the use of a simulation center with microscope setup and access to trained simulation staff could increase the output of this method and streamline the process. Video capture of the interaction could then also be used for resident instruction. Consistency across evaluators would be evaluated to limit rater bias and ensure alignment with the grading rubric.

Achieving competency in laboratory management, including QA, is an important aspect of any pathology graduate education. The Academy of Directors of Surgical and Anatomic Pathology lists basic QA knowledge on its list of competencies for advancing AP residents.¹³ In AP, most residents will encounter at least some QA incidents ranging from mislabeled specimens and slides to missing tissue. Most of these events are preanalytic and may be resolved without patient harm (“near misses”) or any sacrifice in quality care when identified promptly. While QA events are tracked for the purposes of departmental quality improvement, recently trained pathologists and employers both cite laboratory management skills, including quality management, as an area of weakness,¹⁴ indicating a possible deficiency of training throughout many residency programs. There are few methods to assess a resident’s competence in handling them, as multiple-choice questions fail to discern if a resident would be able to come up with a plan on their own, rather than selecting the best option from a provided list. These aspects make OSCE-style examinations a good option for performance-based assessment of resident physicians specifically related to preanalytic issues in AP. In this study, it was evident that when faced with common preanalytic issues, that resident struggled in identifying the issue and providing potential solutions. This observation informed our curricular change to directly address resident education of preanalytic problems in the AP laboratory.

Conclusion

The use of simulation in this study as a method for standardized assessment provides a starting point for the evaluation of pathology resident milestone competency and QA knowledge. We were able to successfully identify specific areas for improvement for several of our residents and provide corrective action. OSPE examination could play a useful role within pathology postgraduate medical training, especially within the framework of the ACGME’s new accreditation policies. In the current era of accountability set forth by the ACGME, standardized evaluation of these competencies holds great importance for postgraduate pathology training. The use of the OSPE would help ensure that the transition to practice is not solely based on time spent in a residency program and the successful completion of a single standardized examination but rather the ability to successfully navigate situations expected of an attending physician in an open-ended situation, mimicking real-world experience more closely than a multiple-choice examination could.

References