Abstract

Background

The 21st Century Cures Act mandates patients’ access to their electronic health record (EHR) notes. To our knowledge, no previous work has systematically invited patients to proactively report diagnostic concerns while documenting and tracking their diagnostic experiences through EHR-based clinician note review.

Objective

To test if patients can identify concerns about their diagnosis through structured evaluation of their online visit notes.

Methods

In a large integrated health system, patients aged 18–85 years actively using the patient portal and seen between October 2019 and February 2020 were invited to respond to an online questionnaire if an EHR algorithm detected any recent unexpected return visit following an initial primary care consultation (“at-risk” visit). We developed and tested an instrument (Safer Dx Patient Instrument) to help patients identify concerns related to several dimensions of the diagnostic process based on notes review and recall of recent “at-risk” visits. Additional questions assessed patients’ trust in their providers and their general feelings about the visit. The primary outcome was a self-reported diagnostic concern. Multivariate logistic regression tested whether the primary outcome was predicted by instrument variables.

Results

Of 293 566 visits, the algorithm identified 1282 eligible patients, of whom 486 responded. After applying exclusion criteria, 418 patients were included in the analysis. Fifty-one patients (12.2%) identified a diagnostic concern. Patients were more likely to report a concern if they disagreed with statements “the care plan the provider developed for me addressed all my medical concerns” [odds ratio (OR), 2.65; 95% confidence interval [CI], 1.45–4.87) and “I trust the provider that I saw during my visit” (OR, 2.10; 95% CI, 1.19–3.71) and agreed with the statement “I did not have a good feeling about my visit” (OR, 1.48; 95% CI, 1.09–2.01).

Conclusion

Patients can identify diagnostic concerns based on a proactive online structured evaluation of visit notes. This surveillance strategy could potentially improve transparency in the diagnostic process.

INTRODUCTION

According to the National Academies of Sciences, Engineering, and Medicine (NASEM) report Improving Diagnosis in Healthcare,1 most people will experience at least one diagnostic error in their lifetime. Lack of adequate data sources and insufficient measurement methods provide a limited view of the problem.1–4 The NASEM report highlights the need to monitor the diagnostic process (ie, “a complex, collaborative activity that involves clinical reasoning and information gathering to determine a patient’s health problem”1) and to create further opportunities to identify, learn from, and reduce diagnostic errors. Among the report’s recommendations is to increase engagement of patients and families to contribute valuable input to facilitate accurate and timely diagnosis.1 Prior research indicates that patients can identify concerns related to the diagnostic process5–7 and often disclose information that existing measurement methods fail to capture.8–11 In the inpatient setting, there has been movement toward understanding patients’ needs for engagement in quality- and safety-focused informatics interventions.12,13 However, patients remain on the periphery of diagnostic safety initiatives.

Implementation of the 21st Century Cures Act14 mandates patients’ access to their clinical notes, diagnostic test results, and other information in their electronic health records (EHRs) beginning in 2021. Increased transparency of medical information and evolving capabilities of patient-facing technologies could provide a novel opportunity to engage patients in safety surveillance.15 For example, patients have reported medication concerns and incorrect documentation while accessing their providers’ notes online (eg, open notes) through secure web-based portals.5,15–19 In one study, 1 in 5 patients who read their notes found a mistake—many of which were related to the diagnostic process.5 To our knowledge, no previous work has systematically invited patients to identify concerns about diagnosis in their recent visits. This type of approach can enhance safety surveillance from patients and raise the bar for transparency of the diagnostic process.

Patients also need structured tools for reliable reporting that yield information that is meaningful to clinicians and health systems for improving safety. Our long-term goal is to develop a proactive surveillance strategy that helps identify diagnostic safety concerns in patients. However, methods to study if and how patients can evaluate the diagnostic process via note review have not been developed. Such methods could uncover patients’ feedback about the diagnostic process both when things go well and when they do not. As a first step to achieving the goal of a proactive surveillance safety strategy related to diagnosis, we developed and tested methods to allow patients to systematically identify any diagnostic concerns while accessing and reviewing their recent visit notes and to identify predictors of patient-reported diagnostic concerns.

MATERIALS AND METHODS

Participants and setting

Our study team partnered with Geisinger, a large integrated healthcare organization in central, south-central, and northeast Pennsylvania, spanning 45 counties, mostly rural, serving approximately 4.2 million residents. The study was approved by the local Institutional Review Boards.

To ensure an appropriate target population for testing, we first applied methods to select patients who were more likely to experience diagnostic concerns. We use the term “diagnostic concern” rather than diagnostic error to account for variations in how patients may self-report and perceive diagnostic accuracy and because diagnosis is an evolving process involving uncertainty. In the context of this study, a diagnostic concern is present when a patient disagrees with the statement, “I feel I was correctly diagnosed during my first visit.” We used an electronic trigger algorithm (e-trigger) based on unexpected visit patterns, an approach that has been tested in prior work.20,21 E-triggers mine EHR databases to identify signals for likely adverse events that can be confirmed with additional review.21,22 The trigger was based on the rationale that an unexpected hospitalization or return to clinic, urgent care, or ED visit after an initial primary care visit may indicate that a diagnosis was missed during the first visit. All planned/previously scheduled office visits or elective admissions were excluded to increase the potential that the visit was “unexpected.” We applied this algorithm in Geisinger’s enterprise data warehouse from October 2019 to February 2020 and identified patients with an index primary care visit followed by an unplanned return visit (such as an emergency department, primary care or urgent care visit, or nonelective hospitalization) within 14 days. We defined this cohort “at-risk” for diagnostic concerns because the diagnosis may have been missed or evolved at the first visit.21

Patients aged 18–85 years old were included if they met e-trigger criteria and if they had previously logged on to MyGeisinger, a patient portal based on the Epic EHR, at least once to view clinical notes. We applied the electronic algorithm to Geisinger’s enterprise data warehouse daily to identify patients as soon as they become eligible, thus detecting “at-risk” patients in real time as possible. After confirming eligibility criteria, we contacted patients via a secure electronic message with an invitation to complete a newly developed instrument, the Safer Dx Patient Instrument, about their experience of diagnosis at the time of the index visit, including an assessment of the diagnostic process. A follow-up message was sent to patients who did not respond after 2 weeks. We excluded patient caregivers (n = 17), patients who were unsure (n = 49), or did not respond (n = 1) to whether or not the diagnosis was correct, and one patient who was unable to view the clinician’s notes in MyGeisinger. Patients received a $25 gift card for instrument completion.

Development and testing of the Safer Dx Patient Instrument

We adapted the Revised Safer Dx Instrument,23 a medical record review tool developed to enable clinicians to determine the presence or absence of diagnostic error for a specific episode of care. The Safer Dx Instrument has been previously validated and applied in multiple settings to evaluate the diagnostic process.24 When used for review of high-risk records (eg, records flagged by electronic algorithms), the instrument can help clinicians reliably identify missed opportunities in diagnosis.21 We adapted this instrument to be used similarly by patients.

The Safer Dx Instrument evaluates 5 dimensions of the diagnostic process25,26: (1) the patient-provider encounter (history, physical exam, symptom presentation, determinations regarding need for further evaluation, testing and/or referral), (2) performance and interpretation of diagnostic tests, (3) follow-up and tracking of diagnostic information over time, (4) subspecialty and referral-specific factors, and (5) patient-related factors. Early drafts of the Safer Dx Patient Instrument were created, reviewed, and revised over multiple meetings among the authorship team and assessed by a psychometrician (CS). A patient advocate and a health literacy expert from Geisinger reviewed multiple versions of the instrument to ensure questions were patient-centered and that questions and formatting were appropriate for the patient population. The items were adapted to assess the 5 dimensions of the diagnostic process from the perspective of the patient, including questions about the reason for the visit, the accuracy of patient symptoms documented, relevance and accuracy of the physical exam documented, testing concerns, adequacy of follow-up instructions, adequacy and completeness of the care plan to address all medical concerns, and assessment of the diagnosis (Table 1).

Table 1.

The Safer Dx Patient Instrument: items for determining presence or absence of diagnostic concern in a primary care encounter

QuestionsResponse
1. Please tell us why you visited the doctor on this day.Open ended
2. The notes captured information related to my symptoms accuratelyaLikert1 2 3 4 5
Please describe any inaccuracies:Open ended
3. The provider conducted a physical exam relevant to my symptoms during my visit.aLikert
1 2 3 4 5
4. The notes captured my physical exam accurately.aLikert
Please describe any inaccuracies:1 2 3 4 5
Open ended
5. Did the provider order any tests?a,bYes/no
 a. Based on the notes reviewed, I had concerns about the test(s) my doctor ordered (eg, lab imaging or any procedure).Likert
1 2 3 4 5
 Please explain:Open ended
 b. Based on the notes review, I had concerns about my test results.Likert
1 2 3 4 5
 Please explain:Open ended
6. The notes reflected what I was supposed to do if my symptoms did not get better.aLikert
1 2 3 4 5
7. The care plan the provider developed for me addressed all my medical concerns.aLikert
1 2 3 4 5
8. I feel I was correctly diagnosed during my first visit.bLikert
1 2 3 4 5
If not, please explain:Open-ended
 a. The notes reflected multiple alternative diagnoses that were discussed with me.Likert
1 2 3 4 5
 b. I received a new diagnosis at my follow-up visit.Likert
 Please explain:1 2 3 4 5
9. I trust the provider that I saw during my visit.Likert
1 2 3 4 5
10. I did not have a good feeling about my visit.Likert
1 2 3 4 5
11. How often do you need to have someone help you when you read instructions, pamphlets, or other written material from the provider or pharmacy?Likert (1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always)
1 2 3 4 5
QuestionsResponse
1. Please tell us why you visited the doctor on this day.Open ended
2. The notes captured information related to my symptoms accuratelyaLikert1 2 3 4 5
Please describe any inaccuracies:Open ended
3. The provider conducted a physical exam relevant to my symptoms during my visit.aLikert
1 2 3 4 5
4. The notes captured my physical exam accurately.aLikert
Please describe any inaccuracies:1 2 3 4 5
Open ended
5. Did the provider order any tests?a,bYes/no
 a. Based on the notes reviewed, I had concerns about the test(s) my doctor ordered (eg, lab imaging or any procedure).Likert
1 2 3 4 5
 Please explain:Open ended
 b. Based on the notes review, I had concerns about my test results.Likert
1 2 3 4 5
 Please explain:Open ended
6. The notes reflected what I was supposed to do if my symptoms did not get better.aLikert
1 2 3 4 5
7. The care plan the provider developed for me addressed all my medical concerns.aLikert
1 2 3 4 5
8. I feel I was correctly diagnosed during my first visit.bLikert
1 2 3 4 5
If not, please explain:Open-ended
 a. The notes reflected multiple alternative diagnoses that were discussed with me.Likert
1 2 3 4 5
 b. I received a new diagnosis at my follow-up visit.Likert
 Please explain:1 2 3 4 5
9. I trust the provider that I saw during my visit.Likert
1 2 3 4 5
10. I did not have a good feeling about my visit.Likert
1 2 3 4 5
11. How often do you need to have someone help you when you read instructions, pamphlets, or other written material from the provider or pharmacy?Likert (1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always)
1 2 3 4 5
a

Question related to the dimensions of the diagnostic process.

b

Skip pattern based on response (1 = disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, 5 = agree).

Table 1.

The Safer Dx Patient Instrument: items for determining presence or absence of diagnostic concern in a primary care encounter

QuestionsResponse
1. Please tell us why you visited the doctor on this day.Open ended
2. The notes captured information related to my symptoms accuratelyaLikert1 2 3 4 5
Please describe any inaccuracies:Open ended
3. The provider conducted a physical exam relevant to my symptoms during my visit.aLikert
1 2 3 4 5
4. The notes captured my physical exam accurately.aLikert
Please describe any inaccuracies:1 2 3 4 5
Open ended
5. Did the provider order any tests?a,bYes/no
 a. Based on the notes reviewed, I had concerns about the test(s) my doctor ordered (eg, lab imaging or any procedure).Likert
1 2 3 4 5
 Please explain:Open ended
 b. Based on the notes review, I had concerns about my test results.Likert
1 2 3 4 5
 Please explain:Open ended
6. The notes reflected what I was supposed to do if my symptoms did not get better.aLikert
1 2 3 4 5
7. The care plan the provider developed for me addressed all my medical concerns.aLikert
1 2 3 4 5
8. I feel I was correctly diagnosed during my first visit.bLikert
1 2 3 4 5
If not, please explain:Open-ended
 a. The notes reflected multiple alternative diagnoses that were discussed with me.Likert
1 2 3 4 5
 b. I received a new diagnosis at my follow-up visit.Likert
 Please explain:1 2 3 4 5
9. I trust the provider that I saw during my visit.Likert
1 2 3 4 5
10. I did not have a good feeling about my visit.Likert
1 2 3 4 5
11. How often do you need to have someone help you when you read instructions, pamphlets, or other written material from the provider or pharmacy?Likert (1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always)
1 2 3 4 5
QuestionsResponse
1. Please tell us why you visited the doctor on this day.Open ended
2. The notes captured information related to my symptoms accuratelyaLikert1 2 3 4 5
Please describe any inaccuracies:Open ended
3. The provider conducted a physical exam relevant to my symptoms during my visit.aLikert
1 2 3 4 5
4. The notes captured my physical exam accurately.aLikert
Please describe any inaccuracies:1 2 3 4 5
Open ended
5. Did the provider order any tests?a,bYes/no
 a. Based on the notes reviewed, I had concerns about the test(s) my doctor ordered (eg, lab imaging or any procedure).Likert
1 2 3 4 5
 Please explain:Open ended
 b. Based on the notes review, I had concerns about my test results.Likert
1 2 3 4 5
 Please explain:Open ended
6. The notes reflected what I was supposed to do if my symptoms did not get better.aLikert
1 2 3 4 5
7. The care plan the provider developed for me addressed all my medical concerns.aLikert
1 2 3 4 5
8. I feel I was correctly diagnosed during my first visit.bLikert
1 2 3 4 5
If not, please explain:Open-ended
 a. The notes reflected multiple alternative diagnoses that were discussed with me.Likert
1 2 3 4 5
 b. I received a new diagnosis at my follow-up visit.Likert
 Please explain:1 2 3 4 5
9. I trust the provider that I saw during my visit.Likert
1 2 3 4 5
10. I did not have a good feeling about my visit.Likert
1 2 3 4 5
11. How often do you need to have someone help you when you read instructions, pamphlets, or other written material from the provider or pharmacy?Likert (1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always)
1 2 3 4 5
a

Question related to the dimensions of the diagnostic process.

b

Skip pattern based on response (1 = disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, 5 = agree).

Patients were asked to rate their level of agreement with statements about the diagnostic process using a 5-point scale (agree, somewhat agree, neither agree nor disagree, somewhat disagree and disagree) and were also able to provide open-ended comments on most items. Response choices for one item (“Did the provider order any tests?”) were dichotomous. Skip pattern questions associated with a conditional response were used. A self-reported assessment of the diagnosis was captured through a single question: “I feel I was correctly diagnosed at my first visit.” To assess for diagnostic concerns, patients were given a specific date and instructed to review available notes. We expected patients to use this information to reflect on their visit experience and rely on hindsight knowledge about their care experiences to answer questions. In addition to questions about the diagnostic process, the instrument included items that assessed trust in the clinician, feelings about the visit, and health literacy.27

Initial versions of the instrument were reviewed using the think-aloud method28 with patients at a separate site (n = 5) to assess for clarity. To obtain further feedback, the instrument was also presented to Geisinger’s Patient and Family Advisory Council, the Patient Experience department leadership, and a multidisciplinary system-wide committee focused on improving clinical diagnosis.29 The instrument was piloted at Geisinger twice (n = 27 and n = 59, respectively). In the first pilot, we identified inconsistencies in the data and reworked the skip patterns, while the second pilot ensured item response variability.

Patient involvement

Patients were consulted throughout the study. Initially, a patient advocate met with the first author to discuss the purpose of the study and how to adapt the instrument to capture the diagnostic process while also being patient-centered. They reviewed multiple drafts of the survey. In addition, the Patient and Family Advisory Council reviewed the instrument and suggestions were incorporated.

Patient review of the visit progress note and response

Patients were instructed to review their clinician’s note from the date of the index visit prior to completing the questionnaire, and if possible, to keep the notes accessible while simultaneously answering questions. Data from the questionnaire was collected and managed using DatStat, a web-based application for collecting and managing research and quality improvement study data within Geisinger. In addition, we extracted patient-related information from the Geisinger enterprise data warehouse that consolidates data from the Epic EHR and other sources. All patient-related variables were those that were confirmed or defined at the time of the index visit date. Information obtained included age at the time of the visit, sex, race and ethnicity, and type of health insurance.

Statistical analysis

The primary outcome was self-identified diagnostic concern (defined as disagreement [eg, disagree, somewhat disagree] with the statement “I feel I was correctly diagnosed during my first visit”). Likert scale response data were treated as continuous data30–33 and assumptions of normality and equal variance of residuals were tested and met. Univariate and multivariate logistic regression models were used to identify whether patient-identified diagnostic concerns were associated with responses to other items assessing patients’ perceptions of the 5 different dimensions in the diagnostic process, trust, and feelings about the visit. In addition, we adjusted for potential confounding factors using patient-related variables (ie, age at the time of the visit, sex, race, ethnicity, type of health insurance, and health literacy). Questionnaire responses were examined for any values that were missing, and sampling weights were used to account for missing data and other potential biases.

We used t-tests to compare patient characteristics between patients who identified their diagnosis as correct and those who reported a diagnostic concern. Bivariate correlations were used to determine the relationship between variables and detect any multicollinearity in our data. To further assess the potential for multicollinearity for each independent variable, we also examined variance inflation factor (VIF) scores (<10). All variables in our analyses were entered sequentially in the model using the forward selection approach to determine which variables to add or drop in the model. Our criterion for entry was at P < .05 and remaining variables that did not meet criteria were removed. We used the Hosmer-Lemeshow test to determine the goodness-of-fit of our model. All analyses were conducted using SAS 9.4 (SAS Institute Inc., Cary, NC, USA).

RESULTS

Patient characteristics

The algorithm was applied to 293 566 primary care visits from October 2019 to February 2020 to identify eligible patients. We sent surveys to 1282 patients who met all inclusion criteria and received 486 responses (response rate: 37.9%). Of these, 469 (96.5%) were completed by patients and 17 (3.5%) by caregivers. After applying exclusion criteria (ie, 17 caregivers, 49 patients were unsure and one was unresponsive to the question of whether the diagnosis was correct, and one who did not have access to notes), our final sample consisted of 418 patients. Of 418 patients, 12.2% (n = 51) had a diagnostic concern.

The average age of patients was 48.6 years (SD ±16.0 years) at the time of the visit. Just over three-fourths of patients were female (76.8%), and almost all were White (95.9%), followed by Black or African American (2.9%), Asian (0.7%), and American Indian or Alaska Native (0.5%). Only 2.6% identified as Hispanic or Latinx. Most had private insurance (83.3%), followed by public insurance (14.1%), federal (1.9%), and no insurance (0.7%). Characteristics of patients with and without self-identified diagnostic concerns were similar, with the exception that latter patients were older (P < .001, Table 2). Half of patients reported that their provider ordered tests during the visit (49.8%; n = 208).

Table 2.

Characteristics of Safer Dx Patient Instrument respondents

VariablesSelf-identified diagnostic concern (n = 51)No diagnostic concern (n = 367)P value
Age at index visit (years), mean (±SD)40.4 (±13.7)49.8 (±16.0)<.001*
Gender.32
 Male9 (17.6)88 (24.0)
 Female42 (82.4)279 (76.0)
Race.24
 White47 (92.2)354 (96.5)
 Black or African American2 (3.9)10 (2.7)
 Asian1 (1.9)2 (0.5)
 American Indian or Alaska Native1 (1.9)1 (0.3)
Ethnicity.75
 Hispanic or Latino1 (2.0)10 (2.7)
 Not Hispanic or Latino50 (98.0)357 (97.3)
Insurance.30
 Private47 (92.2)301 (82.0)
 Public4 (7.8)55 (15.0)
 Federal0 (0.0)8 (2.2)
 None0 (0.0)3 (0.8)
Health literacy.31
 Never needs help with reading39 (76.5)301 (82.0)
 Rarely needs help with reading6 (11.8)47 (12.8)
 Sometimes needs help with reading4 (7.8)14 (3.8)
 Often needs help with reading2 (3.9)5 (1.4)
 Always needs help with reading0 (0.0)0 (0.0)
VariablesSelf-identified diagnostic concern (n = 51)No diagnostic concern (n = 367)P value
Age at index visit (years), mean (±SD)40.4 (±13.7)49.8 (±16.0)<.001*
Gender.32
 Male9 (17.6)88 (24.0)
 Female42 (82.4)279 (76.0)
Race.24
 White47 (92.2)354 (96.5)
 Black or African American2 (3.9)10 (2.7)
 Asian1 (1.9)2 (0.5)
 American Indian or Alaska Native1 (1.9)1 (0.3)
Ethnicity.75
 Hispanic or Latino1 (2.0)10 (2.7)
 Not Hispanic or Latino50 (98.0)357 (97.3)
Insurance.30
 Private47 (92.2)301 (82.0)
 Public4 (7.8)55 (15.0)
 Federal0 (0.0)8 (2.2)
 None0 (0.0)3 (0.8)
Health literacy.31
 Never needs help with reading39 (76.5)301 (82.0)
 Rarely needs help with reading6 (11.8)47 (12.8)
 Sometimes needs help with reading4 (7.8)14 (3.8)
 Often needs help with reading2 (3.9)5 (1.4)
 Always needs help with reading0 (0.0)0 (0.0)
*

Significant at 0.05.

Table 2.

Characteristics of Safer Dx Patient Instrument respondents

VariablesSelf-identified diagnostic concern (n = 51)No diagnostic concern (n = 367)P value
Age at index visit (years), mean (±SD)40.4 (±13.7)49.8 (±16.0)<.001*
Gender.32
 Male9 (17.6)88 (24.0)
 Female42 (82.4)279 (76.0)
Race.24
 White47 (92.2)354 (96.5)
 Black or African American2 (3.9)10 (2.7)
 Asian1 (1.9)2 (0.5)
 American Indian or Alaska Native1 (1.9)1 (0.3)
Ethnicity.75
 Hispanic or Latino1 (2.0)10 (2.7)
 Not Hispanic or Latino50 (98.0)357 (97.3)
Insurance.30
 Private47 (92.2)301 (82.0)
 Public4 (7.8)55 (15.0)
 Federal0 (0.0)8 (2.2)
 None0 (0.0)3 (0.8)
Health literacy.31
 Never needs help with reading39 (76.5)301 (82.0)
 Rarely needs help with reading6 (11.8)47 (12.8)
 Sometimes needs help with reading4 (7.8)14 (3.8)
 Often needs help with reading2 (3.9)5 (1.4)
 Always needs help with reading0 (0.0)0 (0.0)
VariablesSelf-identified diagnostic concern (n = 51)No diagnostic concern (n = 367)P value
Age at index visit (years), mean (±SD)40.4 (±13.7)49.8 (±16.0)<.001*
Gender.32
 Male9 (17.6)88 (24.0)
 Female42 (82.4)279 (76.0)
Race.24
 White47 (92.2)354 (96.5)
 Black or African American2 (3.9)10 (2.7)
 Asian1 (1.9)2 (0.5)
 American Indian or Alaska Native1 (1.9)1 (0.3)
Ethnicity.75
 Hispanic or Latino1 (2.0)10 (2.7)
 Not Hispanic or Latino50 (98.0)357 (97.3)
Insurance.30
 Private47 (92.2)301 (82.0)
 Public4 (7.8)55 (15.0)
 Federal0 (0.0)8 (2.2)
 None0 (0.0)3 (0.8)
Health literacy.31
 Never needs help with reading39 (76.5)301 (82.0)
 Rarely needs help with reading6 (11.8)47 (12.8)
 Sometimes needs help with reading4 (7.8)14 (3.8)
 Often needs help with reading2 (3.9)5 (1.4)
 Always needs help with reading0 (0.0)0 (0.0)
*

Significant at 0.05.

Safer Dx Patient Instrument responses

There were significant differences in responses to what dimensions of the diagnostic process were involved across patients with self-identified diagnostic concerns. Patients who identified diagnostic concerns reported more breakdowns in diagnostic processes: symptoms were not captured adequately [24.5% (n = 12) vs 2.4% (n = 9) for patients reporting correct diagnosis, P < .001], physical exam was not relevant to symptoms [11.8% (n = 6) vs 2.2% (n = 8), P < .001], the notes did not capture the physical exam accurately [19.6% (n = 10) vs 3.5% (n = 13), P < .001], the notes did not reflect what to do if symptoms did not get better [35.3% (n = 18) vs 5.4% (n = 20), P < .00], and disagreement that the care plan addressed all medical concerns [39.2% (n = 20) vs 1.4% (n = 5), P < .001]. Patients identifying a diagnostic concern also indicated lower agreement with items assessing the adequacy of the care plan, trust in the provider, and positive perception of the visit (Table 3). About one-third of patients who identified a diagnostic concern also indicated they received a new diagnosis at the follow-up visit.

Table 3.

Predictors in the Safer Dx Patient Instrument for reported diagnostic accuracy with patient characteristics (n = 418)

VariablesAll responses (n = 418)Self-identified diagnostic concern (n = 51)No diagnostic concern (n = 367)P value
Safer Dx Patient Instrument
The notes captured information related to my symptoms accuratelya<.001*
 Agree348 (83.3)22 (43.1)326 (89.3)
 Somewhat agree33 (7.9)9 (17.7)24 (6.6)
 Neither agree nor disagree14 (3.3)8 (15.7)6 (1.6)
 Somewhat disagree8 (1.9)5 (9.8)3 (0.8)
 Disagree13 (3.1)7 (13.7)6 (1.6)
The provider conducted a physical exam relevant to my symptoms during my visita<.001*
 Agree367 (87.8)31 (62.0)336 (92.6)
 Somewhat agree27 (6.5)10 (20.0)17 (4.7)
 Neither agree nor disagree5 (1.2)3 (6.0)2 (0.6)
 Somewhat disagree6 (1.4)4 (8.0)2 (0.6)
 Disagree8 (1.9)2 (4.0)6 (1.7)
The notes captured my physical exam accuratelya<.001*
 Agree354 (84.7)27 (52.9)327 (89.8)
 Somewhat agree26 (6.2)8 (15.7)18 (5.0)
 Neither agree nor disagree12 (2.9)6 (11.8)6 (1.7)
 Somewhat disagree8 (1.9)3 (5.9)5 (1.4)
 Disagree15 (3.6)7 (13.7)8 (2.2)
Did the provider order any tests?a0.44
 Yes208 (49.8)28 (54.9)180 (49.2)
 No209 (50.0)23 (45.1)186 (50.8)
Based on notes reviewed, I had concerns about test(s) my doctor ordered (eg, lab, imaging or any procedure)a0.02
 Agree28 (6.7)2 (7.1)26 (14.5)
 Somewhat agree16 (3.8)5 (17.9)11 (6.2)
 Neither agree nor disagree9 (2.2)9 (32.1)26 (14.5)
 Somewhat disagree5 (1.2)0 (0.0)5 (2.8)
 Disagree123 (29.4)12 (42.9)111 (62.0)
Based on notes reviewed, I had concerns about my test results0.02
 Agree38 (9.1)5 (17.9)33 (18.9)
 Somewhat agree22 (5.3)8 (28.6)14 (8.0)
 Neither agree nor disagree37 (8.9)5 (17.9)32 (18.3)
 Somewhat disagree4 (1.0)1 (3.6)3 (1.7)
 Disagree102 (24.4)9 (32.1)93 (53.1)
The notes reflected what I was supposed to do if symptoms did not get bettera<.001*
 Agree313 (74.9)19 (37.3)294 (80.8)
 Somewhat agree28 (6.7)5 (9.8)23 (6.3)
 Neither agree nor disagree36 (8.6)9 (17.7)27 (7.4)
 Somewhat disagree14 (3.3)8 (15.7)6 (1.7)
 Disagree24 (5.7)10 (19.6)14 (3.9)
The care plan the provider developed for me addressed all my medical concernsa<.001*
 Agree326 (78.0)10 (19.6)316 (86.1)
 Somewhat agree48 (11.5)9 (17.7)39 (10.6)
 Neither agree nor disagree19 (4.5)12 (23.5)7 (1.9)
 Somewhat disagree7 (1.7)4 (7.8)3 (0.8)
 Disagree18 (4.3)16 (31.4)2 (0.5)
The notes reflected multiple alternative diagnoses that were discussed with me
 Agree4 (7.8)0 (0.0)
 Somewhat agree6 (11.8)0 (0.0)
 Neither agree nor disagree20 (39.2)0 (0.0)
 Somewhat disagree3 (5.9)0 (0.0)
 Disagree18 (35.3)0 (0.0)
I received a new diagnosis at my follow-up visit
 Agree17 (34.0)0 (0.0)
 Somewhat agree8 (16.0)0 (0.0)
 Neither agree nor disagree11 (22.0)0 (0.0)
 Somewhat disagree2 (4.0)0 (0.0)
 Disagree12 (24.0)0 (0.0)
I trust the provider that I saw during my visit<.001*
 Agree348 (83.3)17 (33.3)331 (90.9)
 Somewhat agree32 (7.7)6 (11.8)26 (7.1)
 Neither agree nor disagree19 (4.5)13 (25.5)6 (1.7)
 Somewhat disagree7 (1.7)6 (11.8)1 (0.3)
 Disagree9 (2.2)9 (17.7)0 (0.0)
I did not have a good feeling about my visit<.001*
 Agree9 (2.2)9 (17.7)0 (0.0)
 Somewhat agree7 (1.7)6 (11.8)1 (0.3)
 Neither agree nor disagree19 (4.5)13 (25.5)6 (1.7)
 Somewhat disagree32 (7.7)6 (11.8)26 (7.1)
 Disagree348 (83.3)17 (33.3)331 (90.9)
VariablesAll responses (n = 418)Self-identified diagnostic concern (n = 51)No diagnostic concern (n = 367)P value
Safer Dx Patient Instrument
The notes captured information related to my symptoms accuratelya<.001*
 Agree348 (83.3)22 (43.1)326 (89.3)
 Somewhat agree33 (7.9)9 (17.7)24 (6.6)
 Neither agree nor disagree14 (3.3)8 (15.7)6 (1.6)
 Somewhat disagree8 (1.9)5 (9.8)3 (0.8)
 Disagree13 (3.1)7 (13.7)6 (1.6)
The provider conducted a physical exam relevant to my symptoms during my visita<.001*
 Agree367 (87.8)31 (62.0)336 (92.6)
 Somewhat agree27 (6.5)10 (20.0)17 (4.7)
 Neither agree nor disagree5 (1.2)3 (6.0)2 (0.6)
 Somewhat disagree6 (1.4)4 (8.0)2 (0.6)
 Disagree8 (1.9)2 (4.0)6 (1.7)
The notes captured my physical exam accuratelya<.001*
 Agree354 (84.7)27 (52.9)327 (89.8)
 Somewhat agree26 (6.2)8 (15.7)18 (5.0)
 Neither agree nor disagree12 (2.9)6 (11.8)6 (1.7)
 Somewhat disagree8 (1.9)3 (5.9)5 (1.4)
 Disagree15 (3.6)7 (13.7)8 (2.2)
Did the provider order any tests?a0.44
 Yes208 (49.8)28 (54.9)180 (49.2)
 No209 (50.0)23 (45.1)186 (50.8)
Based on notes reviewed, I had concerns about test(s) my doctor ordered (eg, lab, imaging or any procedure)a0.02
 Agree28 (6.7)2 (7.1)26 (14.5)
 Somewhat agree16 (3.8)5 (17.9)11 (6.2)
 Neither agree nor disagree9 (2.2)9 (32.1)26 (14.5)
 Somewhat disagree5 (1.2)0 (0.0)5 (2.8)
 Disagree123 (29.4)12 (42.9)111 (62.0)
Based on notes reviewed, I had concerns about my test results0.02
 Agree38 (9.1)5 (17.9)33 (18.9)
 Somewhat agree22 (5.3)8 (28.6)14 (8.0)
 Neither agree nor disagree37 (8.9)5 (17.9)32 (18.3)
 Somewhat disagree4 (1.0)1 (3.6)3 (1.7)
 Disagree102 (24.4)9 (32.1)93 (53.1)
The notes reflected what I was supposed to do if symptoms did not get bettera<.001*
 Agree313 (74.9)19 (37.3)294 (80.8)
 Somewhat agree28 (6.7)5 (9.8)23 (6.3)
 Neither agree nor disagree36 (8.6)9 (17.7)27 (7.4)
 Somewhat disagree14 (3.3)8 (15.7)6 (1.7)
 Disagree24 (5.7)10 (19.6)14 (3.9)
The care plan the provider developed for me addressed all my medical concernsa<.001*
 Agree326 (78.0)10 (19.6)316 (86.1)
 Somewhat agree48 (11.5)9 (17.7)39 (10.6)
 Neither agree nor disagree19 (4.5)12 (23.5)7 (1.9)
 Somewhat disagree7 (1.7)4 (7.8)3 (0.8)
 Disagree18 (4.3)16 (31.4)2 (0.5)
The notes reflected multiple alternative diagnoses that were discussed with me
 Agree4 (7.8)0 (0.0)
 Somewhat agree6 (11.8)0 (0.0)
 Neither agree nor disagree20 (39.2)0 (0.0)
 Somewhat disagree3 (5.9)0 (0.0)
 Disagree18 (35.3)0 (0.0)
I received a new diagnosis at my follow-up visit
 Agree17 (34.0)0 (0.0)
 Somewhat agree8 (16.0)0 (0.0)
 Neither agree nor disagree11 (22.0)0 (0.0)
 Somewhat disagree2 (4.0)0 (0.0)
 Disagree12 (24.0)0 (0.0)
I trust the provider that I saw during my visit<.001*
 Agree348 (83.3)17 (33.3)331 (90.9)
 Somewhat agree32 (7.7)6 (11.8)26 (7.1)
 Neither agree nor disagree19 (4.5)13 (25.5)6 (1.7)
 Somewhat disagree7 (1.7)6 (11.8)1 (0.3)
 Disagree9 (2.2)9 (17.7)0 (0.0)
I did not have a good feeling about my visit<.001*
 Agree9 (2.2)9 (17.7)0 (0.0)
 Somewhat agree7 (1.7)6 (11.8)1 (0.3)
 Neither agree nor disagree19 (4.5)13 (25.5)6 (1.7)
 Somewhat disagree32 (7.7)6 (11.8)26 (7.1)
 Disagree348 (83.3)17 (33.3)331 (90.9)
a

Questions related to the dimensions of the diagnostic process.

*

Significant at 0.05.

Table 3.

Predictors in the Safer Dx Patient Instrument for reported diagnostic accuracy with patient characteristics (n = 418)

VariablesAll responses (n = 418)Self-identified diagnostic concern (n = 51)No diagnostic concern (n = 367)P value
Safer Dx Patient Instrument
The notes captured information related to my symptoms accuratelya<.001*
 Agree348 (83.3)22 (43.1)326 (89.3)
 Somewhat agree33 (7.9)9 (17.7)24 (6.6)
 Neither agree nor disagree14 (3.3)8 (15.7)6 (1.6)
 Somewhat disagree8 (1.9)5 (9.8)3 (0.8)
 Disagree13 (3.1)7 (13.7)6 (1.6)
The provider conducted a physical exam relevant to my symptoms during my visita<.001*
 Agree367 (87.8)31 (62.0)336 (92.6)
 Somewhat agree27 (6.5)10 (20.0)17 (4.7)
 Neither agree nor disagree5 (1.2)3 (6.0)2 (0.6)
 Somewhat disagree6 (1.4)4 (8.0)2 (0.6)
 Disagree8 (1.9)2 (4.0)6 (1.7)
The notes captured my physical exam accuratelya<.001*
 Agree354 (84.7)27 (52.9)327 (89.8)
 Somewhat agree26 (6.2)8 (15.7)18 (5.0)
 Neither agree nor disagree12 (2.9)6 (11.8)6 (1.7)
 Somewhat disagree8 (1.9)3 (5.9)5 (1.4)
 Disagree15 (3.6)7 (13.7)8 (2.2)
Did the provider order any tests?a0.44
 Yes208 (49.8)28 (54.9)180 (49.2)
 No209 (50.0)23 (45.1)186 (50.8)
Based on notes reviewed, I had concerns about test(s) my doctor ordered (eg, lab, imaging or any procedure)a0.02
 Agree28 (6.7)2 (7.1)26 (14.5)
 Somewhat agree16 (3.8)5 (17.9)11 (6.2)
 Neither agree nor disagree9 (2.2)9 (32.1)26 (14.5)
 Somewhat disagree5 (1.2)0 (0.0)5 (2.8)
 Disagree123 (29.4)12 (42.9)111 (62.0)
Based on notes reviewed, I had concerns about my test results0.02
 Agree38 (9.1)5 (17.9)33 (18.9)
 Somewhat agree22 (5.3)8 (28.6)14 (8.0)
 Neither agree nor disagree37 (8.9)5 (17.9)32 (18.3)
 Somewhat disagree4 (1.0)1 (3.6)3 (1.7)
 Disagree102 (24.4)9 (32.1)93 (53.1)
The notes reflected what I was supposed to do if symptoms did not get bettera<.001*
 Agree313 (74.9)19 (37.3)294 (80.8)
 Somewhat agree28 (6.7)5 (9.8)23 (6.3)
 Neither agree nor disagree36 (8.6)9 (17.7)27 (7.4)
 Somewhat disagree14 (3.3)8 (15.7)6 (1.7)
 Disagree24 (5.7)10 (19.6)14 (3.9)
The care plan the provider developed for me addressed all my medical concernsa<.001*
 Agree326 (78.0)10 (19.6)316 (86.1)
 Somewhat agree48 (11.5)9 (17.7)39 (10.6)
 Neither agree nor disagree19 (4.5)12 (23.5)7 (1.9)
 Somewhat disagree7 (1.7)4 (7.8)3 (0.8)
 Disagree18 (4.3)16 (31.4)2 (0.5)
The notes reflected multiple alternative diagnoses that were discussed with me
 Agree4 (7.8)0 (0.0)
 Somewhat agree6 (11.8)0 (0.0)
 Neither agree nor disagree20 (39.2)0 (0.0)
 Somewhat disagree3 (5.9)0 (0.0)
 Disagree18 (35.3)0 (0.0)
I received a new diagnosis at my follow-up visit
 Agree17 (34.0)0 (0.0)
 Somewhat agree8 (16.0)0 (0.0)
 Neither agree nor disagree11 (22.0)0 (0.0)
 Somewhat disagree2 (4.0)0 (0.0)
 Disagree12 (24.0)0 (0.0)
I trust the provider that I saw during my visit<.001*
 Agree348 (83.3)17 (33.3)331 (90.9)
 Somewhat agree32 (7.7)6 (11.8)26 (7.1)
 Neither agree nor disagree19 (4.5)13 (25.5)6 (1.7)
 Somewhat disagree7 (1.7)6 (11.8)1 (0.3)
 Disagree9 (2.2)9 (17.7)0 (0.0)
I did not have a good feeling about my visit<.001*
 Agree9 (2.2)9 (17.7)0 (0.0)
 Somewhat agree7 (1.7)6 (11.8)1 (0.3)
 Neither agree nor disagree19 (4.5)13 (25.5)6 (1.7)
 Somewhat disagree32 (7.7)6 (11.8)26 (7.1)
 Disagree348 (83.3)17 (33.3)331 (90.9)
VariablesAll responses (n = 418)Self-identified diagnostic concern (n = 51)No diagnostic concern (n = 367)P value
Safer Dx Patient Instrument
The notes captured information related to my symptoms accuratelya<.001*
 Agree348 (83.3)22 (43.1)326 (89.3)
 Somewhat agree33 (7.9)9 (17.7)24 (6.6)
 Neither agree nor disagree14 (3.3)8 (15.7)6 (1.6)
 Somewhat disagree8 (1.9)5 (9.8)3 (0.8)
 Disagree13 (3.1)7 (13.7)6 (1.6)
The provider conducted a physical exam relevant to my symptoms during my visita<.001*
 Agree367 (87.8)31 (62.0)336 (92.6)
 Somewhat agree27 (6.5)10 (20.0)17 (4.7)
 Neither agree nor disagree5 (1.2)3 (6.0)2 (0.6)
 Somewhat disagree6 (1.4)4 (8.0)2 (0.6)
 Disagree8 (1.9)2 (4.0)6 (1.7)
The notes captured my physical exam accuratelya<.001*
 Agree354 (84.7)27 (52.9)327 (89.8)
 Somewhat agree26 (6.2)8 (15.7)18 (5.0)
 Neither agree nor disagree12 (2.9)6 (11.8)6 (1.7)
 Somewhat disagree8 (1.9)3 (5.9)5 (1.4)
 Disagree15 (3.6)7 (13.7)8 (2.2)
Did the provider order any tests?a0.44
 Yes208 (49.8)28 (54.9)180 (49.2)
 No209 (50.0)23 (45.1)186 (50.8)
Based on notes reviewed, I had concerns about test(s) my doctor ordered (eg, lab, imaging or any procedure)a0.02
 Agree28 (6.7)2 (7.1)26 (14.5)
 Somewhat agree16 (3.8)5 (17.9)11 (6.2)
 Neither agree nor disagree9 (2.2)9 (32.1)26 (14.5)
 Somewhat disagree5 (1.2)0 (0.0)5 (2.8)
 Disagree123 (29.4)12 (42.9)111 (62.0)
Based on notes reviewed, I had concerns about my test results0.02
 Agree38 (9.1)5 (17.9)33 (18.9)
 Somewhat agree22 (5.3)8 (28.6)14 (8.0)
 Neither agree nor disagree37 (8.9)5 (17.9)32 (18.3)
 Somewhat disagree4 (1.0)1 (3.6)3 (1.7)
 Disagree102 (24.4)9 (32.1)93 (53.1)
The notes reflected what I was supposed to do if symptoms did not get bettera<.001*
 Agree313 (74.9)19 (37.3)294 (80.8)
 Somewhat agree28 (6.7)5 (9.8)23 (6.3)
 Neither agree nor disagree36 (8.6)9 (17.7)27 (7.4)
 Somewhat disagree14 (3.3)8 (15.7)6 (1.7)
 Disagree24 (5.7)10 (19.6)14 (3.9)
The care plan the provider developed for me addressed all my medical concernsa<.001*
 Agree326 (78.0)10 (19.6)316 (86.1)
 Somewhat agree48 (11.5)9 (17.7)39 (10.6)
 Neither agree nor disagree19 (4.5)12 (23.5)7 (1.9)
 Somewhat disagree7 (1.7)4 (7.8)3 (0.8)
 Disagree18 (4.3)16 (31.4)2 (0.5)
The notes reflected multiple alternative diagnoses that were discussed with me
 Agree4 (7.8)0 (0.0)
 Somewhat agree6 (11.8)0 (0.0)
 Neither agree nor disagree20 (39.2)0 (0.0)
 Somewhat disagree3 (5.9)0 (0.0)
 Disagree18 (35.3)0 (0.0)
I received a new diagnosis at my follow-up visit
 Agree17 (34.0)0 (0.0)
 Somewhat agree8 (16.0)0 (0.0)
 Neither agree nor disagree11 (22.0)0 (0.0)
 Somewhat disagree2 (4.0)0 (0.0)
 Disagree12 (24.0)0 (0.0)
I trust the provider that I saw during my visit<.001*
 Agree348 (83.3)17 (33.3)331 (90.9)
 Somewhat agree32 (7.7)6 (11.8)26 (7.1)
 Neither agree nor disagree19 (4.5)13 (25.5)6 (1.7)
 Somewhat disagree7 (1.7)6 (11.8)1 (0.3)
 Disagree9 (2.2)9 (17.7)0 (0.0)
I did not have a good feeling about my visit<.001*
 Agree9 (2.2)9 (17.7)0 (0.0)
 Somewhat agree7 (1.7)6 (11.8)1 (0.3)
 Neither agree nor disagree19 (4.5)13 (25.5)6 (1.7)
 Somewhat disagree32 (7.7)6 (11.8)26 (7.1)
 Disagree348 (83.3)17 (33.3)331 (90.9)
a

Questions related to the dimensions of the diagnostic process.

*

Significant at 0.05.

Predictors of patient-reported diagnostic concerns

In univariate analyses, we found the following diagnostic processes to be associated with patient-identified diagnostic concerns: accuracy of patient symptoms documented, accuracy of physical exam according to symptoms, accuracy of the physical exam documented, adequacy of follow-up instructions, and adequacy of care plan to address all concerns. Items related to testing were not statistically significant.

A multivariate logistic regression analysis (Table 4) showed that patients who disagreed with the statement “the care plan the provider developed for me addressed all my medical concerns” were almost 3 times more likely to identify a diagnostic concern (Table 4). Additionally, patients who disagreed with the statement “I trust the provider that I saw during my visit” were 2.1 times as likely to identify a diagnostic concern (95% confidence interval [CI], 1.19–3.71). Patients who agreed with the statement “I did not have a good feeling about the visit” were 1.5 times as likely to identify a diagnostic concern (95% CI, 1.09–2.01). Regarding patient characteristics, patients who are not of Hispanic or Latinx ethnicity were almost 8 times more likely to identify a diagnostic concern compared to patients who are Hispanic or Latinx ethnicity (95% CI, 1.56–039.63).

Table 4.

Predictors in the Safer Dx Patient Instrument for reported diagnostic accuracy

Diagnostic process variableUnivariate logistic regression
Multivariate logistic regression
naOdds ratio95% CI limitsP value (Pr > |t|)Adjusted odds ratio95% CI limitsP value (Pr > |t|)
The notes captured information related to my symptoms accurately4162.501.843.39<.001*2.130.964.73.06
The provider conducted a physical exam relevant to my symptoms during my visit4131.981.412.77<.001*1.050.661.67.84
The notes captured my physical exam accurately4151.981.552.54<.001*0.390.170.90.03
Did the provider order any tests?4171.260.702.27.44
Based on notes reviewed, I had concerns about test(s) my doctor ordered (eg, lab, imaging or any procedure)207b0.850.681.07.17
The notes reflected what I was supposed to do if symptoms did not get better4152.001.622.47<.001*1.100.631.90.75
The care plan the provider developed for me addressed all my medical concerns4184.532.976.89<.001*2.651.454.87.002*
I trust the provider that I saw during my visit4155.943.839.21<.001*2.101.193.71.01*
I did not have a good feeling about my visit4172.131.762.57<.001*1.481.092.01.01*
Patient characteristics
 Age at index visit4180.960.940.98<.001*0.980.941.01.14
 Gender418
  Male (reference group)97
  Female3211.470.693.16.321.750.585.30.32
 Race418
  White (reference group)401
  Other race172.320.727.45.160.840.282.51.76
 Ethnicity418
  Hispanic or Latino (reference group)11
  Non-Hispanic or Latino4071.400.1711.28.757.861.5639.63.01*
 Insurance418
  Private3482.580.897.43.080.940.184.97.94
  Public or Other (reference group)70
 Health literacy417
  Never needs help with reading (reference group)339
  Rarely needs help with reading530.980.392.47.0021.110.284.44.005
  Sometimes needs help with reading182.200.687.09<.001*2.920.7511.45<.001
  Often needs help with reading73.080.5716.65<.001*0.310.061.77.45
Diagnostic process variableUnivariate logistic regression
Multivariate logistic regression
naOdds ratio95% CI limitsP value (Pr > |t|)Adjusted odds ratio95% CI limitsP value (Pr > |t|)
The notes captured information related to my symptoms accurately4162.501.843.39<.001*2.130.964.73.06
The provider conducted a physical exam relevant to my symptoms during my visit4131.981.412.77<.001*1.050.661.67.84
The notes captured my physical exam accurately4151.981.552.54<.001*0.390.170.90.03
Did the provider order any tests?4171.260.702.27.44
Based on notes reviewed, I had concerns about test(s) my doctor ordered (eg, lab, imaging or any procedure)207b0.850.681.07.17
The notes reflected what I was supposed to do if symptoms did not get better4152.001.622.47<.001*1.100.631.90.75
The care plan the provider developed for me addressed all my medical concerns4184.532.976.89<.001*2.651.454.87.002*
I trust the provider that I saw during my visit4155.943.839.21<.001*2.101.193.71.01*
I did not have a good feeling about my visit4172.131.762.57<.001*1.481.092.01.01*
Patient characteristics
 Age at index visit4180.960.940.98<.001*0.980.941.01.14
 Gender418
  Male (reference group)97
  Female3211.470.693.16.321.750.585.30.32
 Race418
  White (reference group)401
  Other race172.320.727.45.160.840.282.51.76
 Ethnicity418
  Hispanic or Latino (reference group)11
  Non-Hispanic or Latino4071.400.1711.28.757.861.5639.63.01*
 Insurance418
  Private3482.580.897.43.080.940.184.97.94
  Public or Other (reference group)70
 Health literacy417
  Never needs help with reading (reference group)339
  Rarely needs help with reading530.980.392.47.0021.110.284.44.005
  Sometimes needs help with reading182.200.687.09<.001*2.920.7511.45<.001
  Often needs help with reading73.080.5716.65<.001*0.310.061.77.45
a

Sampling weights were used to account for missing responses.

b

Based on if patient responded “Yes” to having tests ordered in previous question.

*

Significant at .05.

Table 4.

Predictors in the Safer Dx Patient Instrument for reported diagnostic accuracy

Diagnostic process variableUnivariate logistic regression
Multivariate logistic regression
naOdds ratio95% CI limitsP value (Pr > |t|)Adjusted odds ratio95% CI limitsP value (Pr > |t|)
The notes captured information related to my symptoms accurately4162.501.843.39<.001*2.130.964.73.06
The provider conducted a physical exam relevant to my symptoms during my visit4131.981.412.77<.001*1.050.661.67.84
The notes captured my physical exam accurately4151.981.552.54<.001*0.390.170.90.03
Did the provider order any tests?4171.260.702.27.44
Based on notes reviewed, I had concerns about test(s) my doctor ordered (eg, lab, imaging or any procedure)207b0.850.681.07.17
The notes reflected what I was supposed to do if symptoms did not get better4152.001.622.47<.001*1.100.631.90.75
The care plan the provider developed for me addressed all my medical concerns4184.532.976.89<.001*2.651.454.87.002*
I trust the provider that I saw during my visit4155.943.839.21<.001*2.101.193.71.01*
I did not have a good feeling about my visit4172.131.762.57<.001*1.481.092.01.01*
Patient characteristics
 Age at index visit4180.960.940.98<.001*0.980.941.01.14
 Gender418
  Male (reference group)97
  Female3211.470.693.16.321.750.585.30.32
 Race418
  White (reference group)401
  Other race172.320.727.45.160.840.282.51.76
 Ethnicity418
  Hispanic or Latino (reference group)11
  Non-Hispanic or Latino4071.400.1711.28.757.861.5639.63.01*
 Insurance418
  Private3482.580.897.43.080.940.184.97.94
  Public or Other (reference group)70
 Health literacy417
  Never needs help with reading (reference group)339
  Rarely needs help with reading530.980.392.47.0021.110.284.44.005
  Sometimes needs help with reading182.200.687.09<.001*2.920.7511.45<.001
  Often needs help with reading73.080.5716.65<.001*0.310.061.77.45
Diagnostic process variableUnivariate logistic regression
Multivariate logistic regression
naOdds ratio95% CI limitsP value (Pr > |t|)Adjusted odds ratio95% CI limitsP value (Pr > |t|)
The notes captured information related to my symptoms accurately4162.501.843.39<.001*2.130.964.73.06
The provider conducted a physical exam relevant to my symptoms during my visit4131.981.412.77<.001*1.050.661.67.84
The notes captured my physical exam accurately4151.981.552.54<.001*0.390.170.90.03
Did the provider order any tests?4171.260.702.27.44
Based on notes reviewed, I had concerns about test(s) my doctor ordered (eg, lab, imaging or any procedure)207b0.850.681.07.17
The notes reflected what I was supposed to do if symptoms did not get better4152.001.622.47<.001*1.100.631.90.75
The care plan the provider developed for me addressed all my medical concerns4184.532.976.89<.001*2.651.454.87.002*
I trust the provider that I saw during my visit4155.943.839.21<.001*2.101.193.71.01*
I did not have a good feeling about my visit4172.131.762.57<.001*1.481.092.01.01*
Patient characteristics
 Age at index visit4180.960.940.98<.001*0.980.941.01.14
 Gender418
  Male (reference group)97
  Female3211.470.693.16.321.750.585.30.32
 Race418
  White (reference group)401
  Other race172.320.727.45.160.840.282.51.76
 Ethnicity418
  Hispanic or Latino (reference group)11
  Non-Hispanic or Latino4071.400.1711.28.757.861.5639.63.01*
 Insurance418
  Private3482.580.897.43.080.940.184.97.94
  Public or Other (reference group)70
 Health literacy417
  Never needs help with reading (reference group)339
  Rarely needs help with reading530.980.392.47.0021.110.284.44.005
  Sometimes needs help with reading182.200.687.09<.001*2.920.7511.45<.001
  Often needs help with reading73.080.5716.65<.001*0.310.061.77.45
a

Sampling weights were used to account for missing responses.

b

Based on if patient responded “Yes” to having tests ordered in previous question.

*

Significant at .05.

DISCUSSION

As a first step toward using open notes to engage patients in diagnostic safety surveillance, we tested methods to solicit patients’ feedback about diagnostic concerns on a near real-time basis. At-risk patients were able to review their visit notes, identify breakdowns in the diagnostic process, and self-report any diagnostic concerns. We found that patients who identified diagnostic concerns indicated more concerns about the adequacy of care planning, trust, and reported bad feelings about a visit.

This study builds on the NASEM report recommendations to develop patient-centered methods to measure diagnostic safety.1 The use of open notes has been associated with improved patient engagement and quality of care16,34–38 and this work builds on prior OpenNotes efforts to identify patient safety issues in the medical record.5,16–18 When paired with e-triggers to identify patients potentially vulnerable to diagnostic errors, use of the Safer Dx Patient Instrument may enable more proactive patient-centered measurement of patient-reported diagnostic concerns. Notably, this article focused on patient evaluations and perceptions of the diagnostic process, rather than the diagnostic process from a provider perspective. However, patient concerns, satisfaction, and perceptions are influential predictors of subsequent trust in the medical system, as well as follow-up visits and in some cases care outcomes.39 Systematic evaluation of visit notes by patients has potential to uncover underlying patient-centered contributory factors that affect diagnosis which is otherwise harder to gather. With additional development and testing, this strategy could be used in the future as part of larger organizational initiatives to identify and learn from patient-reported diagnostic concerns and promote organizational learning. For instance, the Safer Dx Patient Instrument could provide near real-time data to healthcare organizations about patients’ experiences of the diagnostic process and intervene to avoid any potential safety issues or to address quality of care issues. Feedback from patients could be useful to both clinicians and health care organizations to improve practice40,41 (ie, patient-clinician communication) and potentially improve diagnostic performance.42 Such feedback programs are essential for the development of a learning health system to improve patient safety.43

Strengths of this approach include providing patients a means to identify where along the diagnostic pathway their concerns may have emerged—symptom accuracy, physical exam relevance and accuracy, follow-up, and care planning. For instance, one dimension of the diagnostic process, care planning (eg, whether the care plan was comprehensive and addressed all medical concerns) was found to be significantly associated with self-identified diagnostic concerns. Discordance between care planning expectations and symptoms/diagnosis is important because if there is no shared understanding about diagnosis with patients, care will not be sensitive to patients’ preferences44 or patients may be left feeling their diagnostic safety was compromised. A surveillance strategy using the Safer Dx Patient Instrument is sensitive to patients’ experiences by helping uncover negative experiences or patient-perceived care breakdowns that are rarely documented in the medical record45 and otherwise invisible to the health system because many patients are hesitant to speak up.

We also found that trust and having a bad feeling about a visit were associated with self-identified diagnostic concerns. Patients and their families experiences of misdiagnosis are associated with reduced trust in their current clinicians.46 Interpersonal trust has long been considered an essential aspect of the patient-physician relationship47,48 and increased trust is associated with better patient outcomes.49 As Table 3 shows, patient concerns about elements of the diagnostic process, such as whether tests were conducted or whether notes reflected what they described accurately, were less strongly related to perceived diagnostic accuracy than trust in the provider. A shared-decision making model adapted specifically for the diagnostic process50 may facilitate trust. Trust could be further reinforced through transparency—clinicians could encourage their patients to review their visit notes online and ensure the visit note documentation is comprehensive and accurate and includes relevant discussion around patient values/goals for care.51 We recommend future research seek to determine predictors of trust in one’s provider since trust appears pivotal for patient perceptions of diagnostic accuracy in our study.

Several limitations warrant discussion. Despite efforts to encourage participating through reminders and an incentive, our response rate was low. The study is limited by small sample size and should be interpreted with caution. Additionally, our sample lacked diversity and included patients who used the portal and were also identified as “at-risk” through one specific algorithm; therefore, the results and frequency of diagnostic concerns may not be generalizable. However, our sample of respondents largely reflected the general demographics of MyGeisinger users. In 2020, close to half of MyGeisinger users are between 18 and 50, are 93% White and non-Hispanic (reflective of the geographic areas of service). While 60% of MyGeisinger users are female, 76.7% of respondents were female. We did not conduct chart reviews on our sample; therefore, we cannot confirm whether the diagnostic concerns identified were safety concerns. We also did not verify whether the index visit and the follow-up visit were related. We cannot account for how patients may have interpreted terms, such as diagnosis, follow-up, and care planning. Further study is required on the relationship between trust and diagnostic concerns. We cannot assess directionality (eg, trust impacts diagnostic concerns or diagnostic concerns impact trust), or if review of the notes impacts trust (ie, evolution of a diagnosis between index visit and follow-up).

Similar to other diagnostic safety studies, hindsight bias remains a possibility and is not limited to not remembering correctly, but rather new information that may shape patients views about their provider retrospectively. It is also possible that some patients did not view their notes as instructed and relied solely on memory. At Geisinger, as of December 2020, only 16% of the more than 2.3 million notes shared by the system through the patient portal were viewed by the patients. However, with the US federal mandate of guaranteed access, patient awareness, and active use of open notes is expected to grow. Future methods must reach a diverse set of patients, and thus we recommend replication and extensions of our findings in stratified samples that include a higher representation of diverse patient populations. Nonresponse bias could not be assessed, and it is possible that nonresponders may have had different experiences of the diagnostic process. Finally, upon further investigation, we observed a probable suppression effect in our model between the item “the notes captured my physical exam accurately” and patients’ trust in their providers. The change of direction in multivariate logistic regression for the item “the notes captured my physical exam accurately” is likely due to relationships between predictors and should not be overinterpreted. We thus recommend future research on open notes to determine when and how perceptions of exam and interaction components relate to patient perceptions of overall outcomes.

CONCLUSION

The movement toward full transparency in 2021 via 21st Century Cures Act14 is a unique opportunity to leverage patient’s access to their clinician’s notes as a method for patient-centered evaluation of quality and safety of care. The Safer Dx Patient Instrument provides at-risk patients an opportunity to self-report diagnostic concerns based on the evaluation of their visit notes. With additional development, testing, and evaluation with diverse patient populations, the proactive surveillance method we tested can guide patients in assessing the diagnostic process. The methods outlined herein may have the potential to engage patients in safety by encouraging self-reporting of diagnostic concerns and improving transparency of diagnostic processes, thus improving diagnostic safety.

FUNDING

This work was supported by the Gordon and Betty Moore Foundation (GBMF 5498) and partially supported by the Houston Veterans Administration (VA) Health Services Research and Development (HSR&D) Center for Innovations in Quality, Effectiveness and Safety (CIN 13-413). TDG was additionally supported by an Agency for Healthcare Research and Quality Mentored Career Development Award (K01-HS025474). HS was additionally supported by the VA HSR&D Service (IIR 17-127 and the Presidential Early Career Award for Scientists and Engineers USA 14-274), the Agency for Healthcare Research and Quality (R01HS27363), the Gordon and Betty Moore Foundation (GBMF 8838), and the CanTest Research Collaborative funded by a Cancer Research UK Population Research Catalyst award (C8640/A23385). DTC was additionally supported by the Office of Academic Affiliations VA Advanced Fellowship Program in HSR&D.

AUTHOR CONTRIBUTIONS

TDG is the guarantor, initiated the study, supervised the collaborative project, designed the data collection tool, monitored data collection for the entire study, developed the statistical analysis plan, and drafted and revised the paper. DTC designed the data collection tool, developed the statistical analysis plan, cleaned and analyzed data, and drafted and revised the paper. DKU implemented the study, designed the data collection tool, monitored data collection for the entire study, and revised the paper. SK implemented the study, monitored data collection for the entire study, and revised the paper. TMS implemented the study, maintained compliance records, monitored data collection for the entire study, and revised the paper. CSp designed the data collection tool, developed the statistical analysis plan, advised on data analysis, and revised the paper. CSc designed the data collection tool, advised on data analysis, and revised the paper. DT designed the data collection tool, advised on data analysis, and revised the paper. HS designed the data collection tool, developed the statistical analysis plan, advised on data analysis, and revised the paper.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

ACKNOWLEDGMENTS

We thank the “Geisinger-In-Motion” team, specifically Amanda Jones and Jason Lobianco, who implemented the instrument; the Committee to Improve Clinical Diagnosis, the Patient and Family Advisory Council, the Patient Experience Department and Greg F. Burke, MD (Chief Patient Experience Officer), Rebecca Ruckno, and Hellene Epstein for providing feedback on the Safer Dx Patient Instrument; the Geisinger Enterprise Analytics team who implemented the e-trigger; and Andrea Bradford, PhD for editing of this manuscript.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY

The data collected in this study is not available for public use.

REFERENCES

1

National Academies of Sciences, Engineering, and Medicine. Improving Diagnosis in Health Care. Washington, DC: National Academies Press; 2015.

2

Graber
M
,
Gordon
R
,
Franklin
N.
 
Reducing diagnostic errors in medicine: what’s the goal?
 
Acad Med
 
2002
;
77
(
10
):
981
92
. https://journals.lww.com/academicmedicine/Fulltext/2002/10000/Reducing_Diagnostic_Errors_in_Medicine__What_s_the.9.aspx Accessed August 10, 2020.

3

McGlynn
EA
,
McDonald
KM
,
Cassel
CK.
 
Measurement is essential for improving diagnosis and reducing diagnostic error: a report from the institute of medicine
.
JAMA
 
2015
;
314
(
23
):
2501
2
.

4

Singh
H
,
Graber
ML
,
Hofer
TP.
 
Measures to improve diagnostic safety in clinical practice
.
J Patient Saf
 
2019
;
15
(
4
):
311
6
.

5

Bell
SK
,
Delbanco
T
,
Elmore
JG
, et al.  
Frequency and types of patient-reported errors in electronic health record ambulatory care notes
.
JAMA Netw Open
 
2020
;
3
(
6
):
e205867
.

6

Giardina
TD
,
Haskell
H
,
Menon
S
, et al.  
Learning from patients’ experiences related to diagnostic errors is essential for progress in patient safety
.
Health Aff (Millwood)
 
2018
;
37
(
11
):
1821
7
.

7

Giardina
TD
,
Korukonda
S
,
Shahid
U
, et al.  
Use of patient complaints to identify diagnosis-related safety concerns: a mixed-method evaluation
.
BMJ Qual Saf
 
2021
;
30
(
12
):
996
1001
.

8

Gallagher
TH
,
Mazor
KM.
 
Taking complaints seriously: using the patient safety lens
.
BMJ Qual Saf
 
2015
;
24
(
6
):
352
5
.

9

Mazor
KM
,
Roblin
DW
,
Greene
SM
, et al.  
Toward patient-centered cancer care: patient perceptions of problematic events, impact, and response
.
J Clin Oncol
 
2012
;
30
(
15
):
1784
90
.

10

Kinnunen
U-M
,
Saranto
K.
 
It is time for self-incident-reporting for patients and their families in every health care organization: a literature review
.
Stud Health Technol Inform
 
2013
;
192
:
92
6
.

11

Haldar
S
,
Mishra
SR
,
Kim
Y
,
Hartzler
A
,
Pollack
AH
,
Pratt
W.
 
Use and impact of an online community for hospital patients
.
J Am Med Inform Assoc
 
2020
;
27
(
4
):
549
57
.

12

Willis
MA
,
Hein
LB
,
Hu
Z
, et al.  
Feeling better on hemodialysis: user-centered design requirements for promoting patient involvement in the prevention of treatment complications
.
J Am Med Inform Assoc
 
2021
;
28
(
8
):
1612
31
.

13

Haldar
S
,
Mishra
SR
,
Pollack
AH
,
Pratt
W.
 
Informatics opportunities to involve patients in hospital safety: a conceptual model
.
J Am Med Inform Assoc
 
2020
;
27
(
2
):
202
11
.

14

ONC’s Cures Act Final Rule. https://www.healthit.gov/curesrule/ Accessed November 30, 2020.

15

Dullabh
PM
,
Sondheimer
NK
,
Katsh
E
,
Evans
MA.
 
How patients can improve the accuracy of their medical records
.
EGEMS (Wash DC)
 
2014
;
2
(
3
):
1080
.

16

Bell
SK
,
Gerard
M
,
Fossa
A
, et al.  
A patient feedback reporting tool for OpenNotes: implications for patient-clinician safety and quality partnerships
.
BMJ Qual Saf
 
2017
;
26
(
4
):
312
22
.

17

DesRoches
CM
,
Bell
SK
,
Dong
Z
, et al.  
Patients managing medications and reading their visit notes: a survey of OpenNotes participants
.
Ann Intern Med
 
2019
;
171
(
1
):
69
71
.

18

Wright
E
,
Darer
J
,
Tang
X
, et al.  
Sharing physician notes through an electronic portal is associated with improved medication adherence: quasi-experimental study
.
J Med Internet Res
 
2015
;
17
(
10
):
e226
.

19

Kayastha
N
,
Pollak
KI
,
LeBlanc
TW.
 
Open oncology notes: a qualitative study of oncology patients’ experiences reading their cancer care notes
.
J Oncol Pract
 
2018
;
14
(
4
):
e251
8
.

20

Singh
H
,
Giardina
TD
,
Forjuoh
SN
, et al.  
Electronic health record-based surveillance of diagnostic errors in primary care
.
BMJ Qual Saf
 
2012
;
21
(
2
):
93
100
.

21

Murphy
DR
,
Meyer
AN
,
Sittig
DF
,
Meeks
DW
,
Thomas
EJ
,
Singh
H.
 
Application of electronic trigger tools to identify targets for improving diagnostic safety
.
BMJ Qual Saf
 
2019
;
28
(
2
):
151
9
.

22

Vaghani
V
,
Wei
L
,
Mushtaq
U
,
Sittig
DF
,
Bradford
A
,
Singh
H.
 
Validation of an electronic trigger to measure missed diagnosis of stroke in emergency departments
.
JAMIA
 
2021
; 28 (10):
2202
11
.

23

Al-Mutairi
A
,
Meyer
AND
,
Thomas
EJ
, et al.  
Accuracy of the Safer Dx Instrument to identify diagnostic errors in primary care
.
J Gen Intern Med
 
2016
;
31
(
6
):
602
8
.

24

Davalos
MC
,
Samuels
K
,
Meyer
AND
, et al.  
Finding diagnostic errors in children admitted to the PICU
.
Pediatr Crit Care Med
 
2017
;
18
(
3
):
265
71
.

25

Singh
H
,
Sittig
DF.
 
Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework
.
BMJ Qual Saf
 
2015
;
24
(
2
):
103
10
.

26

Singh
H
,
Giardina
TD
,
Meyer
A
,
Forjuoh
SN
,
Reis
MD
,
Thomas
EJ.
 
Types and origins of diagnostic errors in primary care settings
.
JAMA Intern Med
 
2013
;
173
(
6
):
418
25
.

27

Morris
NS
,
MacLean
CD
,
Chew
LD
,
Littenberg
B.
 
The single item literacy screener: evaluation of a brief instrument to identify limited reading ability
.
BMC Fam Pract
 
2006
;
7
(
1
):
21
.

28

Charters
E.
 
The use of think-aloud methods in qualitative research an introduction to think-aloud methods
.
Brock Educ J Educ Res Pract
 
2003
;
12
(
2
):
68
82
. doi:.

29

Singh
H
,
Upadhyay
DK
,
Torretti
D.
 
Developing health care organizations that pursue learning and exploration of diagnostic excellence: an action plan
.
Acad Med
 
2020
;
95
(
8
):
1172
8
.

30

Johnson
DR
,
Creech
JC.
 
Ordinal measures in multiple indicator models: a simulation study of categorization error
.
Am Sociol Rev
 
1983
;
48
(
3
):
398
407
.

31

Norman
G.
 
Likert scales, levels of measurement and the “laws” of statistics
.
Adv Health Sci Educ Theory Pract
 
2010
;
15
(
5
):
625
32
.

32

Sullivan
GM
,
Artino
AR.
 
Analyzing and interpreting data from Likert-type scales
.
J Grad Med Educ
 
2013
;
5
(
4
):
541
2
.

33

Zumbo
BD
,
Zimmerman
DW.
 
Is the selection of statistical methods governed by level of measurement?
 
Can Psychol Can
 
1993
;
34
(
4
):
390
400
.

34

Bell
SK
,
Folcarelli
P
,
Fossa
A
, et al.  
Tackling ambulatory safety risks through patient engagement: what 10,000 patients and families say about safety-related knowledge, behaviors, and attitudes after reading visit notes
.
J Patient Saf
 
2021
;
17
(
8
):
e791
9
.

35

Nazi
KM
,
Turvey
CL
,
Klein
DM
,
Hogan
TP
,
Woods
SS.
 
VA OpenNotes: exploring the experiences of early patient adopters with access to clinical notes
.
J Am Med Inform Assoc
 
2015
;
22
(
2
):
380
9
.

36

Wolff
JL
,
Darer
JD
,
Berger
A
, et al.  
Inviting patients and care partners to read doctors’ notes: OpenNotes and shared access to electronic medical records
.
J Am Med Inform Assoc
 
2017
;
24
(
e1
):
e166
72
.

37

Walker
J
,
Leveille
S
,
Bell
S
, et al.  
OpenNotes after 7 years: patient experiences with ongoing access to their clinicians’ outpatient visit notes
.
J Med Internet Res
 
2019
;
21
(
5
):
e13876
.doi:

38

Bell
SK
,
Mejilla
R
,
Anselmo
M
, et al.  
When doctors share visit notes with patients: a study of patient and doctor perceptions of documentation errors, safety opportunities and the patient–doctor relationship
.
BMJ Qual Saf
 
2017
;
26
(
4
):
262
70
.

39

Gupta
D
,
Rodeghier
M
,
Lis
CG.
 
Patient satisfaction with service quality as a predictor of survival outcomes in breast cancer
.
Support Care Cancer
 
2014
;
22
(
1
):
129
34
.

40

Ivers
N
,
Jamtvedt
G
,
Flottorp
S
, et al.  
Audit and feedback: effects on professional practice and healthcare outcomes
.
Cochrane Database Syst Rev
 
2012
; (
6
):
CD000259
.

41

Lane
K
,
Rhodes
M
,
Olson
A.
 
Diagnostic reasoning feedback: improving diagnostic reasoning skills and feedback satisfaction through structured peer feedback
.
MedEdPORTAL
 
2016
;
12.
doi:10.15766/mep_2374-8265.10340.

42

Larson
DB
,
Donnelly
LF
,
Podberesky
DJ
,
Merrow
AC
,
Sharpe
RE
,
Kruskal
JB.
 
Peer feedback, learning, and improvement: answering the call of the institute of medicine report on diagnostic error
.
Radiology
 
2017
;
283
(
1
):
231
41
.

43

Meyer
AND
,
Upadhyay
DK
,
Collins
CA
, et al.  
A program to provide clinicians with feedback on their diagnostic performance in a learning health system
.
Jt Comm J Qual Patient Saf
 
2021
; 47 (2): 120–6. doi:.

44

Schubart
JR
,
Toran
L
,
Whitehead
M
,
Levi
BH
,
Green
MJ.
 
Informed decision making in advance care planning: concordance of patient self-reported diagnosis with physician diagnosis
.
Support Care Cancer
 
2013
;
21
(
2
):
637
41
.

45

Khan
A
,
Coffey
M
,
Litterer
KP
, et al. ; the Patient and Family Centered I-PASS Study Group.
Families as partners in hospital error and adverse event surveillance
.
JAMA Pediatr
 
2017
;
171
(
4
):
372
81
.

46

Suzuki R, Yajima N, Sakurai K,

et al.
Association of patients’ past misdiagnosis experiences with trust in their current physician among Japanese adults [published online ahead of print June 22, 2021]. J Gen Intern Med. 2021; doi:10.1007/s11606-021-06950-y.

47

Hendren
EM
,
Kumagai
AK.
 
A matter of trust
.
Acad Med
 
2019
;
94
(
9
):
1270
2
.

48

Phillips‐Salimi
CR
,
Haase
JE
,
Kooken
WC.
 
Connectedness in the context of patient–provider relationships: a concept analysis
.
J Adv Nurs
 
2012
;
68
(
1
):
230
45
.

49

Birkhäuer
J
,
Gaab
J
,
Kossowsky
J
, et al.  
Trust in the health care professional and health outcome: a meta-analysis
.
PLoS One
 
2017
;
12
(
2
):
e0170988
.

50

Berger
ZD
,
Brito
JP
,
Ospina
NS
, et al.  
Patient centred diagnosis: sharing diagnostic decisions with patients in clinical practice
.
BMJ
 
2017
;
359
:
j4218
.

51

Mangus
CW
,
Singh
H
,
Mahajan
P.
 
Health Information Technology for Engaging Patients in Diagnostic Decision Making in Emergency Departments
.
Rockville, MD
: Agency for Healthcare Research and Quality;
2021
. AHRQ Publication No. 20(21)-0040-4-EF. http://www.ahrq.gov/patient-safety/reports/issue-briefs/healthit-ed.html Accessed February 6, 2021.

This is an Open Access article distributed under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs licence (https://creativecommons.org/licenses/by-nc-nd/4.0/), which permits non-commercial reproduction and distribution of the work, in any medium, provided the original work is not altered or transformed in any way, and that the work is properly cited. For commercial re-use, please contact journals.permissions@oup.com

Supplementary data