Physician Interpretation of Data of Uncertain Clinical Utility in Oncology Prescription Drug Promotion

Abstract

Background

Little is known about how physicians interpret data displays that depict preliminary or exploratory clinical data in physician-targeted sales aids for oncology drugs. Using three factorial experiments, we examined whether disclosures of data limitations and clinical uncertainty adequately communicate the limitations and practical utility of this type of data.

Subjects, Materials, and Methods

The studies used a 2 (disclosure of data limitations: technical, nontechnical) × 2 (disclosure of clinical uncertainty: present, absent) + 1 (control: no disclosure) between-subjects experimental design to examine the impact of disclosures as they relate to presentations of preliminary or exploratory data in promotional communications for oncology products. In each experiment, we randomized oncologists and primary care physicians with oncology experience to view one version of a two-page sales aid. Following this exposure, physicians completed a web-based survey. The design was replicated in three concurrently conducted experiments using sales aids for different fictitious oncology drugs, each featuring one of three common data displays: a forest plot (n = 495), a Kaplan-Meier curve (n = 504), or a bar chart (n = 532).

Results

Results provide initial evidence that in some contexts disclosures can improve understanding of the clinical utility of certain information about a drug and the limitations of results presented in a data display. Disclosures can also temper perceptions of how much evidence is presented that supports a conclusion that the drug is an appropriate treatment. In terms of the language used in the disclosure of data limitations, physicians in all three experiments strongly preferred the nontechnical disclosures.

Conclusion

The findings from the three experiments in this study suggest that disclosures have the potential to increase relevant knowledge, but more research is needed to establish best practice recommendations for using disclosures to convey contextual information relevant for interpreting data displays in promotional communications.

Implications for Practice

This article reports the results from three large, online experimental studies that address a growing concern that drug companies often share favorable clinical trial results with physicians in promotional materials that lack important context for physicians to interpret the data. This series of studies investigates whether strategic use of two types of disclosures (disclosure of data limitations and a disclosure of clinical uncertainty) improves understanding and reduces misinterpretations among physicians. The results from these studies help identify communication factors that impact how physicians critically appraise preliminary or exploratory clinical trial data to inform policy and regulatory efforts.

Introduction

The market for cancer drugs is among the most competitive in the pharmaceutical industry, and companies invest heavily in promoting favorable clinical trial results to physicians [1]. Pharmaceutical companies typically assess many endpoints in addition to the primary clinical outcome of interest to further explore the effects of their products. In many cases, these endpoints are strictly exploratory and support only the reporting of descriptive results. Nonetheless, such endpoints representing exploratory analyses are often presented to physicians via pharmaceutical sales aids. Drug promotional communications, such as sales aids, have been found to contain selectively extracted preliminary and exploratory data from clinical studies that, when taken out of context, may exaggerate the drug's benefits [2]. Furthermore, internal documents from the pharmaceutical industry illustrate that omission and spin of negative data are not uncommon in marketing materials [3], and a review of warning letters issued by the U.S. Food and Drug Administration (FDA) found that brochures and sales aids accounted for the largest proportion of regulatory violations [4].

Physicians usually do not believe that their prescribing behaviors are influenced by promotional communications [5]; however, medical experts, like other decision makers, are susceptible to cognitive biases [6, 7]. For example, variations in how results from clinical trials are presented have been shown to influence impressions of treatment benefit [8, 9]. Despite the advanced education of physicians, a systematic review found that many physicians lack the biostatistical background to properly assess methodological deficits in clinical studies [10]. When assessing clinical trial results, physicians need to be able to understand how the data were analyzed and what those analyses support in terms of conclusions about benefit, yet evidence suggests that physicians have difficulty understanding clinical trial reports [9, 11, 12].

When used properly, data displays can be powerful tools to aid understanding of clinical trial findings [13]. However, the type of data displays commonly used in reporting clinical trial data (and in prescription drug promotional communications) involve aspects of statistics that have been shown to be easily misunderstood [14]. Despite calls to enhance the quality of data visualizations, there remains room for improvement [14–20]. Given that data displays may be misleading without careful interpretation, it is imperative to examine strategies that would facilitate physicians' ability to understand and appropriately frame studies. For example, using disclosure statements is one potential strategy for providing relevant contextual information concerning the analysis, results, limitations, or degree of clinical uncertainty in a data display. Best practice recommendations and research findings suggest that—when noticed—figure captions and disclosures for data visualizations can effectively convey information and can help prevent misinterpretation [17, 21]. Accordingly, it is reasonable to expect that disclosures of methodological and practical limitations could help physicians interpret results presented in data displays, particularly when endpoints are strictly exploratory. Yet such disclosures are not always included in pharmaceutical promotional communications.

Although a few randomized controlled trials have been conducted on the topic of research report interpretation [22–24], there are no published data to date regarding prescribers' use and understanding of data displays in conjunction with qualifying disclosures. In this study, we examined how physicians interpret presentations of data with uncertain clinical utility in promotional communications for three different fictitious oncology drugs using three common data displays: a forest plot, Kaplan-Meier curve, and bar chart. We experimentally manipulated two types of disclosure. The first type of disclosure was a disclosure of data limitations. This disclosure presented additional statistical and methodological information about the data display, and we varied whether the disclosure used nontechnical or technical language. In addition to evaluating the impact of the data limitations disclosures generally, we explored whether the additional detail in a disclosure that used technical language had the unintended effect of reducing informativeness of the data display for physicians who may not have sufficient experience or time to critically evaluate the information [11, 25]. For the second type of disclosure, a disclosure of clinical uncertainty, we examined the impact of presenting a statement that describes the data as having uncertain clinical utility, with the expectation that such a statement will increase understanding and temper perceptions of clinical benefit. Insofar as data display disclosures provide important contextual information for critically appraising clinical trial data, we would expect greater comprehension of the limitations of the data; greater understanding of the evidentiary support for the displayed outcomes; and tempered perceptions of the benefits described in the data display as well as the overall benefits of the drug when physicians are exposed to disclosures versus not exposed to a disclosure (control group).

Materials and Methods

Study Design and Experimental Stimuli

The study used a 2 (disclosure of data limitations: technical, nontechnical) × 2 (disclosure of clinical uncertainty: present, absent) + 1 (control: no disclosure) between-subjects experimental design. The design was replicated in three concurrently conducted experiments. Each experiment examined a unique data display presented in a two-page sales aid for a cancer drug that was based on real sales aids submitted to the FDA upon marketplace dissemination (see Table 1). We limited the sales aids to the presentation of the data display and information about the drug's indication and did not include data on other clinical outcomes or information about contraindications or adverse reactions. In versions that included a disclosure of data limitations (either technical or nontechnical), the disclosure was formatted like a figure caption and positioned below the data display. In versions that included the disclosure of clinical uncertainty (“This presentation includes exploratory information of uncertain clinical utility and should be interpreted cautiously when used to make treatment decisions.”), the statement was positioned in a text box above the data display. (See supplemental online Figs. 1–15 for all sales aids). A CONSORT outlining physician flow through the study are provided in supplemental online Figure 1.

Participants

The data used in the current study were part of a larger study in which data of 2,131 physicians were collected. Eligible physicians spent at least 20% of their time on direct patient care and were licensed to prescribe medication. They either specialized in oncology, hematology, medical oncology, or pediatric oncology or were primary care physicians (PCPs) who specialized in family, general, obstetrics and gynecology, or internal medicine. For the analysis presented in this manuscript, we further required PCPs to have some level of oncology experience based on the number of oncology medications they prescribed in the past month (PCPs who reported prescribing 0 oncology medications were excluded from the analysis), resulting in a sample of 1,531 physicians. For completion of the 15-minute survey, PCPs were provided an honorarium of $40, and oncologists were provided an honorarium of $50.

Study Procedures

Participants were invited to participate by a secure, nonidentifiable hyperlink within an email. Physicians who were eligible and consented were randomly assigned (using a computer algorithm) to one of the experimental arms and directed to the appropriate stimuli and questionnaire. Participants answered questions related to perceived benefits and improvement in clinical outcomes and understanding of the amount of evidence provided. When participants reached the comprehension items, instructions at the top of the page told participants that they could view the sales aid that they saw earlier while answering those items. This was intended to better reflect realistic scenarios where physicians might keep and refer back to sales aids that are left by pharmaceutical representatives. After the comprehension items, participants answered questions about which disclosure they preferred, and we also captured information about their clinical background and other descriptive variables.

Data collection was completed in April 2020. The study was reviewed and approved by RTI International's Institutional Review Board and was granted an exemption from FDA's Research Involving Human Subjects Committee.

Measures

Before conducting the main study, we pilot-tested the questionnaire and experimental stimuli. This step included revising/adapting existing items; getting expert appraisal; conducting cognitive interviewing with a small group (n = 9) of oncologists, physicians, and advanced practice practitioners to confirm understandability; and pretesting the materials for reliability and validity with oncologists and physicians (n = 95) reflective of the sample used in the main study. Our questions were part of a larger survey (see supplemental online Table 1). Comprehension of analysis, results, and limitations of the data was measured with five items that used either true/false or multiple-choice formats. All items were recoded as either 1 (correct) or 0 (incorrect). The total score represented the number of items answered correctly (possible range 0 to 5).

Comprehension of drug utility for the outcomes presented in the data display was measured with one item (“The data presented in the sales aid clearly translates into clinical benefit.”). Response options were true or false (correct).

Perceived evidentiary support for the displayed outcomes was measured with one item that asked participants to rate how much evidence they thought the sales aid provided in terms of whether the drug was suitable for treating patients with the appropriate indication. Response options included insufficient, preliminary, and strong. The item was recoded into two levels: insufficient/preliminary (correct) and strong (incorrect, reflecting a misunderstanding of the strength and reliability of the data).

Perceived drug benefits was measured on a 5-point Likert-type scale ranging from 1 (not at all) to 5 (extremely), and perceived endpoint improvement was measured with a 6-point bipolar item, responses ranging from 1(completely disagree) to 6 (completely agree) (adapted from Kesselheim et al. [26]).

At the end of the survey, preference for nontechnical or technical language was measured by showing participants the text for the technical and nontechnical disclosures and asking them to select their preferred version. They could also select a no-preference option.

We also assessed the following potential covariates: trust in pharmaceutical marketing (adapted from Huh et al. [27]), willingness to try unproven treatments with terminal patients, confidence in biostatistics understanding (adapted from Susarla and Redett [28]), and level of oncology experience (oncologists versus PCPs who prescribed at least one oncology medication in the last month) and collected demographic information and information about access to pharmaceutical sales representatives.

Statistical Analysis

For each dependent variable, we took a two-step approach to the analysis. First, we conducted two-way analysis of variance (ANOVA), analysis of covariance, or logistic regression analyses that excluded the control group to test for differences by type of data limitations disclosure and disclosure of clinical uncertainty and to determine planned contrasts of interest for subsequent models. Next, we conducted one-way models including the control group to test for differences across all five experimental groups. For brevity, we report significant results from the two-way models in the first step and focus on subgroup comparisons from the one-way models from the second step. When appropriate, we conducted follow-up planned contrasts using adjusted Bonferroni p values to reduce the probability of a type 1 error. We conducted all analyses using SAS Enterprise Guide (version 7.13) and SPSS (version 25.0).

Results

Table 2 contains the demographic characteristics of participants in the three experiments.

Effects of Sales Aid Variations

Analysis, Results, and Limitations of Data

On average, participants correctly answered approximately half of the comprehension questions (see Table 3). In the model excluding the control group for experiment 2, there was a significant interaction between the disclosure of clinical uncertainty and type of data display (F  _1,400 = 4.38, p = .037, η² = .01). However, in subsequent one-way ANOVA models that included the control group, experimental condition was not significantly related to data display comprehension in experiment 1 or 2. In experiment 3, experimental condition was significantly related to comprehension of the data display in the model that included the control group (F  _4,527 = 7.96, p < .001, η² = .06). The planned contrast for this model involved a comparison of the control group against the remaining four groups that included a disclosure of data limitations (i.e., all active experimental arms combined). Participants who were exposed to a disclosure of data limitations (mean, 3.38; SE = 0.05) correctly answered more of the comprehension questions than participants assigned to the control group (mean, 2.71; SE = 0.11) (F  _1,523 = 30.01, p < .001, η² = .05).

Clinical Utility of the Drug for the Presented Outcome

Across the three experiments, 31.0%–56.0% of participants correctly indicated that the data presented in the sales aid did not clearly translate into clinical benefit (see Table 3). In logistic regression models that included the control group, we tested whether participants who were exposed to a disclosure of data limitations would be more likely to comprehend the implications for clinical utility of data presented in the sales aid than participants who were not exposed to a disclosure (i.e., control group). In experiments 1 and 3, exposure to a disclosure did not affect comprehension of clinical utility, but experimental condition was significantly related to comprehension of clinical utility in the model for experiment 2 (Wald χ²  ₄ = 11.22, p = .024). Participants were more likely to correctly interpret the clinical utility of the benefit outcome presented in the sales aid if they were assigned to one of the active arms of the experiment (43.6%) than participants assigned to the control group (31.0%; odds ratio [OR], 1.70; 95% confidence interval [CI], 1.07–2.72; Wald χ²  ₁ = 4.98; p = .026).

Perceived Evidentiary Support

Across all three experiments, 10.7%–33.3% of participants incorrectly indicated that the sales aid provided strong evidence that the drug is an appropriate treatment (see Table 3). In logistic regression models that excluded the control group for experiment 3, there was a significant main effect of the disclosure of clinical uncertainty, such that participants exposed to the disclosure were less likely to say the evidence was strong (11.8%) than participants who did not see the disclosure (22.2%; OR, 0.42; 95% CI, 0.21–0.85; Wald χ²  ₁ = 5.75; p = .017). In subsequent models that included the control group, experimental condition was significantly related to the perceived evidentiary support in experiment 3, (Wald χ²  ₄ = 10.11, p = .039), but there were no differences in the likelihood of indicating the evidence was strong relative to the control group (17.3%) among participants in the active experimental arms who were exposed to the disclosure of clinical uncertainty or not exposed to it (Wald χ²  ₁ = 1.83, p = .176 and Wald χ²  ₁ = 0.94, p = .332, respectively).

Perceived Drug Benefit

On average, participants tended to believe that the drug would be moderately beneficial for treating patients based on the limited information in the sales aid (see Table 3). In models that excluded the control group, Levene's test for equality of error variances was significant in the model that excluded the control group in experiment 2 (F  _3,400 = 2.97, p = .032), so we incorporated Huber-White robust SEs when we estimated this model. We found no significant effects on perceived drug benefit from the disclosure of clinical uncertainty or type of data limitations disclosure in models that excluded the control group (p = .091–.855) or by experimental condition in models that included the control group (p = .175–.947).

Perceived Improvement in Clinical Endpoint

On average, participants tended to agree that the drug improved outcomes for the displayed clinical endpoint (see Table 3). Experimental condition was significantly related to perceived improvement in experiment 1 only (F  _4,494 = 3.32, p = .011, η² = .02). Participants who were exposed to a disclosure of data limitations (mean, 4.25; SE = 0.04) thought it was less likely that the drug improved the outcome for the displayed clinical endpoint than participants assigned to the control group (mean, 4.55; SE = 0.07; F  _1,490 = 8.85, p = .003, η² = .02).

Preferences for Nontechnical Versus Technical Disclosures

In all three experiments, the majority of participants preferred the nontechnical specific disclosure over the technical one: experiment 1 (69.3% [n = 343] vs. 16.6% [n = 82], z = 29.20, p < .001), experiment 2 (77.0% [n = 388] vs. 14.9% [n = 75], z = 37.53, p < .001), and experiment 3 (46.2% [n = 246] vs. 35.2% [n = 187], z = 4.79, p < .001).

Discussion

Even commonly used data displays can be confusing, and the conclusions physicians draw from them are sometimes unsupported. Our research shows that, in some cases, a disclosure directly laying out caveats and assumptions or summarizing key takeaways can improve comprehension and lead to more appropriate assessments of clinical data. Also, there were no instances in which physicians' understanding was hindered by the disclosures. Our results support recommendations to accompany data visualizations with a disclosure containing relevant contextual information related to the limitations of the analysis and results [17, 21].

Although physicians' overall perceptions of the benefits conferred by the drug were not affected by the presence or absence of a disclosure of data limitations, the remaining outcome variables examined in this study were significantly affected by the data limitations disclosures in at least one of the experiments, emphasizing the potential for this kind of information format to facilitate interpretation.

First, we found evidence that a disclosure of data limitations can help physicians understand the limitations, analyses, and results presented in a data display. Participants exposed to a disclosure in experiment 3 understood the methodological limitations of the pooled analysis presented in the data display better than participants who did not see the disclosure of data limitations. This finding was not replicated in the other two studies. This finding could be reflective of the nature of each display, which is why we investigated three distinct data displays. For example, with the bar chart used in experiment 3, the only way for people to know some of the important study caveats was through the disclosure. In other words, the disclosure was more important for correctly answering the comprehension questions in experiment 3 than it was in the other two experiments.

Even though the data displays presented data of uncertain clinical utility, more than 40% of physicians in each of the three experiments had an inflated sense that the data clearly indicated that the drug had clinical benefit for the displayed outcome. Only participants in experiment 2 who were exposed to a disclosure of data limitations were less likely than participants assigned to the control group to report that data in the sales aid clearly translated into clinical benefit. In experiment 2, a Kaplan-Meier curve presented results from a statistically nonsignificant interim analysis of overall survival and the last sentence of the data disclosure said: “a conclusion that Xedaliti confers a benefit in terms of OS cannot be made at this time.” The other two experiments did not have a statement that directly commented on clinical benefit. This difference may suggest that explicit language about how the results translate to clinical benefit is more effective than simply noting data limitations or instructing physicians to interpret data cautiously.

Turning to the perceived amount of evidence supporting clinical benefit for the fictitious drugs in the sales aids, as many as a 25% of physicians incorrectly rated the amount of evidence in the sales aids as strong. In experiment 3, participants who were exposed to the disclosure of clinical uncertainty in addition to the disclosure of data limitations were less likely to say the data in the sales aid offered strong evidence in support of clinical benefit than participants who did not see a disclosure of clinical uncertainty. However, there was no difference with respect to the control group. Future research could examine the effect of presenting a disclosure of clinical uncertainty alone, rather than in combination with a disclosure of data limitations, on offsetting any misleading impressions people might form from the data displays.

On average, physicians in all three experiments tended to agree that the drug improved outcomes for the displayed clinical endpoint. In experiment 1, physicians who were exposed to the sales aids with a disclosure of data limitations exhibited more cautious interpretations about the likelihood that their patients would experience an improvement in progression free survival by taking the drug compared with physicians assigned to the control group. It is unclear why the data limitations disclosure did not similarly affect perceptions of improved outcomes on the clinical endpoints for the other experiments. One reason may be because of the content of the disclosure of data limitations for experiment 1, which directly stated that drawing conclusions about the efficacy of the drug in various patient subgroups may not be appropriate.

Physicians' preference for the nontechnical disclosure versus the technical further suggests that this audience values ease of interpretation even if the type of data limitations disclosure did not affect any of the key outcomes. Omitting extraneous statistical terminology and minimizing the need to draw from biostatistical background information to form conclusions, as in our nontechnical disclosure, is responsive to audience preferences. Finally, unlike the disclosure of data limitations, which requires tailoring to each data display, the disclosure of clinical uncertainty uses general language that can apply across many types of data displays. Our evidence suggests that in some situations the disclosure of clinical uncertainty helped physicians accurately interpret the strength of evidence. More research could explore when such a statement is likely to improve understanding and clinical decision-making.

A limitation of the study is that it was not designed to be representative of the target population of oncologists and PCPs with oncology experience and did not use probability-based sampling. Rather, the three randomized factorial experiments were designed to test the effects of different types of data display disclosures in a controlled setting.

Our findings underscore that disclosures of data limitations and a disclosure of clinical uncertainty improve understanding of data displays under some circumstances, but we did not attempt to draw statistical insights across experiments about which sales aid characteristics were influential in this regard. Although the experiments were purposefully designed to explore three unique types of data displays with disclosures that differed in length and content, we did not systematically manipulate these differences across experiments. Instead, the sales aids were modeled off sales aids produced by pharmaceutical companies. A potential avenue for future research would be to examine the characteristics (e.g., type of data display, type of limitations of the data in the data display, wording of disclosures) that make disclosures more or less important for understanding data presented in sales aids. Finally, because we were especially interested in physicians' interpretations of the data displays, we did not include important safety information about the drugs or other clinical outcomes data that would normally be included in this kind of promotional document.

Conclusion

A failure to recognize the limitations of data can lead to misinterpretations of the clinical utility of that data. The findings from the three experiments in this study suggest that disclosures have the potential to increase relevant knowledge and certain aspects of disclosures may make them more likely to contribute to a clear understanding of data displays. These results could be used to inform best practice recommendations from FDA on how to include disclosures of material information in a manner that will render that information clear and conspicuous to audiences. However, more research is needed to inform specific best practice recommendations for the content and format of disclosures used to convey important information about data displays in promotional communications. Although additional research will aid policy makers and the pharmaceutical industry in this space, the current findings can benefit pharmaceutical firms now as they work to comply with FDA statutes (21 U.S.C. 352[a]; 21 U.S.C. 352[c]; 21 U.S.C. 321[n]) requiring the disclosure of information that is material in light of the representations made in their promotional communications.

Acknowledgments

We would like to thank the following current or former U.S. Food and Drug Administration employees for their feedback on study stimuli and questionnaire development: Kathleen Davis, Jessica Cleck-Derenick. We would like to thank the following employees of RTI International for their assistance: Marissa Gargano (assistance with statistical analyses), Molly Lynch (cognitive interviews), Amanda Smith (survey programming), Shari Lambert (stimuli creation), and we thank Emily Geisen, formerly of RTI, for her assistance on the questionnaire development, and cognitive interviews.

This research was funded by contract HHSF223201510002B from the Office of Prescription Drug Promotion, Center for Drug Evaluation and Research, U.S. Food and Drug Administration. Drs O'Donoghue and Aikin are employed by the funder.

Author Contributions

Conception/design: Vanessa Boudewyns, Amie C. O'Donoghue, Ryan S. Paquin, Kathryn J. Aikin

Provision of study material or patients: Vanessa Boudewyns, Amie C. O'Donoghue, Kathryn J. Aikin

Collection and/or assembly of data: Vanessa Boudewyns, Kate Ferriola-Bruckenstein

Data analysis and interpretation: Vanessa Boudewyns, Ryan S. Paquin, Victoria M. Scott

Manuscript writing: Vanessa Boudewyns, Ryan S. Paquin, Amie C. O'Donoghue, Kathryn J. Aikin, Kate Ferriola-Bruckenstein, Victoria M. Scott

Final approval of manuscript: Vanessa Boudewyns, Ryan S. Paquin, Amie C. O'Donoghue, Kathryn J. Aikin, K. Ferriola-Bruckenstein, Victoria M. Scott

Disclosures

The authors indicated no financial relationships.

References

Author notes