Abstract
The National Comprehensive Cancer Network (NCCN) guidelines are among the most widely used guidance in oncology. It is critical to understand the extent to which the recommendations in these guidelines are supported by evidence and to investigate whether these recommendations have been influenced by payments from industry to authors.
We examined the quality and consistency of evidence, as scored by guidelines authors, for systemic treatment incorporated in the NCCN guidelines. Payments data in 2015 were manually abstracted using the Open Payments database, which discloses all payments between the industry and American physicians. Correlations between the percentage of authors who received payments and the proportion of recommendations developed from low‐level evidence per guideline were calculated using Spearman rank correlation.
In total, 1,782 recommendations were identified in 29 guidelines, of which 1,282 (71.9%) were based on low‐quality or low‐consistency evidence (low‐level evidence), including “case reports or clinical experience only” (18.9%). A substantial proportion (31/143, 21.7%) of category 1 (the highest level) recommendations were based on low‐level evidence. The majority of authors (87.1%) received payments from industry. However, no association was found between the prevalence of payments among authors and the percentage of recommendations developed from low‐level evidence per guideline.
The majority of systemic treatment recommendations in the NCCN guidelines are based on low‐level evidence, including more than one in five category 1 recommendations. Payments from industry were prevalent among authors. However, industrial payments among authors were not associated with inclusion of regimen/agent for which there is no conclusive evidence in the guidelines.
The authors found that the majority (71.9%) of systemic treatment recommendations issued in the current National Comprehensive Cancer Network guidelines were based on low‐level evidence. Physicians should remain cautious when using current guidelines as the sole source guiding patient care decisions.
Abstract
摘要
背景?国家综合癌症网络 (NCCN) 指南是肿瘤学领域使用最广泛的指南之一?了解此指南中的建议得到证据支持的程度并调查这些建议是否受到行业给作者付款的影响?这一点至关重要?
材料和方法?按照指南作者的评分?我们针对 NCCN 指南中的系统治疗检查了证据的质量和一致性?我们使用开放付款数据库手动提取 2015 年的付款数据?该数据库披露了制药行业和美国医生之间的所有付款?我们使用Spearman 等级相关来计算收到付款的作者的百分比与根据每个指南的低级证据提出的建议的百分比之间的相关性?
结果?我们在 29 个指南中找到 1 782 条建议?其中的 1 282 条建议 (71.9%) 以低质量或低一致性的证据(低级证据)为依据?包括“仅有病例报告或临床经验” (18.9%)?相当大比例(31/143?21.7%)的 1 类(最高级)建议以低级证据为依据?大部分作者 (87.1%) 会收到来自制药行业的付款?不过?我们未发现作者间付款的普及程度与根据每个指南的低级证据提出的建议的百分比之间存在关联?
结论?NCCN 指南中的大多数系统治疗建议以低级证据为依据?包括五分之一以上的 1 类建议?制药行业付款在作者间较为普遍?但是?作者间的制药行业付款与指南中包含没有决定性证据的疗法/药剂之情况无关?
实践意义:作者发现?现行国家综合癌症网络指南中发布的大多数系统治疗建议 (71.9%) 以低级证据为依据?医生在将现行指南作为指导患者医疗决策的唯一来源时?应该保持谨慎的态度?
Introduction
Clinical practice guidelines are aimed at reducing undesirable variations in practice, defining standards of care, and improving patient outcomes. Physicians may assume that following such guidelines means practicing evidence‐based medicine. However, guideline recommendations imply not only an evaluation of the evidence but also a value judgment based on personal or organizational preferences regarding the various risks and benefits of a medical intervention for a given population. Recently, studies have found that the scientific evidence underlying recommendations in many guidelines was not conclusive [1–4].
The National Comprehensive Cancer Network (NCCN) is a not‐for‐profit alliance of 27 leading cancer centers in the U.S. and among a few leading professional oncology organizations in the world [5]. Within the field of oncology, the NCCN guidelines are among the most comprehensive and widely used guidance in clinical practice. By December 2014, NCCN guidelines were downloaded more than 6 million times [5]. Moreover, the NCCN guidelines have been increasingly incorporated and adopted into physician performance or institutional quality measures [6–12]. In 2011, Poonacha and colleagues analyzed the distribution of strength of recommendations in the NCCN guidelines [13]. However, at that time, the NCCN did not separate the level of evidence from the strength of recommendations [14]. Since 2015, the NCCN began grading five key components of value, namely, quality of evidence, consistency of evidence, efficacy, safety, and affordability of regimen/agent, for systemic treatment in addition to the strength of recommendation (NCCN Guidelines with Evidence Blocks). Therefore, the first objective of the current study was to analyze scientific evidence underlying systemic treatment recommendations in the NCCN guidelines.
Financial conflicts of interest (FCOIs) among authors of guidelines have the potential to influence treatment recommendations [15–17]. Especially in situations in which high‐level evidence is unavailable, experts’ opinions may largely decide whether to recommend a particular intervention in the guidelines. This issue is more relevant to the NCCN guidelines because the Centers for Medicare & Medicaid Services (CMS) and most private payers in the U.S. have accepted the NCCN guidelines as a mandated reference to determine payment for “off‐label” use of anticancer drugs that are not approved for the specific indication by the U.S. Food and Drug Administration [18,19]. Since 2013, the Open Payments Provision (Sunshine Act) required that all U.S. drug and device manufacturers disclose transfers of financial value >$10 to physicians and teaching hospitals, facilitating systematic study of FCOIs and terminating reliance on self‐reporting [20]. Using the Open Payments data, one pioneer study found that the majority (86%) of the authors of the NCCN guidelines on four common cancers, that is, lung, colon, breast, and prostate cancer, received payments from the industry, which has raised significant concerns about whether the payments could have influenced the recommendations in the NCCN guidelines [21]. To address this issue, the second objective of this study was to systematically analyze payments from industry to authors of the NCCN guidelines and to investigate the association between payments from industry and recommendations in these guidelines.
Materials and Methods
Level of Evidence and Strength of Recommendations
From the NCCN website (https://www.nccn.org/evidenceblocks/default.aspx), we obtained the latest versions of the relevant “NCCN Guidelines with Evidence Blocks” as of the end of 2016. In these guidelines, each of the five dimensions, namely, quality of evidence, consistency of evidence, efficacy, safety, and affordability of regimen/agent, were scored by NCCN guidelines authors using a standardized scale from 1 to 5 points (1, least favorable; 5, most favorable; Table 1). We tallied the categories of these five dimensions as well as the strength of recommendations for each guideline on the basis of the following areas of recommendations: risk reduction, initial treatment, salvage treatment, and symptom relief. To simplify the categorization, a score of ≤3 for quality and consistency of evidence was defined as low‐quality and low‐consistency evidence, respectively. The NCCN categories for strength of recommendations are as follows: category I, high level of evidence with uniform consensus; category IIA, lower level of evidence with uniform consensus; category IIB, lower level of evidence without uniform consensus but no major disagreement; category III, any level of evidence but with major disagreement.
National Comprehensive Cancer Network scores of quality/consistency of evidence, efficacy, safety, and affordability of regimen/agent
Includes drug cost, supportive care, infusions, and management of toxicity.
Abbreviations: ADLs, activities of daily living; RCTs, randomized clinical trials.
National Comprehensive Cancer Network scores of quality/consistency of evidence, efficacy, safety, and affordability of regimen/agent
Includes drug cost, supportive care, infusions, and management of toxicity.
Abbreviations: ADLs, activities of daily living; RCTs, randomized clinical trials.
Payment Data
In the Open Payments database, payments are categorized as general payments (GP, including gifts, consultancy and/or speaker fees, and food and beverage), research payments (RP, including any direct compensation, funding for coordination or implementation, or study participant expense), and ownership/investment interests (including stock options, partnership shares) [20,21].
We identified the authors of “NCCN Guidelines with Evidence Blocks,” and, using the public Open Payments database (https://www.cms.gov/openpayments/), payments from industry to each author were manually abstracted. We included all payments made in 2015, except disputed payments. Authors active on multiple guidelines were counted only once.
Statistical Analysis
We tabulated the distribution of quality and consistency of evidence for systemic treatment recommendations in each guideline and overall. For FCOIs, we listed the GP, RP, and ownership/investment separately. In addition, the proportion of authors receiving total payments value greater than $10,000 in 2015 was evaluated because the U.S. Department of Health and Human Services specifically identifies payments exceeding this threshold as significant conflicts of interest [20]. To assess whether the payments to authors were associated with inclusion of regimen/agent for which there is no conclusive evidence in the guidelines, correlations between the proportions of recommendations based on low‐level (low‐quality or low‐consistency) evidence and the percentages of authors who received payment per guideline were calculated using the Spearman rank correlation. In sensitivity analysis, we excluded payments that fell below different dollar value thresholds (e.g., <$100), as physicians receiving such payments may be unaware of and/or less influenced by such payments. A two‐sided α of <.05 was considered significant. All data used in the current study were included in the supplemental online data with this article.
Results
A total of 1,782 systemic regimen/agent recommendations were identified in 29 NCCN Guidelines with Evidence Blocks, with a median of 57 (range, 5–161) recommendations per guideline (supplemental online Fig. 1). Non‐small cell lung cancer had the most recommendations (161), whereas breast cancer risk reduction had the least recommendations (5). Twenty‐three (79.3%) guidelines had more than 30 recommendations. About two thirds of the recommendations were related to salvage treatment (1,159, 65.0%), and one third (573, 32.2%) were related to initial treatment. Only a few recommendations were related to symptom relief (45, 2.5%) and risk reduction (5, 0.3%).
Level of Evidence Underlying Recommendations
The majority (1,360/1,782, 70.7%) of all recommendations were based on low‐quality evidence, including low‐quality randomized trials or well‐designed nonrandomized trials (924, 51.9%), and case reports or clinical experience only (336, 18.9%). The evidence base was highly or mainly consistent for only 650 (36.5%) recommendations, whereas nearly two thirds of the recommendations (1,132, 63.5%) were based on evidence that “may be consistent” (942, 52.9%) or was “inconsistent” (190, 10.7%). Overall, 1,282 of 1,782 (71.9%) recommendations were based on low‐quality or low‐consistency evidence (low‐level evidence); the remaining 500 (28.1%) recommendations were based on high‐quality and high‐consistency evidence (high‐level evidence; Fig. 1).
Level of evidence underlying systemic treatment recommendations (n = 1,782) in the National Comprehensive Cancer Network guidelines. Evidence of low quality or low consistency (score ≤3, graded by guidelines authors) was defined as low‐level evidence. The strength of recommendations was category III for only three recommendations, which were not shown.
The strength of recommendations was category I for 143 (8.0%) and category IIA for 1,581 (88.7%) recommendations, respectively, which meant that the panels reached unanimous agreement regarding this vast majority (96.7%) of recommendations. A total of 55 (3.1%) recommendations were of category IIB, and only 3 recommendations (0.2%) were of category III. Most category II recommendations were developed from low‐level evidence (76.2% for IIA and 81.8% for IIB). Moreover, more than one in five (21.7%) category 1 recommendations were based on low‐level evidence (Fig. 1).
Among all the guidelines, a median of 78.9% of the recommendations were based on low‐level evidence (supplemental online Fig. 1). Regarding the category of recommendations, the majority of the recommendations related to initial treatment, salvage treatment, and symptom relief were derived from low‐level evidence (68.6%, 73.3%, and 86.7%, respectively), whereas all five recommendations related to risk reduction were based on high‐level evidence (Fig. 1).
In total, these 1,782 recommendations were related to 304 clinical circumstances. A median of 4 (range, 1–31) regimens/agents were recommended for each circumstance. Among 141 (46.4%) circumstances with ≥5 recommended regimens/agents, 58 circumstances (41.1%) did not have recommendations developed from high‐level evidence. Among the 144 circumstances with at least one regimen/agent derived from high‐level evidence, 43 (29.9%) and 19 (13.2%) circumstances, respectively, had ≥5 or ≥10 recommended regimens/agents based on low‐level evidence. For example, a total of 31 regimens/agents were recommended (all category IIA) for platinum‐resistant high‐grade serous ovarian cancer, including 2 regimens (liposomal doxorubicin or paclitaxel plus bevacizumab in patients not previously treated by bevacizumab) with high‐level evidence and 29 other regimens/agents with low‐level evidence. Only a few recommendations derived from low‐level evidence had high efficacy (score ≥4), and this value was significantly lower than the proportion of high‐efficacy treatments among recommendations derived from high‐level evidence (14.3% vs. 70.2%, difference: −55.9%, 95% confidence interval [CI]: −60.3% to −51.5%, p < .001; Fig. 2). Moreover, a larger proportion of recommendations based on low‐level evidence had high toxicity (score of safety ≤3) as compared with recommendations based on high‐level evidence (87.4% vs. 81.4%, difference: 6.0%, 95% CI: 2.1%–9.9%, p = .001). However, a smaller proportion of recommendations developed from low‐level evidence were expensive (score of affordability ≤3) as compared with recommendations developed from high‐level evidence (80.7% vs. 86.2%, difference: −5.5%, 95% CI: −9.2% to −1.8%, p = .007).
Comparison of efficacy score between recommendations developed from low‐level and high‐level evidence in the National Comprehensive Cancer Network guidelines. The efficacy of regimen/agent and the quality and consistency of evidence were scored by guidelines authors. Evidence of low quality or low consistency (score ≤3) was defined as low‐level evidence.
Payments from Industry to Authors
We collected data on payments received from the industry in 2015 among authors of 23 guidelines with >30 recommendations. Overall, 572 authors were in the panels of these 23 guidelines, with a median of 32 authors (range, 27–39) per guideline. Among these authors, 30 (5.2%) were not physicians, including NCCN staff and patient advocates. Among the remaining 542 physician authors, most (472, 87.1%) were reported to have received payments from industry (Table 2). The total value of all type of payments was $119 million, including $4.6 million for GP, $115 million for RP, and $33,000 for ownership or investment interest. More than half (280/542, 51.7%) of all authors received a personal total value of more than $10,000. A greater proportion of authors received GP (425/542, 78.4%) compared with RP (292/542, 53.9%). One third of authors (179/542, 33.0%) received GP without any RP. The median value was $667 for GP and $2,346 for RP per author. Only three (0.6%) members had ownership or investment interest in the companies reported.
| Variables | Payments in 2015 | |
|---|---|---|
| General | Research | |
| Authors receiving, n (%) | 425 (78.4) | 292 (53.9) |
| Authors receiving >$100, n (%) | 368 (67.9) | 291 (53.7) |
| Authors receiving >$1,000, n (%) | 259 (47.8) | 280 (51.7) |
| Authors receiving >$10,000, n (%) | 120 (22.1) | 245 (45.2) |
| Percentiles per author, $ | ||
| 25 | 23 | 0 |
| 50 | 667 | 2,346 |
| 75 | 8,578 | 195,639 |
| Range per author, $ | 0–179,988 | 0–553,945 |
| Variables | Payments in 2015 | |
|---|---|---|
| General | Research | |
| Authors receiving, n (%) | 425 (78.4) | 292 (53.9) |
| Authors receiving >$100, n (%) | 368 (67.9) | 291 (53.7) |
| Authors receiving >$1,000, n (%) | 259 (47.8) | 280 (51.7) |
| Authors receiving >$10,000, n (%) | 120 (22.1) | 245 (45.2) |
| Percentiles per author, $ | ||
| 25 | 23 | 0 |
| 50 | 667 | 2,346 |
| 75 | 8,578 | 195,639 |
| Range per author, $ | 0–179,988 | 0–553,945 |
Only three (0.6%) members reported ownership or investment interest in companies.
| Variables | Payments in 2015 | |
|---|---|---|
| General | Research | |
| Authors receiving, n (%) | 425 (78.4) | 292 (53.9) |
| Authors receiving >$100, n (%) | 368 (67.9) | 291 (53.7) |
| Authors receiving >$1,000, n (%) | 259 (47.8) | 280 (51.7) |
| Authors receiving >$10,000, n (%) | 120 (22.1) | 245 (45.2) |
| Percentiles per author, $ | ||
| 25 | 23 | 0 |
| 50 | 667 | 2,346 |
| 75 | 8,578 | 195,639 |
| Range per author, $ | 0–179,988 | 0–553,945 |
| Variables | Payments in 2015 | |
|---|---|---|
| General | Research | |
| Authors receiving, n (%) | 425 (78.4) | 292 (53.9) |
| Authors receiving >$100, n (%) | 368 (67.9) | 291 (53.7) |
| Authors receiving >$1,000, n (%) | 259 (47.8) | 280 (51.7) |
| Authors receiving >$10,000, n (%) | 120 (22.1) | 245 (45.2) |
| Percentiles per author, $ | ||
| 25 | 23 | 0 |
| 50 | 667 | 2,346 |
| 75 | 8,578 | 195,639 |
| Range per author, $ | 0–179,988 | 0–553,945 |
Only three (0.6%) members reported ownership or investment interest in companies.
Association Between Payments to Authors and Evidence Underlying Recommendations
There was no association between the prevalence of GP or RP among authors and the percentage of recommendations derived from low‐level evidence per guideline (all p > .05; Fig. 3). The analyses were repeated excluding all cases for which the total dollar value of GP or RP between industry and a given author was less than $100 or $1,000, and the results were similar.
Comparison of the prevalence of payments from industry among authors and the proportion of recommendations derived from low‐quality or low‐consistency evidence (score ≤3) per guideline. The quality and consistency of evidence were scored by guidelines authors.
Abbreviations: GP, general payments; RP, research payments.
Discussion
In this study, we found that the majority (71.9%) of recommendations in the NCCN guidelines were developed from low‐level evidence, including more than one in five (21.7%) category 1 recommendations. For example, 51.9% of the recommendations were based on “low‐quality randomized trials or well‐designed nonrandomized trials,” and 18.9% were based on “case reports or clinical experience only.” However, it should be emphasized that this is not the mistake of guidelines or the guideline development process. In many clinical circumstances, we will never have ideal evidence; we must use the data we have to best address the needs of patients. The NCCN panels’ charge was to derive evidence‐based or, when evidence was insufficient, consensus‐based treatment guidelines [22]. Our results showed that the panels reached uniform consensus regarding 96.7% of recommendations. The panel members should be commended for their great efforts to put the best available evidence in perspective. More importantly, there were studies, albeit limited, that showed that compliance with guidelines, including the NCCN guidelines, was associated with better outcome [23,24].
Our results highlighted the knowledge gap regarding systemic treatment in oncology. The lack of high‐level evidence was not astonishing, considering previous studies on the characteristics of oncology trials [25]. By analyzing 96,346 trials registered in clinicaltrials.gov, Califf and collagues found that although oncology trials (8,992, 21.9%) were the most common among trials of all specialties, they were more likely to be early‐phase, single‐group, or nonrandomized trials compared with cardiovascular and mental health trials [25]. This might explain the shortage of high‐level evidence in the NCCN guidelines.
We also found that a median of 4 regimens/agents (range, 1–31) were recommended per clinical circumstance. These results highlighted the unmet needs of comparative effectiveness studies. The industry typically does not pursue study topics about comparisons of different, but already approved, drugs. Thus, the government agencies, academia, and cooperative groups are expected to find ways to inspire comparative effectiveness studies, in order to generate high‐level evidence and improve the use of existing treatments for effectively leveraging limited resources in health care. On the other hand, among the 144 circumstances with at least one regimen/agent with high‐level evidence, 43 (29.9%) and 19 (13.2%) circumstances, respectively, had ≥5 or ≥10 recommended regimens/agents based on low‐level evidence. It was arguable to recommend so many regimens/agents, which might put oncologists in potential ethical conflict with patients. Unlike other drugs, chemotherapeutic agents are bought and sold in the doctor's office, and today more than half the revenue of an oncology office may come from chemotherapy sales [26]. Thus, there is financial incentive for oncologists to use more costly drugs when many regimens/agents were recommended in the guidelines and deemed “acceptable.” Guidelines allowing only a few regimens for patients with non‐small cell lung cancer, as compared with the 16 possible drugs and many more combinations included in NCCN guidelines, has been shown to result in equal survival, less use of chemotherapy near the end of life, and lower costs than usual care [27].
This study highlights some concerns regarding the implementation of guidelines. First, because the majority of the recommendations in the NCCN guidelines are developed from low‐level evidence, these recommendations should not be regarded as the ultimate standard of care. Actually, the NCCN clearly stated that “the NCCN guidelines are a statement of evidence and consensus of the authors regarding their views of the currently accepted approaches to treatment. Any clinician seeking to apply or consult the NCCN guidelines is expected to use independent medical judgement in the context of individual clinical circumstances to determine any patients’ care or treatment.” [28]. Quality measures based on guidelines should pay special attention to evidence supporting the recommendations. Second, physicians should remain cautious when using current guidelines as the sole source guiding patient care decisions but use their best clinical judgment and pay attention to patient preferences to make the most appropriate treatment decisions for individual patients. Finally, the off‐label use supported by guidelines should be rigorously scrutinized [18]. There are widespread concerns about the influence of industry‐related payments on recommendations in guidelines, which is likely to result from unintentional bias [16]. However, a systemic review in 2011 found that there are only case studies of the effect of FCOIs on guideline recommendations [29]. To our knowledge, this is the first study to analyze the association between payments from industry and evidence underlying recommendations in the NCCN guidelines. Using the most reliable and transparent data on FCOIs to date, we found that although FCOIs were prevalent among authors of NCCN guidelines, similar to the results in a previous pilot study [21], the prevalence of GP or RP among authors was not found to be associated with inclusion of systemic treatment based on low‐level evidence in the guidelines.
This study has some limitations. First, the NCCN guidelines currently only provide grading of evidence for systemic treatment. However, similar situations were highly likely to exist in other treatment modalities, including radiotherapy and surgery. For example, our previous study found that the number of radiotherapy trials was very limited, despite that as many as 50% of patients with cancer receive radiotherapy [30]. Second, because the Sunshine Act was aimed at disclosing payments between physicians and the industry, we were unable to collect data on FCOIs among patients’ advocates and the NCCN staff [31]. However, they only made up a minority (5.2%) of panel members. Third, because the NCCN guidelines didn't provide an explicit link between the recommendations and the supporting evidence like the European Society for Medical Oncology did, we were not able to examine the scoring of evidence by guideline authors [32,33]. Thus, we could not rule out the possibility that payments from industry to authors might have introduced bias in the evidence scoring process.
Conclusion
Recommendations for systemic treatment issued in the current NCCN guidelines are largely developed from low‐level evidence, including more than one in five treatments with category 1 recommendation. Recommendations developed from low‐level evidence were more likely to have low efficacy and be more toxic to patients compared with recommendations developed from high‐level evidence. Payments from industry were prevalent among authors. However, it was reassuring that industrial payments among authors were not associated with inclusion of treatments for which there is no conclusive evidence in the guidelines.
Acknowledgments
This work was supported by Natural Science Foundation of Guang Dong Province (No. 2017A030312003), the Health & Medical Collaborative Innovation Project of Guangzhou City, People's Republic of China (201803040003), the Innovation Team Development Plan of the Ministry of Education (No. IRT_17R110), and the Overseas Expertise Introduction Project for Discipline Innovation (111 Project, B14035). We thank the English‐speaking professional with a science background at Elixigen Corporation (Huntington Beach, California) for proofreading this manuscript.
Author Contributions
Conception/design: Xu Liu, Jun Ma
Provision of study material or patients: Xu Liu, Ling‐Long Tang, Yan‐Ping Mao
Collection and/or assembly of data: Xu Liu, Ling‐Long Tang, Yan‐Ping Mao
Data analysis and interpretation: Xu Liu, Ling‐Long Tang, Yan‐Ping Mao, Qing Liu, Ying Sun, Lei Chen, Jin‐Ching Lin, Jun Ma
Manuscript writing: Xu Liu, Ling‐Long Tang, Yan‐Ping Mao, Qing Liu, Ying Sun, Lei Chen, Jin‐Ching Lin, Jun Ma
Final approval of manuscript: Xu Liu, Ling‐Long Tang, Yan‐Ping Mao, Qing Liu, Ying Sun, Lei Chen, Jin‐Ching Lin, Jun Ma
Disclosures
The authors indicated no financial relationships.
References
Author notes
Contributed equally
Disclosures of potential conflicts of interest may be found at the end of this article.
