The Inconsistent and Inadequate Reporting of Immune-Related Adverse Events in PD-1/PD-L1 Inhibitors: A Systematic Review of Randomized Controlled Clinical Trials Open Access

Abstract

Background

Immune-related adverse events (irAEs) are of great interest and importance in clinical practice, and many deficiencies and controversies have been noted in the reporting of irAEs. Herein, we aimed to evaluate the current status of irAE reporting in randomized controlled clinical trials (RCTs) of PD-1/PD-L1 inhibitors and to attempt to explain and solve the current pitfalls associated with this reporting.

Materials and Methods

We conducted a systematic review across multiple databases, including PubMed, Web of Science, Embase, and the Cochrane Library. The RCTs that compared PD-1/PD-L1 inhibitors with standard treatments were included. The Harms extension of the Consolidated Standards of Reporting Trials (CONSORT) was used to evaluate the completeness of irAE reporting.

Results

A total of 44 articles and 23,759 patients were included in the analysis. The terminology of the irAEs changed over time (p = .01) and was different among immune checkpoint inhibitors (ICIs) (p = .005). Twenty-two of the studies provided a definition of irAE, but only four of them concretely addressed this definition. The incidence of any grade of irAEs ranged from 16.9% to 96%, whereas grade 3–4 irAE ranged from 2% to 23%. The RCTs with combined therapy exhibited a higher incidence of grade 3–4 irAEs (p = .012). Thirty-two studies reported irAEs in the control arms, whereas seven studies reported irAEs only in the experimental arms. Respiratory, endocrine, and gastrointestinal disorders were the most commonly reported irAEs. IrAEs were generally neglected in the introduction or conclusion sections in all of the study reviews and were never subjected to subgroup analyses. Moreover, withdrawals due to severe irAEs, as well as clarifications of the irAE collection methods, were also poorly reported. RCTs using combination therapies in the experimental arms were associated with a higher reporting quality (p = .032). However, the completeness of the reporting did not improve over the last 5 years (p = .076).

Conclusion

The reporting of irAEs was inadequate, and there are still inconsistencies and controversies in the reporting of irAEs. In the future, authors should be encouraged to adhere to the Harms extension of the CONSORT statement.

Implications for Practice

PD-1/PD-L1 inhibitors profoundly changed the landscape of cancer treatment, and thousands of randomized controlled clinical trials (RCTs) were active or completed over the past decade. However, different from chemotherapy or targeted therapy, the profile of immune-related adverse effects (irAE) was unique. An understanding of irAEs is developed mainly from clinical trials; however, inconsistencies and controversies between trials were noted. This study primarily reviewed the evolution of irAE terminology and definitions and evaluated the reporting quality of each RCT. It was found that RCTs using combined immunotherapy were associated with higher quality of irAE reporting. This article identifies the controversies and deficiencies in current irAE reporting and provides possible explanations and suggestions for these inadequacies.

Introduction

Immunotherapies targeting PD-1 and PD-L1 have dramatically changed the field of cancer treatment [1]. Thousands of clinical trials are currently active or have been completed over the last decade. However, unlike targeted therapy or conventional chemotherapy, immune checkpoint inhibitors (ICIs) exhibit completely different adverse effect profiles, such as endocrine dysfunctions and other autoimmune-like disorders, because of the intrinsic biological traits of the drugs [2]. However, concomitant immune-related adverse events (irAEs) are of great interest and importance for clinical practice because the management of irAEs is totally different from conventional chemotherapy or targeted therapy, in which steroids instead become the main approach. Nonetheless, the recognition of irAEs is still largely dependent on reports from clinical trials in most oncology centers. Therefore, the completeness and quality of the reporting becomes of great importance to clinicians.

For a long time, RCTs, which are the gold standard in evaluating medical interventions, have devoted more attention to information concerning efficacy and survival rather than harmful factors or safety parameters [3]. In response to inadequate adverse event (AE) reporting situations and ethical necessities, the Consolidated Standards of Reporting Trials (CONSORT) group published a Harms extension of the CONSORT statement [4]. Similarly, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) group also developed a checklist for systematic reviews on reporting adverse effects [5]. Consequently, the reporting quality of RCTs has improved over the last decade, because of the efforts of the CONSORT group [6, 7].

In contrast, irAE reporting has not yet been standardized. Some RCTs have not reported irAEs in the main manuscript or even in the supplemental materials, thus leading to a biased understanding of the drugs [8–10]. Moreover, except for the term “irAE,” which is now widely accepted, different terms have been used for the terminology describing irAEs, such as “adverse events of special interest” or “treatment-related select adverse events” [11, 12]. The lack of a uniform definition underlies this phenomenon. Furthermore, the question of whether irAEs could exist in the controlled arms of studies (where no ICIs are used) is another source of controversy. Additionally, the incidence of irAEs can vary dramatically in different trials (as high as 96% and as low as 16.9%, depending on the study). Hence, scientific evidence and ethical necessities have urgently facilitated the need for a transparent and standardized AE reporting system, with both incidence and severity factors being included [3].

Herein, to better depict the current state of the reporting of irAEs, we conducted a systematic review. The following aspects were of great concern in our investigation: (a) the evolution of the definition and terminology of irAEs; (b) the quantitative results of irAE reporting, as well as the controversies or shortcomings that need improvement; and (c) the changes in the completeness and quality of irAE reporting over time.

Material and Methods

Our study was guided by the PRISMA harm statement. The term “IrAE” was used to include all of the other substitution terms, if not otherwise specified.

Search Methods and Inclusion Criteria

A systematic search was conducted by the researchers (X. T. and Z. Z). PubMed, Embase, Web of Science, and the Cochrane Library were searched for relevant published articles from January 2015 to January 2020. “Pembrolizumab,” “nivolumab,” “atezolizumab,” “avelumab,” “durvalumab” and “randomized controlled clinical trials” were used as keywords. The study inclusion criteria were as follows: (a) prospective phase II or III RCTs; (b) solid tumors or hematologic malignancies that were treated with combined or single PD-1/PD-L1 inhibitors; and (c) full-text articles published in English. RCTs that met the following criteria were excluded: (a) PD-1/PD-L1 inhibitors were combined with other biological regimens that may also affect the immune system, such as CTLA-4 inhibitors and interleukins; (b) articles that only included a protocol or abstract, or were lacking fully reported results; and (c) articles with subgroup analysis results that were separately reported.

Data Extraction

The manuscript and supplemental materials were carefully read and examined by the researchers. Basic information, such as author, year of publication, interventions of each arm, blindness, regions in which the RCT was conducted, cancer types, and treatment lines, was documented. Furthermore, the incidence of any irAE and grade 3–5 irAE in both arms and other features, such as whether the irAE was tabulated or displayed in the main manuscript or supplemental materials, were also of interest in this study. Similarly, the individual adverse effects in each article were recorded. Only irAEs with a list of symptoms were considered to be specific or well defined, and the symptoms were grouped into different types according to the National Cancer Institute Common Terminology Criteria for Adverse Events (CTCAE) version 4.0. If both “adverse events of special interests” and “immune-mediated adverse effects” were mentioned and presented in the article, then the latter description was instead extracted. Studies that only reported part of the irAE were not eligible for irAE extraction, because of a reporting bias. The adjusted Harms extension of CONSORT checklist was used for the completeness and evaluation of irAE reporting (Supplemental online Table 1).

Statistical Analysis

The quality of the irAE reporting was expressed by the use of the adjusted checklist score (ACS). The ACS was calculated for each study by dividing the score of the checklist (each item was valued at 1 point) by the total number of items. To investigate the changes in the irAE reporting quality over time, a box plot (showing the relationship between the published year and ACS) as well as a univariate regression analysis were conducted. Other covariates were also included in the linear regression analysis, including single or combined therapy, treatment line, type of ICI used, sample size, number of trial arms, experimental design (blindness), tumor site, geographical region, impact factor of journal, blindness design, and whether or not the study had a positive result for the primary endpoint. The variables that were associated with the ACS at p < .20 in the univariate analysis were further examined with the use of a multivariate regression analysis. A stepwise elimination process was subsequently used to develop a mathematical model. The χ² test was used to compare the irAE types in the different ICI types, and a Fisher's exact test was then used in appropriate cases. A Spearman analysis was used to measure the relationship between the name of the irAE and the published year or drug intervention (studies with mixed name usages or studies published in 2020 were excluded from this set of analyses). For the continuous, normally distributed variables, a Student's t test was used, whereas for the nonnormally distributed variables, the Mann-Whitney U test was used. Values of p < .05 were considered to be statistically significant.

Results

Search Results

The diagram of the research study is presented in Figure 1. In total, 44 RCTs employing nivolumab (n = 10), pembrolizumab (n = 17), atezolizumab (n = 11), avelumab (n = 3), and durvalumab (n = 3) as interventions (with a total of 23,759 patients) were included for the analysis. The years of publication ranged from 2015 to 2020. Only one study was not supported by industry. The general demographics and descriptive statistics of these trials are shown in Table 1 and Supplemental online Table 2.

The Evolution of the irAE Concept

Among the 44 studies that were included in our systematic review, 39 of them had a distinct statement of the term “irAE” in the manuscript. The “irAE” nomenclature differed among the studies, with definitions including “treatment-related select adverse events” (6/39, 15.4%), “adverse events of special interest” (12/39, 30.7%) or “immune-mediated adverse events” (16/39, 41.0%), whereas 5 (12.8%) studies used a mixed terminology (both “adverse events of special interest” and “irAE”) for their descriptions. There was a significant correlation between the name of the “irAE” and the publication year (p = .01), as well as the ICIs that were used in the RCT (p = .005). The name “immune-mediated adverse events,” which is now well accepted, was widely used in the RCTs that used pembrolizumab (11/17, 64.7%), atezolizumab (6/11, 54.5%), and durvalumab (3/3, 100%), whereas the term “treatment-related select adverse events” was mostly used in studies that used nivolumab (6/10, 60%) and in the publication year 2015, when ICI became a more common therapeutic approach.

Twenty-two studies had a casual (18/44, 40.9%) or concrete (4/44, 9.1%) definition for the term “irAE.” Among the casual definitions, the term most often used was the “immunological cause,” although there was no specific explanation for it. Herein, we describe one explicit definition of immune-mediated adverse events as follows: immune-mediated adverse events were defined as adverse events—specified by the sponsor (nonserious and serious) and included by the investigator—associated with pembrolizumab exposure that were consistent with immune phenomena and had a potentially immunological cause regardless of attribution to study treatment or immune relatedness.

irAEs

Except for one study that did not mention the irAE grading instrument, all of the other studies used the National Cancer Institute CTCAE for the safety assessments. IrAE was listed in 39 studies, but one study was excluded from analysis because of a reporting bias. Two studies poorly defined the irAE type. Thirty-six of the studies tabulated the data in detail, and 17 of these studies presented it in the main manuscript, whereas the other studies exhibited the results in the supplementary materials. Twenty-two studies reported the exact incidence of total irAEs by the number of patients, whereas 17 of the studies only reported the incidence by individual cases of irAEs. The incidence of any grade of irAE that was reported in the experimental arm ranged from 16.9% to 96% (median, 28.5%), and the incidence of grade 3–4 irAE ranged from 2% to 23% (median: 7.5%). There were no significant differences in any grade of irAE between the combined therapy and the single agent therapy (p = 0.12); however, the combined immunotherapy therapy exhibited a higher incidence of grade 3 or 4 irAEs (p = .012).

Interestingly, despite the absence of ICI therapy in the controlled arm, 32 studies still reported irAEs, of which 4 studies chose the placebo or best support care (BSC) as the control arms. Moreover, seven studies did not intentionally report irAE incidence in the control arm, with these studies only presenting these data for the experimental arm.

A total of 13 different types of irAE were reported. The number of irAE types that were reported in the studies ranged from 2 to 11 (median, 7). The proportion of the trials reporting different irAE types is presented in Figure 2. The most often reported irAEs were respiratory, thoracic, and mediastinal disorders (36/38, 94.7%), endocrine disorders (35/38, 92.1%), and gastrointestinal disorders (34/38, 89.5%). In contrast, infections and infestations were reported in only one RCT (2.6%). There was no difference between the PD-1 and PD-L1 inhibitors in the irAE types that were reported ≥10 times (χ2 = 7.974, p = .436).

Completeness of irAE Reporting

The results of compliance with the Harms extension of the CONSORT are presented in Figure 3. The highest degree of compliance was found in the domains that were related to the provision of denominators for analysis, with 100% of the studies providing the baseline number of patients that were enrolled in each arm. Similarly, most of the studies provided quantitative data (39/44, 88.6%), the number of irAEs (39/44, 88.6%), and tabulated details (36/44, 81.8%). A total of 97.7% (38/44) of the studies described the instrument that was used for grading or grouping, and 84.1% (37/44) of the studies measured the severity of the irAEs. However, only a few studies provided the methods of collection (7/44, 15.9%), with 11.4% (5/44) of the studies specifically describing the method, and the timing of the irAE collection was stated in 43.2% (19/44) of the studies. Only 6.8% (3/44) of the included RCTs provided an analysis plan, and 11.4% (5/44) of the studies indicated the justification for the selection of the irAEs. Grade 3–4 irAEs were reported in most of the studies, with only three studies having a specific statement of withdrawals that were related to the irAEs. Only one of the studies declared irAEs in the abstract. None of the studies mentioned AE, “safety,” or other words in the title. Similarly, the term “irAE” was generally not mentioned in the introduction or conclusion sections, and not a single study listed a specific irAE in the methods section, with no subgroup analysis being conducted for irAEs in any of the studies.

Changes in the ACS values over time are presented in Figure 4. Univariate regression analysis showed that combined therapy was associated with higher ACS values (regression coefficient = 0.145, SE = 0.062, p = .033). There was also a significant trend for ACS values to change over time with the year of publication (p = .076). Multivariate regression analysis confirmed the significance of combined therapy. All the other covariates failed to show significance with the ACS (Table 2).

Discussion

ICIs have profoundly changed the management of oncological pathologies. The accurate recognition and diagnosis of ICI therapy is of great clinical importance. To our knowledge, this is the first study to investigate the current situation of the reporting of irAEs. We found that there was an obvious evolution of the terminology for irAEs. Respiratory, endocrine, and gastrointestinal disorders were the most often reported irAE types. However, the reporting of irAE remains incomplete. Additionally, the reporting quality of irAEs was associated with combined immunotherapy and did not improve during the last 5 years.

In our study, a significant inconsistency in irAE terminology was noted among the different publications, and the reported frequencies varied over time and across regimens. These incongruities, as well as the incompleteness and lack of standardization, could be due to the several reasons. First, the novelty of the immunotherapy hinders the diagnosis of irAEs. The PD-1/PD-L1 inhibitor RCT was first conducted in 2015, at which time the profile and definition of irAEs were not explicit. Even to date, the exact criteria of irAEs are uncertain, with an increasing number of newly identified irAEs constantly added to the list (such as myocarditis) [13]. Given the difficulties of diagnosis, reports of irAEs are frequently incomplete. Second, RCTs have been developed and founded by different industries, which may also influence the reporting of irAEs. Third, 16 out of 44 studies used combined immunotherapy in the experimental arm, thus making it even harder for clinicians to distinguish whether the adverse effect was immune-related or due to the combined agent. Thus, the recognition and incidence of irAEs may vary greatly across different studies. Finally, as most of the included studies were phase III RCTs, reporting was more focused on the efficacy and survival data, and safety was not a primary endpoint. The importance of irAE reporting may also be underestimated. The lack of unified criteria for irAEs may be the result of the limited knowledge on the mechanisms that drive irAEs. Presently, we can be certain that the introduction of ICIs can influence the function and balance of the immune system, with effects involving autoantibodies, autoreactive T cells and the release of cytokines [14]. Although the delicate system underlying these manifestations remains unknown, we suggest that all RCTs should make a detailed statement on the definition of irAEs and provide the spectrum of symptoms that are of special interest during their clinical trial procedures, which may shed light on the understanding of irAEs in future studies.

Interestingly, some RCTs reported irAEs in the controlled arms, wherein ICIs were not administered. In contrast, seven of the RCTs included in this study only reported irAEs in the experimental arms, ignoring irAEs in the controlled arms. In double/triple-blind RCTs, wherein investigators cannot distinguish between the groups, it is generally accepted that irAEs should be reported in both arms; however, such practices remain controversial in other open-label clinical trials. In our strict interpretation of irAEs, such an approach seems inappropriate. The basic definition of irAE implies adverse events that are related to the reactivation of the immune system following the administration of ICIs. We defined symptoms of irAEs, such as pneumonia or diarrhea, but we should not neglect the prefix “immune-related,” which does not exist in the controlled arm. Chemotherapy-associated adverse events are generally not related to autoimmune symptoms; instead, these events lead to bone marrow depletion and immune suppression. In contrast, evidence of other irAE symptoms in the control arm, such as endocrine symptoms, is harder to understand. These symptoms may not be driven by treatment, and their incidence might be exaggerated. For example, interstitial pneumonia may result from virus infections or other autoimmune diseases, but it presents similar features in computed tomography scans, with steroid therapy leading to similar results and the resolution of radiological alterations. Thus, in this case, irAEs can be mistakenly diagnosed. Unfortunately, bronchoscopy examinations, which can provide strong evidence for diagnosis, were not conducted in the entire population, and less accurate lab tests and clinical experiences were relied upon in most circumstances. Additionally, in cachexic patients, the identification of treatment-related irAEs is extremely difficult. The recognition of irAEs during clinical practice may be difficult, but it is of great necessity because the treatment of the irAEs can be totally different. We suggest that symptoms such as pneumonia should be reported separately in the AE and irAE columns; otherwise, they should be presented as being either “clinically suspected” or “pathologically diagnosed.”

The reporting of irAE types was very homogeneous across the studies, but the incidence of irAEs varied considerably. Thirteen types of irAEs were reported, of which nine types were mentioned in more than 10 articles. The homogeneity may reflect the incomplete recognition of irAEs, wherein the immune-related symptoms could be considered as being nonimmune-mediated. Other less common irAEs, such as central nervous system disorders and myocarditis, exhibit increasing incidence and have attracted increased attention from clinicians [15], which indicates that some progress is occurring in the recognition and understanding of irAEs. However, there is still progress to be made in this aspect.

ACS, which reflects the irAE reporting quality, was associated with combination therapy in the univariate and multivariate linear regression analyses. Interestingly, the incidence of severe irAEs was significantly higher in the combined therapy arm. These results may suggest that the sponsors and investigators were more cautious in reporting irAEs when the incidence of AEs exceeded their expectations. The highest irAE incidence was found in the combination of atezolizumab plus cobimetinib (a MEK inhibitor), with the incidence at 96% [16]; however, whether the MAPK pathway blockade enhanced the incidence of irAEs still requires further validation. In future protocol designs, we should pay more attention to the impact of the interaction between the regimens, not only for the improvement in efficacy but also for the detection of a possibly higher irAE incidence.

JAMA Oncology recently published a meta-analysis summarizing the incidence of AEs in PD-1/PD-L1 inhibitor clinical trials [17]. Wang et al. reported that PD-1 inhibitors were related to a higher incidence of grade 3–4 irAEs than PD-L1 inhibitors, and the most common grade 3–4 irAEs were pneumonitis and dyspnea (respiratory disorders), diarrhea and colitis (gastrointestinal disorders), and alanine transaminase (ALT) or aspartate transaminase (AST) elevations, as well as hepatitis (hepatobiliary disorders). In our study, the top three most commonly reported irAEs were respiratory, endocrine, and gastrointestinal disorders. Endocrine disorders were more commonly reported, but less generic, in severe irAEs, which may be because clinicians are more familiar with endocrine disorders as irAEs and can clinically address them before the development of severe events. Conversely, symptoms such as ALT/AST elevation could be nonspecific and attributed to other factors, such as infections. The accurate diagnosis of immune-mediated hepatitis relies on biopsy procedures, wherein the evidence of lymphocyte aggregation is convincing. However, “extra efforts” harshen the doctors' efforts for diagnosis. Timely endoscopic or biopsy examinations are essential, not only for the sake of the diagnosis but also for the prevention of severe irAEs. Herein, we propose a multiple disciplinary team (MDT) model to manage irAEs, wherein both oncologists and specialists from other departments, such as gastroenterologists and cardiologists, can participate in the diagnosis and treatment process if certain symptoms occur.

All of the studies had deficiencies in the reporting of irAEs. No single study mentioned irAEs in the introduction section, and only a few of them mentioned irAEs in the discussion or abstract sections. Even if the incidence of grade 3–4 irAEs is generally reported, there are few specific data on withdrawal or the influence of severe irAEs on clinical outcomes. Furthermore, subgroup analyses of irAEs would be greatly helpful for clinicians in identifying the sensitive population to irAEs; however, this was largely neglected in the reviewed RCTs in this study. For the past 5 years, the recognition and understanding of irAEs has improved, but the quality of the irAE reporting has exhibited minimal advancement over time. Similarly, in 2013, Péron et al. found that, although the reporting of RCTs had generally improved after the publication of the CONSORT statement, there was no corresponding improvement in AE reporting [3, 6, 7]. This situation is similar in regard to the nononcology fields. In this case, the deficiencies of irAE reporting may be explained by a less comprehensive understanding of irAEs, combined with the fact that most of the studies were phase III RCTs that mainly focused on efficacy but not toxicities. Alternatively, incomplete irAE reporting may result from space limitations during the publication process; however, in our opinion, thorough AE reporting should not be dictated by editorial policies or space limitations. Furthermore, we strongly encourage the use of subgroup analyses in future RCT studies to identify vulnerable populations and to address the urgent clinical need for a better understanding of the impact of sex, age, race, or biological tumor features on irAE occurrence.

Our study was not without limitations. Currently, there is no targeted checklist for the reporting of irAEs. However, the content in the Harms extension of CONSORT checklist comprehensively focuses on reporting completeness and quality, and we thought that it was equally important for RCTs to report irAEs when ICIs were administered. We used a revised edition of the irAE reporting checklist in our study to make it more suitable.

Conclusion

Our study primarily identified inadequate irAE reporting, with significant inconsistencies in terminology and the presence of controversies, such as the reporting of irAEs in the control arm. The reporting quality of irAEs did not improve over time, and future RCTs should be more focused on the recognition and analysis of irAEs, thus improving the reporting frequency with a clear description of the outcomes that are due to severe irAEs. Finally, we propose that symptoms occurring during ICI administration should be carefully examined to determine whether they are immune related; additionally, endoscopic examination, biopsy, and MDT approaches can be helpful in this process.

Acknowledgments

This work was supported by the National Key Research and Development Program of China (No. 2017YFC1308900), National Natural Science Foundation of China (No. 81602057), Clinical Medicine Plus X-Young Scholars Project of Peking University and the Fundamental Research Funds for the Central Universities (No. 20171102), The third round of public welfare development and reform pilot projects of Beijing Municipal Medical Research Institutes (Beijing Medical Research Institute, 2019-1).

All data relevant to the study were included in the manuscript and supplemental materials.

Author Contributions

Conception/design: Shen Lin, Peng Zhi

Collection and/or assembly of data: Xie Tong, Zhang Zhening

Data analysis and interpretation: Xie Tong, Zhang Zhening, Qi Changsong

Manuscript writing: Xie Tong, Zhang Zhening, Li Jian

Final approval of manuscript: Xie Tong, Zhang Zhening, Qi Changsong, Lu Ming, Zhang Xiaotian, Li Jian, Shen Lin, Peng Zhi

Disclosures

The authors indicated no financial relationships.

References

Author notes