Unpublished clinical trials of common rheumatic diseases

Abstract

Objectives

Randomized controlled trials (RCTs) provide high-quality evidence for treatment efficacy, but many RCTs remain unpublished. The objective of this study was to describe the proportion of unpublished RCTs in five rheumatic diseases and to identify factors associated with publication.

Methods

Registered RCTs for five rheumatic diseases (SLE, vasculitis, spondyloarthritis, SS and PsA) with over 30 months since study completion were identified using ClinicalTrials.gov. Index publications were identified by NCT ID numbers and structured text searches of publication databases. The results of unpublished studies were identified in abstracts and press releases; reasons for non-publication were assessed by surveying corresponding authors.

Results

Out of 203 studies that met eligibility criteria, 17.2% remained unpublished, representing data from 4281 trial participants. Higher proportions of published trials were phase 3 RCTs (57.1% vs 28.6% unpublished, P < 0.05) or had a positive primary outcome measure (64.9% vs 25.7% unpublished, P < 0.001). In a multivariable Cox proportional hazards model, a positive outcome was independently associated with publication (hazard ratio 1.55; 95% CI: 1.09, 2.22). Corresponding authors of 10 unpublished trials cited ongoing preparation of the manuscript (50.0%), sponsor/funder issues (40.0%) and unimportant/negative result (20.0%) as reasons for lack of publication.

Conclusions

Nearly one in five RCTs in rheumatology remain unpublished 2 years after trial completion, and publication is associated with positive primary outcome measures. Efforts to encourage universal publication of rheumatology RCTs and reanalysis of previously unpublished trials should be undertaken.

Introduction

Randomized controlled trials (RCTs) provide high quality evidence of a health intervention’s efficacy [1], but many RCTs go unpublished [2–5]. Because trials with negative outcomes are more likely to be unpublished, systemic bias—also known as ‘publication bias’—may inflate the presumed efficacy of therapeutic options or mask potential harms [2–8]. These problems have been observed in several high-profile cases, including reboxetine and lorcainide [7, 9–14]. Selective publication of three reboxetine trials showed non-inferiority in remission and response of major depression when compared with selective serotonin reuptake inhibitors, but inclusion of the remaining five unpublished trials revealed this to be unsupported [14]. A trial of the antiarrhythmic drug lorcainide was discontinued after its harmful effects were discovered but never published, and the subsequent continued use of similar antiarrhythmics likely resulted in unnecessary deaths [7, 9–13, 15]. Unpublished trials also represent substantial waste, both of time and resources, for physicians and research centres [6]. Perhaps most importantly, failing to publish patient data from clinical trials trivializes the sacrifice of time and the assumption of risk made by patients who enrolled.

Unpublished trials have also been identified in the field of rheumatology. In a study of rheumatoid arthritis trials, over a third of studies were not published within 30 months of trial completion, and RCTs with a positive outcome were more likely to be published [16]. However, this study was limited to rheumatoid arthritis, where a remarkable number of successful therapeutic options have emerged over the past two decades. Furthermore, studies since 2010 were not assessed, and an investigation into the reasons for unpublished data was not performed. The objectives for this study were to assess the proportion of completed RCTs that have been published, the time to publication after trial completion, study characteristics associated with these outcomes, and reasons for lack of publication among an additional five common rheumatic diseases: lupus, vasculitis, spondyloarthritis, Sjögren’s syndrome and psoriatic arthritis.

Methods

Included studies

ClinicalTrials.gov was searched for studies assessing treatments of five common rheumatic diseases (lupus, vasculitis, spondyloarthritis, Sjögren’s syndrome and psoriatic arthritis) that were completed between the date of the site’s first availability to the public, 20 February 2000, and 31 December 2018 (see Supplementary Material for full search strategy documentation, available at Rheumatology online). Trials with the following characteristics were excluded: observational studies, interventional studies with non-random subject allocation, extension studies, phase I or IV designation, or recruitment status as active, terminated, withdrawn, suspended or unknown. Trials were also excluded if their primary completion date was after 31 December 2018 to provide a minimum of 30 months for publication.

ClinicalTrials.gov data extraction

The following information was extracted from ClinicalTrials.gov records by one reviewer (C.P.) for each eligible RCT: study identifier (NCT ID), title, first submitted date, start date (the date that the enrolment of participants began), primary completion date (the date of collection of primary outcome measure for the last enrolled patient), study sponsor (industry, non-profit), study phase, disease studied, intervention, study arms(s), comparative effectiveness, masking (open label/blinded assessor, single blind, double blind), enrolment and whether or not the results were submitted to ClinicalTrials.gov.

Experimental interventions were categorized as follows: conventional synthetic DMARDs, biologic or targeted synthetic DMARDs (b/tsDMARDs), non-steroidal anti-inflammatory drugs, corticosteroids, phosphodiesterase (PDE) inhibitors, other drug class (drugs such as antineoplastics or analgesics that are not classifiable in the categories described above), or non-pharmacological intervention (such as counselling).

Index publication identification and extraction

Index publications were identified using the following approach. First, the ClinicalTrials.gov record was reviewed for links to publications. Next, NCT ID numbers were searched using the NCBI Medline Search engine PubMed. Finally, the specific disease term, the designated investigational agent, the principal investigator (PI) and/or identifying trial titles were included in a text search of PubMed, Google Scholar, Scopus and Web of Science. If the first reviewer (C.P.) did not identify any publications, a second reviewer (S.T.) conducted an independent search using the same methodology. Index RCTs were defined by the first study published after trial completion. If no index RCT was identified, the trial was considered unpublished. All searches were finalized on 10 September 2021.

For each published manuscript, C.P. and S.T. performed an independent parallel review to identify the following information: PMID, date of first PubMed publication, PI and study outcome. A study outcome was considered positive if any experimental interventional arm had a statistically significant result for the primary outcome measure, defined by P < 0.05.

Identification of unpublished data

Unpublished outcomes were identified using the following methodology. First, abstracts were identified using a search of PubMed, Google Scholar, Scopus and Web of Science using the investigational agent and the specified disease term. Second, the same search was performed on website search engines of the ACR and EULAR. Third, a Google search of the same parameters was performed to identify industry press releases and/or review papers. Fourth, the results section from ClinicalTrials.gov was reviewed. C.P. and S.T. performed an independent parallel review to identify positive study outcomes, which were defined by any experimental interventional arm having a statistically significant result for the primary outcome (P < 0.05). Press releases that described the outcome negatively, such as ‘unsuccessful’ or ‘negative’, were considered negative.

In cases where a study could not be matched to a publication, the primary author of the originating abstract, the PI or the funding agency listed on the ClinicalTrials.gov website was contacted (if identified) once per week for up to 5 weeks by email. The email contained an explanation of the study as well as a survey that inquired about whether the data had been published and, if it had not been published, why it remained unpublished (see Supplementary Material for the survey instrument, available at Rheumatology online).

Statistical analyses

Descriptive statistics were used to describe the extraction variables. Categorical variables were compared using Fisher’s exact test and continuous variables were compared using the Wilcoxon rank sum test. The time to publication was first evaluated in independent Cox proportional hazard models and graphed using the Kaplan–Meier estimate. All variables with P ≤ 0.10 were then included in an overall model to address potential confounding. Effect sizes were reported using hazard ratios and 95% CIs. All P-values were two-sided and considered statistically significant if <0.05 with no adjustment for multiple comparisons. All analyses were performed on R version 4.04 (R Foundation for Statistical Computing, Vienna, Austria). Data will be made available upon reasonable request.

Results

Of 341 trials identified in the initial ClinicalTrials.gov search, 203 (59.5%) met eligibility criteria and were included in subsequent analysis (Table 1). Of these, 168 out of 203 RCTs (82.8%) had been published, 72 (35.5%) were registered after their start date and 20 (9.9%) were registered after the study was completed (Table 1). RCTs assessed 70 (34.5%) treatments for lupus, 43 (21.2%) for spondyloarthritis, 40 (19.7%) for psoriatic arthritis, 31 (15.3%) for vasculitis and 19 (9.4%) for Sjögren’s syndrome (Table 1). Among RCTs, 140 (69.0%) assessed b/tsDMARDS, and 22 (10.8%) were comparative effectiveness trials (Table 1). Of the 35 unpublished trials, 21 (60%) did not post results to ClinicalTrials.gov (Table 1). There was a total of 4281 patients whose data have not been published.

Out of the 203 RCTs, 168 (82.8%) were published (Table 1). Characteristics associated with trial publication included a positive primary outcome measure (64.9% vs 25.7% unpublished, P < 0.001), phase III trial designation (57.1% vs 28.6% unpublished, P < 0.05), use of a b/tsDMARD intervention (71.4% vs 57.1% unpublished), posting of results to ClinicalTrials.gov (66.7% vs 34.3% unpublished, P < 0.05) and higher enrolment (median 199 vs median 57 for unpublished, P < 0.001) (Table 1). Among diseases, published trials were more likely to have assessed psoriatic arthritis (23.2% vs 2.9% unpublished), spondyloarthritis (23.2% vs 11.4% unpublished) and vasculitis (16.1% vs 11.4% unpublished), while unpublished trials were more likely to have assessed SS (6.0% vs 25.7% unpublished) and lupus (31.5% vs 48.6% unpublished) (Table 1, Fig. 1B). Industry sponsorship was not associated with publication (71.4% vs 68.6% for unpublished studies, P = 0.84) (Table 1).

The median duration between study start and study completion was 762 days (IQR 614) (Table 1) and the median time from study completion to study publication for published studies was 766 days (IQR 472). Among published RCTs, 43% of RCTs had been published within 2 years. In multiple unadjusted Cox proportional hazards regressions, the time to publication was associated with a positive study outcome (hazard ratio [HR] 1.56; 95% CI: 1.12, 2.16), registration before study completion (HR 1.74; 95% CI: 0.99, 3.09), phase II–III trial status (HR 0.75; 95% CI: 0.37, 1.52), phase III trial status (HR 1.28; 95% CI: 0.93, 1.77), results being posted on ClinicalTrials.gov (HR 1.54; 95% CI: 1.10, 2.16), and results submitted to ClinicalTrials.gov but not yet posted (HR 1.19; 95% CI: 0.47, 2.98) (Table 2). In a regression that included all variables with P ≤ 0.10, a positive study outcome was independently associated with the time to publication (HR 1.55; 95% CI: 1.09, 2.22) (Table 2, Fig. 2).

Among the 191 RCTs where outcomes could be identified, 118 (61.8%) had a statistically significant result (P < 0.05) for the primary outcome measure and were considered positive (Supplementary Table S1, available at Rheumatology online). Trials that met their primary outcome measure were more likely to have assessed psoriatic arthritis (33.1% vs 1.4% negative) and spondyloarthritis (28.0% vs 11.0% negative), while trials that failed to meet their primary outcome measure were more likely to have assessed Sjögren’s syndrome (1.7% vs 16.4% negative) and lupus (22.9% vs 53.4% negative) (Supplementary Table S1, available at Rheumatology online, Fig. 1A). Positive trials were more likely to be industry-sponsored (79.7% vs 60.3% negative, P < 0.05), phase III (61.9% vs 41.1% negative) and post results to ClinicalTrials.gov (72.0% vs 53.4% negative, P < 0.001) (Supplementary Table S1, available at Rheumatology online). The proportion of trials that met their primary outcome measure increased over time, from 5.9% prior to 2005 to 13.6% between 2005 and 2009, 32.2% between 2010 and 2014, and 48.3% between 2015 and 2018 (Supplementary Table S1, available at Rheumatology online).

Out of 35 unpublished trials, 34 (97.1%) had PI or sponsor that could be contacted, 10 (28.6%) of which responded to our survey. Respondents were able to pick more than one reason for lack of publication. The most commonly cited reason was ongoing preparation or review of the manuscript (50.0%), followed by sponsor or funder problems (40.0%), unimportant or negative result (20.0%), journal rejection (10.0%), and poor-quality study or design (10.0%). Half of the respondents agreed that lack of publication was primarily the decision of the trial sponsor (37.5% strongly agreed, 12.5% agreed) and the majority of respondents strongly disagreed that lack of publication was primarily the decision of the PI (87.5% strongly disagreed vs 12.5% agreed).

Discussion

In this assessment of over 200 trials of five common rheumatic diseases, nearly one in five rheumatology RCTs remained unpublished more than 30 months after completion. These unpublished trials included data for 35 experimental agents and from over 4000 patient participants. Published trials were more likely to have a positive primary outcome measure and higher enrolment. The proportion of trials that were published varied substantially by disease, with almost half of Sjögren’s syndrome trials going unpublished. Additionally, positive RCTs were published significantly more quickly than negative RCTs.

This study extends prior work in rheumatoid arthritis to the modern era and to five other rheumatic diseases, confirming that many rheumatology RCTs remain unpublished 30 months or more after completion [16]. These findings are similar to other fields, which have observed rates between 25% and 35% [17], but were notably somewhat lower than prior studies. Optimistically, this could be attributable to improvements over time, as our analysis was the most recent on this subject. Regulatory bodies have taken substantial steps to encourage transparency and have implemented requirements for data sharing, so it seems plausible that the lower rate of unpublished studies in our analysis could reflect genuine improvements over time. However, these requirements have not been universally expanded to all trials, nor has there been consistent enforcement of this regulation. The majority of the unpublished trials in our analysis also failed to post any data to ClinicalTrials.gov.

We also noted substantial differences between diseases, with spondyloarthritis, psoriatic arthritis and vasculitis having higher rates of publication than lupus and Sjögren’s syndrome. This discrepancy between diseases is surprising, but given our observation that positive outcomes were associated with publication, it seems likely that the relative lack of success in recent trials of lupus and Sjögren’s syndrome contributed. The high rate of unsuccessful trials in lupus has previously been attributed to disease heterogeneity, imprecise diagnosis, high response rates in placebo groups, poor performance of treatment responder indices and inadequately powered trials [18–22]. Our data provide some support for concerns about trial size as well, as Sjögren’s syndrome trials enrolled one-fifth as many patients on average as psoriatic arthritis trials.

The consequences of unpublished rheumatology trials lie beyond the scope of this work, but prior high-profile instances of unpublished trials have been associated with substantial harm to patients. In one notable example, a trial of the antiarrhythmic drug lorcainide was discontinued after its harmful effects were discovered, but the results were never published [7]. Ten years later, increased mortality was observed among patients treated with similar class IC drugs, including encainide and flecainide [7, 13]. In another example, data revealing a safety signal for the antidepressant drug paroxetine were suppressed, and a subsequent systematic review found that a purportedly favourable treatment profile was undermined by the unpublished results [23]. Such concerns may be of particular importance in rheumatology, where safety signals that affect the immune system can be life threatening. In the study described here, multiple investigational agents were also under investigation for other diseases. At a minimum, subsequent studies using pooled safety data from both published and unpublished trials are necessary.

Similar to other studies, a positive primary outcome measure was associated with higher odds of publication and a faster time to publication [3, 7]. Favouring publication of positive or statistically significant trials distorts the presumed efficacy of medical interventions, which can result in over-use of low value therapies. The US spent more than $1.3 billion stockpiling oseltamivir prior to the 2009 influenza A (H1N1) pandemic, for instance, but its effectiveness in reducing hospitalizations or complications was called into question after subsequent assessments identified substantial publication bias [24–26]. In our study, the majority of unpublished trials investigated novel therapies, including 20 b/tsDMARDs. Many of these treatments could supplant established therapies, which have well-defined safety profiles and generally affordable costs. If publication bias distorts the perceived benefit of such therapies, patients and society may shoulder the burden of higher financial costs without commensurate benefits [7, 27, 28].

In an effort to better understand why the included studies went unpublished, corresponding authors of unpublished trials were surveyed. Five reported that the manuscript was being prepared or reviewed, suggesting that some unpublished manuscripts may ultimately be published. With regard to the reasons that manuscripts went unpublished, over half of respondents to our survey reported that the reason was due to a decision by the study sponsor. A belief that the results were unimportant or negative was also cited. This is overall similar to a prior study that identified reasons for manuscripts to go unpublished, which also included a lack of time or a low priority, studies being incomplete or ongoing, or studies that were still in preparation or under review [29]. It was noted that of the four respondents who reported issues with the sponsor or funding leading to non-publication, all of them were industry-sponsored. One respondent referenced inadequate recruitment, which is a common issue affecting clinical trials. However, all of the studies we included had finished recruitment and were listed as ‘completed’ on ClinicalTrials.gov, so this should not have been a barrier for the majority of authors.

Regardless of the reasons, interventions to improve rates of reporting and publication should be considered. First, the federal government could increase regulation and enforcement of the posting of results to ClinicalTrials.gov. Second, trial databases could require public reporting of the reasons for non-publication, which at a minimum would improve transparency. Third, organizations that provide funding for clinical trials could require an explicit statement that trials will be published in a timely manner before performing functions such as approval, marketing authorization or reimbursement for medical products [30]. Fourth, professional organizations such as the ACR and EULAR should encourage that results be published. Fifth, reanalysis of patient level data from previously unpublished trials should be performed. The results of such analyses—were they to meaningfully influence the interpretation of already-published data—should be used to reassess FDA approvals. The Restoring Invisible and Abandoned Trials (RIAT) movement has made efforts to perform these analyses and has created an audit trail to increase transparency [31]. Finally, it is possible that negative trials are more likely to be rejected by journals; if that is the case; the importance of publishing negative trials should be re-emphasized among journal editors.

This study has several limitations. Our search was limited to ClinicalTrials.gov and may not be representative of other trial registration databases, such as EudraCT or the World Health Organization registry. We also limited the scope to the study of five common inflammatory diseases treated by rheumatologists, excluding rheumatoid arthritis, and our results may not be representative of the entire field [32, 33]. Vasculitis trials were limited to those defined by 2012 Chapel Hill criteria [34]. In addition, we excluded all trial designs that were phase I or IV or utilized non-random subject allocation. We were unable to ascertain the outcome of 12 RCTs. When contacting PIs and study sponsors of unpublished RCTs, only publication status was explicitly requested. Given the low number of survey respondents, this study is not sufficiently powered to make definitive statements about why studies remained unpublished.

In conclusion, this study identified a substantial number of unpublished trials in rheumatic diseases, which is similar to other fields [17]. Associations between positive outcome measures and publication suggest that systemic publication bias likely exists among rheumatology trials, which is likely higher among disease states that have a higher proportion of unpublished trials, such as systemic lupus erythematosus and Sjögren’s syndrome. In addition, this study identified an association between positive outcomes and shorter times to publication among rheumatology RCTs. Efforts to encourage universal publication of rheumatology RCTs and reanalysis of previously unpublished trials are urgently needed. We owe it to our patients.

Supplementary material

Supplementary material is available at Rheumatology online.

Data availability

Data will be made available upon reasonable request.

Contribution statement

C.P., E.V., K.H., S.T., A.G., S.S. and M.P. all made substantial contributions to the conception or design of this manuscript. All authors made substantial contributions to the acquisition, analysis and interpretation of data for the work, drafted the work or revised it critically, gave final approval of this version to be published, and agree to be accountable for all aspects of the work.

Funding

No specific funding was received from any bodies in the public, commercial or not-for-profit sectors to carry out the work described in this article.

Disclosure statement: M.P. participates in clinical trials with Abbvie and AstraZeneca and receives consulting payments from Novartis. The other authors have declared no conflicts of interesnt.

Ethical approval information: This study did not use patient data and institutional review board approval was not required.

Patient and public involvement: There was no patient or public involvement with this manuscript.

Acknowledgements

C.P. is supported by the Medical College of Wisconsin Medical Student Summer Research Program (MSSRP). E.V. is supported by the Medical College of Wisconsin Medical Student Summer Research Program (MSSRP) and the National Institute on Aging Training Grant T35AG029793 (Meurer). K.H. is supported by the Medical College of Wisconsin Medical Student Summer Research Program (MSSRP) and the Center for Immunology Summer Research Program Fellowship. A.D. is supported by the Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Health Care Delivery and the Rheumatology Research Foundation Scientist Development Grant. M.P. is supported by a Rheumatology Research Foundation Scientist Development Grant.

References