Transparency in Infectious Disease Research: Meta-research Survey of Specialty Journals Free

Abstract

Background

Infectious diseases carry large global burdens and have implications for society at large. Therefore, reproducible, transparent research is extremely important.

Methods

We evaluated transparency indicators (code and data sharing, registration, and conflict and funding disclosures) in the 5340 PubMed Central Open Access articles published in 2019 or 2021 in the 9 most cited specialty journals in infectious diseases using the text-mining R package, rtransparent.

Results

A total of 5340 articles were evaluated (1860 published in 2019 and 3480 in 2021 [of which 1828 were on coronavirus disease 2019, or COVID-19]). Text mining identified code sharing in 98 (2%) articles, data sharing in 498 (9%), registration in 446 (8%), conflict of interest disclosures in 4209 (79%), and funding disclosures in 4866 (91%). There were substantial differences across the 9 journals: 1%–9% for code sharing, 5%–25% for data sharing, 1%–31% for registration, 7%–100% for conflicts of interest, and 65%–100% for funding disclosures. Validation-corrected imputed estimates were 3%, 11%, 8%, 79%, and 92%, respectively. There were no major differences between articles published in 2019 and non-COVID-19 articles in 2021. In 2021, non-COVID-19 articles had more data sharing (12%) than COVID-19 articles (4%).

Conclusions

Data sharing, code sharing, and registration are very uncommon in infectious disease specialty journals. Increased transparency is required.

(See the Editorial Commentary by Clancy et al on pages 225–6.)

Infectious diseases is an important field in medicine, epidemiology, and public health across a spectrum that spans basic science, translational research, clinical, and population-level applications. The global burden of infectious disease has been large, with a higher share in less developed countries [1–4]. With the advent of the coronavirus disease 2019 (COVID-19) pandemic, the developed world was sensitized to the field with awakened interest and its funding and research output increased rapidly [5–7]. The rigor and reliability of the evidence generated in the field of infectious diseases therefore has major implications for the health of individuals, populations, and societies at large. In this regard, transparency features, such as sharing of data and code, availability of registered protocols, and reporting of conflicts of interest and funding, can be fundamental in evaluating the evidence obtained by research investigations in infectious diseases [8–10]. Previous work has assessed these transparency indicators in depth in infectious disease models specifically, a type of research that became highly popular and influential during the COVID-19 pandemic [11]. It was found that the majority of such epidemiological modeling studies do not share their data and/or their code, and very few have registered protocols. The vast majority of published articles have conflict of interest and funding statements, but it is not clear whether the information reported is complete [12, 13].

Infectious disease research, nevertheless, encompasses a very large range of study designs and research efforts. It is unknown whether transparency is high for these diverse types of designs and whether there are some study features and characteristics that may be associated with better or worse performance of the published articles in terms of transparency indicators. More importantly, the COVID-19 pandemic was a crash test for many scientific fields, and most prominently this applied par excellence to the field of infectious diseases. Massive publication volume may not necessarily have been accompanied by high quality and/or transparency [5, 14, 15]. It is important to study whether the infectious diseases literature published during the pandemic was different in this regard compared to the prepandemic articles published in the same journals that specialize in infectious diseases.

Here, we present the results of meta-epidemiological assessment of transparency indicators of recent articles published in the major infectious disease specialty journals, comparing the transparency performance of these journals in the prepandemic (2019) and in the COVID-19 pandemic (2021) periods.

METHODS

The protocol registration is available at https://doi.org/10.17605/OSF.IO/FYZPX. Amendments to the original protocol appear in the Supplementary Text.

Journals and Articles

We examined the articles published in 2019 and 2021 by the 9 specialty journals that received the largest total number of citations in infectious disease research, according to the InCites Journal Citation Reports [16] in the respective specialty category. We avoided 2020 since early months had no/few COVID-19 articles, whereas subsequent months had many COVID-19 articles rushed quickly to publication, perhaps with different research practice standards. We focused on articles indexed in the PubMed Central Open Access subset of PubMed, similar to previous work, since they can be massively downloaded for text mining of transparency indicators. We excluded letters, editorials, and study protocols. The specific journals and search queries (overall and for COVID-19 articles) appear in Supplementary Table 1.

Data

For each eligible article, we used PubMed to extract information on meta-data (PMID, PMCID, publication year, journal name, affiliation) and the R package rtransparent [17] to extract transparency indicators (code sharing, data sharing protocol registration, conflicts of interest, and funding statements). rtransparent was also used to retrieve whether a journal was original research or a review, so as to be eligible. rtransparent searches through the full text of the articles for specific words or phrases that strongly suggest that the aforementioned transparency indicators are present in that particular article. The program uses regular expressions to adjust for variations in expressions. For details on the exact code and terms/phrases captured by rtransparent, see Serghiou et al [17].

To validate the performance of the automated text mining algorithms, we evaluated 200 randomly selected articles manually for the presence of each of the 5 text mined indicators. This aimed to identify for each indicator the number of false positives (indicator identified in text mining, but not manually) and false negatives (indicator not identified in text mining but identified manually). This information allowed a correction of the estimates of the proportion of articles that satisfy each of the transparency indicators. As in previous work [11], the corrected proportion C(i) of publications satisfying an indicator i was obtained by U(i) × TP + (1 − U(i)) × FN, where U(i) is the uncorrected proportion detected by the automated algorithm, TP is the proportion of true positives (proportion of those manually verified to satisfy the indicator among those identified by the algorithm as satisfying the indicator), and FN is the proportion of false negatives (proportion of those manually found to satisfy the indicator among those categorized by the algorithm not to satisfy the indicator).

In the 200 randomly selected articles, moreover, whenever there was a conflict of interest disclosure statement, we noted whether no conflict was disclosed (eg, “the authors have no conflicts to disclose”). Similarly, for funding, we noted whether the statement stated that no funding was received or specific funding was disclosed.

From a random sample of the 200 articles, we also manually extracted additional characteristics: meta-data (country of the first and last author; length of the article; number of tables, figures, and appendices); study design; type of pathogen/disease/drug or vaccine studied; and, for clinical studies, whether a study was interventional or not, and its primary focus if interventional (therapy or prevention). Finally, we extracted information about the study population including age group, human versus animal work, and sample size (for systematic reviews, we used the number of included articles). See the protocol (https://doi.org/10.17605/OSF.IO/FYZPX) for definitions and details on items extracted.

Comparisons and Statistical Analysis

We considered 3 primary comparisons that were conducted using Fisher exact tests for each transparency indicator separately. For analysis of associations, we used a statistical significance threshold of .005 [18]. The analysis code was written in R 4.2.1 software [19].

We compared transparency indicators for 2019 versus 2021, 2019 versus 2021 non-COVID-19, and 2021 COVID-19 versus 2021 non-COVID-19 articles in order to evaluate whether there has been an improvement over time, and whether COVID-19 articles differed from non-COVID-19 ones. These comparisons were performed with and without stratification by journal, since it was possible that differences are seen across specific journals. Fisher exact tests were used to test our unstratified comparisons. The tests were performed for each transparency indicator separately. To stratify and account for journals, we had originally anticipated to perform random-effects meta-analyses for odds ratios, stratifying the overall sample by journal. However, there was no significant between-journal heterogeneity for almost all analyses (with rare exceptions), and some cells in 2 × 2 tables for some journals had very small numbers and zeros. Therefore, we preferred as the default the fixed-effects analysis with a 0.5 correction for 2 × 2 tables with zero cells, but random effects are also presented.

Secondary comparisons of the transparency across the manually extracted characteristics were performed. The comparisons were descriptive rather than testing specific hypotheses and they focused on code sharing, data sharing, and registration since it was expected (and indeed documented eventually) that the vast majority of articles would have a placeholder statement for conflict of interest and funding.

RESULTS

The search query retrieved 5701 articles from the PubMed Central Open Access subset of which 361 were neither an original article nor a systematic review (using rtransparent classification of articles) and were excluded. Of the 5340 eligible articles, 1860 were published in 2019 and 3480 in 2021. Nineteen articles published in 2019 were retrieved by the COVID-19 search query; upon in-depth examination, none proved to be COVID-19 research but rather reflected a 1% false COVID-19 assignment error. In total, 1828 of 3480 articles published in 2021 were retrieved with the COVID-19 query. The proportion of COVID-19 publications in 2021 ranged from 0% for Infection and Immunity to 88% for International Journal of Infectious Diseases.

Transparency Indicators

In the 5340 articles, 98 (2%) shared code, 498 (9%) shared data, 446 (8%) were registered, 4209 (79%) contained a conflict of interest statement, and 4866 (91%) included a funding statement.

On manual validation, the false-positive rate of the automated algorithms was 0/3 (0%) for code sharing, 1/12 (8%) for data sharing, 1/19 (5%) for registration, 0/145 (0%) for conflict of interest disclosures, and 0/166 (0%) for funding disclosures. The respective false-negative rates were 2/180 (1%), 4/171 (2%), 1/164 (1%), 1/38 (3%), and 1/17 (6%). Forty-one of 182 (23%) of funding disclosures practically stated that there was no funding for the study. One hundred seventeen of 162 (72%) of the conflict of interest statements practically declared that there was no conflict of interest. Adjusting for the manual validation results, the corrected proportions of transparency indicators were 3% for code sharing, 11% for data sharing, 8% for registrations, 79% for conflict of interest disclosures, and 92% for funding disclosures.

When stratifying the transparency indicators across journals, there were observable differences across publications in the different journals, with some outliers. For instance, Lancet Infectious Diseases had a rate of 9% for articles that shared code while the others had rates around 2%. For data sharing, Infection and Immunity was an outlier with 25% data-sharing articles, while Emerging Infectious Diseases and Journal of Antimicrobial Chemotherapy were at 14%–15% and all other journals just had 5%–10% data sharing. There was also a clear difference in registration rates, with the highest rates in Lancet Infectious Diseases (31%) and the lowest in Infection and Immunity (1%). Rates of conflict of interest disclosures varied widely, from 7% in Emerging Infectious Diseases to 100% in BMC Infectious Diseases. Funding disclosures varied modestly, from 65% in Emerging Infectious Diseases to 100% in BMC Infectious Diseases (Table 1).

The primary comparisons of transparency indicators between 2019 and 2021 showed a statistically significant increase in code sharing, a decrease in data sharing, and a decrease in funding disclosures, but the differences were very modest in absolute magnitude (all ≤4%). Comparing the samples from 2019 and the non-COVID-19 articles from 2021 yielded similar rates of transparency in these 2 years across all 5 transparency indicators. Finally, the comparison of the 2021 non-COVID-19 articles to the COVID-19 articles published in the same year showed a statistically significantly higher rate of data sharing and a lower rate of conflict of interest disclosures in non-COVID-19 articles (Table 2).

Meta-analysis of the data stratified per journal showed no statistically significant heterogeneity in any of the comparisons with the exception of funding disclosures (I² = 72%; 95% confidence intervals [CI] 44%–86%) and registration (I² = 65%; 95% CI 22%–85%) for the comparison of COVID-19 versus non-COVID-19 articles in 2021. Meta-analytic results were largely similar to those inferred by Fisher exact tests without stratifying for journal (Table 3; Supplementary Figures 1–30).

In Table 4 we have tabulated the transparency according to the groups of the main comparisons and journal and performed stratified Fisher exact tests for all hypotheses and all journals separately for each indicator.

Manually Assessed Randomly Selected Articles

In the random sample of 200 articles that were examined in depth, the most common countries of first and last author were the United States and China and they collectively accounted for about a third of the articles. Most studies were classified as observational (34%) or epidemiologic surveillance (22%). The large majority of studies (88%) addressed a specific pathogen. Most studies (55%) used neither P values nor confidence intervals (CIs) in their abstract. Most studies were on humans (80%), and age groups represented were very diverse. One hundred sixty-six of the 200 (83%) studies could be characterized as clinical studies. Among the clinical studies, 42 (25%) were interventional and 124 (75%) were noninterventional. The primary focus of the study was epidemiology in 76 (38%) articles, therapy in 34 (17%), diagnosis in 21 (11%), and prevention in 14 (7%) articles. Thirteen (7%) had a focus on risk and 9 (5%) had a pathophysiological focus. Details on the characteristics of the manually assessed random sample appear in Supplementary Table 2. No characteristic was associated with markedly higher rates of code sharing or data sharing, except for code sharing for predictive models (28%), but numbers are very thin to make robust inferences. For registration, 10/10 (100%) clinical trials and 4/13 (31%) systematic reviews were registered, but only 3 other studies were registered among the remaining 177 articles.

DISCUSSION

This evaluation of 5340 articles in infectious disease research published in 2019 and in 2021 in the top 9 specialty journals with respects to impact showed that only a small minority of articles shared code or data. Study registration was also rare, and it pertained almost exclusively to clinical trials and some systematic reviews. Conflict of interest and funding disclosures were, conversely, very common. There was no major change in absolute magnitude in the proportion of articles that satisfied these transparency indicators during 2021 (a pandemic year) versus 2019 (a prepandemic years). COVID-19 articles had modestly lower rates of data sharing.

Previously [11], we had studied 1338 infectious disease modeling articles from 2019 and 2021 and we had showed that approximately a quarter of publications shared code and a modestly higher proportion shared data. High rates of conflict of interest and funding disclosures were also observed (around 90% for both), while registrations were below 1%. The disparities are likely due to the difference in the study characteristics; infectious disease models were very sparse in our current sample. Most of the studies that we evaluated were clinical, and registration is more common in clinical trials and clinical research overall [20]. Still, registration is distinctly uncommon outside of clinical trials, with the exception of some systematic reviews. Moreover, code and data sharing are expected to be more common in modeling studies, where such sharing is indispensable, while these research practices remain quite rare in most clinical and epidemiological research that comprises the vast volume of articles published in infectious disease journals.

The numbers regarding code sharing are similar to the overall assessments previously performed by Serghiou et al [17], but the rate of data sharing and registration was higher when studying the entire PubMed Central Open Access subset across all scientific disciplines. Iqbal et al [21], showed similar rates of transparency in publications indexed in 2014 when they studied a random sample of 441 PubMed articles between 2000 and 2014. Overall, infectious disease specialty journals seem to be performing below the average of biomedical journals in data sharing and registration.

Code and data sharing are critical elements of computational reproducibility. This sharing is an essential part of most conducted research as it allows the reanalysis and the assessment of the methods section for potential errors or nondisclosed analytical approaches that may affect the study results. We should acknowledge that code sharing may not be applicable to some types of infectious disease articles where there are no quantitative computational parts. Also, data sharing may need to take into account the specific circumstances of the research—for example, consent requirements and the need for deidentification—that may be common in infectious disease research, especially clinical studies. Nevertheless, more transparent sharing in general is desirable. Sharing practices can also maximize the future use of the data and enhance the value of the collected information [22]. The absence of sharing affects the trustworthiness of the analytical approaches in published research. Many efforts have been directed toward increasing reproducibility through increased sharing. These include the changes in journal policy for Science regarding data sharing and the new regulations regarding data sharing that the National Institutes of Health is now implementing [8, 23, 24], further showing the importance of this aspect of transparency.

Registration, on the other hand, is key in research that aims to inform clinical practice and policy and to avoid “vibration of effects” [25, 26], that is, instability in the results due to post hoc, selective choices of statistical analyses and reporting bias, choosing which analyses and outcomes to report. A sensible prospective registration process may increase the reliability of the results from observational as well as interventional research and one may compare notes between registered protocols and subsequent results that are made available [10, 27–30]. Nevertheless, registration remains very uncommon outside of randomized trials and systematic reviews. Differences in proportions of registration across infectious disease journals reflects mostly whether they publish many clinical trials and systematic reviews. When concepts and analyses can be executed promptly, investigators may fear that their ideas may be scooped by others upon registration. However, platforms like Open Science Framework (OSF) allow investigators to register a time-stamped protocol without making it publicly visible until they feel comfortable (eg, upon acceptance or publication of the manuscript).

Our meta-research assessment has certain limitations. First, the existence of transparency indicators in the text of a published article does not guarantee the informational value of the statements. For example, for code sharing and data sharing, one would have to examine in depth whether the code and the data are not only accessible but functional and contain all the information needed to use them.

Second, the veracity of some statements of transparency may also be brought into question, especially for conflict of interest and funding disclosures. Disclosures are notoriously difficult to verify for their completeness and accuracy, despite the emerging availability of some resources such as the Centers for Medicare and Medicaid Services Open Payments database [31] in the United States. Many assessments have been performed with these data and show a worrying amount of undisclosed conflicts in opioid prescription [32], dermatology [12], and otolaryngology [13] guidelines, among others. The research and clinical practice of infectious diseases is not exempt from this pattern. One well-known example is the so called “Lyme wars,” where conflicts originally remained unreported in clinical practice guidelines [33]. Other empirical evaluations have also shown large rates of conflicts in infectious diseases in specific settings or countries (eg, in Japan) [34]. One option is that journals could spot-check reported disclosures against Open Payment databases to identify delinquent authors before publication. However, many authors are not covered by the currently available databases. Improved centralized databases of industry relationships for both clinicians and other researchers would be worth developing.

Third, infectious disease journals vary widely in scope and types of studies that they publish. Infectious disease research spans a wide range of investigations, from in vitro studies of mice to clinical research. Transparency and reproducibility challenges may differ for different types of research. We used a large sample and diverse journals so as to capture this diversity, but some types of studies may have been underrepresented. The 9 examined journals also publish a lion's share of the articles published by infectious disease specialty journals. However, some very influential infectious disease–related studies are not published in the specialty journals of the field, but in general medical and general science journals (eg, Nature or Science), and many articles on infections may appear in specialty journals outside of those focusing on infections. Transparency indicators may exhibit different patterns in these journals.

Fourth, only PubMed Central Open Access subset publications were included and not the full list of PubMed publications from the 9 journals. There may be specific tendencies in the open access subset that leads to a misrepresentation of proportions of transparency. Open access articles may be more transparent overall; if so, we may have overestimated the rate of transparency in the field. Furthermore, the overall rate of open access status has increased with the COVID-19 pandemic and the funders’ and publishers’ open sharing commitments that came as a response to the pandemic. Prior estimates have shown upward of 97% of COVID-19 articles being open access publications [35–37], outpacing any other field's openness and further skewing the sample.

Fifth, we sampled from 2 recent years, 2019 and 2021, where 2021 is a unique pandemic year and therefore these numbers possibly do not reflect true yearly trends but mostly the COVIDization of science that has been observed across fields and disciplines and its repercussions [5, 7, 38, 39]. However, we did use a comparison focusing only on the non-COVID-19 articles in these 2 years.

Finally, some infectious disease studies are purely nonquantitative and coding analyses would make little or no sense. Also, registration is not necessarily feasible or appropriate for all studies. Therefore, one should not expect these transparency indicators to be feasible to attain in 100% of the published infectious disease articles, even under perfect transparency settings. The range of research practices relevant to transparency is wide and some aspects and details go beyond what our assessment could perform. For example, the Transparency and Openness Promotion (TOP) guidance encompasses 8 elements: citation, data transparency, analytic methods (code) transparency, research materials transparency, design and analysis transparency, preregistration of studies, preregistration of analysis plans, and replication [40]. Each of these elements can have different levels of implementation. While in principle more transparency is welcome, not all elements are equally easy, relevant, and useful to achieve. For example, according to TOP, the highest level of replication is registered reports, but these account for <1% of the literature and their extra benefit can be debated [41].

More funders are mandating sharing of data and code, and the Office of Science and Technology Policy recent statement on these issues is relevant (https://www.whitehouse.gov/ostp/news-updates/2022/08/25/ostp-issues-guidance-to-make-federally-funded-research-freely-available-without-delay/). Lack of transparency poses obstacles to reliable and rigorous scientific research. Infectious disease specialty journals would benefit from more transparency.

Supplementary Data

Supplementary materials are available at The Journal of Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Data availability. The code and data generated and analyzed for this study, as well as the protocol, are available at https://doi.org/10.17605/OSF.IO/FYZPX.

Financial support. The Meta-Research Innovation Center at Stanford (METRICS) has been funded by the Laura and John Arnold Foundation.

References

Author notes