Which Topics Drive Dissemination? Alternative Bibliometrics Analysis of the Highest-Ranking Articles in 3 Infectious Diseases Journals Before COVID-19

Abstract

Traditional assessment of research publication impact has relied on conventional bibliometrics, including metrics like journal impact factors, article citations, and individual H-index [1]. These conventional bibliometrics have been used for making career-defining decisions on promotion/tenure and as the key basis for measuring the return on investment put into research. These were reasonable tools when the primary mode of dissemination was physical journals delivered to individuals. With the evolution of digital platforms and diverse modes of scientific dissemination, some limitations of these traditional print-based metrics have become more apparent. Scientific research is now disseminated across additional platforms like electronic journal articles, social media, blogs, popular electronic press, and other digital media [2]. Therefore, alternative bibliometrics (altmetrics) have been developed to account for these newer types of dissemination to help describe the reach and impact of research publications earlier than traditional metrics [1, 3, 4].

At a specialty level, many infectious diseases (ID) journals are well read, but it is not known which topics are most likely to be read or disseminated. However, since better dissemination leads to more citations [5], this information is useful for both the authors and the journal editors. Despite the growing utilization of altmetrics, the field of ID has seen limited exploration in this regard [6–12]. The influx of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2)–related publications during the coronavirus disease 2019 (COVID-19) pandemic underscored the impact of research dissemination on both conventional and alternative bibliometrics [13–15]. This pre-COVID-19 assessment aims to assess the distribution of Altmetric Attention Scores of general ID topics within 3 ID journals prior to the pandemic. By examining the altmetrics of ID publications, this study contributes to an improved understanding of research impact in the evolving landscape of scholarly communication, as we transition from the pandemic-dominated editorial space to a scholarly environment where COVID-19 captures decreasing amounts of attention.

METHODS

This is a cross-sectional study of the top 200 articles in 3 general ID journals via Altmetric (www.altmetric.com). One of the authors (W. N. N.) was granted access to the Altmetric Database through a research question request. The targeted general ID journals for this study included Lancet Infectious Diseases (Lancet ID), Clinical Infectious Diseases (CID), and Open Forum Infectious Diseases (OFID). These 3 journals were chosen based on feasibility of manual review of articles and primary content of generalized clinical ID publications. Bibliometric information for the top 200 publications with highest altmetric scores across the 3 journals was downloaded from the Altmetric Database on 7 January 2019.

Sixteen infection categories were predefined by ID experts (J. R. M. and K. A. C.) based on expert opinion of categories as no formal categorization was available to replicate: skin/soft tissue and bone/joint, gastrointestinal/genitourinary, respiratory, central nervous system, other bacterial infections (including bacteremia, undifferentiated sepsis, excluding infections belonging to previously named organ systems/syndromes), HIV/sexually transmitted infections/viral hepatitis, mycobacterial diseases, non-HIV immunocompromised hosts, drug-resistant organisms, fungal, antimicrobial stewardship and infection control, vaccines and pediatric infectious diseases, travel/tropical medicine and vector-borne/parasitic infections, microbiology diagnostics, outbreaks, and a general “other” category that included articles not strictly fitting any of the previous categories, including animal studies, ID compensation, education, or workforce studies. Society guidelines and podcast/interview transcripts were also included in this general “other” category. Articles could be assigned to >1 category (eg, a study discussing tuberculosis in people with HIV would be categorized as “HIV” and “mycobacterial diseases”).

Each article title and abstract (where available) were initially reviewed and categorized by 2 independent reviewers (S. G. and R. C. M.). Where category agreement was not achieved on initial review of an article, a second adjudication was performed by additional reviewers (J. R. M. and K. A. C.) with discussion to achieve consensus. The analysis was descriptive, including number and distribution of articles, as well as summative, median, maximum, and minimum altmetric scores within infection categories and each journal. The University of Nebraska Medical Center Institutional Review Board (IRB) did not require IRB approval as this project was not classified as human subjects research.

RESULTS

Table 1 shows the conventional and alternative bibliometrics of the 3 journals evaluated. Lancet ID had the highest impact factor at the time of the study (25.1), followed by CID (9.1) and OFID (3.4). The summative and median altmetric scores of the top 200 articles in each journal were highest for Lancet ID (56 505 and 182.5, respectively), followed by CID (43 945 and 145) and OFID (10 555 and 20.5). Figure 1 shows the distribution of articles in each topic for the 3 journals. The most common topics by number of articles published in each journal (Table 1) were travel/tropical medicine/vector-borne/parasite infections in Lancet ID (63/200), vaccines and pediatric infectious diseases in CID (43/200), and antimicrobial stewardship/infection prevention and control in OFID (60/200).

Figure 1.

Category distribution of top 200 articles by Altmetric Attention Scores for Lancet Infectious Diseases, Clinical Infectious Diseases, and Open Forum Infectious Diseases. Abbreviations: HIV, human immunodeficiency virus; ID, infectious diseases.

Open in new tabDownload slide

Table 2 describes the altmetric scores for each topic, combined across the 3 journals. The top 3 topics (excluding “other”) across all journals by number of articles published were antimicrobial stewardship/infection prevention and control (128), travel/tropical medicine/vector-borne/parasite infections (108), and vaccines and pediatric infectious diseases (98). The top 3 topics by summative altmetric scores (excluding “other”) were travel/tropical medicine/vector borne/parasite infections (24 770), vaccines and pediatric infectious diseases (21 685), and antimicrobial stewardship/infection prevention and control (20 763). The top 3 topics by median altmetric scores for each article were outbreaks (264.5), travel/tropical medicine/vector-borne/parasite infections (157), and vaccines and pediatric infectious diseases (138).

DISCUSSION

This study provides an in-depth description of the characteristics of 3 general ID journal articles’ altmetric rankings in the pre-COVID-19 era. Lancet ID had higher overall median attention scores, which may be expected given its higher impact factor, larger circulation, and longer history of strategic publication dissemination on social media. The most popular topic by number of articles in this journal was also travel/tropical medicine/vector-borne/parasite infections, which may be expected given the global readership of this journal.

There were similarities in the top 3 topics by number of articles published and summative and median altmetric scores. This indicates that the topics themselves were of particular interest to multiple readers and were disseminated extensively. There appeared to be a more even distribution of topics among CID and OFID, which may suggest that while the reach may be smaller, they may appeal to an audience with more varied interests. Of note, the Gold Open Access journal OFID is the youngest journal of the 3 assessed journals (<10 years since launch at time of publication) and seems to have established itself as a home for antimicrobial stewardship/infection prevention and control articles. Altmetrics score trends and dissemination effectiveness in distinct categories may also vary based on the journal's scope and reach to a generalized versus specialized audience. This study was unable to assess this, but it would be interesting to explore if, for example, dissemination of antimicrobial stewardship/infection prevention and control category articles in general journals like CID and OFID compares with altmetrics in specialized journals such as Infection Control & Hospital Epidemiology or the recently launched Antimicrobial Stewardship & Healthcare Epidemiology.

It is interesting to note the top 3 topics by median altmetric scores as outbreaks, travel/tropical medicine/vector-borne/parasite infections, and vaccines and pediatric infectious diseases, which suggests that the individual articles in these topics may have been widely disseminated. Each of these topics may have been highly published and accessed during the COVID-19 pandemic, and more attention has been paid to the growing problem of antimicrobial resistance, so it would not be surprising to see these topic trends continue to persist beyond the effects of the pandemic. Of note, the “other” category numerically had higher altmetric scores than the other categories; however, this category is a combination of multiple articles not strictly fitting any of the previous categories. We did not disaggregate the “other” category and therefore were unable to explore specifically what subtopic drove the scores in this category.

Medical journals can use social media to amplify new and interesting publications, driving activity to their websites and subsequently increasing their own accessibility [9]. The Lancet ID (@TheLancetInfDis) X (formerly Twitter) account launched in March 2014 and at the time of this publication had >66 000 followers. In February 2022, CID (@CIDJournal) and OFID (@OFIDJournal) launched their X (formerly Twitter) accounts and at the time of this publication, CID had >15 900 and OFID had >10 400 followers, respectively. Intentional strategies like social media–based chats rely on individual users engaging and discussing published articles; these can increase the altmetric scores of individual articles [16] and contribute to expanded medical education opportunities, particularly in the field of ID [17]. Multiple factors contribute to altmetrics including mentions on social media, blogs, popular electronic press, and other digital media. Using altmetrics to assess topic interest may be desirable for editorial boards as they curate content and may inform targeted dissemination campaigns designed to drive activity to important, but less-reviewed content [9, 16].

As previously demonstrated in studies within other specialties [6–8, 10, 11], and regarding COVID-19 [13–15], altmetrics may not provide a complete picture of the research quality as the correlation between altmetric scores and subsequent citations has been positive in most studies but weak to absent in some cases [4]. The latter holds true for cases where researchers have used sensationalism as a tool to disseminate articles while compromising on authenticity of results [15]. Therefore, altmetrics should be used as a complement to, rather than a replacement for, traditional bibliometrics and other methods of assessing impact and dissemination.

To our knowledge, this is the first study to characterize publications in ID journals by both topic and alternative metrics. Limitations to this study include that alternative metrics represent dissemination, but not quality of research. Further, dissemination may be driven by numerous factors beyond the pure interest in a topic, such as a controversial topic in the public sphere (eg, vaccines). Therefore, alternative metrics may sometimes represent dissemination beyond those with expertise or true academic interest in the field, journal, or topic. Additionally, since the time of data collection, the included journals may have developed different social media audience sizes or strategies that could impact generalizability of these results. While CID and OFID are 2 of 3 flagship journals of the Infectious Diseases Society of America (IDSA), the third, The Journal of Infectious Diseases (JID), was not included due to the focus on general clinical ID journals. All 3 journals have undergone major editorial changes since data were collected, and shifts in leadership priorities and dissemination strategies may influence these metrics in the future. It may be beneficial to evaluate altmetrics trends for these 3 journals as a unit to support the overall IDSA journals strategy; however, this was beyond the scope of this study. Finally, we are unable to report which specific platforms drove the altmetrics factors in this study or if dissemination of specific influencers independently impacted altmetric scores. Further research is needed to deepen our understanding of how best to utilize alternative metrics as academicians publishing in peer-reviewed journals and for journals to determine how to best leverage alternative metrics and digital sharing strategies for greater readership.

CONCLUSIONS

The methods in which publishers and researchers assess impact and dissemination of research is evolving with increased utilization of digital platforms. Although conventional bibliometrics continue to have a role in understanding this, alternative metrics are providing a more real-time descriptive view of the breadth of dissemination of journal articles. Recognizing key topics of interest via alternative metrics may aid in development of dissemination strategies and consideration of topics to be included in journals.

Notes

Acknowledgments. The authors thank Dr Julie K. Silver for her valuable suggestions during the predevelopment stages of this project.

Author contributions. J. R. M., W. N. N., and K. A. C. were involved in the design of the study. S. G. and R. C. M. were involved in collection of data. J. R. M. and K. A. C. were involved in validation of data. All authors were involved in review of the final draft of the manuscript.

References

Author notes