Diagnostic Accuracy of Health Care Administrative Diagnosis Codes to Identify Nontuberculous Mycobacteria Disease: A Systematic Review

Abstract

Background

Health care administrative database research frequently uses standard medical codes to identify diagnoses or procedures. The aim of this review was to establish the diagnostic accuracy of codes used in administrative data research to identify nontuberculous mycobacterial (NTM) disease, including lung disease (NTMLD).

Methods

We searched Ovid Medline, Embase, Scopus, Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials, and ClinicalTrials.gov from inception to April 2019. We included studies assessing the diagnostic accuracy of International Classification of Diseases, 9th edition, Clinical Modification (ICD-9-CM) diagnosis codes to identify NTM disease and NTMLD. Studies were independently assessed by 2 researchers, and the Quality Assessment of Diagnostic Accuracy Studies 2 tool was used to assess bias and quality.

Results

We identified 5549 unique citations. Of the 96 full-text articles reviewed, 7 eligible studies of moderate quality (3730 participants) were included in our review. The diagnostic accuracy of ICD-9-CM diagnosis codes to identify NTM disease varied widely across studies, with positive predictive values ranging from 38.2% to 100% and sensitivity ranging from 21% to 93%. For NTMLD, 4 studies reported diagnostic accuracy, with positive predictive values ranging from 57% to 64.6% and sensitivity ranging from 21% to 26.9%.

Conclusions

Diagnostic accuracy measures of codes used in health care administrative data to identify patients with NTM varied across studies. Overall the positive predictive value of ICD-9-CM diagnosis codes alone is good, but the sensitivity is low; this method is likely to underestimate case numbers, reflecting the current limitations of coding systems to capture NTM diagnoses.

Nosology, the branch of medical science that deals with the classification of diseases, uses codes to standardize reporting of diseases. The International Classification of Diseases (ICD) codes endorsed by the World Health Organization are the most commonly used worldwide. The intention of the codes is to reflect the patient’s true underlying disease, but their accuracy is highly variable, as the code assignment process is subject to potential sources of error [1–4]. These codes serve administrative purposes and are routinely collected and abstracted for billing and to file medical records or registries; they were not designed specifically for research. In recent years, health care administrative database research has been steadily increasing due to its relatively low cost. Administrative data are particularly useful to study rare disorders with a low incidence. Administrative data research relies on coded diagnoses and/or procedures [5, 6]. However, relying on single codes or a combination to identify and accurately reflect a diagnosis can potentially lead to misclassification and erroneous conclusions. This has been recognized by researchers and has led to the validation of several codes using medical record review as the gold standard [7].

Administrative data research has been particularly useful for nontuberculous mycobacterial (NTM) disease due to its low incidence and lack of mandatory public reporting. Although the true prevalence of NTM disease is unknown, various studies have estimated changes in the prevalence of NTM lung disease (NTMLD), which accounts for 90% of NTM cases [8, 9]. US population-based surveys in the 1980s estimated the prevalence of NTMLD at 1–2 cases per 100 000 persons [8]. Using data from 4 health care delivery systems, the average annual period prevalence for 2004–2006 was 5.5 cases per 100 000 persons [10], and most recently a study using a national managed care claims database estimated an annual incidence of NTMLD in 2015 of 4.7 per 100 000 person-years and an annual prevalence that year of 11.7 per 100 000 persons [11]. The increase in prevalence reported in these studies ranged from 7.5% to 8.2% per year [11, 12]. Because NTM disease is relatively rare and unlike tuberculosis, not subject to mandatory reporting in most states, our current understanding of the epidemiology and geographical distribution of these infections derives mainly from administrative data research [11–14]. With the increasing prevalence of NTMLD, it is foreseeable that administrative data research will continue to address many of the questions that remain unanswered, hence the importance of assessing the accuracy of the codes used to identify NTM disease and particularly NTMLD (Supplementary Table 1) [12, 13]. We aimed to identify studies that validated specific ICD codes or sets of codes for NTM disease using a reference population.

Research Question

What is the diagnostic accuracy of ICD codes or sets of codes to identify NTM disease?

METHODS

Data Sources and Search Strategy

A medical librarian (L.H.Y.) searched the literature for records including the concepts nontuberculous mycobacteria lung disease (NTMLD), pulmonary nontuberculous mycobacteria (PNTM), data, databases, and coding. The librarian created search strategies using a combination of keywords and standardized index terms in Ovid Medline 1946-, Embase 1947-, Scopus 1823-, Cochrane Database of Systematic Reviews (CDSR), Cochrane Central Register of Controlled Trials (CENTRAL), and ClinicalTrials.gov 1997–4/2019. All search strategies were completed in April 2019, and a total of 9988 results were found. Using Endnote, 4624 duplicate records were identified by the automatic duplicate finder, and another 420 duplicates were removed manually, resulting in a total of 4944 unique citations included in the project library. Fully reproducible search strategies for each database can be found in the Supplementary Data. Results were updated on May 14, 2020, by running searches from database inception to May 14, 2020, then de-duplicating total results against the original results to find 605 new citations, to ensure capturing newly published manuscripts as databases do not always add content linearly. The PROSPERO registry was searched to corroborate that no systematic review was available on this topic.

Study Selection

All titles and abstracts were reviewed independently by 2 researchers (C.M.C. and K.M.) for relevance to the research question in the initial screening. To be included in the analysis, studies had to meet the following inclusion criteria: (1) used health administrative codes to identify patients with NTM disease including NTMLD, (2) had a reference data set to confirm the diagnosis of NTM disease, and (3) reported a measure of diagnostic accuracy (sensitivity, specificity, positive predictive value [PPV], and/or negative predictive value [NPV]) of the code(s) used to identify patients with NTM disease or had the information available to calculate at least 1 of them. Manuscripts were excluded if the full-length article was not published in either English or Spanish and if the code(s) used to identify NTM was combined with the reference standard (incorporation bias). Inter-rater reliability analysis between the 2 researchers (C.M.C. and K.M.) was performed with crosstabs and calculation of the kappa statistic using SPSS Software, version 24 (IBM, Armonk, NY, USA).

The reference standard was defined as the data set in which NTM cases were confirmed using the diagnostic criteria outlined by the American Thoracic Society and Infectious Diseases Society of America (ATS/IDSA) clinical guidelines for NTM disease, a combination of clinical (ie, symptoms, radiological findings, and exclusion of other diagnoses) and microbiological criteria (ie, positive cultures from ≥3 separate sputum specimens, single lower respiratory tract specimen or culture from sterile site for extrapulmonary disease) (Supplementary Table 2) [15].

Acceptable reference standards included medical record review, mandatory reported NTM surveillance registries, or a microbiology report data set. The test data set patients were identified as having NTM disease using health care administrative diagnostic codes.

Data Extraction and Quality and Bias Assessment

From the full-text articles selected, the following data were abstracted by 1 author (C.M.C.) using the standardized form: citation (first author, publication year, country, and sample size), health administrative data set (years, population characteristics, and type of data), reference data set, ICD code(s) used, and measurements of diagnostic accuracy (Table 1).

Data were extracted from all studies that used 1 or more diagnostic codes to identify NTM in a data set. The primary outcome measure was the diagnostic accuracy of the code(s) used to identify patients with NTM disease. As a secondary outcome, we examined the diagnostic accuracy of the microbiologic case definitions for NTMLD. The case definition for NTMLD included microbiologic, radiographic, and clinical criteria. Most studies only assessed the microbiologic criteria, which are the most important and generally the most feasible to assess. Due to the heterogeneity of the data sets, study populations, and accuracy estimates, we did not perform a meta-analysis.

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses of Diagnostic Test Accuracy Studies (PRISMA-DTA) guidelines and used the Quality Assessment of Diagnostic Accuracy Studies 2 (QUADAS-2) checklist to assess the risk of bias for each individual study (Supplementary Table 3) [16, 17]. The quality and risk of bias of the included studies were assessed by 3 reviewers (C.M.C., T.C.B., and M.O.). We used Endnote X8 (Rutgers University, New Brunswick, NJ, USA) to maintain the reference lists.

RESULTS

Study Characteristics

After removal of duplicate results, we identified 5549 unique citations. For the initial screen, articles were considered relevant to the research question if they referred to human subjects research and used health care databases, claims, or population-based data to identify cases of NTM disease. After review of the titles and abstracts, we identified 96 citations that used claims or billing data to define cases of NTM disease. The inter-rater reliability between C.M.C. and K.M. was excellent (Kappa 0.861), and disagreement on the articles selected was resolved by consensus. After a review of titles and abstracts, 96 were selected for full-text review, of which 89 were excluded and 7 were selected as eligible for this review (Figure 1).

We identified 7 studies for full review that reported at least 1 diagnostic accuracy measure used in an algorithm to identify patients with NTM disease (Table 1). Only Schweitzer’s study clarified in their methods the diagnostic position of the ICD codes used to identify patients [7]. Of the 7 full-text articles, 4 specifically looked at NTMLD, with Plotinsky’s and Ricotta’s studies further limiting the study sample to only patients with culture-confirmed pulmonary Mycobacterium avium complex (MAC) disease [18]. The study population varied significantly among studies, and 4 of the studies used data from US veterans to determine the diagnostic accuracy measures and had a predominantly male and older population not representative of the general population. Plotinsky’s study selected patients managed exclusively by Infectious Diseases or Pulmonary Medicine specialists, and Schneeweiss et al. selected only hospitalized patients.

Three studies used the presence of the ICD-9-CM diagnosis code 031.X in their algorithm to identify patients with NTM disease, but Jones et al. excluded ICD-9-CM diagnosis codes 031.8 and 031.9 (not specific to NTM). Four studies only used ICD-9-CM diagnosis code 031.0 (pulmonary NTM). Of the 4 studies that relied solely on ICD-9-CM diagnosis codes to screen patients for inclusion in their cohort, 3 did full chart review on the whole cohort and Jones et al. used a randomized sample to serve as the reference data set. Winthrop et al. used either positive microbiological cultures or an ICD-9-CM code to identify eligible patients for their cohort, whereas Prevots’s and Ricotta’s studies used culture results to establish theirs. Prevots’s reference data set also lacked clinical data to fully assess ATS/IDSA NTM diagnostic criteria as the gold standard, and instead used only microbiological (≥2 sputum samples positive for an NTM spp. or a single positive culture from a bronchoscopy) and radiological criteria (available in 81% of the cohort) to verify NTM disease.

Primary Outcome: Validity of the Codes Used to Identify NTM Disease

Algorithms and diagnostic accuracy outcomes to identify NTM disease using ICD-9-CM diagnosis codes varied across studies, with sensitivity ranging from 21% to 93% and PPV ranging from 38.2% to 100% (Tables 1 and 2). The sole use of ICD-9-CM code 031.0 by Ricotta et al. also yielded a low sensitivity (21%). Only 2 studies reported diagnostic accuracy for the combination of ICD-9-CM codes plus microbiological data vs the ICD-9-CM code alone. In Winthrop’s VA cohort (n = 71), sensitivity decreased when ≥1 culture positive for NTM spp. was required together with the ICD-9 code 031.X (65% vs 42%). However, Jones et al. found a substantial increase in sensitivity with the use of microbiological data or the ICD-9-CM diagnosis code compared with the ICD-9-CM diagnosis code alone (93% vs 42.9%, respectively), although the increase in the PPV of using both was only modest (54.3% vs 38.2%). However, PPVs of ICD-9-CM codes alone to identify NTM cases were consistently higher in Schneeweiss et al. (70%), Schweitzer et al. (64.6%), and Winthrop et al. (74%–82%). In Winthrop et al., when both microbiological and ICD-9-CM codes were used, the PPV was similar to that of using the ICD-9-CM code alone (77%–90%).

Jones and colleagues reported the specificity and NPV of using ICD-9 codes to identify NTM cases; both were >99.9% [14]. Only Winthrop’s study looked at the diagnostic accuracy of incorporating prescriptions of drugs used to treat NTM infections in their algorithm. The combination of an ICD-9-CM code plus ≥30 days of a macrolide prescription had the highest PPV (100%) to identify NTM cases. Also, 1 of the cohorts of patients included in Winthrop’s study included only patients with rheumatoid arthritis, limiting the generalizability of their results.

Secondary Outcome: Validity of the Codes Used to Identify NTMLD

Four studies provided diagnostic accuracy measures for using the ICD-9-CM code 031.0 to identify NTMLD, with sensitivity ranging from 21% to 26.9% and PPV ranging from 57% to 64.6% (Table 1). The lowest sensitivity was seen in Ricotta’s study, in which those with disease that was microbiologically confirmed and those with an ICD-9-CM code were compared with the reference standard. This is consistent with previously published data that only 27% of microbiologically confirmed cases are coded for NTMLD, suggesting substantial undercoding of this diagnosis in health care administrative data [19].

Quality Assessment of Studies

The quality of the diagnostic accuracy studies included was assessed using the revised QUADAS-2 tool across 4 domains: patient selection, index test, reference standard, and flow and timing. The overall risk of bias in patient selection was judged to be low, as all but 1 article included consecutive cases and there were no inappropriate patient exclusions (Table 3). Plotinsky et al. only included patients who had been seen by a Pulmonologist or an Infectious Diseases specialist. In terms of flow and timing, there was a low risk of bias, given the retrospective nature of the studies and that the accuracy of identification using ICD-9-CM codes was compared with the same reference standard, the ATS/IDSA diagnostic criteria for NTM infection. However, 2 studies may be at risk of bias with regard to interpretation of the reference standard because they used 1 or 2 of 3 components of the reference standard instead of the full criteria. Ricotta and colleagues used only the microbiological culture criteria of the ATS/IDSA criteria as their gold standard to define NTM infection, whereas Prevots et al. lacked data on the clinical component in their reference data set, and only 81% of their cohort had radiological information. However, microbiological data have the highest PPV of the 3 criteria.

One of the major risks for bias is the applicability of studies that broadly used the entire range of ICD-9-CM codes 031.X for NTM infection and did not differentiate coding for disseminated infection (ICD-9-CM code 031.2) from the NTMLD code 031.0. The underlying predisposing risk factors and prognosis of disseminated vs NTMLD are quite distinct, with disseminated NTM being an acute and potentially life-threatening condition more frequently encountered in the hospital setting and NTMLD being an indolent, chronic condition often seen in the ambulatory setting. Indeed, coding accuracy tends to be lower in the outpatient setting [20]. Another potential source of bias is the position of the diagnostic code (ie, primary vs secondary diagnosis), which was specified in only 1 of the studies included in the final review. There is evidence that many comorbidities tend to be underreported and that omission rates for coding of secondary diagnoses are higher in the outpatient setting [21]. This is often related to a limited number of codes that can recorded in billing claims.

DISCUSSION

To our knowledge, this is the first systematic review to summarize the published literature on the diagnostic accuracy of ICD-CM codes to identify patients with NTM infections in health care administrative databases. Of the 7 studies that met eligibility criteria, only 3 had among their aims to validate the ICD-9-CM code for NTM infection against a reference data set. The rest of the studies reported at least 1 diagnostic accuracy measure for such codes as a secondary measure. Overall, the sensitivity of ICD codes in NTM infections was low (<50%) in 3 of the 4 studies that provided this measure, with the highest sensitivity (65%) seen in a cohort of US veterans. The highest sensitivity (93%) and specificity (>99.9%) to detect NTM cases, seen in Jones et al. study, were achieved through an algorithm involving either ICD-9-CD codes (031.0, 031.1, or 031.2) or microbiology results obtained via natural language processing to identify NTM species, a novel approach to obtain information about NTM cultures. These findings reflect substantial undercoding of NTM infections in which patients need to meet clinical, microbiological, and often radiological criteria to establish an NTM diagnosis. It is unclear why the Jones study produced the lowest PPV, but exclusion of ICD-9 codes 031.8 and 031.9 could have left out true infections that were erroneously coded through the patient’s trajectory through the health care encounter [1]. Also, low PPV of a single ICD-CM code(s) assigned at discharge has been described in other infections in which microbiological or radiological confirmation to establish a diagnosis is more complicated (eg, cellulitis) [4], something that might impact identification of NTM infections, where there is typically a time delay between sampling collection and culture positivity due to the slow-growing nature of NTM.

The different populations included in each study may limit the generalizability of their results. Jones et al. and Schneeweiss and colleagues exclusively looked at US Veterans Affairs patients, and in Winthrop et al. and Schweitzer and colleagues’ cohorts, US veterans represented more than 50% of the patients. This overrepresentation of the US veteran population does not reflect the general population, as veterans tend to be older, to have a higher prevalence of comorbidities, and are predominately male [22], limiting the external validity of these studies. This is important, given that the prevalence of NTM infection seems to be higher among females in almost all age groups [23]. The low sensitivity of the ICD-9-CM code 031.0 reported by Ricotta et al., who used the Premier Healthcare Database, a billing database that includes privately and publicly insured patients and also uninsured patients, may be more representative of the general US population. None of the studies included Medicare data to validate the screening algorithms to identify patients with NTM infections, an important population given that the prevalence of NTM infections, particularly NTMLD, increases dramatically after age 50 and peaks around age 70 years [23]. However, the population included in all studies was primarily older, supporting the observed findings.

There are several limitations to our review. First, the generalizability of the findings described in most studies is limited to that of the population being studied, which in half of the studies included only US veterans. Second, none of the studies included ICD-10-CM codes for NTM infections. Therefore, a validation cohort study assessing the sensitivity and specificity of the ICD-10-CM code A31.X by investigating a random selection of a large number of inpatient and outpatient medical records should be undertaken in the future to clarify the diagnostic accuracy of that code. Third, only 1 study alluded to the differences in the diagnostic coding field location (primary, secondary, or all), a relevant source of misclassification, as comorbidities tend to be underreported and omission rates for coding of secondary diagnoses tend to be higher in the outpatient setting [20]. Finally, risk of misclassification with regard to the reference standard was high in 2 studies that did not use the full ATS/IDSA diagnostic criteria, primarily due to the lack of clinical or radiological data.

In conclusion, ICD-9-CM codes alone have poor sensitivity for identifying confirmed NTM cases, potentially missing a relevant number of NTM cases and thereby underestimating the prevalence of this disease, but they are commonly used because they are more readily available compared with microbiological data. However, their good PPV is reassuring, as research done on populations identified using ICD-9CM codes will most likely obtain relevant clinical conclusions applicable to patients with NTM infection. The composite use of microbiological data and ICD-9-CM codes may increase sensitivity without meaningfully changing the PPV, an approach that should be favored if culture data are available. Although accuracy of diagnoses based on claims data may have potential bias due to patient misclassification, given the low, albeit increasing prevalence of NTM infections, use of health care administrative data sets remains appealing and advantageous to estimate epidemiological trends and examine mortality outcomes. Future studies are needed to address the accuracy of these codes in different health care settings (inpatient vs outpatient), code positions (primary vs secondary), administrative databases (eg, Medicaid), and ICD-10-CM systems.

Supplementary Data

Supplementary materials are available at Open Forum Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Acknowledgments

The authors acknowledge the Author Support team at Becker’s library for their publishing assistance.

Financial support. This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sector.

Potential conflicts of interest. M. A. O. reports grants and personal fees from Pfizer, grants from Merck, outside the submitted work. All other authors: no reported conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.

Author contributions. All authors had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Mejia-Chew, Olsen. Acquisition, analysis, or interpretation of data: Mejia-Chew, Yaeger, Montes, Bailey, Olsen. Drafting of the manuscript: Mejia-Chew, Olsen.

Patient consent. Our study does not include factors necessitating patient consent.

References