Residual Confounding or Lack of Effect?

To the Editor—We read with interest the report authored by Dr. Chu and colleagues investigating differences in outcomes of patients included in the Premier database in 2012 in relation to procalcitonin (PCT) tests ordered by intensive care unit (ICU) physicians. The Premier database is a large, geographically representative database of in-hospital stays in the United States and thus gives insight into real life practices. The results of this report are thus important as they show that PCT—as of 2012—was not used as intended by the recently updated US Food and Drug Administration (FDA) indication as a marker to monitor sepsis patients’ response to antibiotics, with the ultimate aim of reducing antibiotic course duration [1]. The clinical trials that found positive effects of PCT testing on antibiotic consumption and outcomes all involved repeated PCT measurements in their testing protocols [2, 3]. In February of this year, FDA has cleared “the expanded use of the Vidas Brahms PCT Assay to help healthcare providers determine if antibiotic treatment should be started or stopped in patients with lower respiratory tract infections, such as community-acquired pneumonia, and stopped in patients with sepsis. …” (https://www.fda.gov/NewsEvents/Newsroom/PressAnnouncements/ucm543160.htm). This new indication is important because it enables industry and healthcare workers to promote education about the correct use of this marker in antibiotic stewardship programs, which will hopefully lead to its real-world effectiveness as suggested by the clinical trial evidence.

Having said this, we have some concerns regarding the study design and statistical approach that the authors used in this article. In real practice, physicians may use PCT in the sickest patients when there is uncertainty about the correct diagnosis and about appropriate next steps in their therapeutic management. A typical example would be a patient on a long course of antibiotics with ongoing clinical signs of infection (e.g., fever). A comparison of outcomes between sicker patients receiving PCT testing and healthier patients without PCT testing is thus likely to be subject to strong confounding by indication, which would lead to systematic bias in the comparisons. The authors used a multivariate modeling approach to analyze the data and derive estimates of the association between PCT use and the outcomes of interest, whereas prior authors [4] utilized a propensity score (PS) matching approach to study the subset of patients for whom there was balance in baseline covariates. Whereas PS matching excludes patients falling outside the region of overlap in the distribution of the expected probability of exposure (i.e., PCT), and thus excludes patients who—according to the measured baseline factors—are not comparable, the approach used by Chu et al. uses all patients in the analysis dataset. This approach has the appeal of using all patients, but succumbs to the problem of needing to average the estimate of association between exposure and outcome across all levels of the included covariates, even when some levels of the covariates (patients with a certain comorbid condition, say) lack a member from either the PCT or comparator cohorts. This problem is made obvious when examined from the lens of PS deciles, in which estimates of association tend to be drastically different in the bottom and top strata of PS deciles than those in the middle, and are in fact often in the opposite directions [5–7]. Therefore, a PS-matched analysis would have helped to mitigate potential confounding by indication.

Also of concern, a paper authored by Balk and colleagues did a very similar analysis in the same database but came up with opposite conclusions. In their study, they compared outcomes in patients with and without PCT testing using the same Premier database with data from 2011 to 2014 [4]. Use of PCT testing on the first day of ICU admission was associated with significantly lower hospital and ICU length of stay, as well as decreased total, ICU, and pharmacy cost of care. One important difference in the study design was that Chu et al. excluded anyone not receiving an antibiotic, which has the effect of removing subjects from the analysis who had PCT and had zero antibiotic days on therapy (DOT). This decision may have the effect of biasing the DOT upward in the PCT cohort. In real life, physicians may prefer to measure PCT in patients with a prolonged antibiotic course leading to reverse causation, with DOT driving the PCT test rather than the other way around.

Based on these considerations, we feel that the report by Chu has merits in showing potential improvement for current PCT use in the United States, but the putative observations of higher antibiotic exposure and Clostridium difficile risk in the PCT patients require further investigation as these findings may be subject to residual confounding issues.

Note

Potential conflicts of interest. P. S. is supported by the Swiss National Science Foundation (SNSF Professorship, PP00P3_150531) and the Forschungsrat of the Kantonsspital Aarau (1410.000.058 and 1410.000.044). P. S. received research support from BRAHMS/Thermofisher, BioMerieux, Roche, Abbott, Nestle and Novo Nordisk.] P. W. is a full-time employee of Covance, Inc. Both authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.

References