Evaluating Models Used in Clinical Decision Support for Empiric Antibiotic Therapy

Antibiograms, which report antibiotic susceptibility percentages by organism, are most helpful for selecting therapy when the organism is known, or a particular organism or two is thought to be most likely. When starting empiric therapy, however, the provider is often uncertain which of many organisms might be the causative agent and relies instead on institutional guidelines, but this approach ignores substantial information available about the patient. We have built numerous models that incorporate the patient-specific information into clinical decision support (CDS) for empiric therapy selection. Presented here is a method for evaluating a model’s performance.

The method compares the antibiotic proposed by a model that generates empiric therapy recommendations, and the antibiotic actually ordered by the provider, against the susceptibility results for the organism that subsequently grew from culture. As an example of the method, we present data from January to August 2016 on 302 hospitalized patients who on the same day had a urinary culture collected, which subsequently became positive, and a new order for levofloxacin (levo) or nitrofurantoin (nitro).

Of the 302 subsequent isolates, 262 (87%) were susceptible to levo and 256 (85%) were susceptible to nitro. Of the 183 patients who received levo, 157 had a susceptible isolate, and of 119 who received nitro, 103 had a susceptible isolate; thus, overall 260 (86%) of the 302 patients received a drug to which their isolate was susceptible. The empiric therapy model under evaluation predicted nitro as the better drug for 227 patients, of whose isolates 219 were susceptible to nitro, and levo as the better drug for 75 patients, of whose isolates 69 were susceptible to levo. If providers had ordered according to the model, 288 (95%) of patients, 22 more patients, would have received an antibiotic to which their isolate was susceptible.

This evaluation method, which compares what the model suggests with what the provider ordered (that is, with current practice), quantifies the improvement in antibiotic selection that use of the model could make—and the number of patients who might benefit.

S. Overly, Teqqa, LLC: Employee, Salary. J. Mehta, Teqqa, LLC: Employee, Salary. S. Hayes, Teqqa, LLC: Employee, Salary. D. Peterson, Teqqa, LLC: Employee, Salary.