Applying Machine Learning Models Derived From Administrative Claims Data to Predict Medication Nonadherence in Patients Self-Administering Biologic Medications for Inflammatory Bowel Disease

Abstract

Background

Adherence to self-administered biologic therapies is important to induce remission and prevent adverse clinical outcomes in Inflammatory bowel disease (IBD). This study aimed to use administrative claims data and machine learning methods to predict nonadherence in an academic medical center test population.

Methods

A model-training dataset of beneficiaries with IBD and the first unique dispense of a self-administered biologic between June 30, 2016 and June 30, 2019 was extracted from the Commercial Claims and Encounters and Medicare Supplemental Administrative Claims Database. Known correlates of medication nonadherence were identified in the dataset. Nonadherence to biologic therapies was defined as a proportion of days covered ratio <80% at 1 year. A similar dataset was obtained from a tertiary academic medical center's electronic medical record data for use in model testing. A total of 48 machine learning models were trained and assessed utilizing the area under the receiver operating characteristic curve as the primary measure of predictive validity.

Results

The training dataset included 6998 beneficiaries (n = 2680 nonadherent, 38.3%) while the testing dataset included 285 patients (n = 134 nonadherent, 47.0%). When applied to test data, the highest performing models had an area under the receiver operating characteristic curve of 0.55, indicating poor predictive performance. The majority of models trained had low sensitivity and high specificity.

Conclusions

Administrative claims-trained models were unable to predict biologic medication nonadherence in patients with IBD. Future research may benefit from datasets with enriched demographic and clinical data in training predictive models.

Graphical Abstract

Open in new tabDownload slide

Background

Inflammatory bowel disease (IBD), comprised of Crohn disease and ulcerative colitis, affects approximately 6.8 million patients globally, with a prevalence rate of 464.5 cases per 100 000 in the United States.¹ Moderate to severe IBD is often medically managed with biologic therapies, several of which are self-administered in the outpatient setting, including adalimumab, certolizumab, golimumab, ustekinumab, and risankizumab.^2,3 Though biologics are considered cost-effective therapies, particularly when compared to costs associated with poorly controlled IBD, they still contribute significantly to the overall cost of IBD care, averaging $36 051 per patient per year in 2015.⁴

Poor adherence to biological therapy contributes to worse IBD outcomes and higher care-associated costs. Current estimates of nonadherence to biologic therapy range from 17.4% to 45%.^5–8 Several risk factors for biologic nonadherence have been identified, including younger age, female gender, tobacco use, payor type, Crohn's disease diagnosis, and comorbid diagnoses such as anxiety and depression.^5–12 Patient medication utilization patterns are also associated with biologic nonadherence, such as nonadherence to prior IBD therapies, concurrent dual therapy with a biologic and an immunomodulator, and chronic opioid use.^5,9,12–14

The development of predictive models to identify patients at high risk of nonadherence and proactively address barriers to adherence is appealing, as risk factors are often present at baseline. Machine learning models have previously been developed to predict medication nonadherence in other disease states, as well as non-biologic immunomodulator therapy in IBD.^15–18 Furthermore, many of the risk factors for nonadherence can be identified from the data contained in administrative claims databases, providing a sufficiently large dataset for training such a model.^18,19 However, the usefulness of this model hinges on the predictive validity when applied in a clinical environment. This study will examine the utility of several competing machine learning models trained on administrative claims data to predict IBD biologic nonadherence in a tertiary academic medical center patient population. Investigators hypothesized training predictive models on large-scale administrative claims datasets including variables previously associated with nonadherence would produce models to accurately identify nonadherence in individual patient populations.

Materials and Methods

Data Sources and Study Design

Model-training datasets were derived from the Meritive MarketScan Commercial Claims and Encounters and Medicare Supplemental Administrative Claims Database, hereafter referred to as MarketScan. This database contains demographics, records of inpatient and outpatient healthcare encounters, and prescription medication transactions for more than 273 million unique beneficiaries covered by employer health plans.²⁰ This dataset includes International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) and Healthcare Common Procedure Coding System/Current Procedure Terminology (HCPCS/CPT) codes associated with encounter diagnoses and procedures performed, as well as the Generic Product Identifier (GPI) associated with dispensed medications.^21–23

MarketScan beneficiaries with a first unique medication dispense between June 30, 2016 and June 30, 2019 of any self-administered biologic medication with a current FDA-approved indication for IBD (adalimumab, certolizumab, golimumab, and ustekinumab) were considered for inclusion. The first dispense date was considered the index date for study purposes. Otherwise eligible MarketScan beneficiaries were excluded if: (1) they did not have at least 2 outpatient encounter claims or 1 inpatient encounter claim associated with an ICD-10-CM code in a 180-day preindex period indicating IBD (K50*, K51*); (2) they were not continuously enrolled for 180 days prior to and 365 days post-index date, with prescription benefits and no gap in enrollment greater than 30 days; or (3) were <18 years of age at the index date. From this cohort, beneficiary demographics and all encounter-level data (including dates of service and associated ICD-10-CM and HCPCS/CPT) were extracted from the preindex period, while all medication dispensing records were obtained from the pre- and postindex periods.

In order to provide a dataset to test model validity outside of MarketScan beneficiaries, a separate cohort was identified from patients treated at a tertiary academic medical center that serves a rural and disproportionate share population. Eligible patient records were identified as having a first unique biologic dispense date between June 30, 2016 and June 30, 2022. Similar to the training cohort, patients were excluded if (1) they were <18 years of age at index date, or (2) had no diagnosis of IBD during the preindex period. While additional relevant clinical and demographic data was available in electronic medical records, data extraction was performed to mirror only the same data elements available from the MarketScan beneficiary records. Only medication dispensing records from pharmacies affiliated with the health system were available for analysis, in contrast to the training dataset which contained all dispenses where insurance was billed across any location. This study was reviewed and approved by the Medical Institutional Review Board of the academic medical center.

Data Preprocessing

After extraction, raw data was used to generate 2 dataset types for model training and assessment. In the first dataset type (investigator-only), only investigator-selected features of interest were included. Features were identified from a review of the relevant literature and clinical experience of the investigators. Demographic features included index IBD diagnosis, sex, age, Medicare payor, and index biologic. Encounter-level features included preindex diagnoses of anxiety/depression or smoking, number of inpatient and outpatient encounters within the 180-day preindex period, and the Charlson Comorbidity Index (CCI) assessed at the index date.²⁴ CCI was calculated through the use of ICD-10-CM codes via previously published methods.²⁵ Medication-related features included prior IBD therapies utilized in the preindex period, identified nonadherence to prior IBD therapies, chronic opioid use, dual therapy with an immunomodulator in addition to the index biologic, number of unique medications dispensed, and prednisone milligram equivalents of corticosteroids dispensed. Chronic opioid use was defined as greater than or equal to 90-day supply of outpatient opioid dispenses in the preindex period without any 30-day gaps in supply.¹⁴ All features were binary indicators, with the exception of age (categorical), number of inpatient and outpatient encounters, number of unique medications dispensed, and prednisone milligram equivalents (continuous). In the event of missing or incomplete data, the binary indicator was considered negative or zero was substituted for continuous data. Full study definitions of administrative claims codes used to generate investigator-selected features are available in Supplementary Table 1.

In the second dataset type the same investigator-selected features were generated, but additional binary indicators were created utilizing all ICD-10-CM codes and GPI codes from preindex encounters and medication dispenses. Binary indicator features were created for each category of Clinical Classification Software Refined (CCSR), a tool which groups ICD-10-CM codes into clinically relevant categories.²⁶ In addition, binary features for the dispense of a particular medication group (GPI 2-digit) in the preindex period were also created. These additional features were intended to aid in identification of any associations with medication nonadherence not previously identified in the literature. This dataset is henceforward referred to as the investigator + CCSR + GPI2 dataset. In both dataset types, features were standardized to a mean of zero and a standard deviation of one.

Data Labeling

Training and testing datasets were labeled with a binary feature indicating nonadherence derived from a proportion of days covered (PDC) calculation. PDC is an indirect measure of outpatient medication adherence that assesses the degree of adherence in a ratio from 0 to 1 based upon the intervals between medication dispenses and the supply dispensed in a given time period. PDC for this study was calculated as follows:

Where days covered are the number of days in which a subject would have supply under the assumption that they are taking the medication as prescribed and only receiving medication from observed dispenses.^27,28 Dispensed supply of any biologic qualifying for inclusion in this study was considered in the numerator. PDC values were dichotomized to create the nonadherence feature, with a PDC < 0.8 corresponding to nonadherence (a threshold commonly used in prior literature).²⁹

Model Training

The approach to model training is outlined in Figure 1. Eight machine learning classification algorithms (labeled A-H) were used to develop six candidate models (#1-6) each for a total of 48 models (A1—H6). Classifier algorithms chosen included logistic regression, ridge regression, lasso regression, k-nearest neighbors, linear support vector machines, decision trees, gradient boosting decision trees, and neural networks. All classifier algorithms used were obtained from the publicly available Python package scikit-learn, version 1.3.0.³⁰ These algorithms were chosen based on availability within the scikit-learn package, as well as to investigate several competing approaches with distinct advantages in handling the study datasets. In specific, logistic regression was selected as a base model for comparison to other approaches. Given the number of considered features, algorithms with built-in feature selection (ridge, lasso regression) or the ability to capture complex decision boundaries in high-dimensional space (k-nearest neighbors, linear support vector machines) were considered. Additionally, investigators suspected non-linear relationships between features for which decision trees and neural network classifiers may be more suited.

For each classifier algorithm, three models were trained using the investigator-only dataset (models 1–3). In model 1, no feature selection or transformation beyond zero standardization was performed. In model 2, a random forest algorithm was first applied to the dataset to estimate feature importance and only features with importance greater than the mean were included in model training. This preprocessing step was conducted to reduce potential errors introduced by possibly irrelevant features. In model 3, data was transformed the data with principal components analysis prior to model training. This step was taken to reduce dimensionality while preserving variance, in hopes of reducing overfitting and enhancing the generalization of the model. Models 4, 5, and 6 were trained using the same methods as 1, 2, and 3, respectively, but used the investigator + CCSR + GPI2 dataset to incorporate additional and potentially relevant unknown features.

All models were trained using ten-fold cross-validation on the training dataset to obtain the optimal model for each algorithm. The optimal model was defined as having the highest F_β score on the training data, where:

A β = 2 value was chosen for this study in order to minimize false negatives in the model. This was chosen given an assumption that the preference would be to overtreat false positive risks for nonadherence rather than undertreat false negatives. For models in which there were multiple hyperparameters or multiple valid settings for a single hyperparameter, a grid search of all combinations of hyperparameters specified was conducted to obtain the combination that maximized F_β score (Supplementary Table 2).

Model Performance Evaluation and Statistical Analysis

Descriptive statistics were reported for all variables, including the number and percentage of beneficiaries with a given condition for categorical variables, as well as mean, median, standard deviation, and interquartile range for continuous variables, as observed in training and test datasets. To examine relationships between individual features and biological nonadherence at baseline in each dataset, 2 logistic regression models including all investigator-only variables were constructed using training and test datasets. Odds ratios from each of these logistic regression models were reported.

Model predictive validity was assessed via several metrics, including accuracy, area under the receiver operating characteristic curve (AUC), Brier score, F1 & F_β score, negative predictive value, precision, sensitivity, and specificity.³¹ AUC was used as the primary metric of model predictive validity, and used to select the best model within a classifier algorithm for comparison against other classifier algorithms. Metrics were obtained for model performance on both training and testing datasets; however, only metrics derived from testing datasets were utilized in assessing model predictive validity.

Results

Description of Training and Test Datasets

After the application of inclusion and exclusion criteria (Figure 2), 6,998 eligible beneficiaries were included in the training dataset, and 285 patients were included in the test dataset. Rates of nonadherence were higher in the test dataset (n = 134, 47.02%) as compared to the training dataset (n = 2680, 38.3%; Table 1). Observed distributions of the PDC variable were left-skewed (Supplementary Figure 1). In the training dataset, factors associated with significantly lower odds of nonadherence included age groups of 45–54 (odds ratio [OR] 0.863; 95% confidence interval [95%CI] 0.756, 0.986) and 55–64 (OR 0.712; 95% CI: 0.599, 0.846) referent to ≤ 44, diagnosis of Crohn’s disease (OR 0.685; 95% CI: 0.586, 0.801), and prior use of mercaptopurines (OR 0.756; 95% CI: 0.62, 0.92) or azathioprine (OR 0.763; 95% CI: 0.653, 0.891). Features associated with higher odds of nonadherence included female sex (OR 1.131; 95% CI: 1.022, 1.252), index biologic of certolizumab (OR 2.288; 95% CI: 1.612, 3.248) or ustekinumab (OR 1.484; 95% CI: 1.261, 1.746) compared to index adalimumab, diagnosis of tobacco use (OR 1.262; 95% CI: 1.019, 1.563), CCI (OR 1.061; 95% CI: 1.001, 1.125), preindex vedolizumab administration (OR 1.324; 95% CI: 1.011, 1.734), or any inpatient admission (OR 1.149; 95% CI: 1.014, 1.302).

In the test dataset, a single variable was associated with significantly lower odds of nonadherence including Medicare coverage (OR 0.419; 95% CI: 0.203, 0.865). Conversely, several factors were associated with greater odds of nonadherence, including female sex (OR 1.885; 95% CI: 1.116, 3.183), age group 45–54 (OR 2.345; 95% CI: 1.097, 5.017) referent to ≤44, and index biologic of certolizumab (OR 4.827; 95% CI: 1.195, 19.488).

Model Performance

Receiver operating characteristic curves assessing classification performance for all models on training and test datasets are available in Supplementary Figures 2 and 3. Based upon AUC when applied to testing data, the highest-performing models from each algorithm type were selected for comparison (models A3, B3, C3, D6, E3, F3, G1, and H6). Among the selected models, F3 and G1 tied for the highest AUC at 0.55 each, however, the difference from other compared models was minimal (Figure 3).

Model performance metrics on training and test datasets for the selected models is displayed in Table 2. On the training set, high accuracy was observed for models D6 (71.42%) and H6 (99.94%); however, these models had similar accuracy to other candidate models when applied to the test dataset. Confusion matrices for all selected models are provided for visual assessment of model accuracy (Figure 4).

In general, models trended towards low sensitivity and high specificity on the test dataset, with the exception of model G1 (gradient-boosted decision tree; sensitivity = 0.71; specificity = 0.36). Model G1 also featured the highest F1, F_β, and negative predictive value values as compared to other models (0.58, 0.65, and 0.59, respectively). This may be attributable due to the nature of the learning algorithm, which sequentially builds a series of decision trees that improve upon prior errors. This can result in overfitting to the minority/positive case (eg, nonadherent) and result in an increase in true and false positives. In addition to the performance metrics for selected models, assessment of metrics for all candidate models on training and test datasets are provided in Supplementary Tables 3 and 4.

Discussion

The goal of this study was to train machine learning models using administrative claims data to accurately predict nonadherence to self-administered biologic therapies in patients with IBD at an academic health system. While numerous associations with nonadherence to self-administered biologic medications were observed within the initial analysis, contrary to our main hypothesis, machine learning models trained on data available in administrative claims databases failed to reliably predict nonadherence in a test dataset derived from an academic medical center’s patient population. Examining the primary metric of model predictive value (AUC) for this analysis, values of 0.5 to a model that performs in accordance with chance, with values between 0.7 and 1 considered a model with moderate to perfect accuracy.³² The highest-performing models generated by this study had an AUC of 0.55 when applied to the test dataset, suggesting limited usefulness in the prediction of nonadherence. While more acceptable AUC values were observed on the training data for some candidate models, they did not translate to higher performance on the testing dataset, suggesting that those models overfit training data and did not have sufficient bias to generalize to new data.³³

There are several possible explanations for why model predictive performance on unseen data was low. Medication nonadherence is a multifactorial issue, and administrative claims data does not contain data on several known correlates with self-administered biologic nonadherence. Datasets with extended demographics and information on patient-specific social determinants of health, including race/ethnicity, income, and education level might provide additional predictive value to train more accurate models.^5,34–37 In addition, patient health literacy and disease-related knowledge are known to have a significant correlation with medication adherence, although this factor is much more difficult to accurately assess and collect at a large scale.^13,16,38,39 Furthermore, no clinical context is available in administrative claims data, such as provider assessments, imaging and procedure studies, laboratory values, and others that have been used in successfully training models to predict adherence in IBD previously.¹⁶ Other studies using machine learning methodology trained on MarketScan data to predict adherence had similar difficulties identifying nonadherence.¹⁸ Future investigations should consider model training in enriched data sources, including electronic medical records or administrative claims data sources with additional socioeconomic and demographic variables. Additionally, assessments of health literacy or IBD-specific knowledge or beliefs may be necessary for predictive models to accurately classify the risk of nonadherence, although the collection of such data is not widespread.

Upon closer examination of the additional metrics of model fit, in general, models trended towards low sensitivity and high specificity (ie, a tendency towards false positives rather than false negatives). Several of the candidate models in series F and G (decision trees and gradient-boosted decision trees) eschewed this trend, most notably candidate model G1 with higher sensitivity and low specificity. As we ultimately aim to identify patients at risk for nonadherence to biological medications, a model with higher sensitivity is preferred, as potential interventions (additional education/behavioral interventions) are likely to be inexpensive and the risk of positive misidentification is unlikely to pose a significant risk to the patient.^39,40

While model performance in predicting nonadherence in this study was low, potential insights for future studies can be gleaned. In contrast to prior literature suggesting increased risk, a diagnosis of Crohn’s disease was observed to be associated with a significantly lower risk of nonadherence.¹² This suggests that the relationship between IBD diagnosis and biological nonadherence may involve relationships with other relevant correlates and should be more thoroughly explored in additional literature. Additionally, the highest-performing models in this analysis were trained from the investigator-only dataset, and utilized primary components analysis (PCA) dimensionality reduction as part of data preprocessing. The greater performance of the investigator-specified feature set suggests that the additional CCSR and GPI columns added no net information gain at best, and at worst, generated additional noise in the models. Omission of these columns, or a more targeted approach to selection in the future is likely appropriate. Furthermore, use of PCA to reduce training dataset noise and improve future models appears appropriate. In consideration of the numerous causes of nonadherence, dimensionality reduction with PCA or another mechanism is likely necessary.

Strengths of this study include the use of a large training dataset, which typically increases the likelihood of creating more generalizable models. In addition, utilizing multiple machine learning algorithms and feature engineering approaches to detect nonadherence ensures that conclusions are not based upon the failure of a single approach.

Weaknesses of this study include the inability to train models on extended demographic and clinical data due to the limitations of administrative claims data, which may have prevented a suitable model from being constructed. This study was also unable to include an evaluation of dosing interval as a possible predictor of nonadherence.

Additionally, it is possible the use of 2 distinct data sources contributed in part to poor model performance on test datasets. The training and test datasets substantially differed in the distribution of demographic and clinical data. As well, missing external dispense data may have limited identification of medication-related features in the testing dataset. While potentially representative of available data in many practice locations, this limits the predictive potential of a model trained with more comprehensive dispensing histories. Training models on a dataset more similar to (or drawn directly from) the target population in the future may improve predictive performance in that population.

In conclusion, machine learning models trained on administrative claims data were unable to accurately predict medication nonadherence in patients self-administering biologics for IBD in a tertiary academic medical center patient population. Future research into training models should consider training in datasets with additional demographic and disease-state relevant variables, while omitting excessive information on unrelated diagnoses or medication dispenses.

Funding

The project described was supported by the NIH National Center for Advancing Translational Sciences through grant number UL1TR001998. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.

Conflict of Interest

The authors have no relevant competing or financial interests to declare.

Data Availability

Data used in this study was made available to the authors by a third-party license from Meritive™. The Merative MarketScan Research Databases are available for researchers who purchase access to the data and complete the required data use agreement processes.

References