Abstract

Objective

Opioid-based analgesia is routinely used in clinical practice for the management of pain and alleviation of suffering at the end of life. It is well-known that opioid-based medications can be highly addictive, promoting not only abuse but also life-threatening overdoses. The scope of opioid-related adverse events (AEs) beyond these well-known effects remains poorly described. This exploratory analysis investigates potential AEs from drug-drug interactions between opioid and nonopioid medications (ODIs).

Materials and Methods

In this study, we conduct an initial exploration of the association between ODIs and severe AEs using millions of AE reports available in FDA Adverse Event Reporting System (FAERS). The odds ratio (OR)-based analysis and visualization are proposed for single drugs and pairwise ODIs to identify associations between AEs and ODIs of interest. Moreover, the multilabel (multi-AE) learning models are employed to evaluate the feasibility of AE prediction of polypharmacy.

Results

The top 12 most prescribed opioids in the FAERS are identified. The OR-based analysis identifies a diverse set of AEs associated with individual opioids. Moreover, the results indicate many ODIs can increase the risk of severe AEs dramatically. The area under the curve values of multilabel learning models of ODIs for oxycodone varied between 0.81 and 0.88 for 5 severe AEs.

Conclusions

The proposed data analysis and visualization are useful for mining FAERS data to identify novel polypharmacy associated AEs, as shown for ODIs. This approach was successful in recapitulating known drug interactions and also identified new opioid-specific AEs that could impact prescribing practices.

INTRODUCTION

Adverse events (AEs) are unexpected and potentially injurious side effects and harms which occur during typical usage of a medication.1 In the United States, approximately 3–7% of hospitalizations are caused by AEs, 10–20% of hospitalizations include AEs, and 10–20% of these AEs are severe.2 Opioids is a generic term for a class of medications which activate central and peripheral opioid receptors, producing analgesia and other physiological effects, such as decreased heart and respiratory rates,3,4 which also can promote addiction leading to potential overdose and death. In 2016, there were approximately 42 000 deaths involving opioids in the United States, which represents a 21.4% increase from 2015.5 Additional well-known AEs associated with opioids include nausea, vomiting, dizziness, and constipation.6

In practice, opioids are often coprescribed for patients with a complicated medical history who are concurrently taking a wide array of additional medications.7 Beyond the aforementioned well-described AEs; however, little is known about the potential for drug-drug interactions between opioid and nonopioid medications. As a result, opioids are often coprescribed with limited information on opioid-drug interactions (ODIs).

The Food and Drug Administration's (FDA) Adverse Event Reporting System (FAERS)8 is a database that contains reported AEs and medication errors involving FDA-approved drugs. The AE reports in FAERS previously have been mined previously to identify potential AEs for specific drugs.9–11 Using FAERS data, some common severe AEs, such cardiovascular incidents, abuse, death, and overdose have been associated with pain-killers, such as nonsteroidal anti-inflammatory drugs (NSAIDs), acetaminophen, and opioids.12

Despite widespread use of opioids and the significant burden of opioid-related adverse outcomes, to the best of our knowledge; however, the linkage of AEs and ODIs have not been well investigated. It is common for opioids to be prescribed to patients taking other medications (polypharmacy), often in the setting of complex medical conditions (eg, traumatic brain injury7) but medical providers lack general safety parameters to guide decision making when combining opioids with other medications. In this preliminary study, we explore the association between ODIs and severe AEs using AE reports in the FAERS database. Using the odds ratio (OR) analysis of single and pairwise ODIs, we identify a set of opioid-specific ODIs that are highly associated with the risk for a set of severe AEs. We also implement a novel visualization method to show the ODIs associated with increased risk of individual AEs. Lastly, we applied multilabel machine learning models to evaluate the feasibility of predicting the association of a specific AE for the use of multiple medicine including opioids to improve the safety of medical treatment. The rest of the paper is organized as follows. First, we identify the mostly commonly reported opioids in the FAERS database along with their reporting frequency. We then introduce the approaches for data preprocessing and AE identification for the most commonly prescribed opioids, as well as the methods employed for the OR analysis of both single-drug and pairwise ODIs, as well as the evaluation of multilabel learning models to predict the AEs of polypharmacy. In the Results section, we provide the top 20 associated AEs for the top 5 reported opioids, the list and visualization of top-ranked ODIs are associated with each of 18 selected AEs for oxycodone and hydrocodone, as well as AE prediction evaluation using 3 multilabel learning models. The article concludes with the Discussion and Conclusion sections.

MATERIALS AND METHODS

Opioids

In this preliminary study, we conducted our analysis on the top 12 opioids reported in the FAERS database, which is an expanded list of the most commonly prescribed opioids according to the National Institute on Drug Abuse (Table 1).6,7,13

Table 1.

Frequency of opioids-involved adverse events in the FAERS database

OpioidFrequency%
1Oxycodone91 0732.25
2Hydrocodone80 1631.98
3Morphine78 3201.93
4Fentanyl75 8441.87
5Codeine51 3081.27
6Methadone27 2360.67
7Hydromorphone15 9710.39
8Buprenorphine80240.20
9Heroin50840.13
10Dihydrocodeine37550.09
11Tapentadol12870.03
12Alfentanil6630.02
OpioidFrequency%
1Oxycodone91 0732.25
2Hydrocodone80 1631.98
3Morphine78 3201.93
4Fentanyl75 8441.87
5Codeine51 3081.27
6Methadone27 2360.67
7Hydromorphone15 9710.39
8Buprenorphine80240.20
9Heroin50840.13
10Dihydrocodeine37550.09
11Tapentadol12870.03
12Alfentanil6630.02

Abbreviations: FAERS: FDA Adverse Event Reporting System.

Table 1.

Frequency of opioids-involved adverse events in the FAERS database

OpioidFrequency%
1Oxycodone91 0732.25
2Hydrocodone80 1631.98
3Morphine78 3201.93
4Fentanyl75 8441.87
5Codeine51 3081.27
6Methadone27 2360.67
7Hydromorphone15 9710.39
8Buprenorphine80240.20
9Heroin50840.13
10Dihydrocodeine37550.09
11Tapentadol12870.03
12Alfentanil6630.02
OpioidFrequency%
1Oxycodone91 0732.25
2Hydrocodone80 1631.98
3Morphine78 3201.93
4Fentanyl75 8441.87
5Codeine51 3081.27
6Methadone27 2360.67
7Hydromorphone15 9710.39
8Buprenorphine80240.20
9Heroin50840.13
10Dihydrocodeine37550.09
11Tapentadol12870.03
12Alfentanil6630.02

Abbreviations: FAERS: FDA Adverse Event Reporting System.

AE reports in the FAERS database and preprocessing

To study the AEs of opioids induced by ODIs, we collected AE reports from the FAERS database reported over the past 14 years, from October 1, 2003 through September 30, 2017. AEs are defined as the Medical Dictionary for Regulatory Activities (MedDRA).14 MedDRA is a medical terminology dictionary and is the dictionary of AE classification. In total, there were 9 805 596 case records reported with at least one of AE by this criterion. In this study, we limited our scope to FDA-approved small molecular drugs. A list of 2521 FDA-approved small molecule opioids was retrieved from DrugBank (version 5.1.0).15 After comparing drug names in the FAERS database with FDA-approved drugs, we removed 5 756 511 AEs (58.7%) that did not contain any drug in the FDA-approved list. In addition, we limited our interest to drugs reported at least 1000 times with an AE proportion of no less than 0.5% in the FAERS database. In the end, we collected 4 094 084 AE reports associated with 774 drugs and 151 AEs. The procedure of data preprocessing is demonstrated in Figure 1.

Data preprocessing scheme. Data were extracted from the FDA Adverse Event Reporting System (FAERS) from reports dated October 1, 2003 through September 31, 2017. The list of FDA-approved drugs was retrieved from DrugBank database on May 22, 2018 (version 5.1.0; released on 2018-04-02). N denotes the number of reports in FAERS, Q indicates the number of adverse effects (AEs), and P represents the number of drugs co-prescribed with opioids.
Figure 1.

Data preprocessing scheme. Data were extracted from the FDA Adverse Event Reporting System (FAERS) from reports dated October 1, 2003 through September 31, 2017. The list of FDA-approved drugs was retrieved from DrugBank database on May 22, 2018 (version 5.1.0; released on 2018-04-02). N denotes the number of reports in FAERS, Q indicates the number of adverse effects (AEs), and P represents the number of drugs co-prescribed with opioids.

OR analysis to identify and visualize associations between ODIs and increased risk of severe AEs

OR is a technique to quantify the risk of an event with data presented in a 2-by-2 contingency table (see Figure 2). In the contingency table, both the column variable (AE of interest) and the row variable (drug of interest) are binary: present (+) or absent (−). The co-occurrence of any drug-AE pair is categorized into 4 conditions: “–,” “−+,” “+−,” and “++”. Then, we convert the FAERS data into a series of 2-by-2 contingency tables with populated numbers of counts: N11, N12, N21, and N22. The OR is calculated as OR=N11N22N12N21.

Example of 2-by-2 contingency table for computing the odds ratio of a drug (i.e., oxycodone) for inducing an adverse effect (AE) (e.g., renal failure).
Figure 2.

Example of 2-by-2 contingency table for computing the odds ratio of a drug (i.e., oxycodone) for inducing an adverse effect (AE) (e.g., renal failure).

For each AE, we calculated all the ORs of all individual drugs, denoted as Drug.OR for nonopioid drugs and Opioid.OR for individual opioids, and calculated the ORs of all pairwise ODIs (with one opioid drug colisted with another nonopioid drug), denoted as Pair.OR. For example, if oxycodone is colisted with 3 other nonopioid drugs in an AE report, 3 different Pair.OR’s would be calculated, one for each opioid-nonopioid combination. Then, the Fisher’s exact test was performed to confirm the ORs of specific AEs of given ODIs. In order to control the experiment-wise false alarm rate at 0.05, a Bonferroni correction was applied.

To facilitate the visualization of ODIs with increased risk of specific AEs, we implemented a new type of visualization plot (see Figures 3–5). As can be seen in the upperpanels of Figures 3–5, the x-axis and y-axis are the relative OR for both ODIs, Pair.OR/Opioid.OR, and relative OR of drugs, Drug.OR/Opioid.OR. Thus, opioid-specific ODIs (eg, oxycodone [opioid]-anagrelide[drug] interaction, represented by “anagrelide”) that are above the solid-bold line are associated with an increased risk of severe AEs (eg, renal failure). ODIs with more than a 1.5-fold Opioid.OR and a P value < .05, ODIs with an Opioid.OR ≥ 0.75 and ≤ 1.5 and a P value ≥ .05, and ODIs with an Opioid.OR < 0.75 and a P value < .05 are labeled as “+,” “o,” and “Δ” symbols, respectively. Moreover, the OR density plots (lowerpanel) indicate the odd ratios of the specified AEs are increased by interacting with the opioid. In other words, the Pair.OR (ORs of DOIs) has a heavier tailed distribution compared with Drug.OR (ORs of single drugs). The OR of the opioid is also plotted. In addition, names of the top 10 ranked drugs interacting with the given opioid are displayed. The ODI plots can be updated conveniently by changing specific parameters.

Visualization of top 5 ODIs (green nodes) for oxycodone and hydrocodone (purple nodes), respectively, causing increased risk of 5 selected example adverse events (orange nodes) (cardiorespiratory arrest, cardiac arrest, renal failure, diabetes mellitus, and pulmonary embolism). Abbreviation: ODIs: opioid-drug interactions.
Figure 3.

Visualization of top 5 ODIs (green nodes) for oxycodone and hydrocodone (purple nodes), respectively, causing increased risk of 5 selected example adverse events (orange nodes) (cardiorespiratory arrest, cardiac arrest, renal failure, diabetes mellitus, and pulmonary embolism). Abbreviation: ODIs: opioid-drug interactions.

Visualization of top-ranked oxycodone-drug interactions causing increased risk of adverse events. Abbreviations: ADE: •••; OR: odds ratio.
Figure 4.

Visualization of top-ranked oxycodone-drug interactions causing increased risk of adverse events. Abbreviations: ADE: •••; OR: odds ratio.

Visualization of top-ranked hydrocodone-drug interactions causing increased risk of adverse events. Abbreviations: ADE: •••; OR: odds ratio.
Figure 5.

Visualization of top-ranked hydrocodone-drug interactions causing increased risk of adverse events. Abbreviations: ADE: •••; OR: odds ratio.

AE prediction of polypharmacy using multilabel learning models

In addition to the identification of pairwise ODIs using the OR analysis and visualization, we further evaluated the feasibility of predicting ODI AEs containing multiple drugs and oxycodone (the most reported opioid in FAERS).

Problem formulation

Let n be the number of AE reports (n = 91 073), p be the number of total drugs involved in all reports (P = 774), and m represent types of AE. For example, in this preliminary study, 5 type of AEs are selected, that is, renal failure, pulmonary embolism, cardiac arrest, cardiorespiratory arrest, and pneumonia. Mathematically, let X=x1,,xp be the vector of individual reports, where xi=xi1,,xinT and xij{0, 1}, i {1, ,p}, j {1, , n} indicate if a drug included in a report (1) or not (0). Let Y=y1,,ym be the AEs, where yk=yk1,,yknTand ykj{0, 1}, k {1, ,m}, j {1, , n}, which indicates if a report is associated with a given AE (1) or not (0). Then the matrix, X, will be used as the input and Y will be used as the output (AEs labels) of the multi-AE prediction problem.

Three multilabel prediction models

In this study, 3 supervised learning models are employed, that is, the binary relevance method using logistic regression as the base learner (BR.lr),16 the classifier chains method using regression as the base learner (CC.lr),17,18 and the multivariate classification and regression random forest model (RFSRC).19 The “mlr”20 R package was used to call the BR.lr and CC.lr models, and the “randomforestSRC”21 R package was used to call the RFSRC model. We evaluate the prediction models by dividing the dataset using 5-fold cross-validation, that is, the dataset is randomly divided into 5 folds, and 4 folds (80% of the data) are used as training data, and the rest 20% are used as the testing data. The average of the following metrics is used to evaluate the performance of the models.

Prediction evaluation metrics

The prediction performance was evaluated using the following metrics (ie, hamming loss, subset 0/1 loss, f1 score, accuracy, and precision). Let Cxi=y^1i,,y^mi and y(i) = y1i,,ymi represent the predicted and actual AE labels for individual reports, then the evaluation metrics are defined as follows20:

For interpretation purposes, hamming loss is defined as the fraction of the wrongly predicted AE labels (0 or 1) to the total number of AE labels (a report might be associated with multiple AEs). Subset 0/1 loss is defined as the fraction of reports that have at least one AE label predicted wrongly. Accuracy is defined as the fraction of correctly predicted AE labels to the total number of AE labels. Precision is defined as the fraction of true positive and also predicted positive AE labels to the predicted positive AE labels. F1 score is defined as an average of the fraction of true positive and also predicted positive AE labels to the predicted positive AE labels, and the fraction of true positive and also predicted positive AE labels to the true positive AE labels.

RESULTS

Top 20 AEs for the 5 most commonly reported opioids

Table 2 shows the top 20 AEs associated with the top 5 reported opioid drugs in FAERS. Not surprisingly, the top 20 AEs for the individual opioid drugs are different but encompass many of the well-known, opioid AEs, such as abuse, overdose, and vomiting. Aside from the expected AEs associated with opioids, each medication has a unique subset of additional AEs, for instance, epistaxis was more commonly reported with codeine than the other opioids. It suggests that opioid-specific ODIs and associated AEs should be investigated separately.

Table 2.

Top 20 AEs ranked by odds ratios for the most frequent opioids

RankOxycodoneHydrocodoneMorphineFentanylCodeine
1Drug abuseCompleted suicideDrug hypersensitivityDrug abuseDrug hypersensitivity
2Cardiorespiratory arrestDrug abuseHypersensitivityProduct quality issueDrug abuse
3Cardiac arrestCardiorespiratory arrestLethargyToxicity to various agentsHypersensitivity
4ConstipationBack painToxicity to various agentsEmotional distressOverdose
5Completed suicideGastroesophageal reflux diseaseDrug abuseDrug effect decreasedToxicity to various agents
6SinusitisAnxietyCardiac arrestCardiac arrestHallucination
7Toxicity to various agentsEmotional distressSomnolenceOverdoseEmotional distress
8Back painSinusitisOverdoseHyperhidrosisBronchitis
9DehydrationCardiac arrestRespiratory failureBack painCompleted suicide
10Neuropathy peripheralPulmonary embolismHallucinationDepressionDeep vein thrombosis
11Disease progressionAmnesiaGeneral physical health deteriorationDrug ineffectiveEpistaxis
12Pulmonary embolismDepressionBack painConstipationJoint swelling
13InjuryDiabetes mellitusPulmonary embolismRespiratory failureSinusitis
14Decreased appetiteCardiac failure congestiveDeathAmnesiaDysphagia
15Pleural effusionArthralgiaCompleted suicideInjuryDrug interaction
16Abdominal painDeep vein thrombosisDisease progressionRenal impairmentRheumatoid arthritis
17DepressionNeuropathy peripheralConstipationRenal failureHyperhidrosis
18HypokalemiaHypoesthesiaDepressionLoss of consciousnessCerebrovascular accident
19Emotional distressWeight increasedVomitingCardiorespiratory arrestThrombosis
20Memory impairmentGait disturbanceLoss of consciousnessUrinary tract infectionCardiorespiratory arrest
RankOxycodoneHydrocodoneMorphineFentanylCodeine
1Drug abuseCompleted suicideDrug hypersensitivityDrug abuseDrug hypersensitivity
2Cardiorespiratory arrestDrug abuseHypersensitivityProduct quality issueDrug abuse
3Cardiac arrestCardiorespiratory arrestLethargyToxicity to various agentsHypersensitivity
4ConstipationBack painToxicity to various agentsEmotional distressOverdose
5Completed suicideGastroesophageal reflux diseaseDrug abuseDrug effect decreasedToxicity to various agents
6SinusitisAnxietyCardiac arrestCardiac arrestHallucination
7Toxicity to various agentsEmotional distressSomnolenceOverdoseEmotional distress
8Back painSinusitisOverdoseHyperhidrosisBronchitis
9DehydrationCardiac arrestRespiratory failureBack painCompleted suicide
10Neuropathy peripheralPulmonary embolismHallucinationDepressionDeep vein thrombosis
11Disease progressionAmnesiaGeneral physical health deteriorationDrug ineffectiveEpistaxis
12Pulmonary embolismDepressionBack painConstipationJoint swelling
13InjuryDiabetes mellitusPulmonary embolismRespiratory failureSinusitis
14Decreased appetiteCardiac failure congestiveDeathAmnesiaDysphagia
15Pleural effusionArthralgiaCompleted suicideInjuryDrug interaction
16Abdominal painDeep vein thrombosisDisease progressionRenal impairmentRheumatoid arthritis
17DepressionNeuropathy peripheralConstipationRenal failureHyperhidrosis
18HypokalemiaHypoesthesiaDepressionLoss of consciousnessCerebrovascular accident
19Emotional distressWeight increasedVomitingCardiorespiratory arrestThrombosis
20Memory impairmentGait disturbanceLoss of consciousnessUrinary tract infectionCardiorespiratory arrest

Abbreviations: AE: adverse events.

Table 2.

Top 20 AEs ranked by odds ratios for the most frequent opioids

RankOxycodoneHydrocodoneMorphineFentanylCodeine
1Drug abuseCompleted suicideDrug hypersensitivityDrug abuseDrug hypersensitivity
2Cardiorespiratory arrestDrug abuseHypersensitivityProduct quality issueDrug abuse
3Cardiac arrestCardiorespiratory arrestLethargyToxicity to various agentsHypersensitivity
4ConstipationBack painToxicity to various agentsEmotional distressOverdose
5Completed suicideGastroesophageal reflux diseaseDrug abuseDrug effect decreasedToxicity to various agents
6SinusitisAnxietyCardiac arrestCardiac arrestHallucination
7Toxicity to various agentsEmotional distressSomnolenceOverdoseEmotional distress
8Back painSinusitisOverdoseHyperhidrosisBronchitis
9DehydrationCardiac arrestRespiratory failureBack painCompleted suicide
10Neuropathy peripheralPulmonary embolismHallucinationDepressionDeep vein thrombosis
11Disease progressionAmnesiaGeneral physical health deteriorationDrug ineffectiveEpistaxis
12Pulmonary embolismDepressionBack painConstipationJoint swelling
13InjuryDiabetes mellitusPulmonary embolismRespiratory failureSinusitis
14Decreased appetiteCardiac failure congestiveDeathAmnesiaDysphagia
15Pleural effusionArthralgiaCompleted suicideInjuryDrug interaction
16Abdominal painDeep vein thrombosisDisease progressionRenal impairmentRheumatoid arthritis
17DepressionNeuropathy peripheralConstipationRenal failureHyperhidrosis
18HypokalemiaHypoesthesiaDepressionLoss of consciousnessCerebrovascular accident
19Emotional distressWeight increasedVomitingCardiorespiratory arrestThrombosis
20Memory impairmentGait disturbanceLoss of consciousnessUrinary tract infectionCardiorespiratory arrest
RankOxycodoneHydrocodoneMorphineFentanylCodeine
1Drug abuseCompleted suicideDrug hypersensitivityDrug abuseDrug hypersensitivity
2Cardiorespiratory arrestDrug abuseHypersensitivityProduct quality issueDrug abuse
3Cardiac arrestCardiorespiratory arrestLethargyToxicity to various agentsHypersensitivity
4ConstipationBack painToxicity to various agentsEmotional distressOverdose
5Completed suicideGastroesophageal reflux diseaseDrug abuseDrug effect decreasedToxicity to various agents
6SinusitisAnxietyCardiac arrestCardiac arrestHallucination
7Toxicity to various agentsEmotional distressSomnolenceOverdoseEmotional distress
8Back painSinusitisOverdoseHyperhidrosisBronchitis
9DehydrationCardiac arrestRespiratory failureBack painCompleted suicide
10Neuropathy peripheralPulmonary embolismHallucinationDepressionDeep vein thrombosis
11Disease progressionAmnesiaGeneral physical health deteriorationDrug ineffectiveEpistaxis
12Pulmonary embolismDepressionBack painConstipationJoint swelling
13InjuryDiabetes mellitusPulmonary embolismRespiratory failureSinusitis
14Decreased appetiteCardiac failure congestiveDeathAmnesiaDysphagia
15Pleural effusionArthralgiaCompleted suicideInjuryDrug interaction
16Abdominal painDeep vein thrombosisDisease progressionRenal impairmentRheumatoid arthritis
17DepressionNeuropathy peripheralConstipationRenal failureHyperhidrosis
18HypokalemiaHypoesthesiaDepressionLoss of consciousnessCerebrovascular accident
19Emotional distressWeight increasedVomitingCardiorespiratory arrestThrombosis
20Memory impairmentGait disturbanceLoss of consciousnessUrinary tract infectionCardiorespiratory arrest

Abbreviations: AE: adverse events.

Top-ranked ODIs inducing 18 selected severe AEs

In this pilot study, we empirically chose 18 common and severe AEs for further analysis: death, pneumonia, anemia, hypotension, depression, hypertension, myocardial infarction, renal failure, sepsis, overdose, completed suicide, cardiac arrest, hemorrhage, diabetes mellitus, drug abuse, cardiorespiratory arrest, pulmonary embolism, and thrombosis. Tables 3 and 4 show the top 5 ODIs, based on the pair.OR, associated with an increased risk for the 18 selected severe AEs for opioids for oxycodone and hydrocodone, respectively. As can be seen, the top-ranked ODIs are diverse and heterogenous for different AEs, given the same opioid. In addition, we graphically represent the top 5 ODIs and 5 associated AEs in a network map (see Figure 3), which can clearly and intuitively display the ODI-AE associations. As aforementioned, for given specific AEs, we implemented a new way to visualize the pair.OR of opioid-specific ODIs. In Figures 4 and 5, we show the top-ranked ODIs with 5 severe AEs (renal failure, pulmonary embolism, cardiac arrest, cardiorespiratory arrest, and pneumonia) for oxycodone and hydrocodone, respectively. As shown in the figures, diverse and distinct ODI-AE pairs emerge for each opioid, suggesting potential drug interactions that may necessitate different prescribing practices to prevent specific AEs.

Table 3.

Top 5 oxycodone-drug interactions associated with increased risk of 18 selected severe AEs

AEsNumber of reportsNumber of drugs with pair.OR > 2Top 5 drugs interacting with oxycodone associated with the AEs
1Hemorrhage181 606111Anagrelide, aztreonam, nizatidine, norethisterone, sulfadiazine
2Pneumonia129 489272Anagrelide, argatroban, sulbactam, thalidomide, trichlormethiazide
3Death128 89638Carmustine, cimetidine, digitoxin, methamphetamine, palonosetron
4Anemia111 336280Anagrelide, flutamide, mitomycin, procarbazine, thalidomide
5Renal failure105 791239Anagrelide, dobutamine, galantamine, milrinone, rilmenidine
6Depression88 941235Adenosine, ampicillin, etomidate, tetracycline, trichlormethiazide
7Hypertension88 831201Methyldopa, mitomycin, nicardipine, raloxifene, trichlormethiazide
8Hypotension86 104192Calcium chloride, clofarabine, dobutamine, dopamine, trichlormethiazide
9Overdose83 63576Caffeine, deflazacort, etoricoxib, flurazepam, galantamine
10Thrombosis70 395226Adenosine, ampicillin, drospirenone, nicardipine, tetracycline
11Myocardial infarction70 02875Bimatoprost, dobutamine, morniflumate, nicardipine, prasugrel
12Sepsis64 938183Anagrelide, calcium chloride, cefotaxime, cilastatin, vasopressin
13Diabetes mellitus42 013152Deflazacort, dobutamine, nizatidine, perphenazine, trichlormethiazide
14Completed suicide41 43794Aripiprazole, ethanol, milnacipran, phenobarbital, quetiapine
15Pulmonary embolism40 521136Drospirenone, ethinyl estradiol, perphenazine, tipranavir, trichlormethiazide
16Cardiac arrest38 33172Benzodiazepine, dobutamine, ethanol, etomidate, perphenazine
17Drug abuse26 04277Dextromethorphan, doxylamine, methamphetamine, nimesulide, pentazocine
18Cardiorespiratory arrest21 05670Aripiprazole, benzodiazepine, ethanol, methamphetamine, rosiglitazone
AEsNumber of reportsNumber of drugs with pair.OR > 2Top 5 drugs interacting with oxycodone associated with the AEs
1Hemorrhage181 606111Anagrelide, aztreonam, nizatidine, norethisterone, sulfadiazine
2Pneumonia129 489272Anagrelide, argatroban, sulbactam, thalidomide, trichlormethiazide
3Death128 89638Carmustine, cimetidine, digitoxin, methamphetamine, palonosetron
4Anemia111 336280Anagrelide, flutamide, mitomycin, procarbazine, thalidomide
5Renal failure105 791239Anagrelide, dobutamine, galantamine, milrinone, rilmenidine
6Depression88 941235Adenosine, ampicillin, etomidate, tetracycline, trichlormethiazide
7Hypertension88 831201Methyldopa, mitomycin, nicardipine, raloxifene, trichlormethiazide
8Hypotension86 104192Calcium chloride, clofarabine, dobutamine, dopamine, trichlormethiazide
9Overdose83 63576Caffeine, deflazacort, etoricoxib, flurazepam, galantamine
10Thrombosis70 395226Adenosine, ampicillin, drospirenone, nicardipine, tetracycline
11Myocardial infarction70 02875Bimatoprost, dobutamine, morniflumate, nicardipine, prasugrel
12Sepsis64 938183Anagrelide, calcium chloride, cefotaxime, cilastatin, vasopressin
13Diabetes mellitus42 013152Deflazacort, dobutamine, nizatidine, perphenazine, trichlormethiazide
14Completed suicide41 43794Aripiprazole, ethanol, milnacipran, phenobarbital, quetiapine
15Pulmonary embolism40 521136Drospirenone, ethinyl estradiol, perphenazine, tipranavir, trichlormethiazide
16Cardiac arrest38 33172Benzodiazepine, dobutamine, ethanol, etomidate, perphenazine
17Drug abuse26 04277Dextromethorphan, doxylamine, methamphetamine, nimesulide, pentazocine
18Cardiorespiratory arrest21 05670Aripiprazole, benzodiazepine, ethanol, methamphetamine, rosiglitazone

Abbreviations: AE: adverse events; OR: odds ratio.

Table 3.

Top 5 oxycodone-drug interactions associated with increased risk of 18 selected severe AEs

AEsNumber of reportsNumber of drugs with pair.OR > 2Top 5 drugs interacting with oxycodone associated with the AEs
1Hemorrhage181 606111Anagrelide, aztreonam, nizatidine, norethisterone, sulfadiazine
2Pneumonia129 489272Anagrelide, argatroban, sulbactam, thalidomide, trichlormethiazide
3Death128 89638Carmustine, cimetidine, digitoxin, methamphetamine, palonosetron
4Anemia111 336280Anagrelide, flutamide, mitomycin, procarbazine, thalidomide
5Renal failure105 791239Anagrelide, dobutamine, galantamine, milrinone, rilmenidine
6Depression88 941235Adenosine, ampicillin, etomidate, tetracycline, trichlormethiazide
7Hypertension88 831201Methyldopa, mitomycin, nicardipine, raloxifene, trichlormethiazide
8Hypotension86 104192Calcium chloride, clofarabine, dobutamine, dopamine, trichlormethiazide
9Overdose83 63576Caffeine, deflazacort, etoricoxib, flurazepam, galantamine
10Thrombosis70 395226Adenosine, ampicillin, drospirenone, nicardipine, tetracycline
11Myocardial infarction70 02875Bimatoprost, dobutamine, morniflumate, nicardipine, prasugrel
12Sepsis64 938183Anagrelide, calcium chloride, cefotaxime, cilastatin, vasopressin
13Diabetes mellitus42 013152Deflazacort, dobutamine, nizatidine, perphenazine, trichlormethiazide
14Completed suicide41 43794Aripiprazole, ethanol, milnacipran, phenobarbital, quetiapine
15Pulmonary embolism40 521136Drospirenone, ethinyl estradiol, perphenazine, tipranavir, trichlormethiazide
16Cardiac arrest38 33172Benzodiazepine, dobutamine, ethanol, etomidate, perphenazine
17Drug abuse26 04277Dextromethorphan, doxylamine, methamphetamine, nimesulide, pentazocine
18Cardiorespiratory arrest21 05670Aripiprazole, benzodiazepine, ethanol, methamphetamine, rosiglitazone
AEsNumber of reportsNumber of drugs with pair.OR > 2Top 5 drugs interacting with oxycodone associated with the AEs
1Hemorrhage181 606111Anagrelide, aztreonam, nizatidine, norethisterone, sulfadiazine
2Pneumonia129 489272Anagrelide, argatroban, sulbactam, thalidomide, trichlormethiazide
3Death128 89638Carmustine, cimetidine, digitoxin, methamphetamine, palonosetron
4Anemia111 336280Anagrelide, flutamide, mitomycin, procarbazine, thalidomide
5Renal failure105 791239Anagrelide, dobutamine, galantamine, milrinone, rilmenidine
6Depression88 941235Adenosine, ampicillin, etomidate, tetracycline, trichlormethiazide
7Hypertension88 831201Methyldopa, mitomycin, nicardipine, raloxifene, trichlormethiazide
8Hypotension86 104192Calcium chloride, clofarabine, dobutamine, dopamine, trichlormethiazide
9Overdose83 63576Caffeine, deflazacort, etoricoxib, flurazepam, galantamine
10Thrombosis70 395226Adenosine, ampicillin, drospirenone, nicardipine, tetracycline
11Myocardial infarction70 02875Bimatoprost, dobutamine, morniflumate, nicardipine, prasugrel
12Sepsis64 938183Anagrelide, calcium chloride, cefotaxime, cilastatin, vasopressin
13Diabetes mellitus42 013152Deflazacort, dobutamine, nizatidine, perphenazine, trichlormethiazide
14Completed suicide41 43794Aripiprazole, ethanol, milnacipran, phenobarbital, quetiapine
15Pulmonary embolism40 521136Drospirenone, ethinyl estradiol, perphenazine, tipranavir, trichlormethiazide
16Cardiac arrest38 33172Benzodiazepine, dobutamine, ethanol, etomidate, perphenazine
17Drug abuse26 04277Dextromethorphan, doxylamine, methamphetamine, nimesulide, pentazocine
18Cardiorespiratory arrest21 05670Aripiprazole, benzodiazepine, ethanol, methamphetamine, rosiglitazone

Abbreviations: AE: adverse events; OR: odds ratio.

Table 4.

Top 5 hydrocodone-drug interactions associated with increased risk of 18 selected AEs

AEsNumber of reportsNumber of drugs with Pair.OR > 2Top 5 drugs interacting with hydrocodone associated with the AEs
1Hemorrhage181 606142Acenocoumarol, alendronic acid, clofarabine, flutamide, remifentanil
2Pneumonia129 489193Acenocoumarol, bimatoprost, cilastatin, cyclizine, mycophenolic acid
3Death128 89632Cefaclor, clozapine, doxylamine, erlotinib, meprobamate
4Anemia111 336239Flutamide, melphalan, mycophenolic acid, thalidomide, ticlopidine
5Renal failure105 791187Dobutamine, flutamide, milrinone, mycophenolic acid, vasopressin
6Depression88 941266Aminophylline, cyclizine, flutamide, goserelin, thrombin
7Hypertension88 831231Aminophylline, domperidone, flutamide, mitoxantrone, mycophenolic acid
8Hypotension86 104150Calcium chloride, cilastatin, dobutamine, flutamide, norepinephrine
9Overdose83 63535Benzodiazepine, caffeine, meprobamate, nicotine, oxazepam
10Thrombosis70 395206Aminophylline, argatroban, ceftazidime, dacarbazine, mycophenolic acid
11Myocardial infarction70 028128Acetylcysteine, aminophylline, flutamide, morniflumate, nizatidine
12Sepsis64 938117Calcium chloride, micafungin, mitoxantrone, norepinephrine, temsirolimus
13Diabetes mellitus42 013190Aminophylline, fludarabine, fosinopril, mycophenolic acid, orlistat
14Completed suicide41 437144Desipramine, eszopiclone, ethanol, nicardipine, valproic acid
15Pulmonary embolism40 521129Bromocriptine, drospirenone, ethinyl estradiol, remifentanil, vecuronium
16Cardiac arrest38 33179Alendronic acid, dobutamine, mitoxantrone, nicardipine, saxagliptin
17Drug abuse26 04259Dextromethorphan, doxylamine, methamphetamine, oxymorphone, pentazocine
18Cardiorespiratory arrest21 05680Aripiprazole, benzodiazepine, bromocriptine, ethanol, nicardipine
AEsNumber of reportsNumber of drugs with Pair.OR > 2Top 5 drugs interacting with hydrocodone associated with the AEs
1Hemorrhage181 606142Acenocoumarol, alendronic acid, clofarabine, flutamide, remifentanil
2Pneumonia129 489193Acenocoumarol, bimatoprost, cilastatin, cyclizine, mycophenolic acid
3Death128 89632Cefaclor, clozapine, doxylamine, erlotinib, meprobamate
4Anemia111 336239Flutamide, melphalan, mycophenolic acid, thalidomide, ticlopidine
5Renal failure105 791187Dobutamine, flutamide, milrinone, mycophenolic acid, vasopressin
6Depression88 941266Aminophylline, cyclizine, flutamide, goserelin, thrombin
7Hypertension88 831231Aminophylline, domperidone, flutamide, mitoxantrone, mycophenolic acid
8Hypotension86 104150Calcium chloride, cilastatin, dobutamine, flutamide, norepinephrine
9Overdose83 63535Benzodiazepine, caffeine, meprobamate, nicotine, oxazepam
10Thrombosis70 395206Aminophylline, argatroban, ceftazidime, dacarbazine, mycophenolic acid
11Myocardial infarction70 028128Acetylcysteine, aminophylline, flutamide, morniflumate, nizatidine
12Sepsis64 938117Calcium chloride, micafungin, mitoxantrone, norepinephrine, temsirolimus
13Diabetes mellitus42 013190Aminophylline, fludarabine, fosinopril, mycophenolic acid, orlistat
14Completed suicide41 437144Desipramine, eszopiclone, ethanol, nicardipine, valproic acid
15Pulmonary embolism40 521129Bromocriptine, drospirenone, ethinyl estradiol, remifentanil, vecuronium
16Cardiac arrest38 33179Alendronic acid, dobutamine, mitoxantrone, nicardipine, saxagliptin
17Drug abuse26 04259Dextromethorphan, doxylamine, methamphetamine, oxymorphone, pentazocine
18Cardiorespiratory arrest21 05680Aripiprazole, benzodiazepine, bromocriptine, ethanol, nicardipine

Abbreviations: AE: adverse events; OR: odds ratio.

Table 4.

Top 5 hydrocodone-drug interactions associated with increased risk of 18 selected AEs

AEsNumber of reportsNumber of drugs with Pair.OR > 2Top 5 drugs interacting with hydrocodone associated with the AEs
1Hemorrhage181 606142Acenocoumarol, alendronic acid, clofarabine, flutamide, remifentanil
2Pneumonia129 489193Acenocoumarol, bimatoprost, cilastatin, cyclizine, mycophenolic acid
3Death128 89632Cefaclor, clozapine, doxylamine, erlotinib, meprobamate
4Anemia111 336239Flutamide, melphalan, mycophenolic acid, thalidomide, ticlopidine
5Renal failure105 791187Dobutamine, flutamide, milrinone, mycophenolic acid, vasopressin
6Depression88 941266Aminophylline, cyclizine, flutamide, goserelin, thrombin
7Hypertension88 831231Aminophylline, domperidone, flutamide, mitoxantrone, mycophenolic acid
8Hypotension86 104150Calcium chloride, cilastatin, dobutamine, flutamide, norepinephrine
9Overdose83 63535Benzodiazepine, caffeine, meprobamate, nicotine, oxazepam
10Thrombosis70 395206Aminophylline, argatroban, ceftazidime, dacarbazine, mycophenolic acid
11Myocardial infarction70 028128Acetylcysteine, aminophylline, flutamide, morniflumate, nizatidine
12Sepsis64 938117Calcium chloride, micafungin, mitoxantrone, norepinephrine, temsirolimus
13Diabetes mellitus42 013190Aminophylline, fludarabine, fosinopril, mycophenolic acid, orlistat
14Completed suicide41 437144Desipramine, eszopiclone, ethanol, nicardipine, valproic acid
15Pulmonary embolism40 521129Bromocriptine, drospirenone, ethinyl estradiol, remifentanil, vecuronium
16Cardiac arrest38 33179Alendronic acid, dobutamine, mitoxantrone, nicardipine, saxagliptin
17Drug abuse26 04259Dextromethorphan, doxylamine, methamphetamine, oxymorphone, pentazocine
18Cardiorespiratory arrest21 05680Aripiprazole, benzodiazepine, bromocriptine, ethanol, nicardipine
AEsNumber of reportsNumber of drugs with Pair.OR > 2Top 5 drugs interacting with hydrocodone associated with the AEs
1Hemorrhage181 606142Acenocoumarol, alendronic acid, clofarabine, flutamide, remifentanil
2Pneumonia129 489193Acenocoumarol, bimatoprost, cilastatin, cyclizine, mycophenolic acid
3Death128 89632Cefaclor, clozapine, doxylamine, erlotinib, meprobamate
4Anemia111 336239Flutamide, melphalan, mycophenolic acid, thalidomide, ticlopidine
5Renal failure105 791187Dobutamine, flutamide, milrinone, mycophenolic acid, vasopressin
6Depression88 941266Aminophylline, cyclizine, flutamide, goserelin, thrombin
7Hypertension88 831231Aminophylline, domperidone, flutamide, mitoxantrone, mycophenolic acid
8Hypotension86 104150Calcium chloride, cilastatin, dobutamine, flutamide, norepinephrine
9Overdose83 63535Benzodiazepine, caffeine, meprobamate, nicotine, oxazepam
10Thrombosis70 395206Aminophylline, argatroban, ceftazidime, dacarbazine, mycophenolic acid
11Myocardial infarction70 028128Acetylcysteine, aminophylline, flutamide, morniflumate, nizatidine
12Sepsis64 938117Calcium chloride, micafungin, mitoxantrone, norepinephrine, temsirolimus
13Diabetes mellitus42 013190Aminophylline, fludarabine, fosinopril, mycophenolic acid, orlistat
14Completed suicide41 437144Desipramine, eszopiclone, ethanol, nicardipine, valproic acid
15Pulmonary embolism40 521129Bromocriptine, drospirenone, ethinyl estradiol, remifentanil, vecuronium
16Cardiac arrest38 33179Alendronic acid, dobutamine, mitoxantrone, nicardipine, saxagliptin
17Drug abuse26 04259Dextromethorphan, doxylamine, methamphetamine, oxymorphone, pentazocine
18Cardiorespiratory arrest21 05680Aripiprazole, benzodiazepine, bromocriptine, ethanol, nicardipine

Abbreviations: AE: adverse events; OR: odds ratio.

ODI AE prediction for oxycodone

In addition to the OR-based analysis to identify ODIs and associated AEs, we further demonstrated the feasibility of applying multilabel learning models to predict AEs in patients with multiple reports. We selected 5 AEs of interest (ie, renal failure, pulmonary embolism, cardiac arrest, cardiorespiratory arrest, and pneumonia) that were reported with the presence of oxycodone. Three learning methods have been evaluated (ie, BR.lr: Binary relevance with logistic regression; CC.lr: Classifier Chains with logistic regression; and RFSRC: random forest adapted for multilabel classification). We used 5-fold cross validation to evaluate the model performance based on the aforementioned metrics, that is, hamming loss, subset 0/1 loss, f1 score, accuracy, and precision. Figure 6 (upperpanel) shows the average values of these metrics on the 5 selected AEs. In addition, Figure 6 (lower panel) shows the average area under the curve values of the 3 models for the AEs. The results indicate that the random forest SRC model outperformed the other models in all evaluation metrics.

Performance comparison of 3 multilabel classification models. Abbreviations: AUC: area under the curve; RFSRC: regression random forest model.
Figure 6.

Performance comparison of 3 multilabel classification models. Abbreviations: AUC: area under the curve; RFSRC: regression random forest model.

DISCUSSION AND CONCLUSION

Opioids are a commonly prescribed class of medications and are frequently taken in combination with other medications in the management of patients with acute and chronic pain. Although there is considerable potential for AEs, the associated AEs of ODIs has not been well investigated. Herein, we conducted an initial exploration of ODI-AE associations by mining millions of AE reports in the FAERS database. Using this approach to AE identification, we were able to recapitulate the well-known AEs associated with opioids, including constipation, abuse, and cardiopulmonary arrest. Furthermore, through the pairwise analysis of opioid-associated AEs, we were also able to recreate common, other well-known medication AEs, such as estrogen-induced venous thromboembolism, immunosuppressive-therapy associated risk of pneumonia, and higher cardiopulmonary arrest with benzodiazepine use.22–24 Together, this knowledge recreation lends credence to the methodology used in this exploratory analysis.

Furthermore, the results of this study suggest individual opioids have unique AEs, potentially related to drug-specific off-target effects. For instance, fentanyl was associated with hyperhydrosis while hydrocodone was associated with gastroesophageal reflux disease. It is certainly possible that many of the identified associations are the result of underlying patient comorbidities that are being treated rather than a direct medication effect, but it is also possible that opioids potentiate the AEs reported. To further stratify these associations, we will need to consider disease-specific treatments. For instance, a strong association between hydrocodone, riociguat and pulmonary embolism emerged in our cohort. However, riociguat is a medication used to treat inoperable or persistent postsurgical chronic thromboembolic pulmonary hypertension, and as such, it is unlikely that this association represents a true ODI between hydrocodone and riociguat. By encompassing disease-level treatment bundles into this tool, we can further enrich the results produced by this pipeline.

In addition, since the FAERS database does not contain a patient cohort without the associated AE, the relative prevalence of the AEs and ODIs may be over represented in the OR calculation. By incorporating additional patient characteristics, such as age, gender, and comorbidities, the risk of specific AEs for a given ODI can be more accurately measured. Also, it is not a trivial task for the drug name comparison and normalization to make use of the reports more accurately in FEARS database. Moreover, the OR analysis depends on the properly constructed confusions matrix. It might be biased and not accurate to use all reported AEs to calculate the negative cases for OR calculation. Ideally, people who were given the drug but reported no AEs should be used as the negative. However, these data were not included in FAERS. Therefore, in addition to FAERS data, it is important to integrate more electronic health record (EHR) data and also big datasets of claims and pharmacy to identify more and unbiased negative controls to further evaluate the potential ODIs and associated AEs. In the last section of this study, we also create a process to use the FAERS database to train machine learning models to predict potential AEs for given ODIs. These models could further be enhanced using sophisticated deep learning models to integrate chemical structure features to identify ODIs associated with severe AEs. Once validated, it is our hope that this pipeline could be helpful to facilitate the identification of additional drug-drug interactions to improve safe prescribing practices.

FUNDING

This work was supported by the Institute for Informatics (I2), Department of Pediatrics startup funding, Washington University in St. Louis.

AUTHOR CONTRIBUTIONS

JB, JC, PP, and FL contributed the study idea and design. JC, GW, and FL conducted the data analysis and wrote the article. JC, GW, AM, ZV, and FL revised the manuscript.

ACKNOWLEDGMENTS

The authors would like to thank Dr. Kelley Foyil for the editing of the manuscript.

CONFLICT OF INTEREST STATEMENT

None declared.

REFERENCES

1

Nebeker
JR
,
Barach
P
,
Samore
MH.
Clarifying adverse drug events: a clinician’s guide to terminology, documentation, and reporting
.
Ann Intern Med
2004
;
140
(
10
):
795
801
.

2

Bond
CA
,
Raehl
CL.
Clinical pharmacy services, pharmacy staffing, and adverse drug reactions in United States hospitals
.
Pharmacother J Hum Pharmacol Drug Ther
2006
;
26
(
6
):
735
47
.

3

Al-Hasani
R
,
Bruchas
MR.
Molecular mechanisms of opioid receptor-dependent signaling and behavior
.
Anesthesiol J Am Soc Anesthesiol
2011
;
115
:
1363
81
.

4

Pathan
H
,
Williams
J.
Basic opioid pharmacology: an update
.
Br J Pain
2012
;
6
(
1
):
11
16
.

5

Scholl
L
,
Seth
P
,
Kariisa
M
, et al. .
Drug and opioid-involved overdose deaths—United States, 2013–2017
.
MMWR Morb Mortal Wkly Rep
2018
;
67
:
1419
27
.

6

Benyamin
R
,
Trescot
AM
,
Datta
S
, et al. .
Opioid complications and side effects
.
Pain
2008
;
11 (2 Suppl
):
S105
20
.

7

Rudd
RA
,
Seth
P
,
David
F
,
Scholl
L.
Increases in drug and opioid-involved overdose deaths—United States, 2010-2015
.
Mmwr Morb Mortal Wkly Rep
2016
;
65
(
5051
):
1445
52
.

8

 FDA Adverse Event Reporting System. https://open.fda.gov/data/faers/. Accessed June 2018.

9

Sakaeda
T
,
Tamon
A
,
Kadoyama
K
,
Okuno
Y.
Data mining of the public version of the FDA adverse event reporting system
.
Int J Med Sci
2013
;
10
(
7
):
796
803
.

10

Ventola
CL.
Big data and pharmacovigilance: data mining for adverse drug events and interactions
.
P T
2018
;
43
(
6
):
340
51
.

11

Chiang
W-H
,
Schleyer
T
,
Shen
L
,
Li
L
,
Ning
X.
Pattern discovery from high-order drug-drug interaction relations
.
J Healthc Inform Res
2018
;
2
(
3
):
272
304
.

12

Min
J
,
Osborne
V
,
Kowalski
A
,
Prosperi
M.
Reported adverse events with painkillers: data mining of the us food and drug administration adverse events reporting system
.
Drug Saf
2018
;
41
(
3
):
313
20
.

13

Strain
E
,
Saxon
AJ
,
Hermann
R.
Opioid use disorder: Epidemiology, pharmacology, clinical manifestations, course, screening, assessment, and diagnosis
.
UpToDate
2019
. https://www.uptodate.com/contents/opioid-use-disorder-epidemiology-pharmacology-clinical-manifestations-course-screening-assessment-and-diagnosis

14

Brown
EG
,
Wood
L
,
Wood
S.
The medical dictionary for regulatory activities (MedDRA)
.
Drug Safety
1999
;
20
(
2
):
109
17
.

15

Wishart
DS
,
Feunang
YD
,
Guo
AC
, et al. .
DrugBank 5.0: a major update to the DrugBank database for 2018
.
Nucleic Acids Res
2018
;
46
(
D1
):
D1074
D1082
.

16

Stoltzfus
JC.
Logistic regression: a brief primer
.
Acad Emerg Med
2011
;
18
(
10
):
1099
104
.

17

Read
J
,
Pfahringer
B
,
Holmes
G
,
Frank
E.
Classifier Chains for Multi-label Classification. In:
Buntine
W
,
Grobelnik
M
,
MlAEnić
D
,
Shawe-Taylor
J
, eds.
BT—Machine Learning and Knowledge Discovery in Databases
.
Berlin Heidelberg
:
Springer
;
2009
:
254
269
.

18

Read
J
,
Pfahringer
B
,
Holmes
G
,
Frank
E.
Classifier chains for multi-label classification
.
Mach Learn
2011
;
85
(
3
):
333
59
.

19

Ishwaran
H
,
Kogalur
UB
,
Chen
X
,
Minn
AJ.
Random survival forests for high-dimensional data
.
Stat Anal Data Min
2011
;
4
(
1
):
115
32
.

20

Probst
P
,
Au
Q
,
Casalicchio
G
,
Stachl
C
,
Bischl
B.
Multilabel classification with R package mlr;2017. https://arxiv.org/abs/1703.08991.

21

Ishwaran
H
, Kogalur U Fast Unified Random Forests for Survival, Regression, and Classification (RF-SRC). R package version 2.9.2;2019. https://cran.r-project.org/package=randomForestSRC.

22

Miller
J
,
Chan
BKS
,
Nelson
HD.
Postmenopausal estrogen replacement and risk for venous thromboembolism: a systematic review and meta-analysis for the U.S. Preventive Services Task Force
.
Ann Intern Med
2002
;
136
(
9
):
680
90
.

23

Pourfarziani
V
,
Panahi
Y
,
Assari
S
,
Moghani-Lankarani
M
,
Saadat
S-H.
Changing treatment protocol from azathioprine to mycophenolate mofetil: decrease in renal dysfunction, increase in infections
.
Transplant Proc
2007
;
39
(
4
):
1237
40
.

24

Lyons
PG
,
Snyder
A
,
Sokol
S
, et al. .
Association between opioid and benzodiazepine use and clinical deterioration in ward patients
.
J Hosp Med
2017
;
12
(
6
):
428
34
.

Author notes

Jinzhao Chen and Gaoyu Wu contributed equally to this work.

This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/), which permits non-commercial re-use, distribution, and reproduction in any medium, provided the original work is properly cited. For commercial re-use, please contact [email protected]