An update to the HIV-TRePS system: the development and evaluation of new global and local computational models to predict HIV treatment outcomes, with or without a genotype

Abstract

Introduction

The story of ART for HIV infection is one of success and continuing progress, in terms of the number and nature of the inhibitors available, the long-term suppression of the virus they offer in combination and the roll-out of therapy to low- and middle-income countries (LMICs).

There are now approaching 30 antiretroviral drugs licensed that are used in combination therapy to fully suppress virus replication and prevent HIV disease progression.^1–3 However, HIV mutates readily, particularly when patient adherence with drug therapy is poor. This promotes increased viral replication and when treatment fails it is often associated with the development of viral drug resistance. The selection of the next combination of drugs is then critical to overcome any resistance that has emerged and to re-establish long-lasting viral suppression.

In well-resourced countries, an individualized decision is made for each patient based on a range of individual patient data, usually including an expert analysis of the treatment history and the results of one or more genotypic resistance tests.^1–3 However, the resistance tests in common use have a number of shortcomings and are only moderately (typically 60%–65%) predictive of response to treatment.⁴

The challenge of optimizing drug selection in LMICs is even greater as resistance tests are typically unavailable or unaffordable and there is a more limited range of drug options.⁵ In the absence of routine viral load monitoring in many settings, therapy failure is often detected late and regimen switch decisions are based on standard protocols rather than optimized for individuals. The result can be suboptimal regimen selection, failure to achieve resuppression of the virus and selection of additional drug resistance, which may limit future therapeutic options and can be transmitted to others.⁶

Over the past 10 years, the HIV Resistance Response Database Initiative (RDI) has collected biological, clinical and treatment outcome data for ∼190 000 HIV-1 patients from multiple sources around the world. From these data, we have used machine learning to develop models to predict HIV-1 treatment outcomes and to identify optimal, individualized therapies.^7–11 The data used include a range of immunological and virological measurements, the treatment history, the drugs in the new regimen and the output measure of treatment response: the amount of virus in the bloodstream (viral load) under the new treatment. We have developed models that use information from genotypic resistance tests in their predictions and others that do not, for use in LMICS where such tests are not generally available. Previous models tested with independent test sets predicted virological response with an overall accuracy of 80% with a genotype and 74% without and the area under the receiver-operating characteristic (ROC) curve (AUC) was 0.83 and 0.80, respectively (A. D. Revell, D. Wang and B. A. Larder, unpublished results).¹¹

The models are used to power an online treatment decision support tool, the HIV Treatment Response Prediction System (HIV-TRePS). In order to keep this system as current as possible in terms of the inclusion of new drugs and reflection of current clinical practice, it is essential that new models are developed, including the latest data available, on a fairly frequent basis. Here, we report on the development of three new sets of random forest (RF) models that estimate the probability of combinations of antiretroviral drugs reducing the plasma viral load to undetectable (<50 copies HIV RNA/mL): (i) global models that do not require a genotype for their predictions of treatment response (global no-genotype models), trained using a large global dataset and intended for use in LMICS without access to genotyping; (ii) local South African models that do not require a genotype (South African no-genotype models) trained using a subset of the above data from patients treated in South Africa; and (iii) global models that use a viral genotype in their predictions (global genotype models), trained using a large global dataset including this information.

The global and South African no-genotype models (i and ii above) enabled us to make a comparison as to whether a global or local modelling solution might be more effective for an individual, resource-constrained country with a substantial HIV epidemic. The accuracy of all the models was ascertained and they were evaluated as potential tools to support optimized, individualized treatment decision-making. This paper represents the update alluded to in our previous publication of modelling.¹¹

Methods

Clinical data

The package of data used for training the models (collected from episodes when ART is changed, for whatever reason) is termed a treatment change episode (TCE).⁷ TCEs were extracted from the RDI database for development of the no-genotype models that had all the following data available (with no missing values) from cases of virological failure: on-treatment baseline plasma viral load (sample taken ≤8 weeks prior to treatment change); on-treatment baseline CD4 cell count (≤12 weeks prior to treatment change); baseline regimen (the drugs the patient was taking prior to the change); ART history; drugs in the new regimen; a follow-up plasma viral load determination taken between 4 and 52 weeks following introduction of the new regimen; and the time to that follow-up viral load (in order that the models can be trained to predict responses at different times). A similar extraction of TCEs was subsequently performed for TCEs that also had an on-treatment genotype [protease and reverse transcriptase (RT) sequence obtained ≤12 weeks prior to treatment change] for the development of the genotype models.

The TCEs were censored using rules established in previous studies and published in detail elsewhere.¹² For example, TCEs involving drugs not adequately represented in the database (<500 cases) were excluded, in this case tipranavir, maraviroc and rilpivirine.

Data partition

The qualifying TCEs were partitioned at random using methods described elsewhere.^8,12 The following datasets resulted (Figure 1). For the global no-genotype models, a training set of 29 574 TCEs and an independent test set of 1700 TCEs from 1700 patients not represented in the training set were obtained. Of these, 3179 training cases and 222 test cases were from South Africa and were used to train and test the ‘local’ South African no-genotype models.

Figure 1.

Schematic of data extraction, partition, model training and testing. SA, South African.

Open in new tabDownload slide

The data that included baseline genotypes were partitioned into a training set of 15 130 TCEs and 750 independent test cases for the development and testing of the global genotype models. Partitioning was constrained in order to give a minimum number of 50 test cases including the latest inhibitor to be modelled (elvitegravir).

Computational model development

The three training sets of TCEs were used to train three committees, each of 10 RF models (Figure 1). Each was trained to predict the probability of the follow-up viral load being <50 copies/mL, using methodology described in detail elsewhere.^3,6 The following 43 input variables were used for the development of the global and South African no-genotype models: baseline viral load (log₁₀ copies HIV RNA/mL); baseline CD4 count (cells/mm³); treatment history comprising 20 binary variables coding for any experience of zidovudine, didanosine, stavudine, abacavir, lamivudine, emtricitabine, tenofovir disoproxil fumarate, efavirenz, nevirapine, etravirine, indinavir, nelfinavir, saquinavir, amprenavir, fosamprenavir, lopinavir, atazanavir, darunavir, enfuvirtide and raltegravir; antiretroviral drugs in the new regimen, covering the same 20 drugs; and time from treatment change to the follow-up viral load (number of days). The output variable was the follow-up viral load coded as a binary variable: ≤1.7 log or 50 copies/mL = 1 (response) and >1.7 log = 0 (failure).

The genotype models used the above variables plus, for the first time, tipranavir as a historical drug and elvitegravir in the new regimen (because these models were developed later and sufficient data were available to include these drugs). In addition, the following 62 mutations, detected in a genotype obtained while on the previous therapy no more than 84 days before the initiation of the new regimen, were used: HIV RT mutations (33)—M41L, E44D, A62V, K65R, D67N, 69 insert, T69D/N, K70R, L74V, V75I, F77L, V90I, A98G, L100I, L101I/E/P, K103N, V106A/M, V106I, V108I, Y115F, F116Y, V118I, E138A/G/K, Q151M, V179D/F/T, Y181C/I/V, M184V/I, Y188C/L/H, G190S/A, L210W, T215F/Y, K219Q/E and P236L; and HIV protease mutations (29)—L10F/I/R/V, V11I, K20M/R, L24I, D30N, V32I, L33F, M36I, M36L/V, M46I/L, I47V, G48V, I50V, I50L, F53L, I54 (any change), Q58E, L63P, A71 (any change), G73 (any change), T74P, L76V, V77I, V82A/F/S, V82T, I84V/A/C, N88D/S, L89V and L90M. The mutations were selected on the basis of the IAS-USA mutation list and previous modelling studies.¹³

Validation and independent testing

Each of the three committees of 10 RF models (global no-genotype, South African no-genotype and global genotype) was developed using a 10× cross-validation scheme.^8,12 For each partition, the model's estimates of the probability of response for the validation cases was compared with the actual response observed in the clinic (binary response variable: response versus failure).

The performance of the three sets of models as predictors of virological response was then evaluated using the independent test cases partitioned from the pool of available TCEs at random prior to training. The models' estimates of the probability of response and the responses observed in the clinics for the independent test cases were used to plot ROC curves and assess the AUC. In addition, the optimum operating point (OOP) for the models was derived during cross-validation and used to obtain the overall accuracy, sensitivity and specificity of the models.

Comparison of the accuracy of the models versus rules-based interpretation of the genotype

Genotypic sensitivity scores (GSS) were obtained for those test cases with genotypes that could be fully interpreted (i.e. all the drugs in the regimen were covered) by three rules-based genotype interpretation systems in common use: ANRS, REGA and Stanford HIVdb. The three systems were accessed online on 28 October 2015 and the GSS calculated by adding the score for each drug in the regimen, as described previously.¹² The GSS were then used as a predictor of response to the regimens and ROC curves calculated, as described for the models, and the performance characteristics compared with those of the global models, both genotype and no-genotype.

In silico analysis to evaluate the potential of the models to help avoid treatment failure

In order to evaluate further the potential clinical utility of the models, we assessed their ability to identify alternative, practical regimens that were predicted to be effective (probability of virological response above the OOP) or more likely to be effective than the regimens actually introduced in the clinic. For the no-genotype models, intended for use in LMICs, a list of three-drug regimens comprising only those drugs represented in the South African data on or before the start date of the new regimen in the clinic, was identified from the RDI database. We entered each of the South African test cases through both the global and local no-genotype models and obtained the models' predictions of response for each regimen on the list, as well as for the new regimen used in the clinic.

For the global genotype models, a list of three-, four- and five-drug regimens was identified from the RDI database. A global set of independent test cases where the new regimen comprised three or more drugs was entered into the models and predictions obtained for the regimens from the drug list that had no more drugs than the regimen used in the clinic.

Results

Characteristics of the datasets

The baseline, treatment and response characteristics of the datasets are summarized in Tables 1 and 2. For the no-genotype datasets, as would be expected, the gender balance is very different in the global datasets and the South African datasets, with the latter having a fairly even split between males and females. The global datasets were drawn from a wide range of institutions and countries in Europe, North America, South America, sub-Saharan Africa and South and South-East Asia.

The datasets have similar baseline viral loads, but the South African patients have substantially lower CD4 counts at baseline than the global set (median of 209 versus 264 for the training datasets). The South African cases were mostly cases moving from first to second line with three drugs in their history, whereas the median number of previous drugs for the global data was four. Thirty-five percent of the global training cases were switching to two NNRTIs and a PI compared with 49% of the South African cases. Twenty-two percent of global cases were switching to two NRTIs and an NNRTI compared with 44% of the South African cases.

The baseline characteristics of the global genotype model data are summarized in Table 2. They resembled those of the global no-genotype data, with the exceptions that the median number of previous drugs was higher, at five versus four, the median baseline viral load was a little higher at 4.13 versus 3.8 log₁₀ copies/mL HIV RNA and median CD4 counts were lower at 218 versus 264 cells/mm³.

Results of the modelling without a genotype

The performance characteristics from the ROC curves of the two sets of 10 individual models during cross-validation and independent testing are summarized in Table 3. The 10 global no-genotype models achieved AUC values during cross-validation ranging from 0.81 to 0.85, with a mean of 0.83. The overall accuracy ranged from 75% to 78% (mean = 76%), the sensitivity ranged from 70% to 72% (mean = 71%) and the specificity from 78% to 83% (mean = 80%). The OOP was 0.44.

The local South African no-genotype models achieved AUC values during cross-validation ranging from 0.73 to 0.90 with a mean of 0.80. The overall accuracy ranged from 71% to 85% (mean = 76%), the sensitivity ranged from 62% to 86% (mean = 70%) and the specificity ranged from 73% to 84% (mean = 79%).

Independent testing of global models

The committee of 10 global no-genotype models achieved an AUC of 0.82 in independent testing with the global test set of 1700 TCEs. The overall accuracy was 74%, the sensitivity 69% and the specificity 77%. The results are listed in Table 3 and the ROC curve for the committee is presented in Figure 2. When tested with the independent set of 222 local South African TCEs, the global models achieved an AUC of 0.79. The overall accuracy was 73%, the sensitivity 68% and the specificity 76%.

Figure 2.

ROC curves for the global and local (South African) models in independent testing with global and local (South African) test data.

Open in new tabDownload slide

Independent testing of the South African no-genotype models

The committee of 10 local South African no-genotype models achieved an AUC of 0.79 when tested with the local South African test cases, identical to that of the global models. The overall accuracy was 74%, the sensitivity 64% and the specificity 79%. The results are listed in Table 3 and the ROC curve for the committee is presented in Figure 2. When tested with the global independent set of 1700 TCEs, the local models achieved an AUC of 0.70. The overall accuracy was 66%, the sensitivity 48% and the specificity 78%. The accuracy of the two sets of models was not significantly different for the South African test cases, but the global models were significantly more accurate for the global test set (P < 0.0001).

Comparing the predictive accuracy of the global no-genotype models versus genotyping

Of the 1700 TCEs in the global no-genotype test set, 680 had genotypes available that were suitable for full interpretation using the three rules-based genotype interpretation systems. The AUC values for the GSS obtained using the three systems were 0.57 (ANRS), 0.57 (Rega) and 0.58 (Stanford HIVdb) (Table 4). All were significantly less accurate predictors of virological response than the global no-genotype models, which achieved an AUC of 0.83 for these cases (P < 0.0001).

Results of the modelling with a genotype

The performance characteristics from the ROC curves of the 10 individual models during cross-validation and independent testing are summarized in Table 5. The 10 global genotype models achieved AUC values during cross-validation ranging from 0.85 to 0.89, with a mean of 0.87. The overall accuracy ranged from 76% to 80% (mean = 79%), the sensitivity ranged from 78% to 84% (mean = 79%) and the specificity from 75% to 82% (mean = 79%). The OOP was 0.38.

Independent testing

The committee of 10 global genotype models achieved an AUC of 0.84 in independent testing with a global test set of 750 TCEs. The overall accuracy was 76%, the sensitivity 79% and the specificity 74%. The results are listed in Table 4. When tested with the 50 independent test cases involving elvitegravir in the new regimen, the models achieved an AUC of 0.82 and overall accuracy of 76%.

Comparing the predictive accuracy of the global genotype models versus genotyping

Of the 750 TCEs in the global test set, 601 had genotypes available that were suitable for full interpretation using the three rules-based genotype interpretation systems. The AUC values for the GSS obtained using the three systems were 0.57 (ANRS), 0.55 (REGA) and 0.58 (Stanford HIVdb) (Table 4). All were significantly less accurate predictors of virological response than the global genotype models, which achieved an AUC of 0.82 for these cases (P < 0.0001).

In silico analysis

The global and South African no-genotype models were then used to identify alternative regimens to the new regimens introduced in the South African clinics, comprising locally available drugs, with the highest estimated probability of virological response. The results are presented in Table 6.

The global models were able to identify alternative regimens that were predicted to be effective for 74% of the South African cases. This was significantly higher than the 64% for the local South African models (P = 0.03). The global models were able to identify alternative regimens with a higher probability of response for all 222 South African test cases and the local South African models for 220 (99%). The difference in probability of response between the alternative regimen with the highest probability of response and the regimen used in the clinic was significantly higher for the global models at 0.18 than the local models at 0.14 (P < 0.001).

Turning to the 141 South African cases that experienced virological failure following their change of regimen in the clinic, the global no-genotype models were able to identify alternative regimens that were predicted to be effective for 62% and the local South African no-genotype models for 50% (P = 0.04). The global models were able to identify alternative regimens with a higher probability of response for all 141 failures and the local South African models for 140 (99%). The difference in probability of response between the alternative regimen with the highest probability of response and the regimen used in the clinic was significantly higher for the global models at 0.20 than the local models at 0.15 (P = 0.007).

The global genotype models identified alternative regimens with no more drugs than the new regimen used in the clinic that were predicted to give a response for 636 (96%) of the 664 global test cases that were used in the analysis. They identified regimens with a higher probability of response for 662 cases (99.7%). For the 424 patients who experienced virological failure following the change to a new regimen in the clinic, the models identified alternative regimens that were predicted to give a response for 401 (95%) and with a higher probability of response than the regimen used in the clinic for all 424 cases.

Discussion

These latest computational models are the most accurate predictors of virological response to combination ART developed to date. Both sets of global models achieved AUC values well in excess of 0.80 in cross-validation and independent testing and there was no significant difference in accuracy between those models that use a genotype and those that do not. The results replicated and reinforced previous findings that computational models, even those that do not use a genotype in their predictions, are substantially more accurate predictors of virological response to combination ART than viral genotyping with rules-based interpretation.¹¹

The results indicate that models developed using large global datasets can perform as accurately as models developed using more limited data from a single setting in predicting responses to HIV therapy for patients from that setting. Moreover, the global models are significantly better able to model the responses to alternative combinations of drugs and identify options with a higher probability of virological response than the regimens used in the clinic: they identified viable alternatives for more cases and with greater improvements in the probability of response than the local models.

The broad range of settings and countries represented in the data used in this study suggests that these findings and the potential utility of these models are generalizable and applicable to all settings. These results taken together suggest that a strategy of developing models using the largest most heterogeneous dataset possible for global use would be superior to one of developing national models using smaller, less heterogeneous datasets.

It is important to note that one of the input variables for these models was the plasma viral load, which previous studies have shown to be very important to the predictive accuracy of the models.¹⁴ Although viral load monitoring is not routine in most resource-limited settings, it is now recommended as the preferred approach to monitoring ART success and diagnosing treatment failure in the latest WHO guidelines.¹⁵ As technological advances enable lower test costs and simpler equipment requiring less infrastructure, maintenance and technical expertise, so the use of viral load in clinical practice is likely to increase.¹⁶ A corollary of this is that use of CD4 counts may become less widespread in such settings, preventing the use of the current models. If this transpires, models could be developed that do not require CD4 counts for their predictions, although results of preliminary studies conducted by the RDI in 2003–04 indicated that the inclusion of CD4 counts contributed to model accuracy (B. A. Larder, D. Wang and A. D. Revell, unpublished results).

Another potential avenue for future research is the development of models that use the duration of previous drug exposures as input variables, instead of the current binary variables. While only a subset of the RDI database includes complete information on the duration of previous therapies, as the database expands this pool may be sufficient for the development and evaluation of models.

The study has some limitations. First, it was retrospective and, as such, no firm claims can be made for the clinical benefit that use of the system as a treatment support tool could provide. Another possible shortcoming, inherent in such studies and discussed in previous publications, is that the cases used are, by definition, those with complete data around a change of therapy and, therefore, may not be truly representative of the general patient population and clinical practice as a whole. Nevertheless, the size of the training dataset, the variety of sources and settings from which the data were collected and the range of clinical practice represented are positive factors in considering the potential robustness and generalizability of the models' performance.

Conclusions

Computational models developed using large global datasets can be as accurate for a patient in a given setting as models developed using data only from that setting and are better able to identify alternatives with a higher probability of virological response than the regimens otherwise used in the clinic.

Global models developed using large heterogeneous clinical datasets are significantly better predictors of response to HIV therapy than genotyping with rules-based interpretation, even when those models do not use a genotype for their predictions. Since use of these models is free of charge, this suggests that scarce funds in LMICs would be better spent on antiretroviral drugs and viral load testing than on genotyping. This would enable a greater range of treatments to be offered, failure to be detected early, and optimal individualized treatment change decisions made using the models.

Full validation of this approach as a clinical tool would require a prospective, controlled clinical trial. Nevertheless, the results suggest that the global models have the potential to reduce virological failure and improve patient outcomes in all parts of the world, with particular utility in resource-limited settings. The models can provide clinicians with a practical tool to support optimized treatment decision-making in the absence of resistance tests and where expertise may be lacking in the context of a public health approach to antiretroviral roll-out and management.

The global models described in this paper are freely available to use online through the HIV-TRePS system at http://www.hivrdi.org/treps.

Funding

This project has been funded in whole or in part with federal funds from the National Cancer Institute, National Institutes of Health, under Contract No. HHSN261200800001E. This research was supported by the National Institute of Allergy and Infectious Diseases.

Transparency declarations

None to declare.

Disclaimer

The content of this publication does not necessarily reflect the views or policies of the Department of Health and Human Services, and mention of trade names, commercial products or organizations does not imply endorsement by the US Government.

Acknowledgements

RDI Data and Study Group

The RDI wishes to thank all the following individuals and institutions for providing the data used in training and testing its models.

Cohorts

Peter Reiss and Ard van Sighem (ATHENA, the Netherlands); Julio Montaner and Richard Harrigan (BC Center for Excellence in HIV & AIDS, Canada); Tobias Rinke de Wit, Raph Hamers and Kim Sigaloff (PASER-M cohort, the Netherlands); Brian Agan, Vincent Marconi and Scott Wegner (US Department of Defense); Wataru Sugiura (National Institute of Health, Japan); Maurizio Zazzi (MASTER, Italy); Rolf Kaiser and Eugen Schuelter (Arevir Cohort, Köln, Germany); Adrian Streinu-Cercel (National Institute of Infectious Diseases ‘Prof. Dr. Matei Balş’, Bucharest, Romania); and Gerardo Alvarez-Uria (VFHCS, India).

Clinics

Jose Gatell and Elisa Lazzari (University Hospital, Barcelona, Spain); Brian Gazzard, Mark Nelson, Anton Pozniak and Sundhiya Mandalia (Chelsea and Westminster Hospital, London, UK); Daniel Webster and Colette Smith (Royal Free Hospital, London, UK); Lidia Ruiz and Bonaventura Clotet (Fundacion Irsi Caixa, Badelona, Spain); Schlomo Staszewski (Hospital of the Johann Wolfgang Goethe-University, Frankfurt, Germany); Carlo Torti (University of Brescia); Cliff Lane and Julie Metcalf (NIH Clinic, Rockville, MD, USA); Maria-Jesus Perez-Elias (Instituto Ramón y Cajal de Investigación Sanitaria, Madrid, Spain); Stefano Vella and Gabrielle Dettorre (Sapienza University, Rome, Italy); Andrew Carr, Richard Norris and Karl Hesse (Immunology B Ambulatory Care Service, St Vincent's Hospital, Sydney, NSW, Australia); Dr Emanuel Vlahakis (Taylor's Square Private Clinic, Darlinghurst, NSW, Australia); Hugo Tempelman and Roos Barth (Ndlovu Care Group, Elandsdoorn, South Africa); Carl Morrow and Robin Wood (Desmond Tutu HIV Centre, University of Cape Town, South Africa); Chris Hoffmann (Arum Institute, Johannesburg, South Africa and Johns Hopkins University, Boston, MA, USA); Luminita Ene (‘Dr. Victor Babes’ Hospital for Infectious and Tropical Diseases, Bucharest, Romania); Gordana Dragovic (University of Belgrade, Belgrade, Serbia); Ricardo Diaz and Cecilia Sucupira (Federal University of Sao Paulo, Sao Paulo, Brazil); Omar Sued and Carina Cesar (Fundación Huésped, Buenos Aires, Argentina); and Juan Sierra Madero (Instituto Nacional de Ciencias Medicas y Nutricion Salvador Zubiran, Mexico City, Mexico).

Clinical trials

Sean Emery and David Cooper (CREST); Carlo Torti (GenPherex); John Baxter (GART, MDR); Laura Monno and Carlo Torti (PhenGen); Jose Gatell and Bonventura Clotet (HAVANA); Gaston Picchio and Marie-Pierre deBethune (DUET 1 & 2 and POWER 3); Maria-Jesus Perez-Elias (RealVirfen); and Sean Emery, Paul Khabo and Lotty Ledwaba (PHIDISA).

References

Author notes