Modeling of randomized hepatitis C vaccine trials: Bridging the gap between controlled human infection models and real-word testing

Abstract

Global elimination of chronic hepatitis C (CHC) remains difficult without an effective vaccine. Since injection drug use is the leading cause of hepatitis C virus (HCV) transmission in Western Europe and North America, people who inject drugs (PWID) are an important population for testing HCV vaccine effectiveness in randomized-clinical trials (RCTs). However, RCTs in PWID are inherently challenging. To accelerate vaccine development, controlled human infection (CHI) models have been suggested as a means to identify effective vaccines. To bridge the gap between CHI models and real-world testing, we developed an agent-based model simulating a two-dose vaccine to prevent CHC in PWID, representing 32,000 PWID in metropolitan Chicago and accounting for networks and HCV infections. We ran 500 trial simulations under 50 and 75% assumed vaccine efficacy (aVE) and sampled HCV infection status of recruited in silico PWID. The mean estimated vaccine efficacy (eVE) for 50 and 75% aVE was 48% (SD ± 12) and 72% (SD ± 11), respectively. For both conditions, the majority of trials (∼71%) resulted in eVEs within 1 SD of the mean, demonstrating a robust trial design. Trials that resulted in eVEs >1 SD from the mean (lowest eVEs of 3 and 35% for 50 and 75% aVE, respectively), were more likely to have imbalances in acute infection rates across trial arms. Modeling indicates robust trial design and high success rates of finding vaccines to be effective in real-life trials in PWID. However, with less effective vaccines (aVEs∼50%) there remains a higher risk of concluding poor vaccine efficacy due to post-randomization imbalances.

Introduction

Chronic hepatitis C virus (CHC) is associated with significant morbidity and mortality in the United States (1) and around the world (2). Syringe sharing among people who inject drugs (PWID) is the primary mode of hepatitis C virus (HCV) transmission in North America and Western Europe (3), with an estimated 67% of all HCV infections attributable to injection drug use in the United States in 2019 (1). The introduction of highly effective direct-acting antivirals (DAAs) paved the way to the ambitious World Health Organization (WHO) goal of viral hepatitis elimination by 2030 (4). However, deficits in the screening and diagnosis and linkage to care, high costs, restricted access, the ongoing opioid epidemic that continues to fuel the HCV epidemic, and significant HCV reinfection rates among PWID have curtailed the impact of this strategy (5, 6). In addition, DAA therapy does not lead to protective immunity, leaving treated people susceptible to reinfection (7). Given these conditions, a safe and efficacious vaccine is an important component for a successful strategy to eliminate HCV (8). It should be noted that effective action plans have been introduced in several countries through price negotiations, community engagement, and education programs (9–11), and there is evidence of success in reaching the WHO elimination goal (12). The United States is still behind on elimination targets, partly due to funding, the absence of a national registry of HCV-positive people and systems to track those treated (9). Overall, the WHO elimination targets continue to be debated in most countries of the world, regardless of income level (13), highlighting the importance of other prevention measures, which would include vaccines.

The recently reported randomized-clinical trial (RCT) was the first phase I/II trial of a prophylactic vaccine for reducing CHC in PWID (14). However, the trial did not result in lower levels of CHC in immunized subjects and additional clinical trials will be needed. Since RCTs in PWID are challenging, long, and expensive to perform, it is imperative to identify good vaccine candidates through experimental studies that would include the qualitative analysis of immune responses induced in animal models and phase 1 studies coupled with theoretical studies using ABMs that can contribute to the design of robust RCTs. Recent discussions to examine candidate vaccines using a controlled human infection (CHI) model (15) could potentially identify vaccines with high efficacy, as previously done for malaria vaccines (16), to be tested in RCTs in PWID. However, there are no theoretical tools to simulate the effect of variables on vaccine outcome, such as post-randomization differences or the frequency of HCV testing post-vaccination.

HCV vaccine development efforts are focused on producing a prophylactic vaccine that would induce immune mediated clearance, preventing CHC, rather than eliciting sterilizing immunity (17–19). This is an acceptable public health goal as only ∼25% of primary infections result in spontaneous clearance (20) and morbidity and mortality are associated with CHC, rather than acute, infection (21). We previously showed through mathematical modeling that even with transient viral replication in vaccines after exposure through injection drug use, HCV transmission among people sharing syringes could be reduced through vaccination (17). Modeling studies have indicated that targeted vaccination (e.g. risk-targeted or sero-targeted), with vaccines of only moderate efficacy (<60%), especially in combination with treatment, could have a significant impact on transmission and CHC rates (22–24).

Agent-based models (ABM) can simulate HCV epidemics among the PWID population (25–27) and have the potential to simulate HCV vaccine trials. In this study, we report the development of an ABM to perform in silico simulations of HCV vaccine RCTs for a two-dose vaccine designed to prevent CHC using various assumed vaccine efficacies (aVE) to evaluate the impact of two parameters, viral sampling and imbalanced acute infection rates between trial arms, on vaccine success. We hypothesized that increased monitoring of clinical trial recruits has the potential to improve detection rates, provide outcomes closer to the aVE and possibly allow reduced sampling sizes. However, monitoring of PWID in clinical trials is challenging and costly, therefore, understanding if increasing the frequency will improve results is important to determine using an ABM before applying it to real-world studies. Post-randomization events, such as loss-to-follow-up or changes in behavior, are an important consideration of vaccine trials and can have an impact on outcomes (28). Imbalances in infection rates between the vaccine and placebo groups is a post-randomization event that can also impact outcomes but has not been widely studied, therefore we wished to analyze this in the context of our in silico trials. Our study utilizes extensive field data on Chicago-area PWID to construct an in silico PWID population and accounts for geographic distribution, injection behaviors, syringe sharing network, and new HCV infections, as well as details of the trial such as regular HCV testing, loss to follow-up and statistical analysis. We ran 500 trial simulations under conditions of 50 and 75% aVE. These aVEs were chosen to represent a vaccine with efficacy that would be considered suitable for an RCT (75%) and an efficacy that may be the minimum acceptable for a vaccine to progress to an RCT (50%).

Results

Simulation outcomes indicate a robust trial design

For clinical trial simulations, we set aVE at 50 and 75% by increasing the probability of viral clearance for vaccinated agents. For the placebo group, the probability of viral clearance was assumed different between women and men with average 34.6% (range 30–40%) and 12.1% (range 10–14%) rates of spontaneous clearance, respectively (Table S3). Just as in a real-world clinical trials, in the in silico trials we sampled HCV infection status of each enrolled PWID monthly. We ran 500 trial simulations under each condition, with different random number seeds. For each trial, we computed outcome measures including the incidence rate ratio (IRR) of CHC under intention to treat (ITT), IRR of CHC among only those in the clinical trial that were infected, hazard ratio (HR) for CHC (Cox proportional hazards model), and HR for time to viral clearance among cases that resolved. Summary statistics, including estimated vaccine efficacy (eVE), for the 500 simulations under each aVE condition are shown in Table 1. As expected from the model design, HCV incidence (3.8–3.97/100 person years [py]) and prevalence (5.82–6.06%) were similar across simulation conditions and treatment arms, while CHC incidence and prevalence were lower in the study arm compared with the placebo arm in both simulation conditions. The average duration of infection among cases that resolved (non-CHC) was about 15 months and did not differ between conditions, also as expected from the software design. Under aVE 50%, the average eVE based on the IRR under ITT was 0.48 (95% CI 0.47–0.49) (Table 1), while under aVE 75%, the average eVE was 0.72 (95% CI 0.71–0.73) (Table 1). The analysis of only infected cases tended to overestimate the VE (shown as eVE among infected in Table 1) (eVE 0.57 and 0.78 for aVE 50 and 75%, respectively) while Cox regression (Cox eVE) yielded estimates very similar to IRR estimates (eVE 0.49 and 0.72 for aVE 50 and 75%, respectively) (Table 1).

Trial simulations demonstrate the risk of low eVE outcomes

The distribution of the eVE in the 500 trial simulations is shown for monthly testing with both aVE conditions in Figs. 1A (aVE 75%) and B (aVE 50%) and descriptive statistics for the simulations are shown in Table 2. The distribution was moderately negatively skewed for both conditions, but both sets of simulations were approximately normally distributed with SD of ∼10% (SD ± 12 and ± 11, for aVE of 50 and 75%, respectively) (Table 2). The majority of trials (∼71%) resulted in outcomes within 1 SD of the mean eVE (means of 48% for 50% aVE and 72% for 75% aVE) (Table 2). Despite this robustness in the trial outcomes, there were simulations with outcomes substantially higher than 1 SD from the mean. For aVE 50%, outcomes ranged from 3 to 75%, while for aVE 75%, outcomes ranged from 35 to 96% (Fig. 1 and Table 2).

We tested the impact of increasing sampling from monthly to biweekly on the distribution of outcomes for the aVE 50% group. With biweekly testing the results were very similar to those obtained with monthly testing (Fig. 1C and Table 2). The mean eVE was 0.486 (95% CI 0.475–0.496) (Table 2), although the distribution was slightly less negatively skewed (Fig. 1C), and 70% of trials were within 1 SD of the eVE mean. A similar range of outcomes for eVE (9 to 79%) (Table 2) was obtained when biweekly testing was applied compared with monthly testing.

Meta-regression analysis demonstrates imbalance in acute infection rates impact outcomes

We performed a meta-regression analysis to understand the cause of outcomes that deviated substantially from the eVE mean for both sets of conditions. The results showed that outcomes significantly higher or lower than the aVE were due to an imbalance in infection rates across arms. Figure 2 shows the relationship between the infection risk ratio (RR) for the study vs. placebo arm and predicted VE for both aVE conditions. The aVE is indicated by the solid blue line. Areas of statistically significant over- and under-estimation are located between the dashed blue lines (30 and 60% for aVE of 50% and 55 and 80% for aVE of 75%). An RR > 1.0 indicates fewer HCV infections in the placebo group compared with the study arm. As shown by the negative slope of the curves, as the RR increases, predicted VE decreases, indicating that VE is more likely to be underestimated. Under aVE 50%, with a 20% greater risk of infection in the study arm (RR = 1.20), the marginal predicted IRR increases from 0.51 (with RR = 1.0) to 0.62 and the predicted VE decreases from 49 to 38% (Fig. 2A). RRs between 0.75 and 1.33 (indicated by vertical lines in Fig. 2A) were not detected as significantly different from an RR of 1 (P < 0.05). The effect of imbalanced infections between the arms under aVE 75% was less pronounced, with predicted VE decreasing from 73% under ideal conditions (RR = 1.0) to 67% with RR = 1.20 (Fig. 2B). Conversely, as the acute infection RR decreased below 1.0 (more acute infections in the study arm vs. the control arm) the predicted VE increases above the aVE for both vaccine conditions (Fig. 2).

Discussion

Vaccines for HCV are critical to the WHO goal of elimination of viral hepatitis, particularly in challenging populations such as PWID (29). To inform the design of clinical trials to test these vaccines, our study performed 1,500 in silico simulations of vaccine RCTs. Such an analysis would not be feasible in the real-world environment and our model offers the opportunity to identify variables that may impact clinical trial outcome. Overall, our model produced estimates that ∼70% of trials under both simulation conditions resulted in outcomes within 1 SD (∼10%) of the aVE. The effect was also observed with biweekly instead of monthly HCV testing, indicating robust trial design and suggesting that under the vaccine parameters applied in this model the return on investment for increased frequency of testing is likely to be negligible.

We observed ∼30% of outcomes with eVE >1SD from the mean and as expected from a normal distribution 5% (aVE 50%) and 3.2% (aVE 75%) with outcomes >2SD. Of note, 5.8% of trial simulations run under aVE 50% resulted in outcomes with eVE <25%, with one as low as 3%, while under aVE 75% the lowest eVE was 35% (Table 2). Our modeling results demonstrated that hard-to-control random variation in HCV infections can account for a significant portion of these extreme outcomes, having a significant undesirable effect on the eVE in a vaccine RCT. It is assumed that this is occurring due to a greater number of infections becoming chronic in the vaccine arm, which results in a downward shift of the eVE, although this does not occur with every simulation. Hence a number of simulations do not show any decrease in eVE when the incidence RR is 1 (Fig. 2) and the variability explains the tails of the distribution seen for both simulation conditions (Fig. 1). A similar scenario could occur in real-world clinical studies, where, by chance, there is an increased number of infections in the vaccine arm, resulting in a larger proportion of chronic infections than might otherwise be expected, despite the immune response induced by the vaccine. As shown in our in silico studies, this would not occur in every clinical study, but our data indicate that any imbalance in infections between the vaccine and control arm should be carefully assessed for impact on outcome. Alternatively, an imbalance in infection rates favoring the vaccine arm could result in overestimating VE, resulting in failure to reject a candidate vaccine with limited efficacy.

Our results suggest that vaccine candidates identified in CHI studies with ≥75% VE will be more robust under post-randomization imbalances and have a higher chance of success in real-world studies. Importantly, large deviations from the aVE occurred even when the imbalance was not detected as significant, especially when the vaccine arm was disadvantaged. We propose that when acute infection rates are even moderately unbalanced across trial arms (e.g. P < 0.20), investigators should take steps to correct for bias, such as using Cox regression to adjust for variables associated with infection. An alternative measure, namely estimating the vaccine effect only among persons in the trial that were infected, could also be useful. Our data indicated that this approach resulted in a slight overestimate of efficacy for the population (Table 1). Despite this, it might be considered an appropriate method to measure efficacy of a vaccine to prevent CHC.

Our study's main limitation is that in silico methods cannot fully capture the complexity of PWID population and vaccine RCTs, and additional factors might affect the vaccine outcome in real-world trials. For example, the immune response induced by the vaccine may be fundamentally different in some individuals due to the extent of drug use, genetic diversity, or underlying conditions that are not recognized at the time of recruitment. We used data from the Chicago area PWID population, but we believe that our findings would generalize to other sites. Higher clearance rates have been reported in younger people (30), but we did not consider age in the current model. However, this could be incorporated into future models. We also did not consider the kinetics of the viral infection in vaccinated individuals (i.e. in host viral load) and did not assume vaccination would result in more rapid clearance of the virus, as demonstrated by similar times to clearance for both arms (Table 1). Data have shown that secondary infections in humans and chimpanzees that have spontaneously cleared an HCV infection are cleared more frequently and more rapidly and result in lower viral load (31–33). If a vaccine induces immune responses that reduce the duration of acute infection from 77 to 127 days to 8 to 48 days (Table S3), rapid clearance could be missed during monthly follow-up testing such that more frequent testing would be warranted. Our model has the capability to test this hypothesis using different patterns of viral load and viral clearance that have been previously demonstrated in reinfected subjects (32). In the long term, vaccine aspects such as possible waning of VE over time (34) could also be incorporated into our model and evaluated.

In summary, we have developed a unique ABM for studying RCTs for HCV vaccines in PWID. Overall, our results demonstrate that in silico simulations can provide an important tool to design RCTs for future vaccines and test parameters that may impact outcome. Specifically, for CHC, we showed that imbalances in infection rates across study arms threaten the accuracy of estimates of vaccine efficacy. Using simulation to explore the properties of various matching strategies could help to inform best practices for achieving balance in HCV vaccine trials.

Materials and methods

Incorporating HCV vaccine trial features in HepCEP

We previously developed the Hepatitis C Elimination in PWID (HepCEP) ABM model to simulate the PWID population in metropolitan Chicago, including their social interactions that result in HCV infection (25, 26, 35). Model features of HepCEP were previously described (25, 26, 35), and are briefly provided herein as follows: attributes of the PWID synthetic population (Table S1), parameters for the generation of the CNEP+ synthetic population (Table S2), geographical environment and syringe-sharing network (Fig. 3), HCV infection parameters (Table S3) and model validation (see Supplementary material). On the basis of this validated model, we developed a prototype ABM simulation that included our previous infection stage progression with enhancements for the vaccine trial simulations (Fig. 4) and design characteristics of a recent HCV vaccine trial (NCT01436357, Table 3) (14). The ABM represents an RCT of a two-dose vaccine (doses at 0 and 2 months) with a specified aVE in preventing CHC but no effect on acute HCV infections (i.e. does not induce sterilizing immunity) or on viral kinetics (i.e. does not result in rapid viral clearance or reduced acute-phase viral titers). The effect of the vaccine is introduced by increasing the probability of viral clearance for agents assigned to the study condition.

Enrollment into the trial occurred over 365 days and was simulated by sampling PWIDs from the in silico Chicago-area PWID population. To achieve a higher likelihood of exposure to HCV we recruited only PWID having at least one receptive syringe sharing injection partner and at least one injection per 30 days. All PWID aged 18 to 59 who tested negative for both HCV RNA and HCV antibodies were eligible for enrollment. To simulate refusal or loss of contact, 20% of the enrollees were dropped from the trial each year. The vaccine and placebo injections were given at 0 and 2 months (Table 3). After the second injection subjects entered the primary follow-up period of 18 months (to avoid calendar complexity, each month in the simulations refers to exactly 28 days). During this period, subjects were tested for HCV RNA and HCV antibodies every 4 weeks (or every 2 weeks in the biweekly schedule). If HCV RNA was detected subjects entered the secondary follow-up phase lasting 9 months to determine whether the infection became CHC or resolved. The trial simulation ended 1,400 days after the first enrollment and any active follow-ups were discontinued.

Vaccine trial simulations

The prototype model was implemented with the Repast Simphony modeling toolkit (2021-06 version), in which each trial simulation takes about 30 min. The Extreme-scale Model Exploration with Swift framework was used for running simulations in a high-performance computing environment (36).

Data collection and analysis

To achieve aVE = 50%, the probability of viral clearance in the vaccinated group was set to 60%; to achieve aVE = 75%, the probability of viral clearance in the vaccinated group was set to 80%. Just as in a real-world clinical trial, in the in silico trial we sampled HCV infection status of each enrolled PWID monthly. In an additional simulation of aVE 50% we adjusted the HCV testing schedule to biweekly (i.e. every 14 days). We ran 500 trial simulations under each condition, with different random number seeds. For each trial, we computed outcome measures including the IRR of CHC under ITT, IRR of CHC among only those in the clinical trial that were infected, HR for CHC (Cox proportional hazards model), and HR for time to viral clearance among cases that resolved. We then used meta-analysis to summarize the outcomes over the 500 simulated trials and conducted random effects meta-regression analyses to test the associations between CHC IRR outcome and HCV infection incidence rate and risk ratio between study arms. The outcome point estimates and confidence intervals were log-transformed for analysis. Analyses were conducted in Stata (StatCorp 17ed).

Supplementary Material

Supplementary material is available at PNAS Nexus online.

Funding

National Institutes of Health grant R01AI158666 (J.O., B.B., and H.D.). Intramural Food and Drug Administration funding to program Z01 BK 04010-18LHV, project 04774 (M.M.). The funders of the study had no role in study design, data analysis, interpretation of the results, writing the report, or in the decision to submit the paper for publication.

Author Contributions

Mary-Ellen Mackesy-Amiti (Formal Analysis, Visualization, Writing—original draft), Alexander Gutfraind (Formal Analysis, Methodology, Writing—original draft), Eric Tatara (Formal analysis, Visualization, Writing—original draft), Nicholson T. Collier (Formal Analysis), Scott J. Cotler (Writing—review and editing), Kimberly Page (Methodology, Writing—review and editing), Jonathan Ozik (Formal Analysis, Methodology, Writing—original draft), Basmattee Boodram (Methodology, Writing—original draft), Marian Major (Visualization, Methodology, Writing—original draft), Harel Dahari (Methodolgy, Writing—original draft).

Data Availability

CNEP+ data used in this study and the simulation software is freely available from https://github.com/sashagutfraind/apk/tree/hepcep. The output data of the trial simulations and statistical analyses are available from https://doi.org/10.17632/5x4kyzrvgw.1.

References

Author notes