Variations in Seasonal Influenza Vaccine Effectiveness due to Study Characteristics: A Systematic Review and Meta-analysis of Test-Negative Design Studies

Abstract

Background

Study characteristics influence vaccine effectiveness (VE) estimation. We examined the influence of some of these on seasonal influenza VE estimates from test-negative design (TND) studies.

Methods

We systematically searched bibliographic databases and websites for full-text publications of TND studies on VE against laboratory-confirmed seasonal influenza in outpatients after the 2009 pandemic influenza. We followed the Cochrane Handbook for Systematic Reviews of Interventions guidelines. We examined influence of source of vaccination information, respiratory specimen swab time, and covariate adjustment on VE. We calculated pooled adjusted VE against H1N1 and H3N2 influenza subtypes, influenza B, and all influenza using an inverse-variance random-effects model.

Results

We included 70 full-text articles. Pooled VE against H1N1 and H3N2 influenza subtypes, influenza B, and all influenza was higher for studies that used self-reported vaccination than for those that used medical records. Pooled VE was higher with respiratory specimen collection within ≤7 days vs ≤4 days of symptom onset, but the opposite was observed for H1N1. Pooled VE was higher for studies that adjusted for age but not for medical conditions compared with those that adjusted for both. There was, however, a lack of statistical significance in almost all differences in pooled VE between compared groups.

Conclusions

The available evidence is not strong enough to conclude that influenza VE from TND studies varies by source of vaccination information, respiratory specimen swab time, or adjustment for age/medical conditions. The evidence is, however, indicative that these factors ought to be considered while designing or evaluating TND studies of influenza VE.

Vaccination is the most effective prevention for seasonal influenza. Observational studies, rather than randomized controlled trials, are used to examine seasonal influenza vaccine effectiveness (VE) due to feasibility and ethical considerations. Continuous changes that occur in influenza viruses (antigenic drift) [1] mean that influenza vaccines have to be re-formulated every influenza season and that vaccine virus strains may be mismatched with circulating virus strains. Influenza VE studies are conducted each season in many jurisdictions worldwide to assess vaccine performance and to inform subsequent influenza season vaccine development.

Studies on influenza VE often have differences in their design. Studies approach participant recruitment differently, and influenza vaccination status may be determined by either self-report or medical record ascertained. Clinic presentation and timing of respiratory specimen swab collection differ across study participants. The characteristics of study participants, such as age and health status, also vary and may impact VE [2]. Adjustment in analysis of VE varies across studies, and adjustment for specific potential confounders such as age and medical conditions may lead to differences in VE estimations. Due to these variations and other factors, influenza VE estimates vary between jurisdictions.

The test-negative design (TND), an observational study design type, is an increasingly popular design for estimating influenza VE [3, 4]. In a TND study, patients presenting with influenza-like symptoms are tested for influenza. Those with a positive test result become the cases, and those with a negative test result become the controls. Influenza VE (represented as a percentage) is calculated as 1 minus the adjusted ratio of the odds of vaccination in those with positive test results to the odds of vaccination in those with negative test results, multiplied by 100. The TND has been credited with reducing biases due to differential health care–seeking behavior between vaccinated and unvaccinated individuals and differential misclassification of influenza infection status [3]. However, if stringent methods for study participants’ enrollment and influenza testing are not applied, the TND may fail to correct for differential health care–seeking behavior among vaccinated and unvaccinated individuals [5].

We systematically identified, critically appraised, and summarized the findings of published TND studies that examined seasonal influenza VE in primary care settings since the 2009 pandemic influenza. We conducted a systematic review and meta-analysis following the Cochrane Handbook for Systematic Reviews of Interventions guidelines [6], and we reported our findings following the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) guidelines [7].

METHODS

Search Strategy and Selection Criteria

We developed and registered a review protocol in the international prospective register of systematic reviews (PROSPERO) before commencement of this review (registration number CRD42017064595). We searched the MEDLINE (Ovid), Embase (Ovid), PubMed, Scopus (Elsevier), Web of Science, and Google Scholar bibliographic databases. Our literature search strategy (Supplementary Table 1) was reviewed by a knowledge synthesis librarian using the PRESS checklist [8]. The literature search was first conducted in April 2017 and updated in July 2018. Corresponding authors of regional influenza surveillance studies were contacted to check if our searches missed any relevant studies. Identified literature citations were imported and screened in a specially designed Microsoft Access 2016 database (Microsoft Corporation, Redmond, WA, USA).

We were interested in community-based TND studies conducted in primary care settings (outpatients) after the 2009 pandemic influenza (from influenza season 2010/2011 onwards). Only studies that reported multivariable-adjusted influenza VE estimates against laboratory-confirmed influenza of any type or subtype were considered for inclusion in the review. We included only studies with influenza confirmation based on reverse transcriptase polymerase chain reaction (RT-PCR) assay or viral culture of a respiratory specimen and only full-text study publications, irrespective of language of publication. We also included only studies in which patients deemed to have received influenza vaccination did so at least 14 days before their symptom onset, and their symptom onset must not have been >7 days before medical consultation, specimen collection, and study enrollment. Studies involving only hospitalized patients and studies that reported results from mixed hospitalized patients and outpatients without reporting separate results for the 2 patient groups were excluded. We also excluded studies based on retrospective analysis of respiratory samples obtained for clinical diagnostic testing. Furthermore, we excluded studies conducted in military barracks, prisons, care homes, schools, and in subgroups such as individuals with chronic diseases. The outcomes of our interest were adjusted influenza VE against the H1N1 and H3N2 influenza subtypes, influenza B, and all influenza. Two reviewers independently screened the identified citations against the eligibility criteria using a 2-stage sifting approach to review titles/abstracts and full-text articles. Disagreements during this process were resolved through discussion between the 2 reviewers or by involvement of a third reviewer. The number of ineligible citations at the title/abstract screening stage and both the number and reasons for ineligibility at the full-text article screening stage were documented.

Data Extraction

We extracted data in MS Excel 2016 (Microsoft Corporation, Redmond, WA, USA). One reviewer independently extracted data from the included articles, and a second reviewer independently checked the extracted data for errors. Disagreements during this process were resolved through discussion between the 2 reviewers or by involvement of a third reviewer. We extracted study details such as name of the first author, publication year, country, and funding source; study characteristics such as influenza season, participant recruitment strategy, number of participants, source of vaccination information, respiratory specimen swab time, influenza vaccine type, influenza diagnostic test, and the adjusted covariates in analysis; study outcome: influenza VE against the H1N1 and H3N2 influenza subtypes, influenza B, and all influenza; and study results: multivariable-adjusted influenza VE and associated 95% confidence interval (CI). Vaccine antigenic similarity with circulating virus strains was determined from articles, where reported. Where incidence of confirmed influenza was reported, we considered the season’s vaccine to be antigenically similar if the strain that caused a majority of the cases (at least 75%) was similar to that contained in the vaccine, antigenically partially similar if there was modest similarity with strains covered in the vaccine, and antigenically dissimilar if circulating strains were not similar to the strains covered in the vaccine.

Data Synthesis and Analysis

The main study characteristics were synthesized in tabular form. We pooled reported multivariable-adjusted influenza VE estimates and their associated 95% CIs using inverse-variance random-effects models implemented in STATA (version 13; StataCorp LP, Texas, USA). Heterogeneity between the pooled adjusted VE estimates was assessed and quantified statistically using the I² statistic [9]. The chi-square statistic (χ ²) was used to assess the statistical significance (P value) of the difference between 2 groups of pooled adjusted results. We assessed publication bias (where appropriate) visually using funnel plots and, statistically, using the Egger’s regression test [10]. Subgroup analysis was conducted according to the source of participants’ influenza vaccination status, respiratory specimen swab time, and whether studies included age or age and medical conditions in their multivariable adjustment models. Subgroup analyses were conducted for all patients, and for each of the following age groups: <5 years, 5 to 17 years, 18 to 49 years, 50 to 64 years, and ≥65 years. We included only results for age groups that clearly fell within these predefined age groups without overlapping with another age group.

RESULTS

From a total of 10 041 identified citations, 70 full-text articles met our eligibility criteria (Figure 1) [11–80]. The main characteristics of these articles are summarized in Table 1. There were 11 articles each from the United States and Spain, 8 articles from Australia, 7 articles from the I-MOVE group (involving multiple European countries), and 6 articles each from the United Kingdom and Canada. There were 3 articles from China and 2 articles each from Germany, Israel, Netherlands, Romania, and South Africa. One article each was from Austria, Croatia, Italy, Japan, New Zealand, Portugal, Taiwan, and Turkey. The sample size from the studies in these articles ranged from 197 to 11 430 participants. All the studies were funded by nonindustry sources, and 1 study received funding from both industry and nonindustry sources.

Pooled Adjusted VE by Method of Confirmation of Vaccination Status

Although not statistically significant, we observed a 10% higher pooled VE against H1N1 (P = .191), 7% against H3N2 (P = .626), and 5% against both influenza B (P = .529) and all influenza (P = .554) (Figure 2) for self-reported vaccination compared with medical record vaccination confirmation (Table 2). Almost all of the studies with self-reported vaccination were, however, from 1 research group in Canada. More of the studies with self-reported vaccination compared with those with medical record vaccination confirmation adjusted for both age and medical conditions. Zero percent (for H1N1), 20% (for H3N2, and influenza B), and 14% (for all influenza) of the studies with self-reported vaccination were from seasons in which vaccine virus strains were antigenically dissimilar to the circulating strains. In contrast, 8.3% (for H1N1), 30.8% (for H3N2), 23.1% (for influenza B), and 16% (for all influenza) of the studies with medical record vaccination confirmation were from seasons in which vaccine virus strains were antigenically dissimilar. Similar observations were made against H1N1 in 18- to 49-year-olds and against all influenza in ≥65-year-olds (Supplementary Table 2).

Pooled Adjusted VE by Timing of Respiratory Specimen Swab Collection

Despite a lack of statistical significance, we observed a 10% higher pooled adjusted VE against H3N2 (P = .596) and influenza B (P = .491), and 8% against all influenza (P = .447) (Figure 3), for swab collection within ≤7 days compared with ≤4 days of symptom onset (Table 2). In contrast, a 5% higher pooled adjusted VE was observed against H1N1 (P = .410) for swab collection within ≤4 days compared with swab collection within ≤7 days of symptom onset. There was no meaningful difference between studies with swab collection within ≤7 days and ≤4 days with regards to adjustment for both age and medical conditions in their analyses. Fifteen percent (for influenza B) and 18.5% (for all influenza) of the studies with swab collection ≤7 days were, however, from seasons in which vaccine virus strains were antigenically dissimilar to the circulating strains. In contrast, 22.2% (for influenza B) and 27.3% (for all influenza) of the studies with swab collection within ≤4 days were from seasons in which vaccine virus strains were antigenically dissimilar. Similarly, 5% (for H1N1) of the studies with swab collection within ≤7 days were from seasons in which vaccine strains were antigenically dissimilar, whereas 0% of the studies with swab collection within ≤4 days were from seasons in which vaccine strains were antigenically dissimilar. Evidence was conflicting across age groups (Supplementary Table 2).

Pooled Adjusted VE by Covariate Adjustment

Notwithstanding a lack of statistical significance apart from for H3N2, we observed a 4% higher pooled adjusted VE against H1N1 (P = .375), 13% against H3N2 (P = .029), 10% against influenza B (P = .144), and 4% against all influenza (P = .427) (Figure 4) for studies that included age among the adjusted covariates compared with those that included both age and medical conditions (Table 2). Three point eight percent (for H1N1), 13% (for H3N2), 13.6% (for influenza B), and 6.7% (for all influenza) of the studies that included age but not medical conditions were, however, from seasons in which vaccine virus strains were antigenically dissimilar to the circulating strains. In contrast, 5.3% (for H1N1), 36.8% (for H3N2), 20% (for influenza B), and 30.6% (for all influenza) of the studies that included age and medical conditions among the adjusted covariates were from seasons in which vaccine virus strains were antigenically dissimilar. Evidence was conflicting across age groups (Supplementary Table 2).

DISCUSSION

Despite a lack of statistical significance, we observed differences in pooled adjusted influenza VE between sources of influenza vaccination confirmation, respiratory specimen swab timing, and adjustments for 2 key confounders in study analysis. In our analysis of all study participants (irrespective of age), small differences were found between self-reported and medical record–confirmed influenza vaccinations, with higher pooled VE observed for self-reported vaccination, contrary to our expectations. However, almost all of the studies for self-reported vaccination were conducted in Canada and by the same group of researchers. We found substantial differences between respiratory specimen swab within ≤7 days and ≤4 days, with higher pooled VE observed for swab within ≤7 days. We also found substantial differences between studies that adjusted for age and those that adjusted for both age and medical conditions, with higher pooled VE observed for studies that adjusted for age. The above findings differed across age groups.

Studies have found that exposure misclassification can lead to significant bias in VE estimation [81, 82]. Self-reported vaccination is susceptible to recall and social desirability (individuals wanting to present a vaccine-compliant image) biases, with the potential for vaccination status misclassification. Smedt and colleagues showed in their simulation study that decreased exposure sensitivity and specificity underestimate true VE when misclassification of exposure (vaccination status) is nondifferential, but that when misclassification is differential, the bias could go in either direction, with the estimated VE deviating largely from the true VE. Compared with vaccination confirmation from medical records, self-reported vaccination usually has a higher sensitivity across various populations [83, 84] but a lower specificity in some population subgroups [85, 86]. Compared with whites, Hispanics were 2.7 times more likely to claim receipt of vaccination (self-report), and compared with younger individuals, self-reported influenza vaccination in the elderly had low specificity [84]. The observed higher pooled adjusted VE for self-reported compared with medical record–confirmed influenza vaccination status in this review, although not expected, may be due to differential misclassification of vaccination status, which Smedt and colleagues showed could either inflate or underestimate the true VE. This becomes more plausible considering that the studies with self-reported vaccination were almost all from Canada and from the same research group. Study center influence such as characteristics of the study participants, participant recruitment strategy, and influenza testing may also explain our findings.

Influenza incubation averages 2 days (range, 1–4 days) [87]. To maximize influenza virus detection from respiratory specimens, it is advocated that, ideally, swabs be collected within <4 days of influenza-like symptom onset. The longer swab collection is from symptom onset, the lower the likelihood of detecting influenza and the greater the potential for false-negative testing. Accurate reporting of symptom onset is therefore important, as a good TND study is predicated on patient symptom onset of ≤7 days. It will also help minimize outcome misclassification bias. False-negative testing among the vaccinated leads to VE overestimation, while false-negative testing among the unvaccinated leads to VE being underestimated. The observed higher pooled adjusted VE for swab collection of ≤7 days compared with ≤4 days in this review may therefore be due to a higher proportion of false negatives among the ≤7 days swab collection group, although this is not confirmable. Additionally, studies that included swab collection within ≤4 days possibly used more stringent swab collection criteria, resulting in reduced precision of VE estimation.

Seasonal influenza VE can vary from person to person. Various individual factors impact the VE [88], and 2 main factors (age and medical conditions) are known to play an important role in determining the likelihood that a vaccine will protect a person against influenza and to what extent. Age-dependent patterns in influenza vaccine protection have been reported from season to season, implicating the potential effect of age-related immune response in seasonal influenza VE [89]. For example, VE in the elderly population is reduced because of lower seroconversion rates that arise due to poorer immunological response to vaccination [90]. How well an individual responds to a vaccine may also be determined by underlying health conditions [91]. The observed higher pooled adjusted VE for studies that included age but not medical conditions compared with those that included both age and medical conditions among adjusted covariates in studies is in line with expectations, as adjusting for both age and medical conditions is likely to diminish VE compared with adjusting for age.

It is widely known that antigenic drift can markedly reduce seasonal influenza VE. For example, Flannery (2016) found that VE against H3N2 was almost 0 for an antigenically drifted genetic group of H3N2 viruses and 44% against a genetic group of H3N2 viruses that were antigenically similar to the seasonal vaccine strains [92]. This may explain the observed higher pooled adjusted VE in the subgroups with lower proportions of studies in which the seasonal influenza vaccine was antigenically dissimilar to the circulating virus strains. Variations in study design, sample size, vaccine type, and the demographic and temporal patterns underlying VE estimates from the included studies may also explain the variations observed in the pooled adjusted VE between compared groups. This, together with vaccine antigenic similarity with the circulating virus strains, may explain the high heterogeneity in many of the pooled adjusted VE. Where there were adequate numbers of studies for exploration of heterogeneity using metaregression, the available covariates tended to be highly collinear, thus limiting the usefulness of metaregression. Second, it was impossible to disentangle the effects of vaccine type and the underlying patient-level variations, as the analysis was conducted at the study level and these were not clearly reported in studies.

To our knowledge, our review is the first to evaluate differences in VE due to source of influenza vaccination status, respiratory specimen swab time, and confounder adjustments in statistical models for analysis. Irving et al. (2009) evaluated influenza vaccination status determined by self-report and by a real-time vaccination registry and found that the sensitivity and specificity of self-reported influenza vaccination compared with vaccination registry records were 95% and 90%, respectively, and that self-reported vaccination status was a sensitive and somewhat specific indicator of actual vaccine status, with misclassification being more common among young people [83]. However, the study did not compare influenza VE from these 2 sources of vaccination. No reviews seem to have compared seasonal influenza VE by respiratory specimen swab time and inclusion of main confounders in statistical models for analysis as we have done.

Our decision to include only influenza seasons after the 2009 pandemic influenza may have limited the number of potentially relevant TND studies for this review. However, it allowed us to focus on studies conducted from when public funding of influenza vaccination increased in most Western jurisdictions. It should be noted that some eligible studies conducted during this stated period may not have been published by the time we conducted our literature search, and therefore would not have been included in this review. Despite growing evidence to suggest that VE may be influenced by prior vaccinations [93, 94], the included studies did not report whether the study participants received the previous season’s influenza vaccination; hence, we could not assess the impact on VE estimates in our analyses. Furthermore, due to insufficient data, we could not examine VE against all outcomes for our subgroup analyses and for all age groups. We could also not separate individual study participants’ effects from study center effects (eg, effectiveness of vaccine policies and programs, participant recruitment strategy, and slight differences in symptom definitions), as the studies were conducted in different jurisdictions with potentially unique jurisdictional characteristics. Finally, we could not assess the reliability of reported estimates from the included studies because we could not ascertain if the studies met all of the assumptions that well-conducted TND studies are expected to meet to ensure that effect size estimates from the studies are not biased [5]. Although many of the studies adjusted for age or age and medical conditions, there were differences in the other covariates adjusted for in the studies. This may have contributed to the high heterogeneity observed in some of our pooled VE estimates.

Our review has many merits. We developed and registered a detailed protocol in PROSPERO before the execution of our search strategy, and we fully complied with the Cochrane Handbook for Systematic Reviews of Interventions guidelines throughout the review. We utilized the expertise of a methodologist trained in evidence synthesis literature searching to develop a comprehensive search strategy for the review, and this was subsequently reviewed by a professional knowledge synthesis librarian using the PRESS checklist. We searched appropriate bibliographic databases for literature and properly screened retrieved citations (against the eligibility) following the standards specified in the Cochrane Handbook for Systematic Reviews of Interventions. Where necessary, we requested additional data from the corresponding authors of the included studies to ensure completeness of the analyzed data. We included only studies in which influenza testing was conducted using the gold standard tests (PCR or viral culture). Furthermore, we examined variations in seasonal influenza VE across all clinically relevant age groups (<5 years, 5 to <18 years, 18 to 49 years, 50 to 64 years, and ≥65 years). We conducted the review to the highest expected standards and have reported in accordance with the PRISMA guidelines.

CONCLUSIONS

The available evidence from TND studies conducted after the 2009 pandemic influenza is not strong enough to conclude that influenza VE varies by source of vaccination status, respiratory specimen swab time, or adjustment for age/medical conditions. However, the evidence is indicative that these factors should be considered while designing or evaluating influenza VE from this study type. There is a need for researchers to ensure that age and medical conditions are both adjusted for in influenza VE estimations from TND studies, while uniformity in covariate adjustments across studies would help reduce heterogeneity and increase precision of pooled VE.

Supplementary Data

Supplementary materials are available at Open Forum Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Acknowledgments

Financial support. No external funding was obtained for this study. S.M.M. is supported, in part, by funding from the Canada Research Chairs Program. G.N.O. is a current recipient of the Manitoba Training Program Fellowship Award and the Evelyn Shapiro Award, both for health services research.

Potential conflicts of interest. S.M.M. has received unrestricted research grants from GlaxoSmithKline, Merck, Sanofi Pasteur, Pfizer, and Roche-Assurex for unrelated studies. The other authors declare that they have no conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.

References