Association between SARS-CoV-2 Neutralizing Antibodies and Commercial Serological Assays

Abstract

Background

Commercially available SARS-CoV-2 serological assays based on different viral antigens have been approved for the qualitative determination of anti-SARS-CoV-2 antibodies. However, there are limited published data associating the results from commercial assays with neutralizing antibodies.

Methods

Sixty-six specimens from 48 patients with PCR-confirmed COVID-19 and a positive result by the Roche Elecsys Anti-SARS-CoV-2, Abbott SARS-CoV-2 IgG, or EUROIMMUN SARS-CoV-2 IgG assays and 5 control specimens were analyzed for the presence of neutralizing antibodies to SARS-CoV-2. Correlation, concordance, positive percent agreement (PPA), and negative percent agreement (NPA) were calculated at several cutoffs. Results were compared in patients categorized by clinical outcomes.

Results

The correlation between SARS-CoV-2 neutralizing titer (EC₅₀) and the Roche, Abbott, and EUROIMMUN assays was 0.29, 0.47, and 0.46, respectively. At an EC₅₀ of 1:32, the concordance kappa with Roche was 0.49 (95% CI; 0.23–0.75), with Abbott was 0.52 (0.28–0.77), and with EUROIMMUN was 0.61 (0.4–0.82). At the same neutralizing titer, the PPA and NPA for the Roche was 100% (94–100) and 56% (30–80); Abbott was 96% (88–99) and 69% (44–86); and EUROIMMUN was 91% (80–96) and 81% (57–93) for distinguishing neutralizing antibodies. Patients who were intubated, had cardiac injury, or acute kidney injury from COVID-19 infection had higher neutralizing titers relative to those with mild symptoms.

Conclusions

COVID-19 patients generate an antibody response to multiple viral proteins such that the calibrator ratios on the Roche, Abbott, and EUROIMMUN assays are all associated with SARS-CoV-2 neutralization. Nevertheless, commercial serological assays have poor NPA for SARS-CoV-2 neutralization, making them imperfect proxies for neutralization.

Introduction

Host cell infections by the recently-emerged severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) begin when the viral spike (S) protein engages the host angiotensin-converting enzyme 2 (ACE2) receptor (1,). The humoral immune response can block infection through neutralizing antibodies, which bind the virus in a manner that prevents host cell infection (2,). For SARS-CoV-2, this may be achieved by interfering with the spike ACE2 receptor interaction, or by disrupting the fusion mechanisms that the virus uses to enter host cell cytoplasm (2).

In the absence of a vaccine, there is considerable interest in identifying high-affinity neutralizing antibodies to SARS-CoV-2 to assess immune status and to evaluate vaccine responses. We previously demonstrated that passive transfer of monoclonal antibodies against SARS-CoV-2 S protein reduced viral titers and pathology in the lungs in a mouse model of SARS-CoV-2 (3). Monoclonal antibodies engineered from neutralizing antibodies, initially identified from convalescent COVID-19 patients, have been advanced as potential antiviral therapeutics (4–6,), and early results from convalescent plasma use in patients indicate a protective effect of antibodies against SARS-CoV-2 (7–10,). While early results are promising, the antibody titer conferring protection remains unclear and the role of neutralizing antibodies in protection has not been fully elucidated (11).

Despite widespread interest in neutralizing antibodies, methods for their detection and quantification are relatively low-throughput and limited to Biosafety Level 3-equipped research laboratories. While high-throughput methods have emerged, most rely on recombinant Vesicular Stomatitis Viruses (VSV) engineered to express a portion of the SARS-CoV-2 viral spike protein, and their subsequent entry into cell lines (12–14). Commercially available serological assays are high-throughput, relatively inexpensive, and use readily available instrumentation. The use of automated serological SARS-CoV-2 assays as a surrogate for neutralizing titers is therefore an attractive option. To date, limited data are available correlating commercially available assays with the presence of neutralizing antibodies.

We previously compared the clinical performance of 3 commercial serological assays (15, 16). Here, we further assess the ability of these assays to predict the presence of neutralizing antibodies.

Materials and Methods

Specimens

This study was approved by the Institutional Review Board of Washington University in St. Louis. Residual EDTA plasma samples from physician-ordered complete blood count studies were utilized. Specimens were obtained from patients with PCR-confirmed COVID-19 and at least one previously positive SARS-CoV-2 serological result. All specimens were stored at -80 °C for 2–6 weeks until thawed and analyzed by serological assays (within 7 days). Specimens were then stored at 4°C for up to 7 days until analysis by the neutralizing assay. Previous studies have demonstrated that IgG, IgA, and IgM antibodies are stable at 4°C for up to 8 months (17). A subset of pre-pandemic samples obtained in 2015 and stored at 80 °C were used as negative controls.

Clinical information

Duration from symptom onset was obtained from two independent assessors by review of the electronic medical record (EMR) and inferred from physician encounter notes. Symptoms included cough, fever, shortness of breath, loss of taste or smell, sore throat, and headache (18). The EMR also was used to collect data on outcomes for each patient. Mortality and intubation were determined by physician encounter notes; acute kidney injury (AKI) was defined using the Risk, Injury, Failure, Loss of kidney function, and End-stage kidney disease (RIFLE) criteria of 2-fold increase in serum creatinine and urine output less than 5 mL/kg/h; cardiac injury was defined as a troponin I concentration > 0.03 ng/mL (Abbott Diagnostics).

INSTRUMENTATION

Specimens were analyzed on 3 commercially available immunoassays and reported previously (15, 16). The Roche Elecsys Anti-SARS-CoV-2 assay was performed on an a Cobas e 601. The Roche assay detects total antibodies (IgG, IgA, IgM) against an epitope of the viral nucleocapsid protein. The Abbott SARS-CoV-2 IgG assay was performed on an i2000 Abbott Architect (Abbott Diagnostics) and detects IgG antibodies against the viral nucleocapsid protein. The EUROIMMUN (EI) SARS-CoV-2 IgG assay was performed on a QUANTA-Lyser 240 (Inova Diagnostics) assay and detects anti-SARS-CoV-2 IgG directed against the S1 domain of viral spike protein. All 3 assays use an assay-specific calibrator to report the ratio of the signal from the specimen to the signal of the calibrator. The results are interpreted as positive or negative relative to a threshold value. For the Roche assay, a positive is a cutoff index (COI) ≥1; for the Abbott assay, a signal to cut-off (S/CO) ≥1.4 is positive and <1.4 is negative; for the EI assay, a ratio ≥ 1.2 is positive, 0.80–1.19 is indeterminate, and < 0.8 is negative. The cutoff of 1.2 was used as a positive result for the EI. All 3 assays specify a positive result as the signal of the sample/the signal of a calibrator, therefore all results are reported here as a ratio.

FOCUS REDUCTION NEUTRALIZATION ASSAYS

Neutralization assays were performed as previously described (19,). Briefly, SARS-CoV-2 strain 2019 n-CoV/USA_WA1/2020 was obtained from the Centers of Disease Control and passaged in Vero E6 cells with DMEM (Corning) supplemented with glucose, L-glutamine, sodium pyruvate, and 10% FBS. Indicated dilutions of plasma were incubated with 10² focus forming units of SARS-CoV-2 for 1 h at 37°C before addition of the antibody virus complex to Vero E6 monolayers at 37°C for 1 h. Cells were overlaid with a 1% w/v methylcellulose in minimum essential medium (MEM) supplemented with 2% fetal bovine serum (FBS) and harvested 30 h later. Methylcellulose overlays were removed and fixed with 4% paraformaldehyde in PBS at room temperature. Plates were then washed and incubated with 1 µg/mL anti-S antibody (CR3022) (20) and HRP-conjugated goat anti-human IgG. Cells infected by SARS-CoV-2 were visualized using TrueBlue peroxidase substrate (KPL) and cell foci were quantified using an ImmunoSpot microanalyzer (Cellular Technologies). For each specimen, a minimum of 8 dilutions of human plasma were performed in duplicate and a standard curve generated. The 1/Log₁₀ plasma dilution (EC₅₀) is the dilution at which 50% of the cells were infected with virus and formed foci (Supplemental Fig. 1).

STATISTICS

Pearson correlation coefficients were calculated between clinical assays and neutralizing titers. Concordance between the assays was calculated using the Cohen Kappa measure. Areas under the curve (AUC) for receiver operator characteristic (ROC) curves were calculated using the Wilson/Brown method. Kappa, positive percent agreement (PPA), and negative percent agreement (NPA) analyses were performed using multiple cutoffs for neutralizing titers owing to a lack of consensus regarding the relevant protective titer. Ideal cutoffs that maximized PPA and NPA were calculated using the Youden Index. Differences between antibody and neutralizing titers categorized by outcomes were calculated using one-way ANOVA and the Sidak multiple comparison test for normally distributed data or a Kruskal-Wallis test with a Dunn multiple comparison test for non-normally distributed data. Normality was assessed using the D’Agostino & Pearson test for normality. For outcome comparisons, all specimens were collected more than 10 days post-symptom onset. All statistical analyses were performed with GraphPad Prism 8 (GraphPad).

Results

40/42 specimens from PCR-confirmed COVID-19 patients with positive antibody results from commercial SARS-CoV-2 assays had neutralizing titers >1:20 by day 14 post-symptom onset (Fig. 1, A). The mean neutralizing titer of specimens from patients 21 days or more from symptom onset was 1:250 (95% CI; 1:149–1:436). In contrast, pre-pandemic control samples were not neutralizing at a titer of 1:20. Neutralizing titers increased subsequently with days post-symptom onset (Supplemental Fig. 2). A subset of patients with serial measurements demonstrated a rapid rise in neutralizing titers between days 5–15 that plateaued at approximately 1:250 and remained increased through the time course tested (Fig. 1, B).

The correlations of the SARS-CoV-2 neutralizing titer with the ratios reported by the Roche, Abbott, and EI assays were 0.29, 0.47, and 0.46, respectively (Fig. 2A–C). Higher neutralizing titers were generally associated with a higher ratio, as measured by all 3 assays. At a cutoff of 1:32 for the neutralizing assay, the concordance kappa with Roche was 0.61 (95% CI; 0.35–0.86), with Abbott was 0.65 (0.42–0.88), and with EI was 0.69 (0.49–0.89). For all three assays, the concordance decreased with an increased threshold for neutralizing titers.

ROC curves to determine the PPA and NPA of a positive antibody result on commercial assays for neutralizing titers ≥ 1:32 revealed an AUC of 0.99 (95% CI; 0.96–1.0), 0.98 (0.95–1.0), and 0.95 (0.91–1.0) for the Roche, Abbott and EI assays respectively (Fig. 3, A). For both the Roche and Abbott assays, the ratio established by the manufacturers produced maximum PPA with decreased NPA for neutralizing antibodies. Lowering the cutoff for EI increased the PPA without negatively impacting NPA. When evaluated for a neutralizing titer of 1:128, the AUC of the Roche assay was 0.86 (95% CI; 0.77–0.95), for the Abbott was 0.82 (0.71–0.94), and for the EI was 0.9 (0.83–0.97) (Fig. 3, B). At this neutralizing titer, the manufacturers’ ratios for a positive result for all 3 assays maximized PPA while reducing NPA for anti-SARS-CoV-2 neutralizing antibodies.

At a neutralizing titer of 1:32, the PPA and NPA for the Roche assay was 100% (95% CI; 94–100) and 56% (30–80) at a ratio of 1.0 (Table 1). The NPA improved to 100% (79–100) with a PPA of 89% (78–96) if the ratio for a positive result on the Roche was increased to 7.2. For the Abbott assay, the PPA was 96% (88–99) and the NPA was 69% (44–86) at a ratio of 1.4. The PPA and NPA for the Abbott changed to 91% (81–96) and 100% (81–100), respectively, if the ratio for a positive result was adjusted to 3.5. For the EI assay, the PPA was 91% (80–96) and the NPA was 81% (57–93) at a cutoff of 1.2. By increasing the ratio for a positive result to 2.9, the PPA decreased to 82% (70–90) and the NPA increased to 100% (81–100). NPA decreased for all 3 assays with increasing cutoff for a protective titer. The ideal cutoff for each assay at a neutralizing titer of 1:128 was 18.0 for the Roche, 4.6 for the Abbott, and 6.6 for the EI assays. PPA remained above 75% for all assays at these cutoffs.

Although not statistically significant, patients who died as a result of COVID-19 had higher mean neutralizing antibody titers (1:576), compared to patients who survived (mean neutralizing titers of 1:162) (Fig. 4, A). Furthermore, patients who were intubated, had cardiac injury, or AKI had significantly higher neutralizing antibody titers relative to those with milder COVID-19 symptoms (Fig. 4B–D). In contrast, no significant associations between patient outcomes and assay signal ratios were observed with the Roche or Abbott assays. Increased ratios were observed with the EI assay in patients who were intubated or had a cardiac injury relative to those that did not. No significant difference was observed with neutralizing titers in male patients or patients >60 years old. Similar trends were observed with the serological assays (Supplemental Fig. 3). If categorized by low (<1:256) or high neutralizing titers (>1:256), there were no significant difference in outcomes between patients. However, there was an increase in the ratio observed in high neutralizing titer patients compared to low titer patients on the Roche assay (43.6, 95% CI; 31.0–56.4 vs. 23.4, 95% CI; 10.5–36.4), on the Abbott assay (6.3, 5.6–6.9 vs. 4.8, 3.6–5.8) and the EI assay (8.5, 7.7–9.2 vs. 5.2, 3.5–6.0) (Supplemental Table 1).

Discussion

The emergence of commercially available serological assays for the detection of antibodies to SARS-CoV-2 has outpaced scientific understanding of their immunological meaning and their value in clinical decision making. Here, we assessed the utility of 3 commercially available clinical assays for correlation with neutralizing antibodies to SARS-CoV-2. We observed modest correlation, but poor concordance and NPA between the Roche, Abbott, and EI SARS-CoV-2 assays for the detection of SARS-CoV-2 neutralizing antibodies. Several studies have demonstrated that neutralizing antibodies are primarily against the S1, S2, and RBD domains of the SARS-CoV-2 spike protein (3, 4). As a result, clinical assays targeting these regions have been hypothesized to better predict neutralizing titers. However, our findings indicate that the Roche (nucleocapsid), Abbott (nucleocapsid), and EI (S1) assays have similar performance for predicting patients with neutralizing antibodies. This implies that patients infected with SARS-CoV-2 develop a broad-based antibody repertoire against multiple proteins and epitopes, but only some of these antibodies have neutralizing properties. The proportion of neutralizing antibodies is highly variable and appears to be dependent on disease severity.

While the World Health Organization (WHO) and the Centers for Disease Control (CDC) have advised against associating immunity with seropositivity (21, 22,), some have proposed that this warning is unnecessarily conservative (23,). Our findings suggest that SARS-CoV-2 serological assays should be interpreted with caution. While the majority of patients with antibodies detected by commercial assays had neutralizing antibodies present by day 14 post-onset of symptoms, about 10% of patients past day 14 had titers that were <1:32. This implies that some patients with previous SARS-CoV-2 infections and positive antibody results by commercial assays may have neutralizing antibodies near the cutoff for a positive result. Although further studies are warranted, these low titers may be inadequate for protection, particularly if neutralizing antibodies are the primary therapeutic benefit of convalescent plasma. While higher reported ratios from all three commercial assays correlated with higher neutralizing titers, this was not universally true. Consistent with this, the correlations between neutralizing titers and serological results were <0.5 on all 3 commercial assays. Nonetheless, we found that higher ratios reported by all 3 commercial assays were associated with higher neutralizing titers. Importantly, all 3 serological assays used in this study currently have Emergency Use Authorization (EUA) to qualitatively determine the presence of antibodies against SARS-CoV-2. While a negative result on SARS-CoV-2 serological assays is likely to be associated with the absence of neutralizing antibody titers, a positive result is not reliable for predicting the presence of neutralizing antibodies. Furthermore, since these assays are under the EUA, they cannot be modified by the laboratory to report quantitative units. Our results argue for a potential utility in reporting the ratio calculated for commercially available assays relative to the calibrator. We, along with others, have previously suggested that commercially available serological assays for SARS-CoV-2 may have utility for identifying convalescent plasma donors (24, 25,). To this end, reporting quantitative units is more likely to identify convalescent patients with higher neutralizing antibody titers than qualitative cutoffs. Furthermore, if neutralizing antibodies are shown to confer protection to SARS-CoV-2, quantitative serological assays may assist in identifying neutralizing titers in mildly symptomatic and asymptomatic populations. Further supporting the use of quantitative serological assays, a recent study demonstrated that patients receiving convalescent plasma with higher levels of antibodies measured by the Ortho-Clinical Diagnostics anti-SARS-CoV-2 IgG assay had improved outcomes relative to patients that received convalescent plasma with medium or low levels (26). Nonetheless, further studies are needed to demonstrate the clinical benefit of using quantitative SARS-CoV-2 serological assays, especially by characterizing this association in a more diverse patient population.

Although studies have reported modest linear correlation (0.6–0.8) between neutralizing SARS-CoV-2 titers with anti-RBD or anti-S IgG (27, 28,), our AUC analysis demonstrates that it is difficult to identify an ideal signal-to-assay ratio cutoff to predict neutralizing antibodies. The NPA for neutralizing antibodies was >90% for all 3 commercial assays only when a 1:20 neutralizing titer was used as a cutoff. It is important to note that this is far below the FDA recommended neutralizing titer for convalescent plasma donors (≥1:160) (29,). At a similar cutoff of 1:128, the NPA for neutralizing titers was below 60% for all 3 of the assays. Furthermore, while it is expected that neutralizing antibodies confer some protection against SARS-CoV-2, the titer required for this protective effect has not been established (11,). Due to the low sensitivity of serological assays for diagnosing early SARS-CoV-2 infection (15, 30,), some studies have suggested lowering the assay cutoff ratios to improve sensitivity (31, 32,). However, if the intended utility of serology is to determine the presence of neutralizing antibodies, our ROC analyses suggest that the assay cutoff should be increased to improve the NPA. Interestingly, some manufacturers are now associating positive serological results with neutralizing antibodies in their validation studies. For instance, the LIAISON SARS-CoV-2 S1/S2 IgG assay claims high agreement with neutralizing antibodies. However, the cutoff titer used for the neutralizing assay was 1:40; far below that recommended by FDA for convalescent plasma therapy (13). If neutralizing antibodies >1:256 are required for protection, then commercial assays at the current cutoffs may have limited utility for identifying patients with protective antibodies; with NPA between 18%–40% for the assays tested in this study.

Here, we observed that higher neutralizing titers are associated with worse clinical outcomes, a finding that was not consistently observed with commercial serological assays. While seemingly counterintuitive, it is consistent with previous literature and may be a result of higher antigen burdens or hyperactive immune responses among other reasons (33–37,). A study of service members in the US Navy with predominantly mild symptoms revealed that about 40% of those with a positive ELISA by the CDC assay had no neutralizing titers at a cutoff of 1:40 (38,). Similarly, a recent study demonstrated neutralizing titers at <1:50 in 33% of recovered patients and below 1:1000 in 79% of patients (27,). Our findings are also consistent with a study assessing the agreement between the EI IgG result and neutralizing titers on predominantly nonhospitalized convalescent plasma donors (36,). The authors demonstrated that at a neutralizing titer of 1:320, the PPA and NPA were 96% and 32%, respectively, and that neutralizing titers were higher in a small cohort of hospitalized patients. Similarly, we demonstrate higher neutralizing titers among patients with worse outcomes in an almost entirely hospitalized cohort. Unique to this study, we also compare commercial tests head-to-head and, by extension, compare serologies to 2 different protein antigens with similar results. Of note, patients that were intubated or had an AKI also had higher results by the EI assay. This correlation with neutralizing antibodies that was not observed on the Abbott or Roche assays may be due to the EI assay targeting the Spike protein, the primary target of neutralizing antibodies. Taken together, previous studies, coupled with the findings presented here, are consistent with the notion that neutralizing antibodies, while an important component of the immune response (3, 4), are unlikely to be the only mechanism of SARS-CoV-2 clearance and protection. Other immune responses such as cellular immunity, T cells, antibody-mediated cellular immunity, and antibody-mediated complement fixation likely play a pivotal role in protection from SARS-CoV-2.

Due to both heavy marketing and misunderstanding of their utility, patients have sought antibody testing for SARS-CoV-2 to determine if they had been previously infected and for peace-of-mind, assuming that they may have some level of protection (the concept of an “immunity passport”). At our institution, around 85% of the SARS-CoV-2 serological tests are performed in the outpatient setting. This implies that the vast majority of these tests may be performed on mildly symptomatic and asymptomatic populations. Therefore, it is crucial that future studies address the correlation between neutralizing titers and commercial assays in the mildly symptomatic and the asymptomatic COVID-19 population. If symptomatic and severely ill patients have the highest titers of neutralizing antibodies, low concordance demonstrated here may be exacerbated by including asymptomatic and mildly symptomatic patients. Furthermore, while neutralizing titers appear to persist in the small group of patients with longitudinal specimens, the duration of follow-up in our study was too short to determine the durability of neutralizing antibodies. Nonetheless, previous studies have demonstrated a reduction in neutralizing titers after 8 weeks post-hospital discharge (34).

There are several limitations associated with this study. The true sensitivity and specificity of neutralizing titers in PCR-confirmed SARS-CoV-2 infected patients could not be accurately determined because specimens were pre-selected for serological positivity by commercially available immunoassays. This approach was chosen given the highly manual nature of testing for neutralizing antibodies and the primary goal of comparing neutralizing antibody titers to commercial assays. Furthermore, while the neutralizing assay utilized is robust and reproducible, it has not been validated for clinical use. In contrast to other studies, this assay uses an infectious strain of SARS-CoV-2 as opposed to pseudotyped rhabodoviruses or lentiviruses that heterologously express the SARS-CoV-2 spike protein. Furthermore, the relatively small number of patients tested means that potentially subtle differences in PPA, NPA, and concordance between the 3 assays could not be distinguished as a result of wide, overlapping confidence intervals. Finally, while others have demonstrated that neutralizing titers appear as early as day 10 post-onset of symptoms, it is possible that assessing patients at later time points (i.e., day 28) would reveal a higher concordance. While the majority of patients tested serially had neutralizing titers that peaked by day 14–15, future studies are needed at later timepoints to assess the correlation of neutralizing antibodies with commercial assays. This includes several months after infection, when other studies have demonstrated the neutralizing response beginning to diminish.

In conclusion, our findings suggest that positive serological results by 3 commercially available assays that measure antibodies against the viral spike or nucleocapsid protein of SARS-CoV-2 have modest correlation with neutralizing antibody titers. COVID-19 patients generate an antibody response to multiple viral proteins such that the quantitative ratios on the Roche, Abbott, and EUROIMMUN assays have comparable associations with SARS-CoV-2 neutralization. Nevertheless, commercial serological assays have poor NPA for SARS-CoV-2 neutralization, making them imperfect proxies for neutralization.

Supplemental Material

Supplemental material is available at Clinical Chemistry online.

Author Declaration

A version of this paper was previously posted as a preprint on bioRxiv as https://biorxiv.org/cgi/content/short/2020.07.01.182220v1.

Author Contributions

All authors confirmed they have contributed to the intellectual content of this paper and have met the following 4 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; (c) final approval of the published article; and (d) agreement to be accountable for all aspects of the article thus ensuring that questions related to the accuracy or integrity of any part of the article are appropriately investigated and resolved.

J.B. Case, statistical analysis; C.W. Farnsworth, statistical analysis, provision of study material or patients.

Authors’ Disclosures or Potential Conflicts of Interest

Upon manuscript submission, all authors completed the author disclosure form. Disclosures and/or potential conflicts of interest:

Employment or Leadership

None declared.

Consultant or Advisory Role

M.S. Diamond, Moderna, Vir Biotechnology, NGM Biopharmaceuticals, INBios.

Stock Ownership

None declared.

Honoraria

None declared.

Research Funding

None declared.

Expert Testimony

None declared.

Patents

None declared.

Role of Sponsor

No sponsor was declared.

References

Author notes