Evaluation of the Clinical Performance of 7 Serological Assays for SARS-CoV-2 for Use in Clinical Laboratories

Abstract

Introduction

Serological assays for the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2), including IgM, IgA, IgG, or total antibodies, are widely available worldwide in a variety of testing formats, making them appealing for clinical and epidemiological use during the ongoing coronavirus disease-2019 (COVID-19) pandemic. However, serological testing has been controversial and there has been confusion about interpretation and applications (1–3). According to several organizations, serology testing is not useful to diagnose acute infection and to establish immunity correlation claims (4, 5). Serological testing may be beneficial in epidemiologic studies, in vaccine development, and for verification response, selection of convalescent plasma donors, and to aid in the diagnosis of a subset of patients negative by molecular testing and children with the multisystem inflammatory syndrome (4–6). However, the value of serology testing on patient management decisions is still largely unknown (4, 5) supporting the need to further our understanding of clinical performance and baseline characteristics of serologic assays.

We characterized the clinical performance of 7 serologic assays with a focus on specificity in a large cohort of samples collected before 2019 and in samples from nucleic acid amplification testing (NAAT)-negative patients, and in acute disease in samples from hospitalized COVID-19-positive patients.

Materials and Methods

This study was approved by the Cleveland Clinic Foundation Institutional Review Board. Samples consisted of 84 residual serum/plasma from 53 COVID-19-positive patients by NAAT 3–21 days postonset of symptoms (POS) (posttest day for 3 with unclear onset), 40 samples from NAAT-negative presurgical patients, and 393 samples collected pre-December 2019 [299 noncrossreactivity: healthy (n = 50), pregnant (n = 10), and patients with various conditions; 94 cross-reactivity: 29 known serological/autoimmune markers (Supplemental Table 1)]. The assays evaluated were: ARCHITECT SARS-CoV-2 IgG on the Architect (Abbott Diagnostics, Abbott Park, IL, US), Access SARS-CoV-2 IgG on the DxI (Beckman Coulter Diagnostics, Brea, CA, US), LIAISON SARS-CoV-2 S1/S2 IgG (DiaSorin Inc, Stillwater, MN, US) on the Liaison XL, Anti-SARS-CoV-2 ELISA IgG and IgA (EUROIMMUN US Inc, Mountain Lakes, NJ, US), and Platelia SARS-CoV-2 Total Ab (Bio-Rad, Hercules, CA, US) on a QUANTA-Lyser 240 (Inova Diagnostics Inc, San Diego, CA, US), and Elecsys Anti-SARS CoV-2 Total on a Cobas 601 (Roche Diagnostics, Indianapolis, IN, US).

Positive agreement (PA) and negative agreement (NA) were calculated relative to NAAT or as negative for the pre-Dec 2019 group. Following the U.S Food and Drug Administration (FDA)’s approach for assays with equivocal results (i.e., Bio-Rad, EUROIMMUN), equivocal results on positive cases were counted as negative for PA, and on negative cases were counted as positive for NA (7); statistics were also calculated in vice versa. Using the former approach, positive and negative predictive values (PPV, NPV) were calculated from the overall %NA and 14+ days %PA. Analysis were carried out using © MedCalc software Ltd (Osten, Belgium).

Results

The IgA test had the lowest %NA at 79.7%, while the other assays demonstrated overall %NA of 97.5%–99.8% (n = 424–433) (Table 1). The %NA was not appreciably different in the pre-December 2019 subcohorts (n = 393; 94 in the cross-reactivity cohort). Ninety-seven specimens had positive/equivocal results by at least 1 assay, 12 by 2 assays (EUROIMMUN IgA, DiaSorin (n = 5); EUROIMMUN IgA, EUROIMMUN IgG (n = 4)), and 3 by 3 assays (EUROIMMUN IgA, EUROIMMUN IgG, DiaSorin) (Table 1 and data not shown). Only 23 corresponded to the cross-reactivity group (n = 94), 14 of which expressed at least 2 known serological/autoimmune makers and 9 expressed more than 2 (Supplemental Table 2). Samples with false positives most commonly expressed Epstein-Barr virus (EBV) IgG, followed by cytomegalovirus IgG, varicella zoster IgG, measles IgG, and anti-hepatitis B surface antibody (Supplemental Table 1). Of 10 samples from pregnant women, 3 were positive by the IgA assay, and of 50 samples from healthy individuals, 8 were positive/equivocal by the EUROIMMUN IgA (n = 5), EUROIMMUNN IgG (n = 2), DiaSorin (n = 1) and Bio-Rad (n = 1) assays.

Test positivity increased with days POS only for the Abbott, Roche, and Bio-Rad assays (Table 2). In samples collected 14–21 days POS (n = 24), %PA ranged from 50.0%–83.3%. The %PA <60% for the Beckman, DiaSorin, and EUROIMMUN IgG assays was unexpected. On days 7–21 (n = 60), the Bio-Rad test demonstrated the highest overall %PA. A large seroprevalence study in the US reported variable disease prevalence with most areas still having a prevalence lower than 10% (8). We therefore calculated PVs for 5% and 10% disease prevalence (Table 2). The Roche assay had the highest PPV at prevalences of 5% and 10% (94.8 and 97.4, respectively), and tied NPVs with the Abbott assay, second to the Bio-Rad assay leading with NPVs of 99.1% and 98.2% for the 5% and 10% prevalences, respectively. The IgA assay had an unacceptably low PPV of 27.9% at a 10% prevalence.

Discussion

Except the IgA assay, all assays met the >95% specificity criteria required by the FDA for Emergency Use Authorization (EUA) and in agreement with other studies (9–14). The %PA for these 7 assays (Table 2) did not meet the FDA’s sensitivity standard of 90% and was lower than reported (9–12). Only the Bio-Rad assay had a %PA above 80% in the 14+ days POS samples. It detected one more case than the Roche and Abbott assays and 3 more than the IgA assay, but 6–8 more than the other 3 assays. On samples collected <14 days POS, the Bio-Rad assay detected 2 more cases than the Roche Total assay and 3 more than the IgA assay but 7–10 more than the other assays. Therrien et al. evaluated these assays on samples collected 2–5 weeks POS (n = 60) and found the IgA and Bio-Rad assays performed similarly, with sensitivities of 85%; 73–95 and 87%; 75–94, respectively (9). They reported performance comparable to our findings for the Roche and Abbott assays but superior %PA for the EUROIMMUN IgG, Beckman and DiaSorin assays ranging from 72%–78% (vs 50%–58% in our study). In samples collected 14 days+ onset of symptoms, Tang and colleagues reported higher sensitivities by 10% and 14% for the Roche and Abbott assays, respectively, and by almost 40% for the EUROIMMUN IgG assay (10, 11). Other studies also reported higher sensitivities for the Beckman, DiaSorin, and EUROIMMUN IgG assays (15, 16). In the context of many unknown factors in the humoral responses of patients with COVID-19, it is difficult to conclusively determine the reason for the low %PA in our study. An obvious limitation of our study is the low number of samples 14+ days POS and only up to 21 days. Moreover, the timing of sample collection was subjectively determined from chart review. The observed differences may be a factor of the heterogeneity of the populations studied and an uncharacterized role of co-morbidities, present in many hospitalized patients with COVID-19, in seroconversion. A phenomenon of seronegative nonresponding patients (i.e., negative by several assays) has been described and although it is not well characterized yet, it may represent at least 5% of 14+ days POS samples (14). In our study, 4/24 samples (16.7%) did not seroconvert by any assay. Lastly, hospitalization rates vary across these studies from approximately 50% to 67% (9, 16), or was not mentioned (10). Our cohort consisted of hospitalized patients (1–38 days), and 50/53 were symptomatic with mild (n = 10), moderate (n = 11), critical (n = 18), and severe (n = 11) disease.

Three strategies are recommended by the Centers for Disease Control and Prevention (CDC) to improve PPV (4). First, to use assays with >95% specificity, but preferably >99.5%. The most stringent criteria were only met by the Roche and Beckman assays. Second, to test patients with a high pre-test probability. The third recommendation is orthogonal testing (i.e., testing positive samples using another test, preferably with a different format or antigen target). For the IgA assay, 12 samples were also positive on at least another assay targeting the viral spike protein (EUROIMUNN IgG and DiaSorin), but none of the assays targeting the nucleocapsid (Abbott, Roche, Bio-Rad) or the receptor binding domain (Beckman). The IgA assay had the lowest PPV, a concerning number of false-positive results, and should not be used in low prevalence settings. In alignment with a study reporting IgA expression in asymptomatic pregnant women (17), 3 of 10 samples from pregnant women were positive for IgA. Although it has been suggested that IgA appears earlier than IgG in patients with COVID-19 (9, 18), our data and longitudinal samples did not support this conclusion (Table 1 and data not shown). It is important to mention that our study calculated PVs using the results from the 14+ POS cohort, which has a very low sample size, and is not well powered to assess PPVs, as indicated by the large 95% confidence intervals.

We found that the Roche, Bio-Rad, and Abbott assays performed better 14+ days POS in hospitalized patients. Using 20–40 samples for EUA assay validation may not be sufficient to establish robust %PA claims (19, 20). In some cases, extending collection to >7 weeks POS was needed to yield sensitivities near 90% (9). Nonetheless, serology testing is not recommended for diagnosing acute COVID-19 infection (4, 5). We concluded with high level of confidence that the Roche, Beckman, Abbott, and Bio-Rad assays detect few false positives and may be useful in seroprevalence studies.

Supplemental Material

Supplemental material is available at The Journal of Applied Laboratory Medicine online.

Nonstandard Abbreviations: SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; COVID-19, coronavirus disease-2019; NAAT, nucleic acid amplification test; POS, post-onset of symptoms; NA, negative agreement; PA, positive agreement; NPV, negative predictive value; PPV, positive predictive value; FDA, US Food and Drug Administration; EUA, Emergency Use Authorization; CDC, Centers for Disease Control and Prevention.

Author Contributions: All authors confirmed they have contributed to the intellectual content of this paper and have met the following 4 requirements: (a) significant contributions to the conception and design, acquisition of data, or analysis and interpretation of data; (b) drafting or revising the article for intellectual content; (c) final approval of the published article; and (d) agreement to be accountable for all aspects of the article thus ensuring that questions related to the accuracy or integrity of any part of the article are appropriately investigated and resolved.

A. Plaga, provision of study material or patients; D. Payto, administrative support, provision of study material or patients; E. Olson, administrative support; M. Strizzi, administrative support; J. Ko, administrative support, provision of study material or patients.

Authors’ Disclosures or Potential Conflicts of Interest: Upon manuscript submission, all authors completed the author disclosure form. Disclosures and/or potential conflicts of interest: Employment or Leadership: M. Strizzi, The Cleveland Clinic Foundation. Consultant or Advisory Role: None declared. Stock Ownership: None declared. Honoraria: None declared. Research Funding: None declared. Expert Testimony: None declared. Patents: None declared.

Role of Sponsor: No sponsor was declared.

REFERENCES

Nonstandard Abbreviations:

 
• SARS-CoV-2

  severe acute respiratory syndrome coronavirus 2

 
• COVID-19

  coronavirus disease-2019

 
• NAAT

  nucleic acid amplification test

 
• POS

  post-onset of symptoms

 
• NA

  negative agreement

 
• PA

  positive agreement

 
• NPV

  negative predictive value

 
• PPV

  positive predictive value

 
• FDA

  US Food and Drug Administration

 
• EUA

  Emergency Use Authorization; CDC, Centers for Disease Control and Prevention