Precision Verification: Effect of Experiment Design on False Acceptance and False Rejection Rates

Abstract

Objectives

We examined the false acceptance rate (FAR) and false rejection rate (FRR) of varying precision verification experimental designs.

Methods

Analysis of variance was applied to derive the subcomponents of imprecision (ie, repeatability, between-run, between-day imprecision) for complex matrix experimental designs (day × run × replicate; day × run). For simple nonmatrix designs (1 day × multiple replicates or multiday × 1 replicate), ordinary standard deviations were calculated. The FAR and FRR in these different scenarios were estimated.

Results

The FRR increased as more samples were included in the precision experiment. The application of an upper verification limit, which seeks to cap FRR at 5% for multiple experiments, significantly increased the FAR. The FRR decreases as the observed imprecision increases relative to the claimed imprecision and when a greater number of days, runs, or replicates are included in the verification design. Increasing the number or days, runs, or replicates also reduces the FAR for between-day imprecision and repeatability.

Conclusions

Design of verification experiments should incorporate the local availability of resources and analytical expertise. The largest imprecision component should be targeted with a greater number of measurements. Consideration of both FAR and FRR should be given when committing a platform into service.

Precision describes the reproducibility characteristic of an analytical system. Simplistically, it describes how tightly quantitative measurements cluster together when testing the same material repeatedly over time. Precision can be divided into several components, including repeatability (ie, within-run), between-run, between-day, and within-laboratory (ie, total) imprecision.¹ It is often expressed as a standard deviation (σ) in the measurand’s units or coefficient of variation (CV) generally as a percentage. The precision profile of an analytical system is the analytical σ or CV at different measurand concentrations and in most cases provided by the manufacturer. Laboratories are required to verify the precision claimed by the manufacturer before committing the system to clinical service.^2,3 It is critically important to ensure that the precision of an analytical system conforms to the manufacturer’s claim, as increased imprecision can lead to an increase risk of erroneous laboratory results, subsequently affcting clinical interpretation and patient management.^4-6

While guidelines exist for precision verification, such as those from the Clinical and Laboratory Standards Institute (CLSI; EP15-A3),¹ these are variably followed.⁷ Variations to these guidelines include the number of days, number of runs per day, and number of replicates per run. Factors influencing the choice of experimental design include financial, operational, and analytical expertise. In general, the greater the number of tests performed, the more resource intensive and complex the exercise becomes. As complexity in the experimental design increases, the greater the requirement for advanced statistical tools for data analysis becomes. For example, an experimental design that considers days, runs per day, and replicates per run requires nested analysis of variance (ANOVA) to derive the corresponding variability of each component.¹

The CLSI guideline EP15 for end-user laboratories recommends a 5 × 5 experimental design, which entails measuring the same material five times a day over 5 days.¹ In addition, EP15 introduces an upper verification limit (UVL) factor that seeks to keep false rejection rates within 5%. A false rejection is the scenario of failing the precision verification excessive by chance alone, when the system is in fact performing within the manufacturer’s precision claim. The main rationale for introducing the UVL is to avoid the operational costs associated with unnecessarily repeating the verification exercise.

However, the CLSI EP15 does not provide information on the related false acceptance, which is the case of having a system passing the precision verification exercise by chance, when in actual fact the precision performance is worse than the stated manufacturer’s claim. From a patient harm-minimization perspective, it is more important to ensure that the false acceptance rate is minimized. A laboratory should give balanced consideration to the rate of both false rejections and false acceptances, to ensure delivery of safe clinical care within available resource constraints. To do so, the laboratory needs to have information on both aspects of false acceptance and rejection rates to make an informed decision on the experimental design and assist in interpretation of the results generated.

The objective of this report is to evaluate the effect of precision verification experimental design on the false acceptance and false rejection rates for instrument/assay verification prior to implementation of an analytical system into clinical practice.

Materials and Methods

Types of Experimental Designs

The precision verification study can be broadly divided into matrix designs, in which several measurements are made each day, in comparison to simplistic designs, in which only one measurement is made daily for several days (ie, days × 1 replicate) or several measurements are made for only 1 day (replicates × 1 day). For the complex matrix designs, the daily measurements can be performed across several runs and several days (ie, days × runs × replicates) or in a single run over several days (ie, days × replicates). Formal definitions of terminology related to precision verification can be found in the Supplemental Material (all supplemental materials can be found at American Journal of Clinical Pathology online).

Day × Replicate Matrix Design

Under this experimental design, a fixed number of replicates are measured consecutively in a single run each day and repeated over multiple days. For day × replicate designs, it is necessary to employ one-way ANOVA analysis to derive the components of imprecision.

For clinical laboratory assay verification, experiments are conducted to determine if the observed standard deviations for within-laboratory imprecision and repeatability imprecision, denoted as $sday$ and $srep$ respectively, are within the claimed specifications.

When the experimental design is balanced (ie, no missing data), the mathematical model for measurements obtained from the precision verification for a day × replicate study is as follows:

where $Yij$ = observed measurement on day i and replicate j; $μ$ = mean, estimated as the average of all readings; $Di$ = error due to day-to-day variation on day i; and $εj(i)$ = within-run error for replicate j on day i.

The results obtained from the verification experiments can be summarized in a one-way ANOVA table:

where $nday$ = number of days (assuming one run per day), $nrep$ = number of repetitions per day, $N=nrepnday$⁠, $σday2$ = day-to-day variance, and $σrep2$ = repeatability variance.

Here, $σday2$ refers to the “pure” between-day variance, that is, between-day variance that has been corrected for the contribution of within-day variance. If the matrix study is balanced, the degrees of freedom for repeatability, $dfrep=N−nday$⁠, can be obtained directly from the ANOVA table.

To determine a variance that involves a linear combination of two or more independent mean squares, the complex estimate of variance will be adopted.⁸ Instead of using the degrees of freedom in the ANOVA table, the effective degrees of freedom for within-laboratory imprecision are derived based on the Satterthwaite formulation^8,9:

Similarly, the effective degrees of freedom for a between-day variance can be derived as

Alternatively, depending on the manufacturer’s claim, $dfWL$ can also be written with $σWLσrep$ instead of $σdayσrep$

where $σWL=σday2+σrep2$

The derivations of these equations are detailed in the Supplementary Materials.

To derive the false acceptance rate when the imprecision is actually greater than the manufacturer’s claim, the ratio $σdayσrep$ is adopted to allow independent examination of the increase in $σday$and $σrep$ while $σWL$ is dependent on $σrep$⁠. Hence, $dfWL,eff$ will not be used in this study, but the formula is listed here for completeness. The statistical derivation of false acceptance rate is described in the section below. The false rejection rate can be derived as described previously.¹

It should be noted that the effective degrees of freedom, df_eff, are derived based on the ANOVA table, which is established from observed experimental data. However, as described in CLSI EP15, df_eff are estimated based on the variances claimed by the instrument manufacturers rather than the observed variances. This approximation will allow users to evaluate the precision verification experimental design and estimate the corresponding false acceptance rate before collecting the data from the full experiments. In practice, the value of df_eff calculated from observed and claimed variance typically does not differ significantly.

Day × Run × Replicate Matrix Design

Under this experimental design, a fixed number of replicates are measured consecutively within a fixed number of times (runs) per day and repeated across multiple days. This experimental design is generally more resource intensive and less commonly employed for imprecision verification. For day × run × replicate designs, it is necessary to employ two-way nested ANOVA analysis to derive the components of imprecision.

The mathematical model for measurements obtained from performance evaluation for a day × run × replicate design is as follows:

where $Yij$ = observed measurement on day i and replicate j; $μ$ = mean, estimated as the average of all readings; $Di$ = error due to day-to-day variation on day i; $Rj(i)$ = error due to run-to-run variation for run j on day i; and $εk(ij)$ = within-run error for replicate k for run j on day i.

The results obtained from the verification experiments can be summarized in the ANOVA table for a two-way nested design.

where

$nday$ = number of days, $nrun$ = number of runs per day, $nrep$ = number of repetitions per run, and $N=ndaynrunnrep$⁠.

The degrees of freedom for repeatability, $dfrep=N−ndaynrun$⁠, can be obtained directly from the ANOVA table. The degrees of freedom for between-day, between-run, and within-laboratory imprecision are derived based on the Satterthwaite formulation¹⁰:

Take note that $dfday,eff$ is dependent on two ratios, $σdayσrep$ and $σrunσrep$⁠.

Although $dfWL,eff$ will not be calculated in the Results section, it is listed here for completeness.

where

The day × run × rep design can be employed to verify the claims on $σday$⁠, $σrun$⁠, and $σrep$ based on the observed standard deviation $sday$⁠, $srun$⁠, and $srep$⁠, respectively. The false acceptance rate and false rejection rate can be derived as described in the section below.

Days × 1 Replicate and Replicates × 1 Day (Nonmatrix) Design

This is the most simplistic experimental design in which a single measurement is performed over multiple days or multiple replicate measurements are performed in a single day. The mean, σ, and CV can be simply calculated across the days (⁠$σday$⁠) or the replicate measurements (⁠$σrep$⁠). The within-laboratory imprecision (⁠$σWL)$ is calculated using all the data (ie, all the replicates and days). This study design does not require ANOVA analysis and complex formulation for determination of the degrees of freedom for each variance component.

Hypothesis Test

The imprecision of the assay can be assessed by comparing the observed standard deviation, s, with the manufacturer’s claimed σ using a hypothesis testing. The null hypothesis $H0$ and the alternative hypothesis $H1$ are as follows:

$H0$ suggests that the observed verification experimental variance $s$ falls within the manufacturer’s claimed $σ$ while $H1$ states that the observed $s$ is larger than the manufacturer’s $σ$⁠. The $χ2$ test of a single variance can be applied to inspect the validity of $H0$ by determining the probability of observing standard deviations of varying magnitude relative to $σ$⁠. The value of $χ2$ is given by

where $dfeff$ = effective degree of freedom.

False Rejection Rate

In its simplest form, a precision verification can reject an assay if any observed $s$ is larger than manufacturer’s $σ$ by direct comparison. However, even when the actual precision of the assay is equal to or better than the supplier’s claim, there is a possibility that the assay may be rejected through random sampling of the actual imprecision, leading to a false rejection. The false rejection rate can be obtained from the critical value from the $χ2$ distribution. When this simple direct comparison of standard deviations is used, the critical value or threshold value for $χ2$ is just the degrees of freedom as $s$ is equal to $σ$ (ie, $H0$ is true).

The area under the curve of the $χ2$ distribution to the right of this critical $χ2$ indicates the probability for obtaining $s$ values that are higher than $σ$ values, which is the false rejection rate $α$⁠, given as (Supplemental Figure 1)

When multiple samples (ie, concentrations) are examined, it is assumed that the assay will be rejected if any one of the samples demonstrates a standard deviation, which is greater than the claimed value. Therefore, the total false rejection rate is given by

Impact of Adopting the UVL on the False Acceptance Rate

CLSI EP15 introduced the UVL to reduce the false rejection rate in assay precision verification experiments.2 The UVL factor, $F$⁠, is a multiplier to the manufacturer’s claimed values introduced to limit the false rejection rate for $σday$ and $σrep$ to 5% each under the day × replicate experimental design.

where $χα,df2$ is the critical $χ2$ value with $α$ of 5% for a single sample experiment. The $dfeff$ of the $χ2$ distribution is determined by the calculation described in the previous sections. The value of $α$is dependent on the number of samples at different concentration levels, $nsamples$⁠, employed in the verification experiments:

If the observed precision values are smaller than the claimed values multiplied by $F$ (ie, $s<Fσ$⁠), assay verification can be accepted. However, the introduction of UVL increases the risk of false acceptance of assays that exceed the manufacturer’s claimed specifications (Supplemental Figure 1). To date, the CLSI EP15 document does not specify this false acceptance rate for a precision experiment with varying degrees of freedom.

False Acceptance Rate

If an assay imprecision is $ψ$ times larger than the claimed value, the corresponding false acceptance rate $β$ can be derived for the acceptance criteria based on simple comparison with CLSI EP15 guidelines.

For direct comparisons, the $β$ value for each sample is given by

If UVL is applied, the $β$ value for each sample is instead given by

When multiple samples are used, it is assumed that the assay will be accepted only when all the samples exhibit standard deviations that are smaller than the claimed value (Supplemental Figure 2). The total false acceptance rate is thus given by

The determination of UVL factor for a day × run × replicate matrix design follows the same procedures as the run × rep design.

Results

Day × Replicate (Balanced ANOVA) Design

Table 1 examines the effect of the number of samples included in the 5-day × 5-replicate design. Under this experimental design, the probability of having an observed imprecision, $s$⁠, that is larger than the claimed $σ$⁠, when the actual imprecision is equal to $σ$⁠, is summarized in the false rejection rate column. When more samples are included in the precision experiment, they increase the false rejection rate as a consequence of the increased number of hypothesis tests being performed. In addition, in Table 1 the UVL keeps the false rejection rate at 5% for experiments involving a different number of samples but significantly increases the false acceptance rate. Also noteworthy is that the false acceptance rate reduces when the observed imprecision increases relative to the manufacturer’s claimed imprecision.

Table 2 displays the effect of the number of days and repetitions on the false acceptance and false rejection rate for experimental designs with three samples. Not surprisingly, the false rejection rate decreases as the observed imprecision increases relative to the claimed imprecision and when a greater number of days or replicates are included in the verification exercise. Increasing the number of days or replicates also reduces the false acceptance for between-day imprecision and repeatability. Nevertheless, there is a diminishing return on false acceptance and rejection rates with each incremental increase in the number of days or replicates.

Table 3 shows a comparison of different experimental designs of day × replicate for three samples with a similar total number of measurements performed. A higher number of day/replicates included in the experimental design have the effect of improving the confidence in the estimation of the imprecision, thereby leading to reduced false acceptance and false rejection rates.

Day × Run × Replicate (Balanced ANOVA) Design

Under this experimental design, the false acceptance rate is similarly reduced by increasing the number of samples, number of days, number of runs, and number of replicates performed, as well as the degree the actual imprecision exceeds the claimed imprecision Table 4 and Table 5. As seen in the day × replicate experimental design above, there is a clear diminishing return on increasing the number of days, runs, or replicates on the false acceptance rate, shown in Tables 4 and 5.

Days × 1 Replicate or Replicates × 1 Day Nonmatrix Design

Finally, some laboratories may not adopt the matrix experimental design (balanced ANOVA). Instead, the between-day imprecision is estimated by measuring the samples once per day across multiple days. The repeatability is estimated by running the samples multiple times in a single day. Under this experimental design, the claimed ratio of σ _day/σ _rep is not relevant here, as the degrees of freedom for each variance component are independent of the claimed ratio. The degrees of freedom for each variance component are simply the number of measurements used in the calculation minus 1. In addition, the role of days and replicates is interchangeable in such designs.

As with the matrix experimental designs, increasing the number of samples, days, and replicates, as well as the magnitude of increase in imprecision relative to the manufacturer’s claim, reduces the false acceptance rate with diminishing returns, as shown in Table 6. When n_day = 1, between-day imprecision cannot be defined; similarly, when n_rep = 1, repeatability within the same day cannot be determined. Therefore, the cases for n_day or n_rep with the value of 1 are not shown in Table 6. In general, for a given total number of measurements, the nonmatrix experimental design performs well compared to the day × replicate (matrix) experimental design when an equal number of measurements are allocated for day and replicate, respectively, as shown in Table 3.

Discussion

Precision verification is a critical preimplementation method evaluation study. However, it can be resource intensive as it involves repeated testing that consumes reagents, quality control materials, pooled patient samples, and other consumables. In addition, it can be operationally demanding as it requires dedicated suitably trained personnel to perform the measurements and collect and analyze the data, resulting in a time-consuming process. While false rejection of the imprecision verification data may result in a new round of verification exercises that is similarly resource and time-consuming, a false acceptance may potentially lead to adverse clinical outcomes. Hence, laboratories should carefully consider the available resources and design the precision verification study to address these analytical and clinical requirements in an informed and balanced manner. Often, the laboratory is constrained by the total number of measurements it can perform for the entire precision verification study.

From the results of this study, several broad recommendations can be made.

Experimental Design

Experimental design can be adjusted based on which of the imprecision components the laboratory wishes to emphasize. In general, more importance should be given to verifying the imprecision component with the largest variation as reported by the manufacturer, which is most often the between-day imprecision. Accordingly, the experimental design should perform the verification over more days, even at the expense of less replicates/runs per day. For example, if the laboratory has determined that it can only perform a total of 20 measurements for a given concentration for a precision verification study, an experimental design with 10 days × 2 replicates will better verify the between-day imprecision compared with a 5-day × 4-replicate experimental design—but at the significant expense of the repeatability component (Table 3).

Importantly, laboratory practitioners should bear in mind the diminishing returns of increasing the number of measurements for a particular experiment component (day, run, replicate, sample). While a matrix experimental design may allow the laboratory to complete the verification exercise more expediently compared with a nonmatrix design, it is generally associated with higher false acceptance rates. The inclusion of more measurements in the verification exercise reduces false acceptance and false rejection rates by reducing the uncertainty and subsequent confidence limits of the estimates.

Statistical Approach

To optimize the data collected, the laboratory should apply the correct statistical approaches. A matrix experimental design is optimally evaluated using ANOVA techniques. Software for analysis of simple matrix designs can be performed using Excel (Microsoft). However, more complex designs may require use of dedicated software packages such as the VCA package¹¹ in the R statistical environment.¹² Nonmatrix experimental designs can be simplistically examined using simple standard deviations and coefficients of variation.

Consider Both False Acceptance and False Rejection Rates

While there is an instinctive preference to reduce false rejection rates and avoid unnecessarily repeating experiments, it is also important to consider the probability of falsely accepting an assay that has increased imprecision. An increased imprecision has significant clinical impact, producing results with greater variability that can lead to an erroneous diagnosis or obscure the trends of a measurement when serially monitored.^4-6 Moreover, it can also increase the frequency of quality control and proficiency testing failures that require significant efforts to investigate, leading to long-term operational inefficiencies.

Replace Missing Values

During an imprecision verification, a measurement value(s) may be missing due to outlier exclusion or missed measurement(s). However, it is important to ensure that the missing values of a verification exercise are suitably replaced. This is particularly important for the matrix experimental designs since most statistical software available to the laboratories assumes a balanced design (ie, no missing values). To replace the missing value(s), the experiment for the entire day should be repeated to ensure the variability of the assay is again adequately captured.

Considerations for Repeating a Failed Imprecision Verification Exercise

When a verification exercise fails to meet the manufacturer’s claim, the laboratory may elect to repeat the entire process. In such instances, the laboratory should select an experimental design that best assesses the particular failed component. For example, if the repeatability failed to verify against the manufacturer’s claim using a matrix experimental design with 10 days × 2 replicates, a repeatability verification exercise by measuring 10 replicates in a single day should be considered.

What to Do When the Precision Verification Fails More Than Once

Assuming that the precision study is conducted properly, and the laboratory is satisfied that there are no pre- or postanalytical issues at fault (eg, instability of materials, erroneous material handling, data entry errors), a repeated failure of the precision verification is likely to indicate that the assay is not performing to the manufacturer’s claims. This is especially true if the verification failed despite application of the UVL, since this limit is meant to cap the false rejection rate at 5%. A repeat failure means that there is only 0.25% (5% × 5%) probability that the assay is performing within the manufacturer’s claim.

References