Deadly statistics: quantifying an ‘unacceptable risk’ in capital punishment^†

1. Introduction

‘The record of how human beings have performed on IQ tests does more than measure us against one another for entry into Harvard law school.’

—James R. Flynn, Intelligence and Human Progress 3 (2013)

For some of us, a single IQ point can mark the difference between life and death. Such is the teaching of Hall v. Florida.¹Hall is the Supreme Court’s third—and most detailed—case to address just when a criminal who might be intellectually disabled can nevertheless be sentenced to death. In Hall, the Court held unconstitutional Florida’s use of a measured IQ score of 70 to mark the upper limit of intellectual disability. Four dissenting Justices were incensed. In an opinion written by Justice Alito,² they interpreted Justice Kennedy’s majority opinion³ to mean that only IQ scores of 75 or more could serve as a cut-off.⁴ Using their own computations of ‘confidence intervals’ (CIs),⁵ they concluded that this rule ‘totally transforms the allocation and nature of the burden of proof’.⁶ The Court reached a rule ‘unhinged from legal logic’,⁷ Justice Alito lambasted, because it ‘misunderstands how the SEM [standard error of measurement] works’⁸ and made ‘factual mistakes that will surely confuse States attempting to comply with its opinion’.⁹

The opinions of this divided Court have escaped probing analysis of their treatment of the statistical issues in ascertaining intellectual disability.¹⁰ Impressed with potential doctrinal implications of the case for criminal jurisprudence more generally, commentators have seen it as supporting challenges to other aspects of capital-sentencing procedures.¹¹ Even more broadly, one astute scholar of mental health law sensed in Hall the seeds of a general ‘scientization of the criminal law’.¹² This commentary on the implications of Hall is intriguing and important, but it leaves undisturbed and unexamined the statistical science in Hall and its relationship to constitutional constraints and legal policies.

Therefore, this article focuses on the statistical issues in the case. It examines the more important statements in the Justices’ opinions on measurement error in IQ testing. These statements are of more than didactic and technical interest. The statistical properties of IQ scores are critical to understanding the capital-sentencing options retained by states after Hall. The Hall majority condemned Florida for ignoring ‘a statistical fact’¹³ that constituted ‘one of the most important concepts in measurement theory’.¹⁴ Without careful analysis of the relevant statistical principles, it is not possible to ascertain how much Hall should constrain states that wish to carve out a range of IQ scores that would preclude further proof of intellectual disability. Furthermore, because IQ scores are of major importance in the broader clinical assessments that courts must review, understanding their statistical properties and limitations is vital for all adjudication of intellectual disability claims.

Section 2 of this article sets the stage for the examination of the exchange among the Justices on the technical concepts such as standard error of measurement (SEM) and CIs. It describes the issue in Hall as it emerged in the Court’s ‘zig-zagging death penalty jurisprudence’¹⁵ and the general divide between the majority and the dissent. It reads the majority opinion as accepting the premise that a state may, without further inquiry, deem all individuals with true IQs above a fixed number as not intellectually disabled but that it must adopt some margin of error in applying this categorical rule to measured scores.

Section 3 explains why the state’s choice of an IQ cut-score, whether conceived of as a true score or framed as a measured one, is inherently arbitrary. It shows how the measured score of 70 selected by Florida as well as the not fully specified but higher numbers demanded by the majority affect the proportion of the population who could be found to be intellectually disabled.

Section 4 analyses SEM and resulting CIs that are central to the Court’s disposition of the case. It clarifies ambiguities in the Court’s opinion and criticizes the dissent’s effort to link a CI with the burden of persuasion. It notes that the statistical apparatus of CIs is not even necessary to establish a cut-off for observed IQ scores that accounts for the SEM. Finally, this section identifies a different statistic—the standard error of estimate (SEE)—that is more appropriate than the Court’s SEM in adjusting for measurement error.

Section 5 moves beyond the situation of a single IQ score and the confines of classical test theory. First, it considers the Justices’ discussion of combining IQ scores to achieve greater precision. It shows that the majority’s comments on the difficulty of aggregation should not bar the use of established methods. Similarly, it suggests that the exclusive reliance on classical test theory and CIs in the opinions should not preclude the use of Bayesian statistical procedures that would permit a more perspicacious treatment of the problem of measurement error in IQ scores than the frequentist one that the Court endorsed.

2. The intellectual disability trilogy

In the space of 25 years, the Supreme Court has moved from permitting the state to execute a man with an IQ in the 50–63 range¹⁶ to barring the state from executing a man with an IQ in the 71–80 range.¹⁷ This transformation began in 1989, in Penry v. Lynaugh.¹⁸ A welter of conflicting opinions in Penry established that before imposing a capital sentence, the sentencing judge or jury must have the opportunity to consider and give appropriate weight¹⁹ to what was then called mental retardation.²⁰ At the same time, the Court declined to hold that the Eighth Amendment’s ban on cruel and unusual punishment insulated all mentally retarded individuals from death at the hands of the state. Writing for the majority, Justice O’Connor concluded that ‘there is insufficient evidence of a national consensus against executing mentally retarded people convicted of capital offenses for us to conclude that it is categorically prohibited by the Eighth Amendment’.²¹ Four dissenting Justices did not dispute this assessment of national opinion, but they insisted that capital punishment is unacceptable for ‘the mentally retarded as a class’²² because their limitations make the death sentence invariably disproportionate.²³

A scant 13 years later, in Atkins v. Virginia,²⁴ the Court found a ‘dramatic shift in the state legislative landscape’.²⁵ The requisite ‘national consensus’²⁶ that it is categorically wrong to execute ‘the mentally retarded’ had coalesced.²⁷ Moreover, Justice Stevens’s majority opinion buttressed this conclusion with the arguments of the dissenters in Penry. The execution of the mentally retarded, he maintained, does not measurably advance the goals of retribution and deterrence.²⁸ In addition, he linked the substantive conclusion that it is wrong to execute anyone who is intellectually disabled to the accuracy of the procedures for identifying death-worthy defendants:

The reduced capacity of mentally retarded offenders provides a second justification for a categorical rule making such offenders ineligible for the death penalty. The risk that the death penalty will be imposed in spite of factors which may call for a less severe penalty … is enhanced, not only by the possibility of false confessions, but also by the lesser ability of mentally retarded defendants to make a persuasive showing of mitigation in the face of prosecutorial evidence of one or more aggravating factors. Mentally retarded defendants may be less able to give meaningful assistance to their counsel and are typically poor witnesses, and their demeanor may create an unwarranted impression of lack of remorse for their crimes. As Penry demonstrated, moreover, reliance on mental retardation as a mitigating factor can be a two-edged sword that may enhance the likelihood that the aggravating factor of future dangerousness will be found by the jury. Mentally retarded defendants in the aggregate face a special risk of wrongful execution.²⁹

But Atkins left unclear and unresolved the question of precisely who is ‘mentally retarded’ for Eighth Amendment purposes. Because Penry permitted capital punishment of unequivocally intellectually disabled offenders, it created little pressure to define intellectual disability carefully. By fashioning a blanket exemption from capital punishment for everyone ‘within the range of mentally retarded offenders about whom there is a national consensus’,³⁰ however, Atkins brought the problem of line drawing to the foreground. Yet, the Court made no effort to define ‘the range of mentally retarded offenders’. Instead, it grandly announced that ‘in determining which offenders are in fact retarded … “we leave to the State(s) the task of developing appropriate ways to enforce the constitutional restriction …” ’.³¹

Still, Atkins dropped some breadcrumbs. A footnote implied that the ‘clinical definitions’ of the American Association on Mental Retardation (AAMR) and the American Psychiatric Association (APA) would do the trick.³² The AAMR’s definition referred to ‘significantly subaverage intellectual functioning, existing concurrently with related limitations in two or more … adaptive skill areas … and manifest[ing] before age 18’.³³ The fourth edition of the APA’s Diagnostic and Statistical Manual of Mental Disorders (DSM-4) was ‘similar’ and added that ‘ “[m]ild” mental retardation is typically used to describe people with an IQ level of 50-55 to approximately 70’.³⁴ In addition, in concluding that execution of the mentally retarded had become ‘truly usual’,³⁵ the Court observed that ‘only five [states] have executed offenders possessing a known IQ less than 70 since we decided Penry’.³⁶

States responded to Atkins’ Olympian command or to earlier public discomfort with executions of the intellectually disabled³⁷ in various ways. Several states enacted laws using a measured IQ score of 70 (or a corresponding number of standard deviations [SDs] below the population mean) as the upper limit for intellectual disability.³⁸ Florida had enacted such a law even before Atkins. Its statute, as interpreted by the state’s Supreme Court, effectively ‘defined intellectual disability to require an IQ test score of 70 or less. If, from test scores, a prisoner is deemed to have an IQ above 70, all further exploration of intellectual disability is foreclosed’.³⁹

Florida applied its statute to Robert Lee Hall. Although Hall’s family and educational history provided distressing evidence of intellectual disability,⁴⁰ the IQ scores deemed admissible at a post-Atkins hearing were too high to permit this compelling evidence to be considered. They ranged from 71 to 80.⁴¹ And so, 12 years after Atkins, the Court squarely confronted the issue of defining intellectual disability in Hall v. Florida. Descending from Olympus to Delphi, Justice Kennedy invoked the APA’s view as expressed in its Diagnostic and Statistical Manual (DSM) that because IQ scores are imprecise, a somewhat higher measured score than 70 could justify clinical inquiry into the individual’s adaptive functioning in school, in the family, and elsewhere.⁴² Thus, professional criteria that had appeared in Atkins to be merely sufficient for a finding of intellectual disability—because they were roughly congruent with legislative and popular views of what it means to be so disabled—turned out to be ‘a fundamental premise of Atkins’.⁴³

It is easy to read this statement in Hall about the DSM as declaring that this branch of death penalty jurisprudence must track the diagnostic criteria of psychiatrists and psychologists.⁴⁴ Adopting this reading of Hall, the dissent scoffed that the majority had equated ‘evolving standards of decency’—which is the traditional test for determining the scope of the Eighth Amendment⁴⁵—with ‘the evolving standards of professional societies’.⁴⁶ But the majority’s language is more restrained. Justice Kennedy cautioned that the professional standards informed but did not dictate the outcome.⁴⁷ Even so, the Court gave great weight to the views of the psychiatrists and psychologists. In this case at least,⁴⁸ their standards informed the Court not only that there was a unanimous expert consensus,⁴⁹ but also that the consensus rested on a ‘statistical fact’—‘one of the most important concepts in measurement theory’⁵⁰—that ‘[e]ach IQ test has a “standard error of measurement”, … often referred to by the abbreviation “SEM” ’.⁵¹ And, most other capital punishment states accepted the professionally prevalent practice of considering adaptive functioning for individuals with IQ scores as high as two SEMs above 70.⁵² From these facts alone, the Court inferred that Florida’s ‘rigid rule … creates an unacceptable risk that persons with intellectual disability will be executed and thus is unconstitutional’.⁵³

Strictly speaking, this conclusion—that a state may not use a measured-score cut-off as low as 70—is as far as the Court’s holding goes. This holding is grounded upon the concept of a ‘true score’. Intuitively, the true score is the defendant’s actual IQ, which might be somewhat different from each and every measurement of it.⁵⁴ The case establishes that (when it is consistent with the unanimous position of the mental health community)⁵⁵ a state may conclusively presume that all defendants whose IQs ‘truly’ are above a specific number (such as 70) are not intellectually disabled—but it may not apply this presumption to defendants with ‘measured’ scores that are only somewhat higher than this targeted number. This holding invites a series of questions: why can any true score be the basis for a ‘conclusive’ presumption of non-disability? How low can this score be? How much higher than the lowest true score cut-off must the lowest ‘measured’ cut-score be? Section 3 responds to the first two questions. Sections 4 and 5 address the last question.

3. The need to allow states to use cut-scores and the meaning of ‘significantly subaverage’

The Hall Court did not reject all IQ cut-offs as unconstitutionally rigid.⁵⁶ One could imagine an opinion striking down the Florida law by asserting that intellectual disability is a single, overarching category to be managed by psychologists or psychiatrists informed by multiple streams of information about the constructs of both intellectual and adaptive functioning, and holding that the Florida law was unconstitutional because it did not allow these experts to present all that information. This outcome would have been even more deferential to the expert community than was Justice Kennedy’s opinion in Hall. It would have resolved the matter by decisively committing every Atkins claim to warmer and fuzzier clinical judgements as opposed to erecting a cold and unyielding statistical barrier.⁵⁷

But could the Eighth Amendment (or any other part of the Constitution) be held to prevent a state from using some cut-score for administrative convenience? Surely, at some point a high IQ score means, ipso facto, there is no significant risk of executing someone who cannot be said to deserve this punishment because of an intellectual incapacity. The ‘unacceptable risk’ approach—which implies that some level of risk is acceptable—was a less-radical response to implementing Atkins. The Hall majority did not dispute Justice Alito’s observation that ‘[a] defendant who does not display significantly subaverage intellectual functioning is [simply] not among the class of defendants we identified in Atkins.’⁵⁸ ‘Significantly subaverage intellectual functioning’ as introduced in Atkins is integral to the conceptualization of intellectual disability in psychology, and a standardized test is a more objective and reliable instrument than is a clinician’s impressions of how far a defendant’s intellect is from the population norm. Medically, a substantial impairment in intellect is necessary to a diagnosis of intellectual disability,⁵⁹ and low IQ scores currently are essential to an expert determination of a substantial impairment.⁶⁰

IQ scores are so important because intellectual disability is not a condition that can be attributed to a single mechanism such as the loss of dopamine-producing brain cells that causes Parkinson’s.⁶¹ It is no less real,⁶² but its very definition is based in large part on its rarity in the population. The question of how rare is rare enough has plagued efforts to develop satisfactory diagnostic guidelines. Over the past 50 years, proposed IQ cut-off scores have varied from one to two SDs below the population mean.⁶³ The popularity of the ‘more traditional’⁶⁴ figure of two SDs seems to reside in a desire to keep the number of diagnoses low, partly to avoid stigmatizing large numbers of minorities in school settings.⁶⁵

As the number of SDs for the cut-off grows, the proportion of people who are subject to the classification shrinks. IQ scores, like height, weight, and many other physical characteristics,⁶⁶ tend to have a ‘normal’ or ‘Gaussian’ distribution in the population.⁶⁷ Consequently, this relationship is strongly non-linear in the vicinity of –2 SD. A score change of a few points can dramatically change the proportion of people who are affected. Because this fact has a good deal to do with the professional preference for one cut-score over another, it is worth pausing to draw a picture of where the cut-scores discussed in Hall lie.

The Florida legislature defined ‘significantly subaverage general intellectual functioning’ as ‘performance that is two or more standard deviations from the mean score on a standardized intelligence test’.⁶⁸ Justice Kennedy translated this into the scale on which IQ scores are reported as follows:

The concept of standard deviation describes how scores are dispersed in a population … . The standard deviation on an IQ test is approximately 15 points, and so two standard deviations is approximately 30 points. Thus a test taker who performs ‘two or more standard deviations from the mean’ will score approximately 30 points below the mean on an IQ test, i.e., a score of approximately 70 points.⁶⁹

The ‘normal’ distribution is the bell-shaped one prominent in elementary statistics courses. The exact shape of all normal distributions can be determined from just two numbers—the mean and the SD. The mean states where the bell sits, and the SD determines how steeply its sides flow down from the top.

A few symbols will help immeasurably in keeping track of the different quantities that are important in Hall. We can use x to refer to all the IQ scores in a population, μ_x to denote their mean, and σ_x to represent their SD. As is conventional in statistics, z will denote the scores expressed in units of SDs.⁷⁰ As shown in Fig. 1, for an IQ test with a population mean of μ_x = 100 and a population SD of σ_x = 15, 2.5% of the scores lie at or below 70 (z = –2, which is to say 2σ_x below the mean). More than twice as many, 5.1%, fall at or below 75 (z = –1.67), and nearly seven times as many, 16.7%, occur at or below 85 (z = 1).

Fig. 1.

Frequency distribution of IQ scores in a population with a mean of 100 and a SD of 15. Note that 2.5% of the scores lie below 71 (the Florida cut-off), 5.1% are below 76 (the Hall Court’s suggestion), and 16.7% are below 86 (an older cut-off) (IQ scores are integers, and the height of each bar is proportional to the relative number of people at each integer. The normal curve is a continuous line that fits these heights. In this figure and throughout this article, I use the area under the curve from x – 0.5 to x + 0.5 to compute the proportion of people with an integral IQ score x.).

Open in new tabDownload slide

It may be worth summarizing what we have learned about the law, psychology, and statistics of IQ cut-scores so far. Both opinions in Hall treat a major deficit in intellectual (as opposed to adaptive) functioning as a necessary but not sufficient condition for death penalty disqualification, and both regard IQ tests as a valid measure of the relevant intellectual functioning.⁷¹ For decades, behavioural scientists and clinicians, seeking to limit the percentage of the population that would be labelled intellectually disabled and appreciating the shape of the normal curve that describes the distribution of IQ in the general population⁷² have deemed a departure of at least twice the population SD σ_x as indicative of the ‘significantly subaverage intellectual functioning’ that is essential to a diagnosis of disability.

The choice of any particular cut-score to mark ‘significantly subaverage’ obviously is arbitrary, for there is no precise point at which the quality ‘significantly subaverage’ springs into existence. There is little or no difference in intellectual functioning in people who are within an IQ point or two of one another. Thus, the choice of any particular number of SDs is largely a convention.⁷³

The inherent room for debate regarding this convention and its accordion-like history⁷⁴ did not trouble the Court in Hall. To the contrary, the majority took the use of two SDs as an unquestioned starting point in defining the kind of intellectual disability that precludes capital punishment. With that value in place, the case turned on a second convention—one pertaining to measurement error. As stated in Section 2, the majority held that a cut-score of z = –2 would be satisfactory for true scores, but was constitutionally deficient for real scores (those with measurement error). Measurement error prevented a state from circumventing the need for a more comprehensive (and softer) clinical evaluation of offenders with IQ scores in the range of z = –1.67 to z = –2.

In the next section, I try to clarify what statistical theory has to say about the constitutionally tolerable location of the threshold for a conclusive presumption that the offender is not intellectually disabled. This section considers, corrects, and builds on the Justices’ treatment of the following statistics and ideas: (1) the meaning of a test-taker’s ‘true score’; (2) estimates of test reliability; (3) estimates of the test’s SEM; and (4) the CI for the test-taker’s true score.

4. True scores and single-measurement error within classical test theory

4.1 First- and second-order questions

Why is z = 1.67 SDs below the mean—the figure that the Court seemed to accept—the highest that the Constitution permits? Why not z = 2.00 as Florida wanted? One possible answer is that the 2.47% of the population whose scores lie between these two values includes too many people who are actually intellectually disabled and hence death disqualified. Presumably, this is what the Court meant when it spoke of ‘an unacceptable risk that persons with intellectual disability will be executed’.⁷⁵ but Justice Kennedy’s opinion does not quantify this risk or the trade-off with the risk of not executing an offender who, under existing law, otherwise deserves to die. The only reason the Court offered to suspect that there is a substantial fraction of intellectually disabled capital offenders in this IQ range is that psychologists and psychiatrists have guidelines that use z = –2 as a benchmark (because scores this low are kind of rare) but then blur the boundary by referring to the possibility of measurement error in individual cases. For example, the American Association on Intellectual Development and Disabilities (AAIDD) states that a ‘significant limitation[] in intellectual functioning’ exists when ‘[a]n IQ score … is approximately two standard deviations below the mean, considering the standard error of measurement for the specific assessment instruments’ used and the instruments’ strengths and limitations’.⁷⁶

The majority ran with this idea. As indicated in Section 3, it adopted the conventional choice of z = –2 (corresponding to 70 points on the usual IQ score scale), for the line that separates the intellectually able from the intellectually disabled but then argued that the ‘inherent error’ in IQ tests means that to be reasonably sure of drawing the line at 70 (the figure selected because the number of people whose scores fall at or below 70 is not too large), it must be drawn at 75. Adjusting for measurement error in this manner represents imposing a second convention (relating to the uncertainty in individual measurements) on top of the first (establishing the departure from the average that suffices to permit a diagnosis of intellectual disability).

This two-part framework presupposes that the choice of z = –2 is at or near the edge of what is constitutionally permissible. If Florida could have drawn the first-order line substantially below this value, then z = –2 already contains a tolerable margin for error. The failure to explain how the AAIDD–APA choice of an IQ score designed to prevent stigmatizing too many people as intellectually disabled captures the reasons expressed in Atkins for regarding mental retardation as a death-disqualifying condition exposes Justice Kennedy’s opinion to the objection that ‘definitions of intellectual disability … are promulgated for use in making a variety of decisions that are quite different from the decision whether the imposition of a death sentence in a particular case would serve a valid penological end’.⁷⁷

Of course, direct attention to the validity of IQ scores for the desired purpose could lead to a higher rather than a lower score for precluding further inquiry into the existence of the disability.⁷⁸ My point is not that a state should be free to impose a stricter limit. It is simply to underscore that the Hall Court does not come to grips with the basic question in defining an IQ cut-score.⁷⁹

Pretermitting that question, the Court pursued the second-order question of measurement error. Let us assume that μ_x – 2σ_x = 70 is as high as one can go in demarcating the range of IQ scores that are so aberrant as to permit the diagnosis of intellectual disability. How does it follow that 75 (or some similar number) is the lower bound for scores that make it unnecessary to consider the adaptive functioning prong of the diagnosis? The Court invoked the ‘ “standard error of measurement”, … often referred to by the abbreviation “SEM” ’.⁸⁰ The majority opinion then quoted the DSM-5, which states, ‘Individuals with intellectual disability have scores of approximately two standard deviations or more below the population mean, including a margin for measurement error (generally ±5 points) … . [T]his involves a score of 65–75 (70 ± 5).’⁸¹

The dissent disputed the assertion that ±5 points is the ‘margin for measurement error’. An exasperated Justice Alito complained that ‘there is no reason to assume a SEM of 5 points’⁸² when ‘we know that the SEM for Hall's most recent IQ test was 2.16 — less than half of the Court's estimate of 5’.⁸³ To evaluate this exchange, and to see whether Hall permits a state to specify a cut-off below 75, we must appreciate how the SEM is used to produce ‘a margin of error’ and what alternatives are available. Unless one believes that anything that the APA and AAIDD do must be followed (even if there are statistically superior approaches that would satisfy the concerns articulated in Atkins), it becomes necessary to consider (1) what figure to use for the SD of errors and (2) how many such standard errors are required to keep the risk of misclassification tolerable. We will see that Justice Kennedy could have avoided the ambiguities detected by Justice Alito by expressing the margin of error in terms of some fixed number of SEMs rather than the number 75. And looking more deeply, we will see that the Court’s SEM is not the best standard error to use for an individual’s score and that there are other reasonable ways to use the most appropriate statistic. A constitutional analysis ought to reflect these statistical facts as well as the popularity of the SEM.

4.2 True scores and measurement error

Individual IQ scores are squishy. They are not perfectly reproducible. Measured scores fluctuate around ‘true scores’, and to the extent that the observations erratically depart from the true value, their ‘reliability’ is less than 1. In psychometrics, classical true score theory defines an individual’s unknown ‘true score’ (which we can denote with the Greek letter tau, τ) as the average score that a given individual would achieve after taking the same test an infinite number of times (without learning anything from taking the test each time and without any other relevant changes). The difference between this test-taker’s score x on a given test administration and the true score τ is the error (e) in that measurement (X). In these symbols, x = τ + e. If the error is positive, the true score is less than the measured score. If it is negative, the true score is greater than the observed score. We know the observed score. We want to infer the true score.⁸⁴

We can at least imagine many measurements of the same individual’s IQ. Some of the errors in these measurements will be large, some small. These errors e will have some distribution with a particular shape that gives rise to a mean and a SD. Classical true score theory posits that the errors e in the individual’s repeated scores are normally distributed with mean 0 and an unknown SD. The mean error (μ_e = 0) indicates that the errors are centred on the true score, and the SD (σ_e) tells how spread out they usually are. If we place true scores τ along the x-axis, the observed scores come from the normal distributions that could be drawn around each true score in Fig. 2.

Fig. 2.

Theoretical distribution of observed scores x shown for one individual with a true score of τ = 70 in a population with normally distributed true scores with mean μ_τ = 100 and SD σ_τ = 15. The individual’s observed score distribution (indicated in the smaller histogram) is normal with mean μ_x = τ = 70 and SD σ_e = 2.16.

Open in new tabDownload slide

These normal distributions attached to the true scores must not be confused with the normal distribution that applies to the true IQs in a population. The error distributions for each possible true score apply to individuals at every true score level. They could be different even for test-takers who have the same true scores. Only one such distribution, for one test-taker, appears in Fig. 2. Different individuals at each true score level would have distributions with larger or smaller widths, since some people would be more consistent in their scores on repeated tests. The mean of each individual’s error distribution is μ_e = 0,⁸⁵ not μ_x = 100, and its SD σ_e for a given individual’s error distribution is much smaller that the SD of true scores across the entire population of heterogeneous people. Smaller, but not zero. To assess the precision of the estimated true score τ of a given individual, we need to estimate σ_e for that individual.

4.3 Reliability and SEM

The Court fully recognized that IQ scores of different individuals are expected to fluctuate randomly around the true scores for these individuals. The majority put the point as follows:

Each IQ test has a ‘standard error of measurement’ … often referred to by the abbreviation ‘SEM’. A test's SEM is a statistical fact, a reflection of the inherent imprecision of the test itself … . An individual's IQ test score on any given exam may fluctuate for a variety of reasons. These include the test-taker's health; practice from earlier tests; the environment or location of the test; the examiner's demeanor; the subjective judgment involved in scoring certain questions on the exam; and simple lucky guessing.⁸⁶

In practice, test developers use less-direct methods to estimate reliability. Roughly speaking, they split the test in half and compare the score on one half with that on the other half.⁹¹ These two internal scores are analogous to test–retest scores. It is as if each person took two tests at once. If the internal reliability is ρ = 0.98,⁹² then the SEM is 15 √0.02 = 2.12.

As this example indicates, the Court’s understanding of the overall SEM as an inherent property of the test is not quite correct.⁹³ The SEM can vary with the group used to estimate test–retest reliability; moreover, other measures of reliability are also employed (to account for measurement error induced by different forms of the same test or scoring by different raters).⁹⁴ As Justice Alito wrote, ‘there is not a single, uniform SEM across IQ tests or even across test-takers. Rather, “the [SEM] varies by test, subgroup, and age group” ’.⁹⁵

This complication, however, does not preclude estimates of σ_e for different test-takers. As is typical in applied statistics, there will be arguments about the application of the general concepts and group statistics to individuals,⁹⁶ but let us assume that one has a good estimate of reliability for the kinds of random measurement error that are of concern for the test and the defendant who has taken it. If the reliability, and hence the overall SEM, accounts for all the important sources of departures from the individual’s true score, what can we say about the individual’s true score? The majority thought that the answer lies in using the overall SEM to form a CI around the observed score.⁹⁷

4.4 CIs from the SEM (SEM-IS)

Justice Kennedy asserted that ‘[t]he SEM allows clinicians to calculate a range within which one may say an individual's true IQ score lies.’⁹⁸ He added that ‘SEM is a unit of measurement: 1 SEM equates to a confidence of 68% that the measured score falls within a given score range, while 2 SEM provides a 95% confidence level that the measured score is within a broader range.’⁹⁹ This exposition is not ideal. Clinicians do not use the SEM to decide whether ‘the measured score falls within a given score range’.¹⁰⁰ The measured score is the IQ score x, and one can say with 100% confidence whether this score falls within any interval one likes. Moreover, as explained below, ‘confidence’ is a term of art—95% ‘confidence’ does not have the simple meaning that a lay reader would ascribe to it (and that the Court may have thought it does).

In an act of statistical one-upmanship, Justice Alito supplied examples of how to compute CIs with the SEM. According to the dissent,

If a test-taker scores a 72 on an IQ test with a SEM of 2, the 66% confidence interval is the range of 70 to 74 (72 ± 2). In this situation, there is approximately a 66% chance that the test-taker's ‘true’ IQ is between 70 and 74; roughly a 17% chance that it is above 74; and roughly a 17% chance that it is 70 or below. Thus, there is about an 83% chance that the score is above 70.

Similarly, using two SEMs, we can build a 95% confidence interval. [L]et us hypothesize a case in which the defendant's obtained score is 74. With the same SEM of 2 as in the prior example, there would be a 95% chance that the true score is between 70 and 78 (74 ± 4); roughly a 2.5% chance that the score is above 78; and about a 2.5% chance that the score is 70 or below. The probability of a true score above 70 would be roughly 97.5%.¹⁰¹

The arithmetic is mostly correct.¹⁰² The semantics are not. It is common to form a CI by adding and subtracting some number of SEMs to the observed score x.¹⁰³ And to increase ‘confidence’, one need only add and subtract a larger number of SEMs.¹⁰⁴ To this extent, the opinion is correct. But ‘confidence’ is an abstruse and technical term.¹⁰⁵ It is not, as Justice Alito proposed, the probability or ‘chance that the test-taker's “true” IQ falls within this range’.¹⁰⁶ In classical true score theory, the true score is a fixed number, not a random variable with probabilities distributed across some range of possible values. In Justice Alito’s second example, 74 ± 4 is one of many possible 95% CIs. If we retested the same person, the second score might be 76, giving rise to a new CI of 76 ± 4. A third test might yield a CI of 80 ± 4. As the number of intervals constructed from the formula CI = x ± 1.96 SEM grows infinitely large, 95% of them will include the true score τ, whether that score is between 70 and 78, whether it is between 72 and 80, whether it is between 76 and 80, or whether it is inside any other conceivable interval. Thus, the CI is useful for indicating the precision of an estimate,¹⁰⁷ but ascertaining the probability that the true score is within a given region requires other methods.¹⁰⁸

The conflation of ‘confidence’ with ‘probability’ leads the dissent to accuse the majority of misallocating the burden of persuasion. Justice Alito wrote that

As Hall concedes, the Eighth Amendment permits States to assign to a defendant the burden of establishing intellectual disability by at least a preponderance of the evidence … . In other words, a defendant can be required to prove that the probability of a 70 or sub–70 IQ is greater than 50%. Under the Court's approach, by contrast, a defendant could prove significantly subaverage intellectual functioning by showing simply that the probability of a ‘true’ IQ of 70 or below is as little as 17% (under a one-SEM rule) or 2.5% (under a two-SEM rule). This totally transforms the allocation and nature of the burden of proof.

[I]t would be simple enough to devise a 51% confidence interval—or a 99% confidence interval for that matter. There is therefore no excuse for mechanically imposing standards that are unhinged from legal logic and that override valid state laws establishing burdens of proof. The appropriate confidence level is ultimately a judgment best left to legislatures, and their judgment has been that a defendant must establish that it is more likely than not that he is intellectually disabled. I would defer to that determination.¹⁰⁹

To continue where Justice Alito left off in his construction of CIs, the 51% CI centred on an observed score extends ±0.69 SDs around that score. For the postulated σ_e = SEM = 2, a score of x = 72 produces a 51% CI with a lower bound of 71¹¹⁰; a score of x = 71 yields a lower bound of 70.¹¹¹ According to Justice Alito’s analysis, therefore, a state that defined ‘significantly subaverage intellectual functioning’ as a true IQ score of 70 could exclude all offenders with observed scores (on that IQ test) of 72 or more from further consideration on the theory that it is probably the case that these offenders’ true scores exceed 70.

There are two problems with this reasoning. First, Justice Alito stops short of the logical conclusion to the argument. On the dissent’s understanding of ‘confidence’ and the burden of persuasion, there is no reason to compute a CI. After all, we are only concerned with errors in the lower tail of the curve. (By definition, a defendant whose true IQ score is higher than the interval estimate for any observed score of at least 70 is not intellectually disabled.) The normal curve is symmetric, so exactly 50% of the area under the curve lies to the left of the measured score. Look back at Fig. 2. When the distribution is centred on 70 itself, 50% of the error distribution dips below 70. When it is centred on 71, less than 50% of the error distribution is below 70. The same is true of the error distribution around an observed score of 72, 73, and so on. Consequently, no CI at all needs to be used. Every measured score greater than 70 has the majority of its measured error in the region above 70. No computation is required.

The second and more fundamental problem is that ‘confidence’ is not a probability statement about a fact subject to the burden of persuasion. As I show in a more detailed essay on ‘confidence’ and the burden of persuasion, depriving the state of the opportunity to fix the line at either 70 (considering only the lower tail of the error distribution) or 72 (considering both tails) does not necessarily conflict with the state’s general use of a 51% burden of persuasion; and it is not at all obvious that the more-probable-than-not standard should apply in this context.¹¹² More defensible procedures for establishing a cut-score for measured IQ scores are available. The next section presents one of them.

4.5 SEM-adjusted maximum score (SEM-AM)

Justice Alito defended the 70-point cut-off as much simpler to apply than the majority’s incorporate-a-margin-of-error approach. He remarked that

it is unclear to me whether the Court concludes that a defendant is constitutionally entitled to introduce non-test evidence of intellectual disability (1) whenever his score is 75 or lower, on the mistaken understanding that the SEM for most tests is 5; (2) when the 66% confidence interval (using one SEM) includes a score of 70; or (3) when the 95% confidence interval (using two SEMs) includes a score of 70.¹¹³

Unfortunately, Justice Kennedy did not clarify which of these possibilities the Court had in mind, but it is doubtful that the references to the 70–75 range and to Hall’s concession that a state could choose 75 as an absolute cut-off are integral parts of its holding. The disability-testing guidelines now tend to be framed in terms of standard errors rather than specific scores.¹¹⁴ The Court simply ‘agree[d] with the medical experts that when a defendant’s IQ test score falls within the test′s acknowledged and inherent margin of error [around the statutorily prescribed cut-off for eligibility], the defendant must be able to present additional evidence of intellectual disability, including testimony regarding adaptive deficits’.¹¹⁵ It is not that hard to use a rule that sets the maximum IQ score for a finding of intellectual disability at the legislative maximum for the true score (such as 70) plus 1.96 SEMs for the test in question.¹¹⁶ For an IQ test with an SEM of 2.16, this maximum would be 70 + 1.96 × 2.16 = 74. A score on this test of 75 or above thus would make an offender ineligible for the Atkins exemption.

We can call such a rule an SEM-adjusted maximum score rule (SEM-AM, for short) because it takes the legislatively determined true score and boosts it some number of SEMs to fix the maximum IQ score from a single test that can place an offender (subject to other evidence) in the intellectually disabled range. In symbols, the adjusted maximum is x_max = τ_leg + k SEM, where τ_leg is the legislative choice for the largest true score consistent with intellectual disability and k is the number of SEMs for the desired ‘confidence’ (68% for k = 1, and 95% for k = 1.96). If SEM were equal to the standard error σ_e for every defendant, one could say that the SEM-AM rule for 95% confidence would keep only about 2.5% of offenders with true IQ scores at τ_leg from producing further evidence of their disability.¹¹⁷

In this way, a state can pick a single number x_max, as Justice Alito wanted. Moreover, this simple procedure does the same thing as the SEM-IS rule the dissent deprecated as unduly complicated. In statistical jargon, the SEM-AM rule is simply a null hypothesis test that tells us whether, at the 0.025 level for a false alarm, an observed score is larger than 70.¹¹⁸ The outcomes of this test for statistical significance are identical to the decisions that follow from using two-sided 95% CIs in the SEM-IS procedure. That is, if we decide that a true score is above 70 if and only if the 95% CI for the measured score x is entirely above 70, we will reject the claim of a true IQ of 70 or less in just those cases in which the measured IQ x exceeds 70 + 1.96 SEM.¹¹⁹ Whether we decide according to the SEM-IS or the SEM-AM rule makes no difference.¹²⁰

Two caveats on these rules are in order. First, we cannot specify the expected proportion of defendants who are falsely classified as ineligible, for that depends on how many convicted capital offenders have true scores of 70, 69, 68, 67, and so on. For example, if we were to assume that 30% have true IQs of τ = 70, that 15% have τ = 69, that 8% have τ = 68, and that 4% have τ = 67, then the expected error rate would be (30% × 0.025) + (15% × 0.0077) + (8% × 0.0020) + (4% × 0.0004) + … = 0.88%.¹²¹

Second, as noted earlier, the SEM is a kind of average error across all IQ scores. As an estimator of σ_e, it works best for estimating the variation in true scores that are near the mean for all of these test-takers. But the only offenders who can seek the exemption from capital punishment are in the vicinity of two or more SDs from the population mean (if that is the targeted true score τ_leg). The most appropriate SEM for an SEM-AM rule might come from reliability statistics for a group of low-scoring test-takers in the region of τ_leg.

4.6 CIs from the standard error of estimate (SEE-IS)

Conditioning has a second consequence. It shifts the point on which the interval sits towards the mean. Instead of using the observed score x as the point estimate, we use ρx + (1 – ρ)μ_x.¹²⁵ This adjustment reflects the common, but often unrecognized phenomenon of regression to the mean.¹²⁶ To the extent that chance affects test scores—the central concern of the Hall Court—some people will get a bonus over what they could earn based on their ability alone, and some will be penalized. Thus, chance will pull or push some people into the two extreme ends of the distribution of scores. On average, the truly high-ability people will stay high and the truly low-ability ones will stay low, but the moderate-ability people who happened to score very high or low on the test the first time around will drift back to the middle of the pack in the scenario of infinite retesting that produces a true score.¹²⁷

As an example of applying these regression-based formulas for estimating the range of true scores that people with a given observed score have, consider an observed score of x = 75 on a test that has a reliability of ρ = 0.97. In the SEM-IS approach, the 95% CI would be 75 ± 1.96 × 15 √(1 – 0.97) = 70–80. A defendant could be said to fall into the potentially disabled zone of 70 or below. In the regression-based SEE-IS approach, the 95% CI is 75.75 ± 1.96 × 15 √0.97(1 – 0.97) = 71–81. The net effect of narrowing and raising the interval in this example is that offenders with single-measured IQ scores of 75 on this test no longer qualify for a more probing inquiry into their possible intellectual disability.

This procedure for correcting IQ scores for regression to the mean is related to more general statistical ideas about shrinkage estimators.¹²⁸ These estimators¹²⁹ compromise the properties of more traditional estimators ‘(maximum likelihood, minimum variance unbiased, least squares, etc.)’ to achieve better precision or other desirable qualities.¹³⁰ They have been the subject of considerable study¹³¹ and application in many fields.¹³²

The SEE-IS approach of using a conditional standard error permits statements about the frequency with which the true score would fall into the CI. For example, one can legitimately report that ‘[f]or people who have a score like yours, we find that X% of them truly fall somewhere between A and B.’¹³³ Following Justice Alito’s guide to CIs, a defendant with a score of 71 on our test with a reliability of 0.98 could argue that about 68% of observed scores of 71 correspond to true scores in the 70–74 range; hence, 16% of them have true scores of 70 or below. Then, shifting to the majority’s reasoning, the defendant could urge that depriving 16% of people who might otherwise be put to death of the opportunity to bring forth proof of an intellectual disability ‘creates an unacceptable risk that persons with intellectual disability will be executed, and thus is unconstitutional’.¹³⁴

5. Other statistical issues in and outside of Hall

The discussion up to this point does not exhaust the set of procedures for constructing CIs that statisticians have devised.¹³⁵Hall should be read as directing experts to handle measurement error in IQ scores in the most professionally responsible manner rather than instantiating any one of the narrow alternatives listed in the dissenting opinion. This would leave room for combining multiple scores and for the consideration of Bayesian procedures that assign probabilities to all possible true scores. This section briefly discusses these two matters.

5.1 Multiple scores

Justice Alito complained that ‘the Court entirely ignores [the fact] that Florida … takes into account the inevitable risk of testing error by permitting defendants to introduce multiple scores’.¹³⁶ Indeed, the trial court in Hall heard testimony about at least four IQ test scores for Hall—71, 72, 73, and 80¹³⁷—and it appears that Hall had taken as many as seven IQ tests over a 40-year period.¹³⁸

Although the majority did not ‘entirely ignore’ multiple scores, Justice Kennedy’s treatment of them is difficult to understand, statistically as well as legally. He wrote that ‘[e]ven when a person has taken multiple tests, each separate score must be assessed using the SEM’,¹³⁹ and he seemed to be referring to the single-score SEM. If this is what ‘the SEM’ means, the dictum cannot be right. Uncertainty about the true score declines as more measurements are made.¹⁴⁰ In technical terms, because the SEM for a single score is greater than the standard error of the average of several scores, using the single-score SEM as a measure of the probable error in the average score would be a mistake. An SEM-based score interval for the average can lie above a figure like 70 even though at least one—or even all—of an offender’s separate intervals dip below 70. Assuming that Hall’s IQ scores are independent, as the dissent implied,¹⁴¹ the 95% CI that applies to his average score is 73–75.¹⁴² Having accounted for the measurement error in the average of the IQ scores, the state should be able to enforce a 70-or-less rule for a true score, barring Hall from producing evidence of his clear deficits in adaptive functioning.

One might argue that the computation of the interval for Hall’s average is oversimplified, for it assumes fully independent scores. Indeed, the majority was worried that ‘the analysis of multiple IQ scores jointly is a complicated endeavor’.¹⁴³ But this argument is a makeweight. Does the majority really believe that the Constitution requires the state to treat a defendant with 10 successive, carefully administered IQ scores of 71 the same as a defendant with but a single score of 71?¹⁴⁴ Not only is this rule intuitively implausible, but analysis of multiple scores is hardly beyond the ken of statisticians.¹⁴⁵ The chapter in a psychology handbook¹⁴⁶ cited in Justice Kennedy’s majority opinion for the view that the problem is unduly complicated¹⁴⁷ actually states that the solution ‘is not nearly so complicated as it might seem at first glance’.¹⁴⁸ It shows how to combine scores with pencil and paper or a spreadsheet.¹⁴⁹

Finally, Justice Kennedy added that ‘because the test itself may be flawed, or administered in a consistently flawed manner, multiple examinations may result in repeated similar scores, so that even a consistent score is not conclusive evidence of intellectual functioning’.¹⁵⁰ Certainly, all IQ tests are imperfect in many ways. In addition to being unable to eliminate random error, they may be culturally biased or ‘biased in favor of neurotypical individuals’.¹⁵¹ Or, a defendant may be aiming to understate his true score.¹⁵² But the only thing that CIs can protect against is random error.¹⁵³ If the test is so flawed in other ways that multiple scores are not usable, then, a fortiori, single scores cannot be used. If the possibility of repeated mistakes in test administration introduces serious bias, then serious bias is at least as great a problem as in interpreting a less-precise single test score.

Given the flimsiness of the Court’s comments about multiple scores, the best interpretation of them is that they merely mean that the option of retesting in the Florida law is not sufficient to handle the problem of measurement error. Under Hall, multiple scores do not obviate the need to consider measurement error through an appropriate interval estimate, but a statistically sound interval estimate based on multiple scores should be admissible to ascertain whether the defendant’s true score is 70 or less.

5.2 Credible regions (Bayesian credible region)

A final issue in the construction of CIs or equivalent decision rules is whether the Constitution prevents a state from going beyond the classical test theory discussed in the Hall opinions and briefs.¹⁵⁴ The SEM-IS, SEM-AM, and SEE-IS approaches to measurement error all embody the ‘frequentist’¹⁵⁵ or ‘classical’¹⁵⁶ theory of statistical inference. Within this framework, a test-taker’s true score is a parameter in a statistical model of random error. The ‘[p]arameters are fixed, unknown constants’, which means that ‘no useful probability statements can be made about parameters’.¹⁵⁷ The best we can do is postulate values for the true score and see how often decisions about the true scores of many defendants, based strictly on the observed score (or multiple scores), would be right or wrong. Suppose, for example, that every capital offender has a true score of 70 or less. If the observed scores are normally distributed with the estimated standard error, then the 95% SEM-IS rule would keep the expected rate of false alarms (decisions wrongly preventing a defendant from demonstrating disability with evidence of deficits in adaptive functioning) to no more than 2.5%. And, in no case will we misclassify a non-disabled offender as potentially disabled—because there are no such offenders. But suppose that 50% of capitally convicted offenders have true scores of exactly 70 and that 50% have true scores of 71. No more than approximately 2.5% of the half with IQs of 70 will be misjudged, but about 41% of the half with IQs of 71 will be deemed disabled (if there is sufficient evidence of deficient adaptive functioning) even though not one of them truly qualifies for the exemption.¹⁵⁸

Calculations of this sort allow us to gauge the performance of any decision rule—its operating characteristics—conditional on the assumed values for the true scores of the offenders. But they cannot tell us the probability that is of interest to the law—the chance that an offender has a true score of 70 or less—or even the proportion of those who are expected to have such low true scores given a defendant’s observed score. With information on the distribution of true IQ scores among convicted capital offenders, however, it is feasible to estimate the latter quantities. Suppose a jurisdiction has recorded the IQ scores of convicted capital defendants over the past decade and that these scores are approximately normally distributed with mean 70 and SD 8. This distribution also describes the distribution of true scores. Some defendants’ true scores will be above their measured ones, but others will be below. Such differences will wash out, and the overall picture of true scores will mirror that of the measured ones. It will be centred at 70 and spread out so as to have most scores between 65 and 75.¹⁵⁹

If this historical pattern applies to the next defendant, we should be suspicious of a measured score x like 80 (or 60) that is in a tail of the prior distribution of true scores τ. If we knew nothing of the past, our best guess for the defendant’s true score would be the measured one,¹⁶⁰ but we ought to factor in the reality that low and high true scores are the exception and hence are less likely to be the explanation for the extreme score. A better estimate for this defendant’s true score τ would be closer to the middle of the pack of true scores—not at the very middle, but somewhere between the observed score x and the average of the previously encountered scores. Moreover, because we are working with more information than just the one newly observed score x, we can have more confidence in the blended estimate than in either the prior mean or the new observation as an estimate of τ. Thus, the interval estimate for the true score can be narrower than the classical CI.

The mathematical recipe for blending the prior distribution of true scores with a newly observed score is known as Bayes’ rule.¹⁶¹ Applying it to a normal prior distribution and an observed score with measurement error yields a ‘Bayesian credible region’¹⁶² (BCR) that has the properties noted above—integrating the prior information shifts the estimate towards the historical mean and produces a narrower interval.¹⁶³ BCRs could be used in the same way as CIs to determine whether the credible range of true scores covers a true score cut-off such as 70. Or, more directly, one can look at the posterior true score distribution and easily compute whether the portion below this cut-off is sufficiently small to reject the claim of a disability.¹⁶⁴ By way of illustration, if the empirical prior distribution were normal with mean 70 and SD 8, then individuals with a measured score of x ≥ 73 would have no more than a 10% risk of having a true score τ ≤ 70.¹⁶⁵ If 10% is an ‘unacceptable risk’, a higher cut-score for measured IQs would have to be selected. A state willing to tolerate only a 1% risk would use a cut-score of at least 76 in this particular illustration.¹⁶⁶ With modern computational methods, Bayes’ rule can be applied to almost any prior distribution, not just the normal ones mentioned here. This form of inference is well established in statistics,¹⁶⁷ psychometrics,¹⁶⁸ and social science.¹⁶⁹ The Supreme Court once characterized it as a ‘more precise method’ than classical hypothesis testing.¹⁷⁰ The approach would enable a state to attain a probability of, say, 95% for a correct decision with respect to the true score cut-off of 70, as the majority in Hall seemed to desire or demand.¹⁷¹

6. Summary and conclusion

The Hall Court holds that (1) a state can categorically deny the intellectual disability exemption to individuals whose measured IQ scores are above some cut-score, and (2) this lowest permissible cut-score must be greater than 70 (on a test normed to have a mean of 100 and a SD of 15). Justice Kennedy’s majority opinion further suggests that in choosing a cut-score, a state must attend in some manner to SEM, but the opinion does little more than gesture to publications on psychometrics, IQ scores, and intellectual disability that emphasize the importance of recognizing imprecision in IQ scores. To provide more specific guidance, this article has presented four procedures for coping with the one aspect of ‘measurement error’¹⁷²—namely, random errors in measuring IQ scores—that the Court invoked to invalidate Florida’s ‘rigid rule’.¹⁷³ The four procedures flow from the fundamental distinction between a test-taker’s unknown true score and a measured score. They all seek to control the ‘unacceptable risk’¹⁷⁴ that an individual with a measured score has a true score that is less than or equal to the maximum true score consistent with a diagnosis of disability. The first three procedures, derived from classical test theory, accomplish this goal indirectly, if at all. The fourth attends directly to the most pertinent probability.

The first procedure (SEM-IS) is the one described in the two opinions. It uses an overall SEM derived from the estimated ‘reliability’ for a specific test and population of test-takers to construct an approximate 95% (or other) CI. If the CI exceeds the targeted true score, the state can deny the Atkins exemption without regard to deficits in adaptive functioning. As the dissent notes, the procedure can be applied with any desired confidence coefficient, although the desired level of ‘confidence’ does not have the meaning the dissent (and probably the majority) ascribed to it.

Secondly, I described an equivalent hypothesis-testing procedure that defines the state’s cut-score for the measured IQ as a legislatively targeted true IQ score (such as 70) plus some number of SEMs. Using this SEM-adjusted score (SEM-AS) to reject claims of disability, this procedure tests whether an observed score is statistically significantly larger than 70. It keeps the risk of categorically rejecting a disability claim from a defendant with a true IQ score of 70 or less to a predetermined maximum. This is exactly what the SEM-IS rule does. But without data on the distribution of true scores among defendants, it is impossible to say how well these procedures limit the risk that individuals with measured scores greater than 70 have true scores at or below 70.¹⁷⁵

The third procedure uses an estimator for true IQ scores based on the distribution of those scores for all individuals with a defendant’s test score. Instead of the Court’s SEM, it uses a SEE that is larger for scores that are far from the population mean. The SEE-IS procedure also shrinks the estimated true score towards the mean.¹⁷⁶

The fourth and final approach to handling measurement error goes beyond the classical test theory on which the first three were based. It would permit a state to use recent data on the IQ scores of convicted capital offenders together with test-reliability data to estimate the probability that a defendant with a measured IQ has a true score in any desired interval. The BCR for the highest posterior density could be inspected to see if it contains any true IQ scores that would permit defendant’s claim of disability to proceed. The meaning of the region would be clear because, unlike the frequentist CI, the BCR does describe probabilities for true scores. Alternatively, the probability of a true score in the qualifying range could be computed directly from the posterior distribution of true scores. This quantity is what Justice Alito thought he was computing. Although it is the quantity of the most legal interest, it cannot be derived from the CI.

In short, Hall requires legislatures and courts to take heed of the statistical principles and methods that should guide the interpretation of IQ scores by psychologists and psychiatrists. The essence of the case is the Court’s refusal to countenance ‘an unacceptable risk that persons with intellectual disability will be executed’.¹⁷⁷ Understandably, the opinions do not strive to be comprehensive in their discussions of how to achieve this result. Unfortunately, to the extent that the Justices do articulate details of statistical procedures for quantifying the uncertainty of psychological measurements, the dicta are not always dependable.¹⁷⁸ By offering some corrections and elaborations, and by demonstrating that there is more than one statistically acceptable way to try to control the risk of error, this article lays the groundwork for more informed use of the statistical properties of IQ scores in adjudicating claims of intellectual disability.

Acknowledgements

I am grateful to Johannes Fredderke, Jay Kadane, Mae Quinn, and an anonymous referee for comments on a draft and to Jim Greiner, Jay Kadane, Jay Koehler, and Hal Stern for comments on a related paper.

Author notes