Coherently updating degrees of belief: Radical Probabilism, the generalization of Bayes’ Theorem and its consequences on evidence evaluation

Abstract

[…] be the sensible man who tailors his beliefs to the available evidence (Hacking, 1967, p. 311).

1. Introduction

The core of the Bayesian perspective to the scientific method can be concisely described in the following terms. Scientific rationality—related to the criteria used to check scientific hypotheses and their plausibility—should refer to a probabilistic framework. We fully endorse this point of view and have, as expressed by Galavotti (1996, p. 253), the ‘conviction that the entire edifice of human knowledge rests on probability judgments, not on certainties’.¹ This means that to rationally assess the plausibility of different hypotheses considered at a certain historical moment, say t, scientists should determine the probability of each hypothesis in the light of the totality of information available at that time and update these probabilities as new information are acquired, for example through laboratory experiments or observations. The study of the rational principles governing the attribution of probabilities to hypotheses and observations, and their updating according to new available information is therefore fundamental.

Two articles published in this journal (Taroni et al., 2016, 2018) have endorsed such a subjectivist interpretation of probability. In (Taroni et al., 2018) attention was stressed upon the fact that such subjective probabilities can be correctly informed by relative frequencies, whenever available, though this does not equate at all to the acceptance of a frequentist interpretation of probability.² Briefly, given a subjectivist interpretation of probability, opinions of a given subject at a given time can be represented in probabilistic form by specifying their degrees of belief; these beliefs must obey the principles of probability calculus in order to be called coherent. This is generally illustrated through the so-called ‘Dutch Book argument’, that is, based on the idea that a given subject can avoid bets that would make him suffer a certain loss if, and only if, his degrees of belief are probabilities and therefore managed by the rules of probabilities. Nevertheless, the requirement of coherence at a particular time says nothing about how a given individual should modify their beliefs in light of newly acquired evidence.

The study of the rational principles underlying the updating of personal beliefs can be seen as complementary and a natural consequence to the arguments developed in (Taroni et al., 2016, 2018). One’s probabilities should be updated on the basis of appropriate principles in order to guarantee a rational approach. Please note that updating one’s mind in the light of newly acquired evidence does not mean changing one’s opinion. de Finetti (1995) illustrated this aspect in the following terms:

If we reason accordingly to Bayes’ theorem we do not change opinion. We keep the same opinion and we update it to the new situation. If yesterday I said ‘Today is Wednesday’, today I say ‘It is Thursday ’. Yet, I have not changed my mind, for the day following Wednesday is indeed Thursday. (at p. 100)

The cardinal principle for the updating of personal beliefs is the one known under the name of ‘Conditioning principle’ which—broadly speaking—states that if the initial opinions of a subject are represented by a probability function $Pr(·)$⁠, subsequently, the subject acquires empirical evidence or observations, say E, then their final opinions should be represented by the conditional probability function $Pr(·|E)$⁠. The conditional probability depends on the evidence E that has been acquired, and not on what could have been observed, but, in fact, it has not.

This principle, which will be described in Sections 3 and 4, can only be applied if the acquired information is made up of ‘certain’ evidence. In many cases, in truth, very often, this information is made up of ‘uncertain’ evidence, so we should speak of acquisition of ‘soft evidence’³ compared to ‘hard evidence’.⁴ The problem is well posed by Schum (1994) who wrote:

If we contemplate using Bayes’ rule, how are we to revise our own prior belief about [hypotheses] H and $H¯$⁠, based upon evidence it is not a proposition but a probability distribution? (at p. 352)

This aspect is strictly related to a doctrine called ‘Radical Probabilism’ that holds that no facts are known for certain.

In most forensic and judicial literature related to probabilistic approaches to evidence evaluation and interpretation, it can be noticed that a discussion about the quantification of degrees of belief in the presence of soft evidence is completely missing. It is therefore a question of exposing how a rational individual can govern the rational update of his state of mind when the acquired information is not constituted by certain evidence (also called ‘hard’ evidence). This article is devoted to considering how rational individuals ought to revise probability judgments in reaction to their experiences in presence of ‘soft’ evidence.

The article provides a comprehensive discussion on this issue and is structured as follows. Section 2 gives an overview of fundamental definitional aspects for the representation of beliefs. Sections 3 and 4 cover the aspects related to the revision of beliefs in presence of hard and soft evidence, respectively, by introducing the concept of probability kinematics, with a simple example developed in Section 5. Section 6 highlights the connection between probability kinematics and Schum’s cascaded inference. Sections 7 and 8 extends further the discussion to evidence evaluation through the use of Bayes’ factor for unequivocal testimony (Section 7) and a Bayes’ factor adapted to take into account for uncertainty on the reported evidence: equivocal evidence (Section 8). Section 9, finally, concludes the article.

2. Logic for reasoning under uncertainty: representation of beliefs and relevant propositions

Let us briefly recall some fundamental definitional aspects suitable for describing how it is possible to ‘measure’ degrees of belief that a certain fact occurred. It is first necessary to be precise about what a ‘degree of belief’ is. A degree of belief is ‘personal’, it is the judgement of a given person at a given time about the truth of a given event or proposition. ‘Evidence’ bearing on that proposition is expressed by means of other propositions. Therefore, it shall be said, on first approximation, that a proposition, say B, is ‘relevant’, according to an opinion, for another proposition A if, and only if, knowing the truth (or the falsity) of B would change the degree of belief in the truth of A.

There is, however, an important distinction between defining the general concept of representation of beliefs and the evaluation of a particular case. It can be established that coherent, quantitative measures of uncertainty about events or propositions of interest must take the form of probabilities (Bernardo and Smith, 1994). These probabilities are subjective, since they are based on personal judgements, they are a personal numerical representation of the uncertainty relation between events. Consider a ‘probability function’, denoted by the symbol $Pr(·)$ where the $(·)$ contains the event or proposition, the probability of which is of interest. Numerical degrees of belief must satisfy, for any propositions A and B, the laws of probability theory. De Groot (1970) proved that the acceptance of certain assumptions concerning the uncertainty relation between events (e.g. the complete comparability of events) implies that for any event (or proposition) A there exist a unique probability $Pr(A)$ satisfying the laws of probability theory (axioms of probability). Briefly, it can be proved that (i) if it is known that a proposition is true (false), then the degree of belief should take the maximum (minimum) numerical value, that is $Pr(A)=1$ (⁠$Pr(A)=0$⁠); (ii) coherent quantitative degrees of belief have a finitely additive structure (i.e. if A and B are incompatible, then $Pr(A∪B)=Pr(A)+Pr(B)$⁠); (iii) events which are practically possible but non certain should be assigned a probability in the interval (0, 1), $0<Pr(A)<1$⁠. The degrees of belief in events not known to be true, are somewhere between the certainty that the event is true and the certainty that it is false. Some more results follow from these axioms. A straightforward consequence of the axiomatic formulation is that if the probability of an event A is assessed, the probability of its complement $A¯$ (i.e. the event that takes place when A is not true) can be simply obtained as $Pr(A¯)=1−Pr(A)$⁠.

This is the simplest example of how probability calculus works as a ‘logic for reasoning under uncertainty’. The logic places constraints on the ways in which numerical degrees of belief may be combined. Notice that the laws of probability require the degrees of belief in any two mutually exclusive and exhaustive events A and in B to be such that they are non-negative and their sum is equal to one. Within these constraints, there is not an obligation for A to take any particular value. Any value between the minimum (0) and the maximum (1) is allowed by the probability axioms. The question is: how is a subjective probability to be determined? How is newly available knowledge to be incorporated? We will focus now on the first question, while the second one will be addressed later in Section 3 and in Section 4. The simplest way to measure a subjective probability is a direct measurement technique based on comparison with standard events with given probabilities (O’Hagan, 1994):

A formal derivation of subjective probability based on this approach would need to assume that any two events can be compared to say which You regard as the more probable, and also that there exists a set of standard events with given probabilities against which any other event can be compared to determine its probability (at p. 98).

A subject is entitled to its own measures of belief, but must be consistent with them. The appropriateness of a set of probabilities held by a particular subject needs to be checked (Ramsey, 1980). Probability values need to be expressed in an operational way that will also make clear what coherence means and what coherent conditions are. de Finetti (1980) framed the operational perspective as follows:

However, it must be stated explicitly how these subjective probabilities are defined, i.e., in order to give an operative (and not an empty verbalistic) definition, it is necessary to indicate a procedure, albeit idealized but not distorted, an (effective or conceptual) experiment for its measurement. (at p. 212)

Therefore, one should keep in mind the distinction between the ‘definition’ and the ‘assignment’ of probability. A description of de Finetti’s perspective has been published by (Dawid and Galavotti, 2009).

Note that coherence on degrees of beliefs is the only condition that should be guaranteed. Assignments that violate the laws of probability are incoherent in the sense that they will lead to a sure loss, no matter which proposition turns out to be true (see, e.g. Jeffrey, 2004).

Another fundamental aspect for the definition of one’s degree of belief is that of ‘relevance’. A proposition B is said to be relevant for another proposition A if and only if the answer to the following question is positive: if it is supposed that B is true, does that supposition change the degree of belief in the truth of A? A judgement of relevance is an exercise in hypothetical reasoning. There is a search for a certain kind of evidence because it is known in advance that it is relevant; if someone submits certain findings maintaining that they constitute relevant evidence, a hypothetical judgement has to be made as to whether or not to accept the claim. In doing that, a distinction has to be drawn, not only between the proposition A and the evidence B for the proposition A, but also between that particular evidence B and whatever else is known.

When a proposition’s degree of belief is assigned, there is always exploitation of available ‘background information’, even though it is not explicit. An assessment of the degree of belief in the proposition ‘this coin lands heads after it is tossed’ is made on the basis of some background information that has been taken for granted: if the coin looks like a common coin from the mint, and there is no reason for doubting that, then it is usually assumed that it is well balanced. Should it be realized, after inspection, that the coin is not a fair coin, this additional information is ‘evidence’ that changes the degree of belief about that coin, even though it is still believed that coins from the mint are well balanced.

A ‘relevant proposition’ is taken to mean a proposition which is not included in the background information. The distinction between ‘evidence’ and ‘background information’ is important, because sometimes it has to be decided that certain propositions are to be considered as evidence, while others are to be considered as part of the background information. For example, suppose a DNA test has been evaluated. Assume that all scientific theories which support the methodology of the analysis are true, that the analysis has been done correctly, and that the chain of custody has not been broken. These assumptions all form part of the background information. Relevant evidence is made up of only those propositions which describe the result of the test, ‘plus’ some other propositions reporting statistical data about the reliability of the evidence. Alternatively, propositions concerning how the analysis has been done, and/or the chain of custody can also be taken to be part of the evidence while scientific theories are to be considered background information. Therefore, as said before, it is useful to make a clear distinction between what is considered, in a particular context, to be ‘evidence’, and what is considered to be ‘background’. The importance of this aspect will be discussed further in Sections 3 and 4. For this reason, background information is introduced explicitly in the notation.

Let $Pr(A|I)$ denote ‘the degree of belief that proposition A is true, given background information I’, and let $Pr(A|B,I)$ denote ‘the degree of belief that proposition A is true, given that proposition B is assumed to be true, and given background information I’. Given that a probability—the measure of uncertainty—as a degree of belief, is conditional on the status of information of a given subject who assigns it,⁵$Pr(A|B,I)$ should be written as $Prs,t(A|B,I)$ where s and t are the information available to subject s at time t, respectively. For ease of notation, in what follows, the subscripts s and t are omitted. In Section 3, it will be shown why time t plays an important role in the beliefs updating procedure.

The purpose here is only to emphasize the point that ‘all subjective probabilities are conditional on available knowledge’. It is obvious that personal beliefs depend upon the particular knowledge one has. If the choice is made to represent degrees of belief by means of probabilities, then it must be kept in mind that it will always be the case that probabilities are necessarily relative to the available knowledge ‘and’ to the assumptions made. Note that the term I for background information is often omitted, and it is solely reported $Pr(A)$⁠. This probability expresses the personal degree of belief that the event A is true, given all available knowledge I. Note that, as expressed in (van Fraassen, 1986):

It should be emphasized that logically speaking—and contrary to many textbook expositions—this conditional probability has nothing to do with learning or opinion change or ‘updating’ on new information. Literally, that my conditional probability $Pr(A|I)$ equals 1/3, for example […] clearly expresses only my present opinion about two eventualities (at p. 18).⁶

Once the occurrence of another event or proposition B (e.g. the scientific evidence) is observed, the degrees of belief can be updated by incorporating the newly available knowledge. This is usually indicated by $Pr(A|B)$⁠, though what is really meant is $Pr(A|B,I)$⁠. The letter I is suppressed for simplicity of notation, but $Pr(A|B)$ must be read as the current degrees of belief about the truth of A, given the evidence B and all background knowledge.

3. Logic for reasoning under uncertainty: probabilities’ updating with certain evidence

Section 2 served the purpose of discussing both definitional and operational aspects relative to the representation of beliefs. It has been clarified that degrees of belief must take the form of subjective probabilities, and that these must obey the laws of probability calculus in order to be considered a coherent representation of beliefs. But there is another question that must be answered: ‘how is a new item of evidence to be incorporated in one’s knowledge?’ The answer is in some cases given by the well known Bayes’ Theorem, as it will be recalled in the current section, but there are instances for which there is not an immediate answer. This will be addressed in Section 4.

Consider a proposition A and another proposition B, relevant for A. As it was highlighted in Section 2, a relevant proposition B, with respect to another proposition A, is a proposition such that the supposition, if it were true, would change the degrees of belief in A. The new degrees of belief are given by $Pr(A|B,I)$⁠, that is the conditional measure of degree of belief (with the information regarding B), to the initial measure of degree of belief (without the information regarding B).

Equation (2) is the simplest algebraic version of the formula first proved in the second half of the 18th century by the Reverend Thomas Bayes. The importance of Bayes’ Theorem is due to the fact that it is a rule for ‘turning around’ conditional probabilities and ‘updating degrees of belief on receiving new evidence’.

The result in the denominator is an expression of the law of total probability also known as ‘Extension of the conversation rule’ (Lindley, 1991). The probability $Pr0(A|B,I)$ is also called the probability of A, ‘conditional on’ B (at time t₀) or the posterior probability of A. $Pr0(A|I)$ and $Pr0(A¯|I)$ are called ‘prior’, or ‘initial’, probabilities. $Pr0(B|A,I)$ is called the ‘likelihood’ of A given B (at time t₀). Analogously, $Pr0(B|A¯,I)$ is the ‘likelihood’ of $A¯$ given B (at time t₀).

This process is called ‘Simple conditioning principle’ (Eagle, 2011; Maher, 1993); it represents the process in which the prior probability of each proposition A is replaced by a posterior probability that coincides with the prior probability of A conditional on B at time t₁. Jeffrey and Hendrickson (1988–1989) noticed that:

In the unlikely event that your judgmental states today and tomorrow are both representable by definite probability distributions—say, P for today’s distribution and Q for tomorrow’s—it may be that tomorrow’s unconditional probabilities Q(H) for hypothesis H are simply today’s conditional probabilities $P(H|A)$ for those hypotheses given the answer A to some question. This is the prized special case of Conditioning $Q(H)=P(H|A)$ (or ‘conditionalization’). (at p. 215)

In order to clarify the general mechanism underlying probabilistic updating, it may be helpful to consider Equation (4) in another way.

Consider all the possible scenarios that can be derived from the combination of two logically compatible propositions A and B at time t₀: $Pr0(A,B|I),Pr0(A,B¯|I),Pr0(A¯,B|I),$ and $Pr0(A¯,B¯|I)$⁠. These scenarios can be represented graphically by means of a ‘probability tree’, as shown in Fig. 1. At time t₀, it is not known if proposition A is true or not; the same for proposition B. Four exclusive and exhaustive scenarios do exist but we do not know which one is true. We just know that the probabilities related to this scenario must sum up to 1, $Pr0(A,B|I)+Pr0(A,B¯|I)+Pr0(A¯,B|I)+Pr0(A¯,B¯|I)=1$⁠.

Fig. 1

The probability tree for propositions A and B at time t₀, given background information I.

Open in new tabDownload slide

A probability tree is a type of graphical model which consists of a series of branches stemming from nodes, usually called random nodes, which represent uncertain events (Shafer, 1996). At every random node there are as many branches as the number of the possible outcomes of the uncertain event. In this context, outcomes of uncertain events are described by propositions and branches containing more than one node correspond to the logical conjunction of as many propositions as the number of nodes. Branches have associated with them probabilities of the corresponding conjunctions of propositions calculated via the multiplication rule, and the probability of each proposition is given by the sum of the probabilities of the branches containing it (extension of the conversation). Following the axioms of probability, the sum of the probabilities of all the branches must add to one.

Consider now time t₁, where proposition B is known to be true. The probability tree at time t₁ is shown in Fig. 2. Since B is known to be true, only two scenarios are possible in the new state of knowledge and thus two of the probability nodes in Fig. 2, inconsistent with B, must be assigned a probability equal to zero: $Pr1(A,B¯|I)=Pr1(A¯,B¯|I)=0$⁠.

Fig. 2

The probability tree for propositions A and B at time t₁, given background information I.

Open in new tabDownload slide

Again, the sum of the probabilities of the branches in Fig. 2 must add up to 1. The original probabilities of the branches must be amended in such a way that their sum turns out to be one: $Pr1(A,B|I)+Pr1(A¯,B|I)=1$⁠.

Results in (6) and (7) allow us to show that the use of Bayes’ Theorem for probability updating is equivalent to the redistribution of probabilities among possible scenarios in a symmetric way; given that the state of information has changed on learning only that B is true, $Pr1(B|I)=1$⁠, and nothing else, there is no reason to make a change biased for or against certain particular scenarios.

4. Logic for reasoning under uncertainty: probabilities updating with uncertain evidence

The ‘Simple conditioning principle’ (Section 3) is not widely applicable. Suppose that a given report expresses some uncertainty and we cannot be certain of evidence B as truth. We wish to use this ‘equivocal’ report in a further inference about propositions A and $A¯$ given that B is not known to be true but there is instead a probability distribution expressing uncertainty about it (Schum, 1994, p. 352). The problem is well posed by Jeffrey (1983)^,7 and reiterated in (Jeffrey, 1987):

H is the hypothesis that it’s hot out. Smith and Jones have each testified as to H’s truth or falsity, but you don’t regard their testimony as completely reliable. Perhaps you’re not even quite sure what either of them said. (You are deep in a tunnel, into which they have shouted their report.) Let E and F be the propositions that Smith and Jones, respectively, said that it is hot. How can you represent your judgment about E and F, and your consequent judgment about H? (at p. 391)

We are so often faced with uncertain evidence. We can more drastically say that examples of this type are the norm in real life situations as in forensic science and in judicial settings where things are partially perceived or remembered. An illustrative example in medicine is described in (Jeffrey and Hendrickson, 1988–1989).

Consider a forensic situation in which our previous propositions A and $A¯$ are substituted with H and $H¯$ (for Hypotheses) as routinely used in legal literature. Notation B is also updated by using letter E (for Evidence) to take into account the scientific information delivered by a forensic scientist through his report.

Assume, for the sake of argument, that the scientist’s degree of belief in the truth of proposition E at time t₁ is higher than his initial degree of belief at time t₀, but it falls short of certainty, that is $Pr0(E|I)<Pr1(E|I)<1$⁠. What is the effect of this uncertain evidence upon the hypotheses H and $H¯$?

The problem is that the scientist ‘cannot’ take the probability of H conditional on E as his new degree of belief, because ‘he does not know E’ for certain. If there is any uncertainty left in the report, the best that the scientist can do is to assess directly the effect of such imperfect information on his degrees of belief, and, indeed, this is all that the scientist is reasonably entitled to do. A probabilistic rule can be formulated that allows him to update directly his degrees of belief once dealing with uncertain evidence E.

Consider, for the sake of clarity, the probability tree of the scientist’s problem given in Fig. 3, i.e. a probability tree containing the nodes H ( ‘Hypothesis’) and E ( ‘Evidence’). The difference with the probability tree in Fig. 2 is that the probabilities of all four scenarios at time t₁ are now greater than zero. The problem is thus: ‘how can a decision-maker redistribute the probabilities of the four different scenarios in such a way that they add up to 1?’

Fig. 3

The probability tree for the hypothesis H and the evidence E at time t₁, given background information I.

Open in new tabDownload slide

In this way, the change in the scientist’s state of information between time t₀ and time t₁ is taken into account about event E. The other probabilities corresponding to all branches of the probability tree can be calculated analogously. It can be easily verified that they sum up to 1.

Equation (10) obeys the same principle as Equation (6). The constant factor now is $Pr1(E|I)/Pr0(E|I)$ instead of $1/Pr0(B|I)$⁠, but the probabilities have been redistributed among the possible scenarios in a symmetric way as before: given that the state of information has changed on learning the new probability $Pr1(E|I)$⁠, and nothing else happened, there are no reasons to make a change biased for or against certain particular scenarios.

Equation (11) is a straightforward ‘generalization of Bayes’ Theorem’, known in the philosophical literature under the name of ‘Jeffrey’s rule’ because it was the philosopher of science Richard Jeffrey who first argued it was a reasonable general updating rule (Jeffrey, 1983). The approach is also known as ‘Generalized conditioning’, ‘Jeffrey conditioning’ or ‘Probability Kinematics’. Jeffrey conditioning can be formally expressed in the following terms under a dichotomous situation:

If a person with a prior such that $0<Pr0(E|I)<1$ has a learning experience whose sole immediate effect is to change their subjective probability for E to $Pr1(E|I)$⁠, then their post-learning posterior for H should be $Pr1(H|I)=Pr1(E|I)×Pr0(H|E,I)+[1−Pr1(E|I)]×Pr0(H|E¯,I)$⁠.

Note that Jeffrey conditioning reduces to ‘Simple conditioning rule’ (Section 3) when $Pr1(E|I)=1$⁠; so, note that the Simple conditioning rule is a special case of Jeffrey’s rule. So, as expressed by(Howson, 1997), Jeffrey conditionalization offers ‘a consistent and certainly convenient release from the apparent dogmatism implicit in ordinary conditionalization. If we regard probabilities of 1 as in practice unattainable, we can view ordinary conditionalization merely as a convenient approximation of Jeffrey conditionalization for a proposition whose probability shifts to almost 1’ (at p. 204).

Schum (1994) explains Jeffrey’s rule of conditioning in the following terms:

According to Jeffrey’s rule, we determine $Pr(H|W$’s probabilistic report$)$ as follows: We first determine $Pr(H|E)$⁠, as if E did occur; this requires the priors $Pr(H), Pr(H¯)$ and the likelihoods $Pr(E|H),Pr(E|H¯)$⁠. We then determine $Pr(H|E¯)$⁠, as if E did not occur. This requires the above priors and the likelihoods $Pr(E¯|H),Pr(E¯|H¯)$⁠. The final step involves taken the weighted average of $Pr(H|E)$ and $Pr(H|E¯)$⁠, where the weights are W’s assessments of the likeliness that he observed E or $E¯$⁠. (at p. 352)

Comments and discussions on Formula (12) can be found in (Bradley, 1994; Diaconis and Zabell, 1982; Good, 1981; Howson and P Urbach, 1993; Jeffrey, 2004; Skyrms, 1987) and a brief historical discussion is presented in Section 6.

5. A simple numerical example

Consider, just for sake of illustration, a simple case involving a medical diagnosis taken from (Aitken et al., 2020). A medical blood test detects certain symptoms of a disease, but unfortunately, the test may not always register the symptoms when they are present, or it may register them when they are absent. Therefore, there is the need to evaluate the accuracy of the test; this can be done by assessing the sensitivity and the specificity of the performed test.

Letting H be the event that a person is affected by a given disease and E stand for the event that the test indicates a positive result, $Pr0(E|H,I)$ and $Pr0(E¯|H¯,I)$ stand for the sensitivity and the specificity of the test, respectively. Suppose now that the sensitivity of the test is equal to 0.95, while the specificity is equal to 0.99. This is equivalent to assess $Pr0(E|H)=0.95$ and $Pr0(E¯|H¯)=0.99$⁠. Suppose also that the prevalence of the disease in the relevant population is known to be 0.1, that is assume $Pr0(Hi|I)=1$⁠.

If the scientist declares unambiguously that the blood test is positive, then the probability that the individual is effectively affected by the disease increases from a prior probability of 0.1 to a posterior probability greater than 0.91.

Results obtained through Jeffrey’s rule by changing prior probability on H are a linear function. Note that once there is no uncertainty about the evidence E, $Pr1(E|I)$ can be set equal to 1, and the posterior probability in (14) is equivalent to that obtained in (13) for unequivocal evidence.

6. Some remarks of interest for the forensic scientist

In the previous section, we approached the problem of handling either hard or soft evidence through the quantification of the posterior probability of the main hypothesis of interest, say H.

Situations involving equivocal evidence are not so rare in forensic science. See, for example, data obtained to support classification of individuals according to an age threshold, given sex and the third molars’ dental maturity on a given scale as presented in (Corradi et al., 2013) in which the forensic scientist has difficulties in classifying third molars into one definite stage, for example due to an unclear radiographic image. Other scenarios involve the case where there is uncertainty about the reported testimony related to the hair colour of a person of interest, or there is uncertainty about animal hair classification by microscopy, or about the result of a test. An example involving how to deal with equivocal testimony about the result of a test is provided in Section 8.

Dodson (1961) gave a rule for beliefs updating based upon probabilistic equivocation of an evidence. He called it ‘Modified Bayesian theorem’ and his solution represented exactly Jeffrey’s conditioning rule. It seems that the model 00was criticized on its axiomatic aspects but it has been developed further in (Gettys and Willke, 1969) independently from Jeffrey (1965) (Jeffrey 1983). Their aim was to propose:

An algorithm […] that relaxes the requirement of Bayes’ theorem that the true data state be known with certainty by postulating a true but unobservable elementary event, w [E in our notation], which gives rise to posterior probabilities which reflect the uncertainty of the data. (at p. 125)

This historical discussion was pointed out by Schum in his discussion paper (Edwards et al., 2008).

It is of interest to take advantage of the works done by Schum on what he called ‘cascade of inference’ (see, e e.g. Schum and DuCharme (1971) and Schum (1994)). Schum described it in the following form (Edwards et al., 2008):

The point that initially stimulated our interest in cascaded inference was that the observation of evidence about an event is not diagnostically equivalent to observation of the event itself. (at p. 235)

In other words, Schum (1994) pointed out that ‘evidence about some event and the actual occurrence of this event are not the same’ (at p. 18). So that an event, say R, represents evidence that an event E happened or is true.

From evidence R, I must ‘infer’ whether or not E actually happened or is true. Under a stipulation that all evidence is inconclusive to some degree, this inference can only be expressed in probabilistic terms (at p. 18).

His work is somehow related to Jeffreys principle. Schum wrote (Edwards et al., 2008):

The Dodson and Gettys-Willke-Jeffrey formulations concern instances in which a source of evidence gives equivocal testimony in the form of a probability estimate, and so we cannot be sure whether or not E occurred. But DuCharme and I were interested in the very many other situations in which a source give unequivocal testimony the E occurred but we are still uncertain about whether or not it actually did occur. This happens whenever the source is less than perfectly credible. So we began by distinguishing between testimonial evidence $E*$ [R in our notation] that event E occurred and event E itself. (at pp. 235–236)

The question that—from the Schum point of view—discriminates between the Gettys-Willke-Jeffrey’s and his own work is ‘who does the credibility assessment of witness W?’ He wrote:

In the Jeffrey situation, [a witness] W provides an assessment of his own credibility as far as his observation was concerned. He is uncertain about whether his observation was E or $E¯$⁠, and he expresses this uncertainty by means of $Pr1(E)$ and $Pr1(E¯)$⁠. In [Schum’s development], we make an assessment of the credibility of W’s unequivocal testimony by means of [$Pr(R|E)$ and $Pr(R¯|E¯)$]. (at p. 353)

7. Bayes’ factor for unequivocal testimony

In forensic science, it is often of interest to assess the value of the evidence. The coherent metrics to assess the value of the evidence is the Bayes’ factor (Good, 1989), often simply called ‘likelihood ratio’ (though the two expressions are not, in general, equivalent and the likelihood ratio just represents a special case of Bayes’ factor).

Our first interest is focused on highlighting the link between Jeffrey’s solution for ‘equivocal’ testimony which is focused on posterior probabilities, and Schum’s works offering solutions for the value of ‘unequivocal’ testimony through cascaded inference, where source inaccuracy is considered in two inferential steps (R to E, and E to H). The potential for a parallelism between the two approaches (Jeffrey’s posterior probability and Schum’s Bayes’ factor) is discussed in (Schaefer and Borcherding, 1973).

Consider a reported testimony R of an event E. The current scenario allowing for unequivocal testimony, but where the credibility of the source is questioned is described in Fig. 4, reporting a probability tree containing nodes H ( ‘Hypothesis’), E ( ‘Evidence’) and R ( ‘Reported testimony’).

Fig. 4

The probability tree for the hypothesis H, the evidence E and the reported testimony R at time t₁, given background information I.

Open in new tabDownload slide

Note that Equation (15) assumes that events R and H are considered as independent conditional upon E. This assumption should be carefully considered as mentioned by Schum (1994, pp. 311–312). An example of the acceptance of such an assumption is presented in (Thompson et al., 2003) dealing with errors in DNA evidence evaluation.

Probabilities in Equation (18)⁸ can be defined as follows:

• $Pr0(E|H)$ represents the sensibility of the analytical method;

• $Pr0(E|H¯)$ represents the complement of the specificity of the analytical method;

• $Pr0(R|E)$⁠, represents the probability to detect correctly a feature when the feature does exist;

• $Pr0(R|E¯)$ represents the probability to detect a given feature when that feature does not exist. This can be defined as a ‘false association’.

These probabilities specify the ‘quality’ of the performed test (points 1 and 2) and that of a given laboratory (points 3 and 4).

This is an extreme situation, often difficult to justify and defend in front of a Court of Justice.

Clearly, whenever the possibility of a laboratory error is accounted for (i.e. $Pr0(R|E)<1$ and $Pr0(R|E¯)>0$⁠), the value of the evidence will be smaller.

Information useful to assign values for $Pr0(R|E)$ and $Pr0(R|E¯)$ can be obtained through results of internal laboratory tests or collaborative tests (i.e. proficiency tests) regularly performed by forensic laboratories. A discussion on the quality and relevance of proficiency tests in forensic science is out of the scope of this paper. The interested reader can refer to (Koehler, 2008, 2013, 2017a,b) for comments and critical discussions on the current state of affairs.

The fact that a scientist offers an unequivocal evidence, by saying that ‘the test result is positive and he is 100% sure of that’, does not mean that there is no uncertainty around the evidence. It is matter of fact that a Court (or a patient in a medical context) should have information on the laboratory performances, not just on the test itself to be able to assess the meaning of the expert’s statement. This can be done by using Equation (18).

8. Bayes’ factor for equivocal testimony

A slightly different situation is that of a scientist saying that he ‘supposes that the test is positive; he is, e.g. 70% sure that the test is positive’. Jeffrey (1983) expressed this possible scenario through the following examples:

The agent inspects a piece of cloth by candlelight, and gets the impression that it is green, although he conceded that it might be blue or even (but very improbably) violet. If G, B and V are the propositions that the cloth is green, blue and violet, respectively, then the outcome of the observation might be that, whereas originally his degrees of belief in G, B and V were.30,.30, and.40, his degrees of belief in those same propositions after the observation are.70,.25, and.05. (at p. 165)

and

It is easy enough to cite examples like 1 [the previous one] for the other senses. Transcribing a lecture in a noisy auditorium, the agent might think he had heard the word “red”, but still think it possible that the word was actually “led”. He might be unsure about whether the meat he is testing is pork or veal […]. (at p. 166).

The idea here is to extend the Bayes’ factor for the assessment of ‘unequivocal’ testimony to consider situations involving ‘equivocal’ testimony.

The probabilities of all other branches can be obtained analogously.

This expression simplifies to (17) when the testimony is unequivocal, i.e. $Pr1(R)=1$⁠.

Consider now the case described in Section 7, where

• $Pr0(E|H)=0.95$⁠,

• $Pr0(E|H¯)=0.01$⁠,

• $Pr0(R|E)=1$⁠,

• $Pr0(R|E¯)=0.04$⁠.

It must be pointed out that the Bayes’ factor in (20) depends on the prior probability $Pr0(H)$⁠. The Bayes’ factor does not in fact simplify to a likelihood ratio: it represents a measure of change in support of competing propositions, rather than a measure of support.

A small Bayes’ factor does not mean that the probability of the hypothesis of interest H is low: it only means that available knowledge (e.g. the reported testimony) lowers the probability of the hypothesis of interest. Take the case where $Pr1(R)=0.7$ and $Pr0(H)=0.9$ in Table 1. The Bayes’ factor in this case is 0.417, however the hypothesis H is more likely than hypothesis $H¯$ (⁠$Pr1(H)=0.79$⁠). In the same way, a large value of the Bayes’ factor does not mean that the probability of the hypothesis of interest is elevated. Undoubtedly, equivocal evidence drastically reduces the value of the evidence as shown in Fig. 5. In fact, the more uncertainty there is about the event R, i.e. the lower is $Pr1(R)$⁠, the more the value of the Bayes’ factor will decrease. Furthermore, the more uncertainty there is about the event R, the lower the impact there will be on the laboratory’s ability to detect correctly a feature when that feature does exist (i.e. the value of $Pr0(R|E)$⁠) for the calculation of the Bayes’ factor.

Fig. 5

Bayes’ factor in Equation (20) for $Pr0(H)=0.5$⁠, values of $Pr0(R|E)$ ranging from 0 until 1 in situations involving unequivocal (⁠$Pr1(R)=1$⁠) and equivocal evidence (⁠$Pr1(R)=0.99, Pr1(R)=0.9$ and $Pr1(R)=0.7$⁠). Note that the solid line indicates the neutral value of the Bayes’ factor, $BF=1$⁠.

Open in new tabDownload slide

9. Conclusion

The classical Bayesian perspective is based on conditioning related to evidence (laboratory experiments or observations in general) taken as certain. What is called ‘Radical Probabilism’ replaces such an extreme view by ‘the conviction that probabilities need not be based on certainties.’ (Galavotti 1996, p. 254). Uncertainty related to the reporting does exist. An extended operational perspective can be adopted. Such an extension (the so-called Dodson-Gettys-Willke-Jeffrey formulation) can be viewed as a generalization of Bayes’ Theorem and its use should be encouraged to deal with real life situations as supported in (van Fraassen, 1986) by strongly affirming that:

Our arguments cannot touch someone who insists that either changing opinion more or less by caprice, or only by rules which take into account factors other than those occurring explicitly in Jeffrey’s problem, is no less rational than by such a rule as we have endeavored to discover. We can only say: if a general rule is to be followed for all cases falling into the scope of the problem as posed, then Jeffrey’s rule as the required generality, and no other rule does. (at p. 23)

Forensic scientists deal with the value of the evidence through a measure called the Bayes’ factor. A cascaded inference taking into account what has been called equivocal evidence has been introduced, theoretically justified and described through a series of examples.

In the authors’ perspective, such a generalization for the evidence evaluation should be applied.

Footnotes

Acknowledgement

The authors would like to thank the Swiss National Science Foundation for its support through grant n. IZSEZ0_19114.

References