Search
Close
Search
Advanced Search
Search Menu
AI Discovery Assistant

ABSTRACT

We look at the distribution of the Bayesian evidence for mock realizations of supernova and baryon acoustic oscillation data. The ratios of Bayesian evidence of different models are often used to perform model selection. The significance of these Bayes factors is then interpreted using scales such as the Jeffreys or Kass & Raftery scale. First, we demonstrate how to use the evidence itself to validate the model, that is to say how well a model fits the data, regardless of how well other models perform. The basic idea is that if, for some real data set, a model’s evidence lies outside the distribution of evidence that result when the same fiducial model that generates the data sets is used for the analysis, then the model in question is robustly ruled out. Further, we show how to assess the significance of a hypothetically computed Bayes factor. We show that the range of the distribution of Bayes factors can greatly depend on the models in question and also the number of data points in the data set. Thus, we have demonstrated that the significance of Bayes factors needs to be calculated for each unique data set.

methods: statistical, cosmological parameters, dark energy

1 INTRODUCTION

Modern cosmology, in its current ‘precision era’, requires a thorough understanding of the employed statistical methodology in order to make robust inferences from the data. This is especially important with regards to understanding the plethora of statistical ‘tensions’ that have cropped up in the past years, which have the potential to challenge the current concordance model of cosmology ΛCDM (cosmological constant dark energy; cold dark matter). The most notorious tension is the ‘H0 tension’ that is the disagreement between the local expansion rate inferred by Planck’s measurement of the cosmic microwave background (CMB; Planck Collaboration VI 2020) and that value directly measured by observations of Cepheids by the SH0ES collaboration (Riess et al. 2019). These tensions are often investigated in terms of a model selection question where the ΛCDM model is compared to some new model (Joachimi et al. 2021; Tröster et al. 2021). The comparison is typically performed by computing the Bayes factor Bij = log Zi/Zj, where Z is the evidence, P(D|Mi, j), the probability of the data, D, given model Mi, j. If Bij > 0, the model i is favoured over model j and vice versa. This quantity is useful since using Bayes’ theorem P(Mi|D) = P(D|Mi)P(Mi)/P(D), and using equal prior probabilities for the models, implies that Bij = log P(Mi|D)/P(Mj|D). Thus, the Bayes factor corresponds to the relative probability of one model over another.

Seminal works that have laid the ground work for the adoption of Bayesian statistics in cosmology include Trotta (2007, 2008) and Liddle (2007). In particular, Trotta (2007) introduces the Savage–Dickey ratio that is an efficient way to calculate the Bayes factor for the case where the two models in question are nested.

There are a few drawbacks to this inference procedure. The first problem is that this procedure is inherently a comparison; it can only determine which model is better or worse, not whether either model is a good fit to the data in an absolute sense. In other words, it is useful only for model selection, not for model validation. This is an important point to keep in mind since there is no reason to believe that the true model of the Universe, whatever it may be, could be guessed at and so included as a discrete choice in this model selection procedure.

The second drawback is a question of significance. The mathematics of the procedure returns a number. It is a further step of human interpretation to decide what that number means in terms of whether we should believe one model is true and the other is false. This interpretation is done with one of the two commonly used scales, the Jeffreys scale (Jeffreys 1961) and the Kass & Raftery scale (Kass & Raftery 1995, see Table 1). However, these scales are not a one size fits all solution. Of course, determining a criteria for mapping the value of a statistic and significance is arbitrary and so either of these scales could serve as the mapping between the statistic and significance. However, physics already employs a criteria to determine whether a signal is significant, the oft-quoted ‘5σ’ p-value. A p-value is the probability of observing a signal given that a null hypothesis is true. This p-value is a frequentist statistic so, at first, it may seem counter intuitive to use a frequentist criteria for a Bayesian calculation. However, it is a meaningful question to ask what is the distribution of any statistic over different realizations of the data, even the Bayesian evidence.

Table 1.
Open in new tab

The Jeffreys and Kass–Raftery scales.

Jeffreys scale Zi/ZjKass–Rafferty scale Zi/ZjInterpretation
1–3.21–3Not worth mentioning
3.2–103–20Positive
10–10020–150Strong
> 100>150Very strong
Jeffreys scale Zi/ZjKass–Rafferty scale Zi/ZjInterpretation
1–3.21–3Not worth mentioning
3.2–103–20Positive
10–10020–150Strong
> 100>150Very strong
Table 1.
Open in new tab

The Jeffreys and Kass–Raftery scales.

Jeffreys scale Zi/ZjKass–Rafferty scale Zi/ZjInterpretation
1–3.21–3Not worth mentioning
3.2–103–20Positive
10–10020–150Strong
> 100>150Very strong
Jeffreys scale Zi/ZjKass–Rafferty scale Zi/ZjInterpretation
1–3.21–3Not worth mentioning
3.2–103–20Positive
10–10020–150Strong
> 100>150Very strong

A third drawback is that Bayesian evidence generally depend on the subjective choice of a prior. Common choices of such a prior includes the flat prior, where the prior probability of a value of a parameter being true is uniform over some domain. This is often a good choice since such a flat prior reflects a maximum amount of agnosticism about the value of a parameter. However, flat priors are not invariant under changes in parametrization, e.g. flat in P(θ) is not flat in P(log θ). Further, any uniform distribution is only well defined if the domain is finite, e.g. we might say Ωm is free to vary in the range [0,1]. However, since the Bayesian evidence is the average of the likelihood over the prior volume, if the volume were expanded to include regions of low likelihood, then the evidence will decrease.

Previous studies have investigated the utility of Bayes factors and the Jeffreys scale for performing model selection (Starkman, Trotta & Vaudrevange 2008; Jenkins & Peacock 2011; Raveri & Hu 2019; Joachimi et al. 2021). For instance, Jenkins & Peacock (2011) treat the evidence ratio as a statistic and point out that it is noisy (see also Joachimi et al. (2021)) and thus selecting models based solely on that number may be insufficient. Both  Jenkins & Peacock (2011), Joachimi et al. (2021) find that the log of the evidence should have a variance that scales with the number of data points. Furthermore, Nesseris & García-Bellido (2013) calculate false-positive and false-negative rates when using the Jeffreys scale to select a particular class of models. Specifically, the models they consider have predictions that are linear in their parameters. The different models considered in that paper are all nested with different numbers of parameters. That is, model M2 has one extra parameter compared to M1 and for a certain value for that extra parameter, reduces to M1. The authors show that the Jeffreys scale can fail to penalize extra degrees of freedom when mock data was generated from the simpler model. They show that the frequency of this failure is influenced by the choice of prior. The authors conclude that the Bayes factor should not be the only tool used for model selection.

Another recent, companion paper points out related short comings of using the Bayes factor. Specifically, Koo et al. (2021a) demonstrate that, if the true model is not an option when calculating Bayes factors, such a scheme will still pick some false model as being best and thus not rule it out. They go on to show that the distribution of likelihoods from the iterative smoothing method  (Shafieloo 2007; Shafieloo et al. 2006) can rule out all false models (Koo et al. (2021b). The current work takes inspiration from the methodology of that paper and shows how to use a models’ evidence alone, not the Bayes factors, to determine if a model is a good fit to the data regardless of how well other models perform.

In contrast to questions of model selection, which are often answered by computing Bayes factors, there are questions of model validation that are answered with other statistical tools. Such ‘goodness-of-fit’ tests can come in frequentist forms, such as the χ2 test, or Bayesian forms, such as the posterior predictive distribution (Doux et al. 2021).

In this paper, we seek to develop a methodology to address these three drawbacks of using Bayes factors for model comparison. We first calculate the distribution of the Bayesian evidence, for data sets commonly used in cosmology, and show how to use this distribution to answer the question of whether a model is a good fit to the data, irrespective of other models. Further, for these same data sets, we show how to assign a frequentist p-value to the Bayes factor to characterize significance when used for model comparison.

2 MOCK DATA

The approach to calculating the distribution of the Bayesian evidence is to generate a hundred mock realizations of the data and then calculate the Bayesian evidence for each one. The data sets that we use for this calculation are mock future supernova (SN) data sets as might be expected from the WFIRST telescope (Green et al. 2012; Spergel et al. 2015) and future baryon acoustic oscillation (BAO) data sets as might be expected from DESI (DESI Collaboration et al. 2016a,b).

Type Ia SN are one of the observational pillars that built the ΛCDM model and directly measure the acceleration of the Universe. Existing SN data sets have all shown broad consistency with the ΛCDM model despite concerted searches for new physics or systematics (Riess et al. 1998, 2007; Perlmutter et al. 1999; Kowalski et al. 2008; Hicken et al. 2009; Amanullah et al. 2010; Suzuki et al. 2012; Betoule et al. 2014; Scolnic et al. 2018; L’Huillier et al. 2019; Koo et al. 2020; Keeley et al. 2021). These successes of SN exist because they are standardizable candles; the measurement and fitting of the light curve of the SN allows us to infer the luminosity of the SN (up to a global calibration; Riess et al. 1998; Perlmutter et al. 1999; Riess et al. 2007; Kowalski et al. 2008; Hicken et al. 2009; Amanullah et al. 2010; Suzuki et al. 2012; Betoule et al. 2014; Scolnic et al. 2018), and thus measuring the flux is equivalent to measuring a luminosity distance, or equivalently a distance moduli:
(1)
Here, μ is the distance modulus, DL is the luminosity distance, mB is the peak B-band flux, MB is the absolute magnitude of SN and serves as the parameter that calibrates the distance redshift relation, and α and β are hyperparameters that control how the light-curve parameters (X1 and C) influence the distance–redshift relation. In our mock data set, we imagine the case that the SN data set is calibrated and thus do not vary these extra parameters. Our mock data set includes 3000 SN in the range within 0.01 < z < 3.0, and the redshifts are uniformly distributed in the log  of this range. The existing Pantheon data set includes SN at z = 2.3 (Scolnic et al. 2018), so it is not unreasonable that future compilations that build off of the Pantheon compilation could extend even further in redshift. The errors on the distance modulus of each SN are of the form σμ = 0.12 + 0.12(z/3.0)2. The mock data sets are generated by adding random Gaussian variables to the model predictions with standard deviation as described. the likelihood is Gaussian when making inferences with this mock data set.
The DESI BAO data set measures the clustering of galaxies both along the line of sight and perpendicular to it and so includes constraints on both the Hubble distance (DH(z)) and the angular diameter distance (DM(z)) individually  (Howlett et al. 2015; Ross et al. 2015; DESI Collaboration et al. 2016a,b; Alam et al. 2017; Beutler et al. 2017; du Mas des Bourboux et al. 2017, 2020; eBOSS Collaboration et al. 2020). Because the absolute distance to these galaxies is unknown, the constraints from BAO analyses are expressed as ratios of distances with respect to the sound horizon at drag epoch, rd:
(2)
The sound horizon can be thought of as a calibration parameter, and as with the SN, we imagine a case where the calibration is known and thus our mock BAO data set is just constraints on DH(z) and DM(z). DESI is forecasted to measure DH(z) and DM(z) in 29 individual redshift bins within 0.05 < z < 3.55. The precision of each of the measurements can vary from less than 1 per cent to more than 10 per cent (see tables 2.3, 2.5, 2.7 in DESI Collaboration et al. 2016a). As with the SN data set, the mock data set is generated by adding a random Gaussian variable to the model predictions and hence the likelihood for this data set is Gaussian. These likelihoods that we use to generate the data are the same as the ones we use to make inferences from the data.

We generate 100 realizations of the combined data sets for each of the three models in question (see Section 3) for a total of 300 mock data sets. The seed is fixed for the different models so the residuals are the same. As in, there are only 100 unique realizations of the residuals.

3 MODELS

We consider three distinct models when computing the distribution of the Bayesian evidence and Bayes factors. Specifically we analyse the ΛCDM model, the transitional dark energy (TDE; Keeley et al. 2019) model, and the phenomenologically emergent dark energy (PEDE; Li & Shafieloo 2019) model. The TDE model is parametrized; thus,
(3)
and we choose w0 = −0.8, w1 = −2.0, zT = 1.0, and Δ = 0.2. The PEDE model has a similar form of the parametrization,
(4)
The parameters of the TDE model are fixed to put it on a level playing field with PEDE (generalized emergent dark energy or GEDE; Li & Shafieloo 2020) is an extension of PEDE that connects ΛCDM and PEDE) and with ΛCDM. Thus for each of the models, the only free parameters are H0 and Ωm. H0 is allowed to vary between 60 and 80 km s−1 Mpc−1 and Ωm is allowed to vary between 0.0 and 1.0. When generating the data, we use H0 = 70 km s−1 Mpc−1 and Ωm = 0.3 as the fiducial parameters for each of the models. These models were specifically chosen to be distinguishable given the mock data sets. This is achievable in part because, for the parameters we chose, the TDE model behaves like a quiescent (w > −1) dark energy in the region most constrained by the data. On the other hand, the PEDE model exhibits a purely phantom (w < −1) behaviour. ΛCDM, of course, has a cosmological constant dark energy (w = −1). Though both of these models can be thought of as nested extensions to ΛCDM when their additional parameters are allowed to vary, these types of models are not commonly investigated alternatives to ΛCDM. These models are further interesting since they can encode an evolution in w(z) at different redshifts than z = 0, as in the case with the Chevalier–Polarski–Linder (CPL; Chevallier & Polarski 2001; Linder 2003) parametrization w(z) = w0 + waz/(1 + z), which is the most common parameterization of evolving dark energy. So, in summary, we are investigating potential cases where, for example, if TDE is the true model of cosmology, calculating the Bayes factor between ΛCDM and PEDE may favour ΛCDM, but neither are the true model. We show how to use the Bayesian evidence itself to show that both ΛCDM and PEDE are bad fits to the data in such a case.

The evidence are calculated using multinest (Feroz & Hobson 2008; Feroz, Hobson & Bridges 2009; Feroz et al. 2019), specifically the importance nested sampling algorithm, with 400 live points, a sampling efficiency of 0.3, and an evidence tolerance of 0.1.

We also investigate the case where the models in question have different degrees of freedom. Specifically, we look at the cases where we allow the curvature to vary in our models, though the data is still generated from the case where the curvature is fixed to be 0. We call these models where the curvature (Ωk) is allowed to vary, kTDE, kPEDE, and kΛCDM.

4 RESULTS

In Fig. 2, we see the distributions of the log evidence for nine different cases (three models were used to generate the data and three models were used for inference). Of particular note are the cases where the model being used to infer the evidence is the one used to generate the data. The distributions for these cases are exactly the same and lie on top of each other in the figure. This is expected since generating these mock data sets is essentially generating random residuals and these random residuals are the same between the different models. Additionally, the fact that the models have the same parameters that are allowed to vary is also necessary for the distributions to be identical. This distribution of the evidence can be used as a robust test of validation, to see if a model is a good fit to the data. To perform this test, we take the evidence calculated with real data and some model to be validated and compare it to the distribution calculated with that same model. If the evidence calculated with real data lies outside this distribution, then we can robustly say that model is ruled out or invalidated, regardless of how well other models perform.

One example mock data set (generated from the fiducial ΛCDM model) along with the predictions of our three considered models, ΛCDM, TDE, and PEDE, that are best fit to the data.
Figure 1.

One example mock data set (generated from the fiducial ΛCDM model) along with the predictions of our three considered models, ΛCDM, TDE, and PEDE, that are best fit to the data.

Open in new tabDownload slide
Distribution of the log evidence for mock data sets generated from the model labelled on the left, inferred using the model labelled on the top. Of particular note is that the distributions for the cases that the model used for the inference is the same as the one used for the data generation are all exactly the same.
Figure 2.

Distribution of the log evidence for mock data sets generated from the model labelled on the left, inferred using the model labelled on the top. Of particular note is that the distributions for the cases that the model used for the inference is the same as the one used for the data generation are all exactly the same.

Open in new tabDownload slide

In Fig. 3, we see the distributions of the differences in the log evidence for each of the three pairwise combination of models for each of the three models used to generate the data sets. To explain the figure’s labels, the difference is such that Δlog Z = log ZL − log ZR so positive values represent a preference for the model on the left. Basic expectations are borne out by this figure. The model used to generate the data always performs better than a false model. Further, since the TDE model is a quintessent dark energy model in the regime where there is the most data, and since the PEDE model is always phantom, ΛCDM is preferred over the TDE model when the PEDE model is true, and ΛCDM is also preferred over the PEDE model when the TDE model is true. Further, PEDE is closer to ΛCDM and thus it is preferred over TDE when ΛCDM is the true model. That Δlog Z = 0 is outside the distribution in all these cases indicates that, for this optimistic forecast for DESI BAO data and WFIRST SN data, these dark energy models should be imminently distinguishable. One point that is worth bringing up is that the range of these distributions is noticeably different. For different model comparisons, the scatter in the Δlog Z values will be different. This goes to show that using a fixed scale like the Jeffreys or the Kass & Rafferty scale cannot serve as one size fits all solutions to the interpretation of Bayes factors. Indeed, for different data sets and models, one would need to calculate these kinds of distributions to make robust inferences about Bayes factors.

Distribution of the difference in the log evidence for mock data sets generated from the model labelled on the left, inferred using the models labelled on the top. The labels on the top refer the log evidence of the first model minus the log evidence of the second model.
Figure 3.

Distribution of the difference in the log evidence for mock data sets generated from the model labelled on the left, inferred using the models labelled on the top. The labels on the top refer the log evidence of the first model minus the log evidence of the second model.

Open in new tabDownload slide

In Fig. 4, we see the distribution of evidence for the specific case of when the data is generated from the TDE model. This figure is useful to make an easy comparison to  Koo et al. (2021b). This figure also highlights one potential advantage or drawback of Bayesian methods over the distribution of likelihoods from the iterated smoothing method. The Bayesian evidence explicitly uses models and so are more interpretive and thus can say things specifically about model quantities such as curvature. However, because Bayesian methods are more interpretive, they can arrive at wrong conclusions. As is the case in this figure, when the wrong dark energy model is used in the inference, Bayesian methods can come to the wrong conclusion about the existence of curvature.

Distribution of the log evidence for mock realizations from the TDE model inferred using different models when curvature is and is not allowed to vary.
Figure 4.

Distribution of the log evidence for mock realizations from the TDE model inferred using different models when curvature is and is not allowed to vary.

Open in new tabDownload slide

In Fig. 5, we see the distribution of the differences in the log evidence for each of the models when curvature is and is not allowed to vary. This is an example of an additional degree of freedom. In all of these cases, the data were generated from models with no amount of curvature. To explain the figure’s labels, the difference is such that Δlog Z = log Zflat − log Zcurved and so positive values in the distribution indicate that the flat model is preferred and allowing for this extra degree of freedom is disfavoured. Further, focusing on the curves where the models in question are different than the ones used to generate the data, using the wrong dark energy model can, in turn, cause wrong conclusions about whether the Universe is curved. There can either be some amount of confusion between the mismatch in the dark energy models and the curvature (cases where the distribution spans Δlog Z = 0) or there can be a significant preference for curvature when none was included in the mock data sets, as in the cases where the data was generated with the TDE model but PEDE or ΛCDM were assumed. Further, in the cases where the model used to generate the data is the same as the model used for inference, along with its extension that includes curvature, the distributions of the evidence are roughly equivalent (they each span Δlog Z ∈ [ − 1.0, 3]). In this specific case of using a known model of dark energy and inferring whether curvature is preferred, one could set the criteria for a |$\gt 99{{\ \rm per\ cent}}$| detection of curvature at Δlog Z < −1.0. Of course, we would not a priori know that we have correctly identified the true model of dark energy. Further, this calculation does not generalize to arbitrary data sets or nested models. For instance, if one were to include different data sets such as weak lensing or the CMB, or were to investigate different models, one would need to recalculate the significance of the Bayes factor value.

Distribution of the difference in the log evidence for mock data sets generated from the model labelled on the left, inferred using the models labelled on the top. The difference in the models as labelled on the top is whether curvature is allowed to vary. The difference in the log evidence is the difference between the flat version of the model minus the version where curvature is free to vary. The right-hand panel is the same as the left-hand panel but zoomed in on the region −1.0 < Δlog Z < 3.5.
Figure 5.

Distribution of the difference in the log evidence for mock data sets generated from the model labelled on the left, inferred using the models labelled on the top. The difference in the models as labelled on the top is whether curvature is allowed to vary. The difference in the log evidence is the difference between the flat version of the model minus the version where curvature is free to vary. The right-hand panel is the same as the left-hand panel but zoomed in on the region −1.0 < Δlog Z < 3.5.

Open in new tabDownload slide

In Fig. 6, we can more clearly see cases where the Jeffreys scale would return wrong conclusions. The mock data used in this analysis is the same as above except now the errors are 10 times larger. To elaborate, the data are generated with uncertainties that are 10 times larger than the normal case, and the likelihood in the analysis for this case uses the same expanded uncertainties. We will focus on the gold curve corresponding to the case where the PEDE model was used to generate the data and the two models being compared are the PEDE and TDE models. In that case in particular, 15 per cent of the time Δlog Z is above log (100) and 3 per cent of the time Δlog Z is below −log (100). Thus there is a non-negligible chance that this analysis would return ‘very strong’ conclusions when a more indeterminate conclusion be returned. It is additionally important to emphasize the pink curve, which corresponds to the case where the PEDE model was used to generate the data and the two models being compared are the ΛCDM and TDE models. In this case, 10 per cent of the time Δlog Z is above log (100) and 3 per cent of the time Δlog Z is below −log (10). Any strong conclusion is inaccurate in this case since the true model is not included in the inference. Similarly, we see for the for all of these cases, the scatter in Δlog Z is large compared the scale the different levels of the Jeffreys scale thus making it an unreliable tool for interpreting Bayes factors. The specific values for the frequencies of these confidently wrong conclusions is not of primary concern, but instead the concern arises from the fact these frequencies are non-negligible.

Same as Fig. 3 except now the errors on the data are 10 times larger. Since the distribution of Δlog Z spans a wide range on either side of 0, there is no insignificant chance of returning a ‘very strong’ conclusion that either model is true. The pink and gold curves are highlighted as discussion will focus on them. The gold curve illustrates this well where there is a non-negligible chance of returning very strong conclusions for both the correct and incorrect model, when a more indeterminate conclusion would be appropriate. Similarly, the case for the pink curve returns very strong or strong conclusions for two wrong models, since of course, as the true model is not included in the inference. The black vertical lines denote the ‘strong’ and ‘very strong’ regions of the Jeffreys scale.
Figure 6.

Same as Fig. 3 except now the errors on the data are 10 times larger. Since the distribution of Δlog Z spans a wide range on either side of 0, there is no insignificant chance of returning a ‘very strong’ conclusion that either model is true. The pink and gold curves are highlighted as discussion will focus on them. The gold curve illustrates this well where there is a non-negligible chance of returning very strong conclusions for both the correct and incorrect model, when a more indeterminate conclusion would be appropriate. Similarly, the case for the pink curve returns very strong or strong conclusions for two wrong models, since of course, as the true model is not included in the inference. The black vertical lines denote the ‘strong’ and ‘very strong’ regions of the Jeffreys scale.

Open in new tabDownload slide

5 DISCUSSION AND CONCLUSIONS

The first takeaway from this paper is that the models presented here should be distinguishable with future WFIRST and DESI data sets.

Secondly, is that we have shown how to use the evidence itself as a criterion for model validation, to judge if a model is a good fit to the data at all, independently of any other model. Using a distribution of the evidence for mock data generated from a model and inferred using the same model, one can see if the evidence calculated from the real data lies outside that distribution. Should the evidence computed with real data for a given model fall outside that range, then that model would be invalidated. The distribution of the evidence can be used to calculate the mapping between the Bayesian evidence and a p-value and determine a model-independent criteria to answer the question: ‘Below what value would the Bayesian evidence have to be to conclude a model is not a good fit to the data?’

A third takeaway is that we have demonstrated interpreting Bayes factors using fixed scales, like the Jeffreys or Kass & Rafferty scales, can yield incorrect conclusions. This is seen in the fact that the distributions of Δlog Z span hundreds of values. Even in the case where the uncertainties in the data is 10 times larger than normal, such that the models should be indistinguishable, the scatter in the Bayes factors is large compared to the range of the Jeffreys scale and will come to correct and incorrect ‘very strong’ conclusions. Thus, blindly using a fixed scale such as the Jeffreys scale can be misleading. To accurately assess the significance of a computed Bayes factor, one should perform these kinds of calculations, generate a sample of mock data sets and calculate the Bayes factors in those cases, and then see where the Bayes factor for the real data set lies in that distribution.

Finally, we point out that the distributions of both log Z and Δlog Z depend greatly on the specifics of the data set, both the size of the covariance matrix and the nature of the data sets (e.g. BAO and SN).

ACKNOWLEDGEMENTS

We would like to thank Benjamin L’Huillier and Hanwool Koo for useful comments on the draft. This work was supported by the high performance computing cluster Seondeok at the Korea Astronomy and Space Science Institute. AS would like to acknowledge the support by National Research Foundation of Korea NRF-2021M3F7A1082053 and Korea Institute for Advanced Study (KIAS) grant funded by the Government of Korea.

DATA AVAILABILITY

The data used in this work are mocks generated by the authors, as described in the text. They are available upon request.

REFERENCES

Alam
S.
et al. ,
2017
,
MNRAS
,
470
,
2617
10.1093/mnras/stx721

Crossref
Search ADS
 

Amanullah
R.
et al. ,
2010
,
ApJ
,
716
,
712
10.1088/0004-637X/716/1/712

Crossref
Search ADS
 

Betoule
M.
et al. ,
2014
,
A&A
,
568
,
A22
10.1051/0004-6361/201423413

Crossref
Search ADS
 

Beutler
F.
et al. ,
2017
,
MNRAS
,
464
,
3409
10.1093/mnras/stw2373

Crossref
Search ADS
 

Chevallier
M.
,
Polarski
D.
,
2001
,
Int. J. Mod. Phys. D
,
10
,
213
10.1142/S0218271801000822

Crossref
Search ADS
 

DESI Collaboration
et al. .,
2016a
,
preprint (arXiv:1611.00036)

DESI Collaboration
et al. .,
2016b
,
preprint (arXiv:1611.00037)

Doux
C.
et al. ,
2021
,
MNRAS
,
503
,
2688
10.1093/mnras/stab526

Crossref
Search ADS
 

du Mas des Bourboux
H.
et al. ,
2017
,
A&A
,
608
,
A130
10.1051/0004-6361/201731731

Crossref
Search ADS
 

du Mas des Bourboux
H.
et al. ,
2020
,
ApJ
,
901
,
39

 

eBOSS Collaboration
et al. .,
2020
,
Phys. Rev. D
,
103
,
083533

 

Feroz
F.
,
Hobson
M. P.
,
2008
,
MNRAS
,
384
,
449
10.1111/j.1365-2966.2007.12353.x

Crossref
Search ADS
 

Feroz
F.
,
Hobson
M. P.
,
Bridges
M.
,
2009
,
MNRAS
,
398
,
1601
10.1111/j.1365-2966.2009.14548.x

Crossref
Search ADS
 

Feroz
F.
,
Hobson
M. P.
,
Cameron
E.
,
Pettitt
A. N.
,
2019
,
Open J. Astrophys.
,
2
,
10
10.21105/astro.1306.2144

Crossref
Search ADS
 

Green
J.
et al. ,
2012
,
preprint (arXiv:1208.4012)

Hicken
M.
,
Wood-Vasey
W. M.
,
Blondin
S.
,
Challis
P.
,
Jha
S.
,
Kelly
P. L.
,
Rest
A.
,
Kirshner
R. P.
,
2009
,
ApJ
,
700
,
1097
10.1088/0004-637X/700/2/1097

Crossref
Search ADS
 

Howlett
C.
,
Ross
A. J.
,
Samushia
L.
,
Percival
W. J.
,
Manera
M.
,
2015
,
MNRAS
,
449
,
848
10.1093/mnras/stu2693

Crossref
Search ADS
 

Jeffreys
H.
,
1961
,
Theory of Probability
.
Oxford Univ. Press
,
Oxford

Google Scholar

OpenURL Placeholder Text

 

Jenkins
C. R.
,
Peacock
J. A.
,
2011
,
MNRAS
,
413
,
2895
10.1111/j.1365-2966.2011.18361.x

Crossref
Search ADS
 

Joachimi
B.
,
Köhlinger
F.
,
Handley
W.
,
Lemos
P.
,
2021
,
A&A
,
647
,
L5
10.1051/0004-6361/202039560

Crossref
Search ADS
 

Kass
R. E.
,
Raftery
A. E.
,
1995
,
J. Am. Stat. Assoc.
,
90
,
773
10.1080/01621459.1995.10476572

Crossref
Search ADS
 

Keeley
R. E.
,
Joudaki
S.
,
Kaplinghat
M.
,
Kirkby
D.
,
2019
,
J. Cosmol. Astropart. Phys.
,
2019
,
035
10.1088/1475-7516/2019/12/035

Crossref
Search ADS
 

Keeley
R. E.
,
Shafieloo
A.
,
Zhao
G.-B.
,
Vazquez
J. A.
,
Koo
H.
,
2021
,
AJ
,
161
,
151
10.3847/1538-3881/abdd2a

Crossref
Search ADS
 

Koo
H.
,
Shafieloo
A.
,
Keeley
R. E.
,
L’Huillier
B.
,
2020
,
ApJ
,
899
,
9
10.3847/1538-4357/ab9c9a

Crossref
Search ADS
 

Koo
H.
,
Keeley
R. E.
,
Shafieloo
A.
,
L’Huillier
B.
,
2021a
,
preprint (arXiv:2110.10977)

Koo
H.
,
Shafieloo
A.
,
Keeley
R. E.
,
L’Huillier
B.
,
2021b
,
J. Cosmol. Astropart. Phys.
,
2021
,
034
10.1088/1475-7516/2021/03/034

Crossref
Search ADS
 

Kowalski
M.
et al. ,
2008
,
ApJ
,
686
,
749
10.1086/589937

Crossref
Search ADS
 

L’Huillier
B.
,
Shafieloo
A.
,
Linder
E. V.
,
Kim
A. G.
,
2019
,
MNRAS
,
485
,
2783
10.1093/mnras/stz589

Crossref
Search ADS
 

Li
X.
,
Shafieloo
A.
,
2019
,
ApJ
,
883
,
L3
10.3847/2041-8213/ab3e09

Crossref
Search ADS
 

Li
X.
,
Shafieloo
A.
,
2020
,
ApJ
,
902
,
6

 

Liddle
A. R.
,
2007
,
MNRAS
,
377
,
L74
10.1111/j.1745-3933.2007.00306.x

Crossref
Search ADS
 

Linder
E. V.
,
2003
,
Phys. Rev. Lett.
,
90
,
091301
10.1103/PhysRevLett.90.091301

Crossref
Search ADS
 

Nesseris
S.
,
García-Bellido
J.
,
2013
,
J. Cosmol. Astropart. Phys.
,
2013
,
036
10.1088/1475-7516/2013/08/036

Crossref
Search ADS
 

Perlmutter
S.
et al. ,
1999
,
ApJ
,
517
,
565
10.1086/307221

Crossref
Search ADS
 

Planck Collaboration
et al. .,
2020
,
A&A
,
641
,
A6
10.1051/0004-6361/201833910

Crossref
Search ADS
 

Raveri
M.
,
Hu
W.
,
2019
,
Phys. Rev. D
,
99
,
043506
10.1103/PhysRevD.99.043506

Crossref
Search ADS
 

Riess
A. G.
et al. ,
1998
,
AJ
,
116
,
1009
10.1086/300499

Crossref
Search ADS
 

Riess
A. G.
et al. ,
2007
,
ApJ
,
659
,
98
10.1086/510378

Crossref
Search ADS
 

Riess
A. G.
,
Casertano
S.
,
Yuan
W.
,
Macri
L. M.
,
Scolnic
D.
,
2019
,
ApJ
,
876
,
85
10.3847/1538-4357/ab1422

Crossref
Search ADS
 

Ross
A. J.
,
Samushia
L.
,
Howlett
C.
,
Percival
W. J.
,
Burden
A.
,
Manera
M.
,
2015
,
MNRAS
,
449
,
835
10.1093/mnras/stv154

Crossref
Search ADS
 

Scolnic
D. M.
et al. ,
2018
,
ApJ
,
859
,
101
10.3847/1538-4357/aab9bb

Crossref
Search ADS
 

Shafieloo
A.
,
2007
,
MNRAS
,
380
,
1573
10.1111/j.1365-2966.2007.12175.x

Crossref
Search ADS
 

Shafieloo
A.
,
Alam
U.
,
Sahni
V.
,
Starobinsky
A. A.
,
2006
,
MNRAS
,
366
,
1081
10.1111/j.1365-2966.2005.09911.x

Crossref
Search ADS
 

Spergel
D.
et al. ,
2015
,
preprint (arXiv:1503.03757)

Starkman
G. D.
,
Trotta
R.
,
Vaudrevange
P. M.
,
2008
,
preprint (arXiv:0811.2415)

Suzuki
N.
et al. ,
2012
,
ApJ
,
746
,
85
10.1088/0004-637X/746/1/85

Crossref
Search ADS
 

Tröster
T.
et al. ,
2021
,
A&A
,
649
,
A88
10.1051/0004-6361/202039805

Crossref
Search ADS
 

Trotta
R.
,
2007
,
MNRAS
,
378
,
72
10.1111/j.1365-2966.2007.11738.x

Crossref
Search ADS
 

Trotta
R.
,
2008
,
Contemp. Phys.
,
49
,
71
10.1080/00107510802066753

Crossref
Search ADS
 

APPENDIX A: SCALINGS WITH SIZE AND TYPES OF DATA SETS

In this Appendix, we will seek to understand the dependence of the mode and variance of the distributions of log Z on the size and kind of the data set. We will focus discussion primarily on the distributions where the models used to generate the data and make inferences from the data are the same since they should be the most generalizable case.

In Fig. A1, we can begin to see the dependence of the kind and size of data sets on the distribution of evidence. We lay out three cases where we have calculated evidence for subsets of the full data set, namely an ‘SN only’ case, a ‘BAO only’ case, and a case where we use every other data point for both kinds of data sets, which we will refer to as the ‘half’ case. To elaborate about the ‘half’ case, we include both SN and BAO information but half of the total data points of each. For the SN part of the ‘half’ case, we include the distance moduli information of the lowest redshift SN, but skip the next lowest, include the third lowest, and so on. The BAO part is similar where we include the measurement of H(z) and DM(z) for the lowest redshift bin, skip the next bin, include the third bin, and so on. This case is the simplest comparison with the full case, since both the BAO and SN data sets are included and since both span similar redshift ranges. In other words, this case should be most similar to a case with half the information and the transformation N → N/2. The point the ‘SN only’ and ‘BAO only’ cases is to demonstrate that the distributions of evidence become more distinguishable than would be expected from just increasing the number of data points. This would have to arise from the fact that degeneracies in model parameters are being broken by the different kinds of data sets.

Distributions of log Z (left-hand panel) and Δlog Z (right-hand panel) for different partitions of our full data set. On the top is the distributions of evidence for only the SN data set, on the middle is the distribution of evidence for only the BAO data set, and on the bottom is the distribution of evidence for half of the full data set such that every other data point is used. The difference in scale corresponds to the difference in the number of data points in the different partitions.
Figure A1.

Distributions of log Z (left-hand panel) and Δlog Z (right-hand panel) for different partitions of our full data set. On the top is the distributions of evidence for only the SN data set, on the middle is the distribution of evidence for only the BAO data set, and on the bottom is the distribution of evidence for half of the full data set such that every other data point is used. The difference in scale corresponds to the difference in the number of data points in the different partitions.

Open in new tabDownload slide

For the full data set, the mode of the in-kind distributions are −1537 and they have a variance of 1107. For the ‘half’ data set, the mode of the in-kind distributions are −760 and they have a variance of 758. For the ‘BAO only’ and ‘SN only cases, we see the mode of the in-kind distributions are −33 and −1506, respectively, and their variances are 36 and 1109, respectively. From this, we see that the difference in the modes of the various cases scales as half the number of data points N/2. This result is similar to the familiar statistics of measurements of multiple Gaussian random variables, with the familiar expectations that the χ2 value of the true value scales as N/2.

Further, the difference between the left-hand panels of Figs A1 and A2, in particular that the distributions are more distinguishable in Fig. 2, arises from the fact that, more than just adding more data, the fact the BAO also constrains H(z) allows it to break degeneracies that is essential for making these distributions distinguishable. In particular, the ‘SN only’ panel of Fig. A1 has essentially the same number of data points as the full case (3000 versus 3058) but the full case is far less distinguishable. Indeed, the ‘SN only’ has a similar distinguishability as the ‘half’ case (see Table A2). Counter-intuitively, the ‘BAO only’ case is more distinguishable than the ‘half’ case and is similar to the full case. Simply adding more of the same kind of data points increases the variance of these distributions more than they separate the modes of the distributions.

Table A1.
Open in new tab

The number of data points for each of the data cases, along with the mode of the in-kind distributions.

CaseNumber of data pointsMode of log Z distribution
Full3058−1537
Half1530−760
SN only3000−1506
BAO only58−33
CaseNumber of data pointsMode of log Z distribution
Full3058−1537
Half1530−760
SN only3000−1506
BAO only58−33
Table A1.
Open in new tab

The number of data points for each of the data cases, along with the mode of the in-kind distributions.

CaseNumber of data pointsMode of log Z distribution
Full3058−1537
Half1530−760
SN only3000−1506
BAO only58−33
CaseNumber of data pointsMode of log Z distribution
Full3058−1537
Half1530−760
SN only3000−1506
BAO only58−33
Table A2.
Open in new tab

The frequency that the not in-kind distributions are beyond the 95 per cent level of the in-kind distributions.

Data: Model:FullHalfSN onlyBAO only
(per cent)(per cent)(per cent)(per cent)
D:ΛCDM M:TDE91571969
D:ΛCDM M:PEDE1114411
D:TDE M:ΛCDM85481183
D:TDE M:PEDE1009834100
D:PEDE M:ΛCDM121158
D:PEDE M:TDE1009980100
Data: Model:FullHalfSN onlyBAO only
(per cent)(per cent)(per cent)(per cent)
D:ΛCDM M:TDE91571969
D:ΛCDM M:PEDE1114411
D:TDE M:ΛCDM85481183
D:TDE M:PEDE1009834100
D:PEDE M:ΛCDM121158
D:PEDE M:TDE1009980100
Table A2.
Open in new tab

The frequency that the not in-kind distributions are beyond the 95 per cent level of the in-kind distributions.

Data: Model:FullHalfSN onlyBAO only
(per cent)(per cent)(per cent)(per cent)
D:ΛCDM M:TDE91571969
D:ΛCDM M:PEDE1114411
D:TDE M:ΛCDM85481183
D:TDE M:PEDE1009834100
D:PEDE M:ΛCDM121158
D:PEDE M:TDE1009980100
Data: Model:FullHalfSN onlyBAO only
(per cent)(per cent)(per cent)(per cent)
D:ΛCDM M:TDE91571969
D:ΛCDM M:PEDE1114411
D:TDE M:ΛCDM85481183
D:TDE M:PEDE1009834100
D:PEDE M:ΛCDM121158
D:PEDE M:TDE1009980100

For completeness, we include the distributions of Δlog Z but we will limit discussion of their scalings because these distributions will naturally depend on not just the number of data points but the differences in what the models predict and thus are not generalizable. We will point out that the variance of these distributions is large, as in the full case, and, for some cases, will occasionally come to strong incorrect conclusions, as in the case with expanded errors.

© 2022 The Author(s) Published by Oxford University Press on behalf of Royal Astronomical Society
This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://dbpia.nl.go.kr/journals/pages/open_access/funder_policies/chorus/standard_publication_model)
Issue Section:
Article
Download all slides