Sebastian Funk, Sam Abbott and Johannes Bracher’s Discussion Contribution to the Papers in Session 2 of The Royal Statistical Society’s Special Topic Meeting on Covid-19 Transmission: 11 June 2021

We thank Teh and co-authors for a stimulating paper. The comparison of the different performance characteristics of R_t estimators is to be applauded. The lack of any more general systematic comparison has been a crucial gap in the literature. At the same time, the chosen approach highlights some difficulties in addressing this gap.

• 1.

  Reproduction numbers are generally not observable and the ability of a model to reconstruct their trajectories can only be assessed in simulation studies. These may not yield sufficient information to accurately assess the ability of a model to estimate R_t in the real world. While simulation studies can be useful to assess the self-consistency of a model and its sensitivity to potential misspecification, their use for comparing estimates between models is more limited unless the data generating process is not one of the models used for estimating.

• 2.

  It is challenging to make a fair comparison between tools that each have much flexibility and can be used and parameterised in a variety of ways that affect the resulting reconstructed R trajectories; identifying whether any resulting performance differences are due to differences in the methods themselves or specific user choices can be difficult.

• 3.

  By definition of the instantaneous reproduction number R_t is the average number of new infections caused by infectious individuals at time t (weighted by their infectiousness). However, this leaves much room for interpretation. It can be defined over any time frame such as daily or weekly, and this choice affects the estimates. Moreover, even within the same model there may be several quantities which could be interpreted as R_t. For example, consider the following discrete-time branching process model, initialised with some starting number of infectious individuals I₀, a mean reproduction number R and over-dispersion k:

  I_t ∼ Poisson (λ_t).

  λ_t = X_{t − 1}I_{t − 1}.

  X_t ∼ gamma (mean = R, var = R/k).

  There are at least three ways one could define an effective reproduction number in this model:

  (a) R_t = I_t/I_{t − 1}.

  (b) R_t = X_{t − 1}.

  (c) R_t = R.

Any assessment of the quality of R_t estimates would thus involve a decision on the timescale at which the reproduction number is considered and what exactly R_t represents and whether models agree that this is being estimated.

We believe that evaluating the performance of published R_t estimates or methods for generating them is valuable, and reiterate our appreciation of the authors’ approach to doing so. Further comparisons should be encouraged by both those developing methods and those consuming them. Ideally, this could be led by an independent arbiter who defines the parameters of the exercise and, in collaboration with the developers, applies the methods to be assessed.