Model averaging to streamline the stock assessment process

Abstract

Introduction

Stock assessment can be defined as the application of quantitative and statistical models to estimate the current and historical status and trends of a fish stock, including abundance, mortality, and productivity (Hilborn and Walters, 1992). The typical stock assessment process involves a team of scientists generating a suite of candidate models, examining their outputs, and then selecting a single model that is considered to be the “best”. The selection process may be based on a mixture of criteria, e.g. the model likelihood or the examination of residuals. The outcomes of this assessment process are often point estimates of the stock status, with minimal presentation of the form and size of the uncertainty of those estimates. In the United States and Europe, the tendency has been to deal with uncertainty through the definition of stock status limits (Shertzer et al., 2008; Ralston et al., 2011).

Stock assessment can be very resource-intensive. A sufficient number of scientists with an appropriately high level of training are required to generate the suite of candidate models, interpret their output, and review the conclusions. Additionally, as each model is examined in detail, the process can be very time-consuming. The magnitude of the resource problem is likely to increase as demand for advice and data availability increases (Methot, 2009; Jardim et al., 2014).

A key step in the stock assessment process is that of moving from the initial suite of candidate models to a single “best” model. This effectively means that the assumptions behind the alternative models are rejected. But these alternative models could provide perfectly valid advice. As Box and Draper (1987) stated, “Remember that all models are wrong, the practical question is how wrong do they have to be to not be useful”. Although only one model may be the most likely, the others may still represent plausible “states of nature” and contribute to the estimation of the uncertainty about it. Many other authors have raised and discussed issues surrounding the basing of conclusions on a single model (Butterworth et al., 1996; Hilborn, 1997; Patterson, 1999; Hill et al., 2007; Simmonds et al., 2011). The major danger is that ignoring model-selection error leads to underreporting of variability and too narrow confidence intervals (Claeskens and Hjort, 2008). Model averaging provides a way to incorporate this error into a single result by combining the results from several models.

We propose the use of model averaging (Claeskens and Hjort, 2008) specifically within the age-structured a4a framework (section 3; Jardim et al., 2014) to address the two stock assessment issues identified above: (i) the need to choose only a single model, and (ii) intensive resource use.

There are several benefits of using a model-averaging approach. It removes the need to select a single model, which, in turn, reduces the need to conduct extensive diagnostic model checks. Consequently, the introduction of model averaging allows for a change in stock assessment practices wherein more time is spent on defining an initial suite of plausible models for each stock. This suite of models must be carefully chosen to represent possible “states of nature”. It may even be possible to define a sufficiently exhaustive set of initial models that would be appropriate to use for a group of stocks, e.g. all demersal species in the North Sea. We believe that by the combined effort of experts in fishery and statistical modelling, model averaging, in the context of statistical age-structured stock assessment models, could become an automated process.

In this paper, we propose a change in the stock assessment process from one where model checking and model selection are the focus to one where designing appropriate models is the most important scientific task. We believe that this change is not a revolution, merely an evolution of what is already done. We envisage a process where the full gamut of fishery disciplines—oceanography, genetics, biology, population biology, gear technology, among others—can have a direct input into the design of stock assessment models. The way in which these diverse groups communicate with the stock assessment group would be to propose issues that they would like to see addressed. These issues will help form a suite of plausible models which, through model averaging, will be the basis of the stock assessment. We recognize that there are unsolved problems even for model selection when modelling overdispersed-count data; therefore, we restrict our attention to lognormally distributed data. First, we describe in more detail the idea of setting up plausible states of nature and techniques for model averaging, and in the final section, we discuss practical issues.

The a4a stock assessment framework

The a4a framework is designed to be a flexible stock assessment modelling tool with an intuitive interface. This is a key part of our model-averaging approach, as the success of the idea depends on fishery scientists being able to easily and efficiently translate ideas into mathematical models of the stock dynamics in the assessment model.

The a4a framework gives easy access to a wide variety of fishing mortality, survey catchability, recruitment, and observation variance models, allowing the user to set up a wide range of population dynamics and fishery models. Obviously, the user must be familiar with linear modelling and working with splines to get the most out of this interface.

Most stock assessment models rely on a few basic assumptions: the data correspond to a fully spatially mixed stock, full spatial coverage in terms of catch estimates and surveys, known natural mortality (M), a constant stock–recruitment relationship, and constant survey catchability. These assumptions, depending on the model, can be relaxed or adapted, but often, it is not straightforward to incorporate flexibility with respect to these assumptions, mainly because most stock assessment models have not been designed as exploratory tools, but rather to provide a single assessment. Modelling frameworks, like a4a (Jardim et al., 2014) or Stock Synthesis 3 (SS3, Methot and Wetzel, 2012), promote the process of exploring different models, opening the possibility of dealing with several distinct models, instead of tweaking small details of a single model.

Plausible states of nature

When dealing with building a model for stock assessment, one is often faced with the fact that more than one hypothesis can be valid, e.g. a regime shift in the North Sea in 1988 changed the temperature that may have affected the mortality rate of salmon (Salmo salar) via reduced growth (Beaugrand and Reid, 2012). Was there a regime shift effect? Does the regime shift affect mortality or reproductive success? Other examples of situations where fishery science could inform a list of plausible scenarios are: In our opinion, these cases fall into two categories: Each of the situations listed above highlights a recognized need by the fishery/scientific community to address multiple hypotheses about the “states of nature”. Some effects may act across a number of stocks, such as temperature effects or variations in primary productivity, while others may be more stock-specific, such as gear mesh size increases affecting smaller or thinner fish. The next step in the process is combining these issues into a set of plausible states of nature which will essentially define the stock assessment models that should be applied. In the model-averaging setting, all of these models will contribute to the final estimates and uncertainty of the state of the stock and to other standard outputs from stock assessment. The weight that each model receives in the final output ultimately depends on the fit of the model to the data. A selection of procedures for weighting models is given in the next section.

• Changes in phytoplankton and zooplankton coincided with an increase in catches of the Western stock of horse mackerel (Trachurus trachurus L.) in the northern North Sea after a northerly expansion from the Bay of Biscay after 1987 (Reid et al., 2001).

• Food availability and climate effects can directly affect recruitment of fish stocks, as shown for North Sea cod (Gadus morhua), which has experienced depressed recruitment since the mid-1980s (Beaugrand et al., 2003; Olsen et al., 2010).

• Cases characterized by spatially variable selectivity tend to confuse standard approaches of interpreting fishing mortality rates in concert with selectivity (Crone et al., 2013). Assuming that differences in spatial and temporal availability of the fish can alter the expected shape of the gear-selectivity curve, there is a need for flexible selectivity patterns. Crone et al. (2013) suggest including both the contact selectivity and availability in stock assessments to model the combined factors that affect fish vulnerability: log Q = log contact selectivity + log availability.

• About the previous point, several authors document shifting stocks, suggesting that migratory and distributional changes have occurred (Nye et al., 2009; Last et al., 2011; Jansen et al., 2012; Poos et al., 2013), and could affect availability to surveys.

• Increasing efficiency of survey vessels is a known issue. For example, one of the two multinational surveys for North Sea cod was removed from the annual stock assessment due to suspected time-varying catchability (ICES, 2011). Temporal trends in survey or commercial catchability can bias the estimates of stock size and fishing mortality in stock assessment models that do not account for them (Wilberg et al., 2010).

• Developing operating models for southern bluefin tuna (Thunnus maccoyii) (R. Hillary, pers. comm.), where uncertainty in assumptions of life history parameters, such as natural mortality and carrying capacity, is incorporated by setting up a grid of plausible input parameter values.

• Anything that could be considered to be on a continuum, which would be dealt with by specifying a range of plausible values. For example, a range of breakpoints in a stock–recruitment relationship, a range of ages above which fishing mortality is constant, a range of M values, or a range of growth scenarios.

• Different hypotheses form categories rather than a range of options. For example, in Wilberg et al. (2010), there are a variety of models proposed for survey catchability with time-step changes, smooth changes, changing catchability with covariates such as spatial patchiness of effort, or spatial extent of fishing effort.

Model averaging

Model averaging is a technique for incorporating model-selection uncertainty into inference (Buckland et al., 1997). From the stock assessment perspective, this can be thought of as the incorporation of uncertainty due to different plausible states of nature, as mentioned above. The purpose of this section is not to be prescriptive in the model-averaging schemes to consider, but rather to highlight that there are a variety of approaches: frequentist and Bayesian, simple and complex.

Model averaging can be thought of as a model-weighting algorithm where the weights are based on the support for the model in the data (Claeskens and Hjort, 2008). We discuss four model-averaging techniques, three chosen as they are relatively easy to implement, and a fourth chosen for its desirable features. The first two methods use fixed weights, while the third and fourth methods acknowledge that model weights, like parameter estimates, are random and use stochastic simulation to build the distribution of the weights. Finally, we mention how expert weights can be incorporated into model-averaging schemes. For further reading on model averaging and model discrimination, we suggest Buckland et al. (1997), Burnham and Anderson (2002), Claeskens and Hjort (2008), and King et al. (2010).

Single-model inference

Frequentist weights

It is suggested to subtract the minimum AIC from each model AIC to avoid numerical issues when taking exponents.

• Simulate for each model M$(θ1,M∗,θ2,M∗,…)$ from an MV normal distribution with mean $(θˆ1,M,θˆ2,M,…)$ and variance $ΣˆM$⁠.

• Calculate $μ∗=∑M∈SμM∗=∑M∈Sf(θ1,M∗,θ2,M∗,…)$⁠.

Bayesian weights

Confidence intervals can be derived in a similar fashion to the simulation approach described for frequentist weights. There are a variety of options for the distribution of the model parameters. One option would be to assume MV normality using the posterior mean and posterior-covariance matrix derived from the MCMC samples. Another option would be to build a copula approximation to the posterior distribution using an estimate of the posterior-variance matrix and kernel-density estimates of the marginal-posterior distributions of the parameters.

Frequentist simulation-based weighting

We now consider two simulation-based approaches to model averaging. Two reasons to do simulation-based model averaging is to (i) account for the fact that the fixed weights in the previous two examples are, themselves, estimates; and (ii) avoid assuming that the estimators are perfectly correlated.

The first is a frequentist approach developed in Buckland et al. (1997), which can be thought of as bootstrapping the full model-selection process, from model fitting through model selection to prediction. It requires that the empirical distribution of the data can be simulated using non-parametric bootstrapping. Single-model inference can then be applied independently to each resample, the estimator $μˆ∗$ calculated for each resample in turn, and the mean and confidence interval of the estimator derived using standard techniques (Davison and Hinkley, 2003). It should be noted that in fishery contexts, it is often not possible to apply non-parametric bootstrapping. Parametric bootstrapping may be considered, but care should be taken to ensure that the parametric model underlying the parametric bootstrap does not overweight one or more of the models under consideration.

Bayesian simulation-based weighting

The fourth and final method we consider is known as reversible jump Markov Cain Monte Carlo (RJMCMC). We mention this approach for completeness, but the details are beyond the scope of this paper. The improvements that RJMCMC brings over the use of fixed weights, e.g. the HME approach, are that the assumption that the estimators are perfectly correlated is not required, and there is no need to use independent samplers for each model. We refer the reader to King et al. (2010) for further details on RJMCMC.

Incorporating expert weights

For the Bayesian approaches, prior weights on models can be applied directly to the weights w_M, as in the Bayesian setting, model weights are analogous to model probabilities.

Approaches to model averaging can be developed in relative isolation relative to the development of plausible models. An efficient division of labour would be to engage statisticians and programmers to develop tailored methods for model averaging of stock assessment models. This would free up stock assessment scientists, population biologists, fishery biologists, oceanographers, and others to collaborate on proposing a set of plausible models; this task would be periodically reviewed.

Discussion and future challenges

The use of model averaging proposed here should allow fishery scientists to concentrate on fishery science. Within this framework, fishery scientists who are not stock assessment experts in the traditional sense can, along with stakeholders, directly contribute to the stock assessment process. By defining a stock assessment as a (potentially large) set of models, specialized and stakeholder knowledge can be more readily incorporated into the definition of a stock assessment through the development of a list of plausible models and hypotheses. It may be that a stock assessment defined by such a group of models provides more consistent year-to-year advice than a stock assessment defined by a single model, which is periodically refined. It is also possible that this approach would receive more support from the stakeholder community.

As stated previously, the key to the success of the ideas in this paper is a flexible and intuitive interface. In a4a (Jardim et al., 2014), this is achieved through the use of equations for fishing mortality, survey catchability, recruitment, and observation variance, specified as linear models and splines. We think this interface is a valuable tool for translating plausible ideas/states of nature into plausible stock assessment models, and could be used to cover many of the examples mentioned in the text.

In terms of implementation, a tier of models could be constructed to give some common structure across stocks. This type of coherence is often sought by having an assessment group be responsible for all flatfish or all demersal fish in an ecoregion. One could set up ecoregion-level scenarios applicable to all stocks, followed by further levels such as demersal and pelagic. This would provide a starting point for any new stock that is assessed, and should improve the coherence and consistency, and perhaps even the transparency, of the assessment process. It may be that the ecoregion level is just too broad to be practical; more appropriate groupings would take into account knowledge of fisheries and the biology of the species, and could be informed by classification analyses.

Model averaging avoids the pitfalls of using a single model: too narrow confidence intervals, overly optimistic tests of significance, and potentially biased results (Claeskens and Hjort, 2008). Dealing with uncertainty in general (model and parameter) in fishery advice is developing, but there needs to be more discussion about the translation of (model and parameter) uncertainty in advice through to the implementation of policy. We do not address these issues here, but instead refer the reader to Hill et al. (2007), who give a thorough discussion of model uncertainty (as well as a good historical perspective) in terms of ecosystem management. Their examples and recommendations apply equally well to single-species stock assessment in our context.

Bimodal distributions may result from the use of model averaging in the face of competing hypotheses. In these situations, taking the posterior model-averaged mean is clearly not a good summary description. A better description would be the marginal density or at least the highest posterior density interval. It is the management procedures requiring a point estimate that stand out as inadequate in such situations, rather than the idea of model averaging. Different models may make different predictions for good reasons. Is it better to select one scenario based on human reasoning rather than average a plausible range of scenarios based on data?

Model averaging is not easy. Simpler methods make more assumptions, while a full Bayesian treatment through reversible jump MCMC is very difficult. The smoothed AIC approach is straightforward, but relies on a normal approximation to the distribution of model-parameter estimates, which could be seriously inadequate. The HME method uses samples from the parameter (posterior) distribution requiring the use of an MCMC algorithm (automatic methods are available for this). However, the HME is notorious for having infinite variance and has been thoroughly discouraged by some leading statisticians (Neal, 2008). Bootstrap-generated weights are appealing, since they are based on few assumptions, and the algorithm is simple. However, it is typically very difficult to generate bootstrap resamples with fishery data due to the complexity of the underlying sampling schemes.

When Bayesian model-averaging is being used, priors on models need to be specified. There are various approaches to this, and perhaps a sensible approach is to set priors on groups of models where all the models in each group will be addressing similar issues. In an advisory body, guidelines for practical issues like this could be developed in conjunction with the model averaging procedure itself.

An evolution of the stock assessment process

We are not suggesting a radical change to the way fisheries are managed. We are not suggesting a move away from single-species assessments, nor are we suggesting a change in the advice. We are lobbying for a change in the typical stock assessment procedure. The most exciting foreseen benefit is that this approach aims to bring improved coherence within an advisory body. There are many ICES Working Groups that attract the best scientists in their field. If we can develop ways to accumulate ideas from a range of sources into a set of stock assessment models, then we could have the basis for a cross-discipline, scientifically defensible, and powerful stock assessment.

Acknowledgements

We thank Ruth King for general discussion on model-averaging methods, and two anonymous reviewers whose helpful comments made for an improved paper.

References

Author notes