• In the 2019 shortfin mako shark stock assessment it was noted that different assumptions and modelling frameworks led to different outcomes.
• Projections, and hence advuce, ignored large uncertainties about growth, age-at-maturity, natural mortality, stock-recruitment relationship, selectivity, and catch rates by fleets.
• Although a variety of diagnostics were used, it was difficult to compare models that use different data sets and structures.

• A cookbook for using model diagnostics in integrated stock assessments Carvalho, F., Winker, H., Courtney, D., Kapur, M., Kell, L., Cardinale, M., Schirripa, M., Kitakado, T., Yemane, D., Piner, K.R. and Maunder, M.N., 2021. Fisheries Research, 240, p.105959.

• Validation of stock assessment methods: is it me or my model talking? Laurence T Kell, Rishi Sharma, Toshihide Kitakado, Henning Winker, Iago Mosqueira, Massimiliano Cardinale, Dan Fu, , ICES Journal of Marine Science, 2021;, fsab104, https://doi.org/10.1093/icesjms/fsab104

The main decisions made in the 2017 assessment were related to the choice of

• Catch scenarios,
  
• Biological parameters
• Indices of abundance

Use prediction skill to validate alternative model, i.e.

• Model Assumptions, structure and fixed parameters
• Biological processes
• Quality of Catch, effort and length statistics

• Trend analysis
• Length-based indicators,
• Stock assessment methods
  • Catch-only
  • Bayesian state space biomass dynamic
  • integrated assessment models.

A variety of diagnostics are available to examine goodness of fit, however, it is difficult to compare models with different data sets or structures

• Residual patterns can be removed by adding more parameters than justified by the data,
• Retrospective patterns can be removed by ignoring the data.
• Hindcasting to estimate prediction skill, a measure of the accuracy of an estimate compared to its observed value unknown by the model.

• Requires assessing whether it is plausible that a system equivalent to the model generated the data
• Empirical data plays an important role in sustainability science, i.e. did your model get it right?
• However, the main quantities of interest in stock assessment, \(SSB\) and \(F\), are latent variables that are not observable.
• So omit observations and then predict the out-of-sample values to estimate prediction skill
• Prediction skill can be used to explore model misspecification and data conflicts, and help to identify alternative hypotheses, and weight ensemble models.

Trends are similar, levels relative to \(B_{MSY}\) are different

• Columns: if North Pacific growth and fecundity assumed then the stock is more productive and is above \(B_{MSY}\).
• Rows: Assumed historical catch had little impact. Was because the working group only considered a limited number of historical scenarios?

In contrast

• Columns: Prior for \(r\) has little effect, model driven by data
• Rows: Catch scenario has biggest impact

Easy to interpret as a score of 0.5 indicates that the model forecasts are twice as accurate as a naïve baseline prediction.

Where a naïve forecast equal to the last observed value, i.e. the weather tomorrow is the same as today.

“Stock assessment is the description of the characteristics of a ‘stock’ so that its biological reaction to being exploited can be rationally predicted and the predictions tested” Sidney Holt

Therefore need

• An objective way to validate models
• To identify the value-of-infomation, i.e. what knowledge and data will improve our ability to estimate historical status and to provide