Selection of Stock Assessment Methods and Reference Points for use in Management Procedures
Estimation of Prediction Skill using Mean Absolute Scaled Error (MASE) and Tuning using Receiver Operating Characteristic (ROC).
L Kell
20 June, 2019
Summary
To evaluate data limited methods we condition an Operating Model (OM) on based life history characteristics and then simulate a variety of data types (i.e. length, catch, and catch per unit effort) which are then used to fit a number of different assessment methods. The predictions from the assessment methods are compared to the OM using the Mean Absolute Scaled Error (MASE) and the ability of the methods to assess stock status to target and limit reference points using Receiver Operating Characteristic (ROC).
- Set up OM and OEM, run contrasting assessment methods and compare projections and prediction skill using MASE and current stock status relative to reference points using ROC.
- The OM and OEM is based on WKLife, i.e. Fischer, DeOliveira and Kell (submitted).
- The justification for the choice of methods is based on Pons, Kell and Cope (submitted), i.e. take the best performing length and catch only methods.
- Compare the Cat 3&4 methods to a Cat 1 based on a biomass dynamic model
Introduction
The provision of fisheries management advice requires fitting a model to historical data to assess current stock status, then predicting the future response of the stock to management. It is important to validate models used by checking that historical, current and future predictions are consistent with reality. This is difficult to do, however, as it is seldom possible to compare quantities derived from a model, such as Spawning Stock Biomass (SSB) and fishing mortality (F), to actual observations. Therefore simulation is commonly used to validate assessment methods used to provide advice. For example where an Operating Model (OM) is used to simulate observations from a fishery and stock using an Observation Error Model (OEM), allowing assessment models to be fitted then projected forward and the predictions compared to the OM.
Methods
Mean Absolute Scaled Error
It is difficult to compare models based on diffent datasets using traditional methods such as AIC. Although Root mean square error (i.e. the square root of the variance of the difference between a prediction and the true value) is often used to indicates how well a model fits it cannot be used to compare across series. We therefore use mean absolute scaled error (MASE) as it is independent of the scale of the data, so can be used to compare forecasts across data sets with different scales. MASE can be used to compare future predcitions to be compare to a naive forecast. In a naive forecast observations from the last time period are used as the current period’s forecast, without adjusting them or attempting to establish causal factors. If a method performs poorly compared to the naive forecast then it has no predictive ability. ## Receiver Operating Characteristic
ROC curves are used to identifty stock status relative to reference points, i.e. how to specify limit and target reference points, ROC is therefore a way of tuning MPs.
For example if the objective is to identify when a target is reached or a limit is exceeded then this can be done by plotting the true positive rate (TPR) against the false positive rate (FPR) at various threshold or model settings. For example if the objective is to avoid \(F_{lim}\) using a length based method, what should be the value of the estimate of Z (i.e. the threshold) at which the limit is exceeded.
Results
Figure 1 Operating model for turbot.
Fits to entire time series
Biomass dynamic model using catch and CPUE
Figure 2, Simulation test of biomass dyanmic assessment for turbot.
Catch only
Length only
Figure 3 Estimates of SPR.
Figure 4, Estimates of F+M
Hindcasts
Biodyn
Figure 5, Simulation test of biomass dyanmic assessment for turbot.
MASE
ROC
Results
Figures
Tables
Software Versions
- R version 3.5.2 (2018-12-20)
- FLCore: 2.6.13
- FLBRP: 2.5.3
- FLasher: 0.5.3.9001
- FLife: 3.2.3.0
- ggplotFL: 2.6.6
- Compiled: Thu Jun 20 11:16:41 2019
Acknowledgements
This vignette and many of the methods documented in it were developed under the MyDas project funded by the Irish exchequer and EMFF 2014-2020. The overall aim of MyDas is to develop and test a range of assessment models and methods to establish Maximum Sustainable Yield (MSY) reference points (or proxy MSY reference points) across the spectrum of data-limited stocks.
References
Session Info
R version 3.5.2 (2018-12-20)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.5 LTS
Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_GB.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_GB.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_GB.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] parallel stats graphics grDevices utils datasets methods
[8] base
other attached packages:
[1] doParallel_1.0.14 foreach_1.4.4 LBSPR_0.1.3
[4] mpb_3.0.1 mydas_1.0.3.0 randtests_1.0
[7] FLife_3.2.3.0 ggplotFL_2.6.6 FLasher_0.5.3.9001
[10] FLFishery_0.1.6 FLBRP_2.5.3 FLCore_2.6.13
[13] iterators_1.0.10 lattice_0.20-38 reshape_0.8.8
[16] dplyr_0.8.0.1 plyr_1.8.4 ggplot2_3.1.1
[19] knitr_1.22
loaded via a namespace (and not attached):
[1] tidyselect_0.2.5 xfun_0.6 purrr_0.3.0
[4] colorspace_1.4-1 htmltools_0.3.6 stats4_3.5.2
[7] yaml_2.2.0 rlang_0.3.4 pillar_1.4.0
[10] glue_1.3.1 withr_2.1.2 RColorBrewer_1.1-2
[13] stringr_1.4.0 munsell_0.5.0 gtable_0.3.0
[16] codetools_0.2-16 evaluate_0.13 Rcpp_1.0.1
[19] scales_1.0.0 plotrix_3.7-4 gridExtra_2.3
[22] digest_0.6.19 stringi_1.4.3 grid_3.5.2
[25] tools_3.5.2 magrittr_1.5 lazyeval_0.2.2
[28] tibble_2.1.1 crayon_1.3.4 tidyr_0.8.2
[31] pkgconfig_2.0.2 MASS_7.3-51.1 Matrix_1.2-15
[34] assertthat_0.2.1 rmarkdown_1.12 rstudioapi_0.10
[37] R6_2.4.0 compiler_3.5.2 BMW X1 XDRIVE18D XLINE AU14GXJ
£125.32