Alright, so it has been a while since there has been updates on the single ESM calibration activities so for those who need a refresher here is goes…

We originally tried to find the Hector parameters that best fit temperature. Although we were able to solve for Hector parameters the values for S and diff returned were unrealistic, which we determined was caused by S and diff trading off with one another. When we were only using temperature data in the MSE S and diff were unidentifiable so we ended up adding heat flux as an additional comparison data.

When we use the heat flux data we ran into two issues.

  1. Not all of the model’s had heat flux values for each experiment and ensemble member.
  2. When we use the historical heat flux data the resulting Hector parameters are still wonky, because Hector tried to capture the all of the variability in heat flux data by adjusting parameter values even though we don’t expect Hector to be able to reproduce the inter-annual variability of the heat flux time series.

Solution

So in order to avoid the obstacles that diff, S, and limited heat flux data present us we decided to use the range of the cmip range for the 2100 heat flux + the mesa function RPL designed for the MCMC as the tie breaker to use in the Hector calibration. We also do not fit volscl for the models that do not provide historical output because often times optim takes volscl to funky town even though it has little to no impact on the performance of Hector’s temperature fit in the future period.

So here are our results I think they look pretty good for the most part and have a few questions for you at the end.

What does the MSE look like?

For each model? What models do the best and the worst?

What is the model that Hector does the best job emulating?

## [1] "MRI-CGCM3"

What is the model that Hector does the worse job emulating?

## [1] "MPI-ESM-P"


What does the values we are trying to minimize look like for each variable / experiment? Is it consistent across models / variables / ensembles?

variable experiment min max mean sd range
heatflux rcp26 1.175 1.176 1.175 0.000 0.000
heatflux rcp45 1.088 1.088 1.088 0.000 0.000
heatflux rcp60 0.837 0.838 0.837 0.000 0.001
heatflux rcp85 1.286 1.287 1.286 0.000 0.001
Tgav historical 0.037 1.802 0.365 0.260 1.765
Tgav rcp26 0.005 0.065 0.029 0.015 0.060
Tgav rcp45 0.003 0.114 0.033 0.028 0.112
Tgav rcp60 0.003 0.084 0.023 0.020 0.081
Tgav rcp85 0.005 0.075 0.025 0.019 0.071

As we expected (because of the way that the mesa function work) the min heatflux value is the same across the models/ensembles.

What do the best fit parameters look like?

So it looks like there are still some unreasonable values of diff, alpha, and volscl.

## # A tibble: 4 x 4
##   param     min    max  mean
##   <chr>   <dbl>  <dbl> <dbl>
## 1 alpha   0.119   2.45  1.48
## 2 diff    1.38  788.   42.8 
## 3 S       1.23    5.38  3.56
## 4 volscl -0.373   3.95  1.18

Which have unreasonable param values?

For alpha? (This is going to be values out side of the range of 0 to 1)

model S diff alpha volscl
ACCESS1-0 4.481406 4.022459 2.169738 1.3063726
ACCESS1-3 4.137873 4.869920 2.012696 1.3027606
CanESM2 5.379942 2.676373 1.845813 1.4425507
CCSM4 3.417640 2.150772 1.361184 0.8032603
CESM1-BGC 4.212259 4.757437 1.254342 1.1186954
CESM1-CAM5 4.672884 3.511973 2.004213 1.0839185
CMCC-CESM 3.066088 2.427470 2.454835 0.1542517
CMCC-CM 3.344814 2.104657 1.797835 -0.3734199
CMCC-CMS 3.492108 1.933353 1.819106 -0.1699221
CNRM-CM5 3.793208 6.027884 1.340480 1.1009000
CSIRO-Mk3-6-0 4.160642 5.036721 1.662681 1.6443379
CSIRO-Mk3L-1-2 2.832724 11.630328 1.152573 0.9342279
EC-EARTH 4.288184 4.615576 1.925060 1.4865312
FGOALS-g2 2.603372 3.416440 1.866707 -0.0187293
GFDL-CM3 3.925983 1.375715 2.226984 0.9854006
GFDL-ESM2G 2.205985 5.230977 1.360018 1.7792677
GISS-E2-R-CC 3.033876 10.034147 1.068817 1.3610505
GISS-E2-R 3.273466 7.933532 1.226466 1.4798282
inmcm4 2.816547 11.726621 1.160714 0.6170523
IPSL-CM5A-MR 4.760392 3.727721 1.142407 1.1579202
MIROC-ESM-CHEM 5.377651 2.763389 1.768058 0.9420724
MIROC-ESM 5.283901 2.968408 1.613244 1.3916866
MIROC5 3.640256 6.619373 1.484628 1.5105629
MPI-ESM-LR 3.231097 2.406721 1.499654 0.8941075
MPI-ESM-MR 3.184624 2.382020 1.812322 0.9785344
MRI-CGCM3 3.207000 8.602656 1.733598 1.2702199
MRI-ESM1 1.228837 788.170015 1.606054 2.9912901
NorESM1-ME 2.847869 2.789965 1.910678 0.3259734

Alright so it looks like the majority of the alpha param values are outside this range. Is that alarming?


For S? (we will say that the normal range is between 0 and 10)

## [1] model  S      diff   alpha  volscl
## <0 rows> (or 0-length row.names)

Well it looks like all of the S values are reasonable!


For diff? (we will say that the normal range is between 0 and 8)

##            model        S       diff     alpha    volscl
## 1 CSIRO-Mk3L-1-2 2.832724  11.630328 1.1525729 0.9342279
## 2   GISS-E2-R-CC 3.033876  10.034147 1.0688171 1.3610505
## 3         inmcm4 2.816547  11.726621 1.1607136 0.6170523
## 4      MPI-ESM-P 1.464755 473.694144 0.1193029 3.9530229
## 5      MRI-CGCM3 3.207000   8.602656 1.7335980 1.2702199
## 6       MRI-ESM1 1.228837 788.170015 1.6060545 2.9912901

Okay so I think that the most alarming diff values are MRI-ESM1 and MPI-ESM-P. But I suspect that this is because these models only reported historical temperature data. May be we cannot fit to only historical temp + 2100 heat flux.

I am not surprised that inmcm4 is in there because it has wonky heat flux data. The other models with diff fits that range from 8 to 11 would need a bit more digging into.

Temp Comparison Plots

All

Calibrated Hector pretty consistently underestimates the ESM temperature data around 1960.

ACCESS1-0

ACCESS1-3

CanESM2

CCSM4

CESM1-BGC

CESM1-CAM5

CESM1-WACCM

CMCC-CESM

CMCC-CM

CMCC-CMS

CNRM-CM5

CSIRO-MK3-6-0

CSIRO-MK3L-1-2

This fit in not great, the temp over the historical period ramps up faster than I would have expected compared to the other ESM historical data. So it is not unsurprising to me that Hector struggles to be as warm as the ESM data.

EC-EARTH

FGOALS-g2

There is almost no volcano dips in this historical period.

GFDL-CM3

GFCL-ESM2G

I think that the RCP 2.6 ESM output looks funky. SO I am not surprised that Hector over estimates RCP 2.6 temp from 2050 to 2100.

GICC-E2-H-CC

GISS-E2-H

GISS-E2-R-CC

GISS-E2-R

inmcm4

IPSL-CM5A-MR

IPSL-CM5B-LR

MIROC-ESM-CHEM

MIROC-ESM

MIROC5

MPR-ESM-LR

MPR-ESM-MR

MPI-ESM-P

This is the model that Hector does the worst job of emulating. Which is not surprising since there is only the historical data to use as comparison data.

MRI-CGCM3

So this is the model with the smallest MSE value.

MRI-ESM1

This diff value is one unrealistic but that is not surprising since the only ESM temp data we have is from the historical period.

NorESM1-ME

Heat Flux Comparison Plots

It looks like most of the Hector values from the best fits fall at the ends of the cmip range.

Questions

  1. I used the Hector cmip range heat flux values from 2100 for all of the scenarios, should I limit the heat flux range to the experiments we have the individual esm temp data for? Do you think that might be a result?
  2. I think that the fits look reasonably good, but there is still a question about why Hector pretty consistently underestimates temp around 1960. Jeff Arnold brought that up the other day do we look into it? I wonder if it has to do with the vol forcing files we are using…