7E1. State the three motivating criteria that define information entropy. Try to express each in your own words.
# The three motivating criteria that define information entropy are listed below:
# (1) be measured on a continuous scale such that the spacing between adjacent values is consistent,
# (2) capture the size of the possibility space such that its value scales with the number of possible outcomes
# (3) be additive for independent events such that it does not matter how the events are divided.7E2. Suppose a coin is weighted such that, when it is tossed and lands on a table, it comes up heads 70% of the time. What is the entropy of this coin?
p <- c(0.7, 1 - 0.7)
(H <- -sum(p * log(p)))## [1] 0.61086437E3. Suppose a four-sided die is loaded such that, when tossed onto a table, it shows “1” 20%, “2” 25%, “3” 25%, and “4” 30% of the time. What is the entropy of this die?
p <- c(0.20, 0.25, 0.25, 0.30)
(H <- -sum(p * log(p)))## [1] 1.3762277E4. Suppose another four-sided die is loaded such that it never shows “4”. The other three sides show equally often. What is the entropy of this die?
p <- c(0.20, 0.25, 0.25, 0.30)
(H <- -sum(p * log(p)))## [1] 1.3762277M1. Write down and compare the definitions of AIC and WAIC. Which of these criteria is most general? Which assumptions are required to transform the more general criterion into a less general one?
# AIC is defined as Dtrain+2p where Dtrain is the in-sample training deviance and p is the number of free parameters estimated in the model.DIC is defined as D¯+pD=D¯+(D¯+D̂ ) where D¯ is the average of the posterior distribution of deviance and D̂ is the deviance calculated at the posterior mean.
#WAIC is defined as −2(lppd−pWAIC)=−2(∑Ni=1logPr(yi)−∑Ni=1V(yi)) where Pr(yi) is the average likelihood of observation i in the training sample and V(yi) is the variance in log-likelihood for observation i in the training sample.
# All definitions involve two components: (1) an estimate or analog of the in-sample training deviance (i.e., Dtrain for AIC,and lppd for WAIC) and (2) an estimate or analog for the number of free parameters estimated in the model (i.e., p in AIC, pD in DIC, and pWAIC in WAIC).
# WAIC is the mose generalthan AIC . To move from WAIC to DIC, the posterior distribution should be assumed to be approximately multivariate Gaussian. To move from DIC to AIC, The priors should be assumed to be flat or overwhelmed by the likelihood.7M2. Explain the difference between model selection and model comparison. What information is lost under model selection?
# Model selection is choosing to retain the model with the lowest information criterion value and to discard all other models with higher values. This practice loses information about relative model accuracy contained in the differences among information criterion values; this is especially problematic when the selected model only outperforms its alternatives to a small degree.
# Model averaging is using Bayesian information criteria to construct a posterior predictive distribution that leverages the uncertainty in multiple models. This practice does not lose information on its own. However, when combined with undisclosed data dredging, it can lead to spurious findings.7M3. When comparing models with an information criterion, why must all models be fit to exactly the same observations? What would happen to the information criterion values, if the models were fit to different numbers of observations? Perform some experiments, if you are not sure.
# Information criteria are based on deviance, which is accrued over observations without being divided by the number of observations. Thus, it is a sum and not an average. So, all else being equal, a model with more observations will have a higher deviance and thus worse accuracy according to information criteria. It would be an unfair comparison to contrast models fit to different numbers of observations.
# To further confirm with experients, we can calculate WAIC for models fit to increasingly small subsamples of the same data. The information criteria should decrease alongside the sample size. As show in the result, the WAIC increased as sample size increased from 300 to 400 and then to 500.
library(rethinking)## Loading required package: rstan## Loading required package: StanHeaders## Loading required package: ggplot2## rstan (Version 2.19.3, GitRev: 2e1f913d3ca3)## For execution on a local, multicore CPU with excess RAM we recommend calling
## options(mc.cores = parallel::detectCores()).
## To avoid recompilation of unchanged Stan programs, we recommend calling
## rstan_options(auto_write = TRUE)## For improved execution time, we recommend calling
## Sys.setenv(LOCAL_CPPFLAGS = '-march=corei7 -mtune=corei7')
## although this causes Stan to throw an error on a few processors.## Loading required package: parallel## Loading required package: dagitty## rethinking (Version 2.01)##
## Attaching package: 'rethinking'## The following object is masked from 'package:stats':
##
## rstudentdata(Howell1)
d <- Howell1[complete.cases(Howell1), ]
d_500 <- d[sample(1:nrow(d), size = 500, replace = FALSE), ]
d_400 <- d[sample(1:nrow(d), size = 400, replace = FALSE), ]
d_300 <- d[sample(1:nrow(d), size = 300, replace = FALSE), ]
m_500 <- map(
alist(
height ~ dnorm(mu, sigma),
mu <- a + b * log(weight)
),
data = d_500,
start = list(a = mean(d_500$height), b = 0, sigma = sd(d_500$height))
)
m_400 <- map(
alist(
height ~ dnorm(mu, sigma),
mu <- a + b * log(weight)
),
data = d_400,
start = list(a = mean(d_400$height), b = 0, sigma = sd(d_400$height))
)
m_300 <- map(
alist(
height ~ dnorm(mu, sigma),
mu <- a + b * log(weight)
),
data = d_300,
start = list(a = mean(d_300$height), b = 0, sigma = sd(d_300$height))
)
(model.compare <- compare(m_500, m_400, m_300))## Warning in compare(m_500, m_400, m_300): Different numbers of observations found for at least two models.
## Model comparison is valid only for models fit to exactly the same observations.
## Number of observations for each model:
## m_500 500
## m_400 400
## m_300 300## Warning in ic_ptw1 - ic_ptw2: longer object length is not a multiple of shorter
## object length
## Warning in ic_ptw1 - ic_ptw2: longer object length is not a multiple of shorter
## object length
## Warning in ic_ptw1 - ic_ptw2: longer object length is not a multiple of shorter
## object length## WAIC SE dWAIC dSE pWAIC weight
## m_300 1822.663 29.91606 0.0000 NA 3.474795 1.000000e+00
## m_400 2453.559 28.47269 630.8961 43.47832 3.085900 1.006120e-137
## m_500 3061.557 34.92783 1238.8949 49.10423 3.189094 9.492727e-2707M4. What happens to the effective number of parameters, as measured by PSIS or WAIC, as a prior becomes more concentrated? Why? Perform some experiments, if you are not sure.
# As the prior becomes more concentrated, the effective number of parameters decreases. In the case of DIC, this is because pD is a measure of how flexible the model is. With more concentrated priors, the model becomes less flexible. In the case of WAIC, this is because pWAIC is a measure of the variance in the log-likelihood for each observation in the training sample. With more concentrated priors, the likelihood will become more concentrated as well and thus variance will decrease.
# To further confirm with experiment, we can calculate WAIC for two models of the same data that only differ in the concentration of their priors. The result showed that the pWAIC decreases as the priors become more concentrated.
d <- Howell1[complete.cases(Howell1), ]
d$height.log <- log(d$height)
d$height.log.z <- (d$height.log - mean(d$height.log)) / sd(d$height.log)
d$weight.log <- log(d$weight)
d$weight.log.z <- (d$weight.log - mean(d$weight.log)) / sd(d$weight.log)
m_wide <- map(
alist(
height.log.z ~ dnorm(mu, sigma),
mu <- a + b * weight.log.z,
a ~ dnorm(0, 10),
b ~ dnorm(1, 10),
sigma ~ dunif(0, 10)
),
data = d
)
m_narrow <- map(
alist(
height.log.z ~ dnorm(mu, sigma),
mu <- a + b * weight.log.z,
a ~ dnorm(0, 0.10),
b ~ dnorm(1, 0.10),
sigma ~ dunif(0, 1)
),
data = d
)
WAIC(m_wide, refresh = 0)## WAIC lppd penalty std_err
## 1 -103.0451 55.58893 4.066397 36.328657M5. Provide an informal explanation of why informative priors reduce overfitting.
# Informative priors reduce overfitting because they constrain the flexibility of the model. The priors make it less likely for extreme parameter values to be assigned high posterior probability. In simpler terms, informative priors reduce overfitting by forcing the model to learn less from the sample data.7M6. Provide an informal explanation of why overly informative priors result in underfitting.
# Overly informative priors result in underfitting because they constrain the flexibility of the model too much. The priors make it less likely for “correct” parameter values to be assigned high posterior probability. In simpler terms, overly informative priors result in underfitting by preventing the model from learning enough from the sample data.