The chapter began with the problem of overfitting, a universal phenomenon by which models with more parameters fit a sample better, even when the additional parameters are meaningless. Two common tools were introduced to address overfitting: regularizing priors and estimates of out-of-sample accuracy (WAIC and PSIS). Regularizing priors reduce overfitting during estimation, and WAIC and PSIS help estimate the degree of overfitting. Practical functions compare in the rethinking package were introduced to help analyze collections of models fit to the same data. If you are after causal estimates, then these tools will mislead you. So models must be designed through some other method, not selected on the basis of out-of-sample predictive accuracy. But any causal estimate will still overfit the sample. So you always have to worry about overfitting, measuring it with WAIC/PSIS and reducing it with regularization.
Place each answer inside the code chunk (grey box). The code chunks should contain a text response or a code that completes/answers the question or activity requested. Make sure to include plots if the question requests them. Problems are labeled Easy (E), Medium (M), and Hard(H).
Finally, upon completion, name your final output .html file as: YourName_ANLY505-Year-Semester.html and publish the assignment to your R Pubs account and submit the link to Canvas. Each question is worth 5 points.
7E1. State the three motivating criteria that define information entropy. Try to express each in your own words.
# 1 it should measure with a continuous scale which the spacing between adjacent values is consistent
# 2 it should capture the size of the possibility space is the value scales with the number of possible outcomes
# 3 it should be flexible to add independent events such that it does not matter how the events are divided7E2. Suppose a coin is weighted such that, when it is tossed and lands on a table, it comes up heads 70% of the time. What is the entropy of this coin?
p <- c(0.7, 0.3)
entropy <- -sum(p * log(p))
entropy## [1] 0.61086437E3. Suppose a four-sided die is loaded such that, when tossed onto a table, it shows “1” 20%, “2” 25%, “3” 25%, and “4” 30% of the time. What is the entropy of this die?
p <- c(0.2, 0.25, 0.25, 0.3)
entropy <- -sum(p * log(p))
entropy## [1] 1.3762277E4. Suppose another four-sided die is loaded such that it never shows “4”. The other three sides show equally often. What is the entropy of this die?
p <- c(1/3, 1/3, 1/3)
entropy <- -sum(p * log(p))
entropy## [1] 1.0986127M1. Write down and compare the definitions of AIC and WAIC. Which of these criteria is most general? Which assumptions are required to transform the more general criterion into a less general one?
# AIC = D_train + 2p, Dtrain = in-sample training deviance and p = number of
# free parameters estimated in the model. WAIC is defined as −2(lppd−pWAIC)
# where Pr(yi) = average likelihood of observation i in the training sample
# and V(yi) = variance in log-likelihood for observation i in the training sample.7M2. Explain the difference between model selection and model comparison. What information is lost under model selection?
# In model selection, we pick the model with best information criteria value.
# In model averaging, the DIC or WAIC are used to make a posterior predictive
# distribution that combines all models. 7M3. When comparing models with an information criterion, why must all models be fit to exactly the same observations? What would happen to the information criterion values, if the models were fit to different numbers of observations? Perform some experiments, if you are not sure.
# If there were different numbers of observations when comparing models,
# it will result in different Information Criterion, thus the relative model
# accuracy will also be lost in model comparison.
library(rethinking)
data(Howell1)
str(Howell1)## 'data.frame': 544 obs. of 4 variables:
## $ height: num 152 140 137 157 145 ...
## $ weight: num 47.8 36.5 31.9 53 41.3 ...
## $ age : num 63 63 65 41 51 35 32 27 19 54 ...
## $ male : int 1 0 0 1 0 1 0 1 0 1 ...d <- Howell1[complete.cases(Howell1), ]
d_500 <- d[sample(1:nrow(d), size = 500, replace = FALSE), ]
d_400 <- d[sample(1:nrow(d), size = 400, replace = FALSE), ]
d_300 <- d[sample(1:nrow(d), size = 300, replace = FALSE), ]
m_500 <- map(
alist(
height ~ dnorm(mu, sigma),
mu <- a + b * log(weight)
),
data = d_500,
start = list(a = mean(d_500$height), b = 0, sigma = sd(d_500$height))
)
m_400 <- map(
alist(
height ~ dnorm(mu, sigma),
mu <- a + b * log(weight)
),
data = d_400,
start = list(a = mean(d_400$height), b = 0, sigma = sd(d_400$height))
)
m_300 <- map(
alist(
height ~ dnorm(mu, sigma),
mu <- a + b * log(weight)
),
data = d_300,
start = list(a = mean(d_300$height), b = 0, sigma = sd(d_300$height))
)
(model.compare <- compare(m_500, m_400, m_300))## Warning in compare(m_500, m_400, m_300): Different numbers of observations found for at least two models.
## Model comparison is valid only for models fit to exactly the same observations.
## Number of observations for each model:
## m_500 500
## m_400 400
## m_300 300## Warning in ic_ptw1 - ic_ptw2: longer object length is not a multiple of shorter
## object length
## Warning in ic_ptw1 - ic_ptw2: longer object length is not a multiple of shorter
## object length
## Warning in ic_ptw1 - ic_ptw2: longer object length is not a multiple of shorter
## object length## WAIC SE dWAIC dSE pWAIC weight
## m_300 1852.926 29.64695 0.0000 NA 3.349455 1.000000e+00
## m_400 2452.219 29.07466 599.2931 42.18261 3.059352 7.330995e-131
## m_500 3063.875 35.59586 1210.9489 50.66083 3.283663 1.111182e-2637M4. What happens to the effective number of parameters, as measured by PSIS or WAIC, as a prior becomes more concentrated? Why? Perform some experiments, if you are not sure.
# Informative priors are are more concentrated, which will lead to a decrease of
# the effective number of parameters. This proportionality can be observed from
# the WAIC mathematical equation:
# WAIC(y, Θ) = -2(lppd - varΘ logp(yi|Θ))
d <- Howell1[complete.cases(Howell1), ]
d$height.log <- log(d$height)
d$height.log.z <- (d$height.log - mean(d$height.log)) / sd(d$height.log)
d$weight.log <- log(d$weight)
d$weight.log.z <- (d$weight.log - mean(d$weight.log)) / sd(d$weight.log)
m_wide <- map(
alist(
height.log.z ~ dnorm(mu, sigma),
mu <- a + b * weight.log.z,
a ~ dnorm(0, 10),
b ~ dnorm(1, 10),
sigma ~ dunif(0, 10)
),
data = d
)
m_narrow <- map(
alist(
height.log.z ~ dnorm(mu, sigma),
mu <- a + b * weight.log.z,
a ~ dnorm(0, 0.10),
b ~ dnorm(1, 0.10),
sigma ~ dunif(0, 1)
),
data = d
)
WAIC(m_wide, refresh = 0)## WAIC lppd penalty std_err
## 1 -103.0002 55.56807 4.067975 36.416397M5. Provide an informal explanation of why informative priors reduce overfitting.
# Informative priors are very conservative, they learn less from the sample,
# so when such a model is applied, since the number of parameters these priors
# look for are limited there is no chance of an over-fitting scenario.7M6. Provide an informal explanation of why overly informative priors result in underfitting.
# Informative priors are conservative which will avoid over-fitting.
# However, overly informative priors will further less the over-fitting,
# which lead to under-fitting.