The chapter began with the problem of overfitting, a universal phenomenon by which models with more parameters fit a sample better, even when the additional parameters are meaningless. Two common tools were introduced to address overfitting: regularizing priors and estimates of out-of-sample accuracy (WAIC and PSIS). Regularizing priors reduce overfitting during estimation, and WAIC and PSIS help estimate the degree of overfitting. Practical functions compare in the rethinking package were introduced to help analyze collections of models fit to the same data. If you are after causal estimates, then these tools will mislead you. So models must be designed through some other method, not selected on the basis of out-of-sample predictive accuracy. But any causal estimate will still overfit the sample. So you always have to worry about overfitting, measuring it with WAIC/PSIS and reducing it with regularization.
Place each answer inside the code chunk (grey box). The code chunks should contain a text response or a code that completes/answers the question or activity requested. Problems are labeled Easy (E), Medium (M), and Hard(H).
Finally, upon completion, name your final output .html file as: YourName_ANLY505-Year-Semester.html and publish the assignment to your R Pubs account and submit the link to Canvas. Each question is worth 5 points.
7E1. State the three motivating criteria that define information entropy. Try to express each in your own words.
# Three motivating criteria that define information entropy:
# 1.Uncertainty should be measured on a continuous scale
# 2.Uncertanity needs increase with number of possible events
# 3.additive for independent events.7E2. Suppose a coin is weighted such that, when it is tossed and lands on a table, it comes up heads 70% of the time. What is the entropy of this coin?
p1 <- 0.7
Hp <- c(p1, 1-p1)
H <- -sum(Hp*log(Hp))
H## [1] 0.61086437E3. Suppose a four-sided die is loaded such that, when tossed onto a table, it shows “1” 20%, “2” 25%, “3” 25%, and “4” 30% of the time. What is the entropy of this die?
Hp <- c(0.20, 0.25, 0.25, 0.30)
H <- -sum(Hp*log(Hp))
H## [1] 1.3762277E4. Suppose another four-sided die is loaded such that it never shows “4”. The other three sides show equally often. What is the entropy of this die?
Hp<-c(1/3, 1/3,1/3)
H <- -sum(Hp * log(Hp))
H## [1] 1.0986127M1. Write down and compare the definitions of AIC and WAIC. Which of these criteria is most general? Which assumptions are required to transform the more general criterion into a less general one?
# AIC: Akaike Information Criterion. It estimate if the model fits. AIC = Dtrain + 2K. K is parameter count and ~=E Dtest.
# WAIC: Widely Applicable IC. WAIC(y,theta) = -2(lppd-sum(Var(theta)*logp(Yi|theta)))
# WAIC is the most general.
# Assumptions required to transform the more general criterion into a less general one:
# 1. Poster distribution is approximately multivariate Gaussian.
# 2. Priors are flat by likelihood.7M2. Explain the difference between model selection and model comparison. What information is lost under model selection?
# Difference between model selection and model comparison:
# Model selection: to choose the model with lowest info criterion.
# Model comparison: to compare with multiple models.
# Information is lost under model selection:
# Relative model accuracy7M3. When comparing models with an information criterion, why must all models be fit to exactly the same observations? What would happen to the information criterion values, if the models were fit to different numbers of observations? Perform some experiments, if you are not sure.
# Information criterion is calculated by sum of Var. If the models were fit to different numbers of observation, the #information criterion will be different. When we compare models, and the models have different observations. The model that ##fits more observations will have less accuracy in info criterion, while the model that fits less observations will have #better information criterion. But in fact, the model should work the same, if the numbers of observations is the same.7M4. What happens to the effective number of parameters, as measured by PSIS or WAIC, as a prior becomes more concentrated? Why? Perform some experiments, if you are not sure.
# As a prior becomes more concentrated, the effective number of parameters will go down.
# PSIS or WAIC: the likelihood will become more concentrated, but the pWAIC will be less.7M5. Provide an informal explanation of why informative priors reduce overfitting.
# Informative priors reduce overfitting, because it limit the model. The model will learn less about the samples, and extreme parameters are not pointed to high posterior prob. So it will not overfit the data.7M6. Provide an informal explanation of why overly informative priors result in underfitting.
# Overly informative priors result in underfitting, because it force the model to fit parameters, and the model can not learn more from the samples themselves.