The chapter began with the problem of overfitting, a universal phenomenon by which models with more parameters fit a sample better, even when the additional parameters are meaningless. Two common tools were introduced to address overfitting: regularizing priors and estimates of out-of-sample accuracy (WAIC and PSIS). Regularizing priors reduce overfitting during estimation, and WAIC and PSIS help estimate the degree of overfitting. Practical functions compare in the rethinking package were introduced to help analyze collections of models fit to the same data. If you are after causal estimates, then these tools will mislead you. So models must be designed through some other method, not selected on the basis of out-of-sample predictive accuracy. But any causal estimate will still overfit the sample. So you always have to worry about overfitting, measuring it with WAIC/PSIS and reducing it with regularization.
Place each answer inside the code chunk (grey box). The code chunks should contain a text response or a code that completes/answers the question or activity requested. Problems are labeled Easy (E), Medium (M), and Hard(H).
Finally, upon completion, name your final output .html file as: YourName_ANLY505-Year-Semester.html and publish the assignment to your R Pubs account and submit the link to Canvas. Each question is worth 5 points.
7E1. State the three motivating criteria that define information entropy. Try to express each in your own words.
# continuous. when we measure the uncertainty, we need to use continuous scale
# Increasing possible events' quantity. Uncertainty will increase when the number of outcome increase
# Additive. Events need to be additive7E2. Suppose a coin is weighted such that, when it is tossed and lands on a table, it comes up heads 70% of the time. What is the entropy of this coin?
p <- c(0.7,0.3)
(H <- -sum(p*log(p)))## [1] 0.6108643# the entropy of this coin is 0.61086437E3. Suppose a four-sided die is loaded such that, when tossed onto a table, it shows “1” 20%, “2” 25%, “3” 25%, and “4” 30% of the time. What is the entropy of this die?
p <- ( c(0.2,0.25,0.25,0.3))
(H <- -sum(p*log(p)))## [1] 1.376227# the entropy of this die is 1.3762277E4. Suppose another four-sided die is loaded such that it never shows “4”. The other three sides show equally often. What is the entropy of this die?
p <- c(1/3,1/3,1/3)
(H <- -sum(p*log(p)))## [1] 1.098612# the entropy is 1.0986137M1. Write down and compare the definitions of AIC and WAIC. Which of these criteria is most general? Which assumptions are required to transform the more general criterion into a less general one?
# AIC: Akaike Information Criterion. AIC = Dtrain +2k(k is parameter count and D is in sample training deviance)
# WAIC: Widely Applicable Information Criterion. WAIC = -2(lppd - penalty_termWAIC), it's popular than AIC
# If transform WAIC to AIC, we'll assume that posterior distribution is Gaussian7M2. Explain the difference between model selection and model comparison. What information is lost under model selection?
# model selection: model with lowest information criterion value and to discard all other models that have higher values. It will lose the information relative model accuracy
# Model comparison: use multiple models in pridiction and explain. It doesn't lose any information7M3. When comparing models with an information criterion, why must all models be fit to exactly the same observations? What would happen to the information criterion values, if the models were fit to different numbers of observations? Perform some experiments, if you are not sure.
# if the model does not fit to exactly the same observations, there will be higher deviance, which will cause less accuracy. Information criterion will decrease with more different numbers of observations.7M4. What happens to the effective number of parameters, as measured by PSIS or WAIC, as a prior becomes more concentrated? Why? Perform some experiments, if you are not sure.
# As for WAIC, the more concentrated priors, the log-likelihood will be more concentrated and the variance will decrease.
# the PSIS will not be affected since it use single data point7M5. Provide an informal explanation of why informative priors reduce overfitting.
# Information prior help to reduce overfitting since it constrain the flexibility of the model 7M6. Provide an informal explanation of why overly informative priors result in underfitting.
# the overly informative prior will limit the model to a large extend and will affect how sample data work.