The chapter began with the problem of overfitting, a universal phenomenon by which models with more parameters fit a sample better, even when the additional parameters are meaningless. Two common tools were introduced to address overfitting: regularizing priors and estimates of out-of-sample accuracy (WAIC and PSIS). Regularizing priors reduce overfitting during estimation, and WAIC and PSIS help estimate the degree of overfitting. Practical functions compare in the rethinking package were introduced to help analyze collections of models fit to the same data. If you are after causal estimates, then these tools will mislead you. So models must be designed through some other method, not selected on the basis of out-of-sample predictive accuracy. But any causal estimate will still overfit the sample. So you always have to worry about overfitting, measuring it with WAIC/PSIS and reducing it with regularization.
Place each answer inside the code chunk (grey box). The code chunks should contain a text response or a code that completes/answers the question or activity requested. Problems are labeled Easy (E), Medium (M), and Hard(H).
Finally, upon completion, name your final output .html file as: YourName_ANLY505-Year-Semester.html and publish the assignment to your R Pubs account and submit the link to Canvas. Each question is worth 5 points.
7E1. State the three motivating criteria that define information entropy. Try to express each in your own words.
#1) the scale being measured should be continous;
#2) capture the possible outcomes's probabilty
#3) should be addictive to independent events7E2. Suppose a coin is weighted such that, when it is tossed and lands on a table, it comes up heads 70% of the time. What is the entropy of this coin?
p=0.7
q=1-p
prob=c(p,q)
entropy=-sum(prob*log(prob))
entropy## [1] 0.6108643#entropy = 0.61086437E3. Suppose a four-sided die is loaded such that, when tossed onto a table, it shows “1” 20%, “2” 25%, “3” 25%, and “4” 30% of the time. What is the entropy of this die?
p=c(0.2,0.25,0.25,0.3)
entropy=-sum(p*log(p))
entropy## [1] 1.376227#entropy = 1.3762277E4. Suppose another four-sided die is loaded such that it never shows “4”. The other three sides show equally often. What is the entropy of this die?
p=c(1/3,1/3,1/3)
entropy=-sum(p*log(p))
entropy## [1] 1.098612#entropy = 1.0986127M1. Write down and compare the definitions of AIC and WAIC. Which of these criteria is most general? Which assumptions are required to transform the more general criterion into a less general one?
# AIC is defined as D_train + 2k where k is the parameter complexity. It balances off the
# model goodness of fit, with the complexity of model captured by the number of parameters.
# WAIC is defined as -2*(lppd - ∑i varθlogp(y_i | θ)), which does not assume Gaussian
# posterior and estimated out of sample log-score.
#WAIC is most general. To transform from WAIC to AIC, assumptions that the posterior distribution is approximately multivariate Gaussian and the priors are flat or overwhelmed by the likelihood should be transformed.7M2. Explain the difference between model selection and model comparison. What information is lost under model selection?
# model selection is using information criterias to select a best model and overcome model overfit.
# model comparison is using different models to understand how different variables will impact the model predictions.
# model selection loses information about the relative model accuracy.7M3. When comparing models with an information criterion, why must all models be fit to exactly the same observations? What would happen to the information criterion values, if the models were fit to different numbers of observations? Perform some experiments, if you are not sure.
#Information criteria based on deviance, which is more accrued than observations without being divided by the number of observations. It is not fair to compare models with different number of observations.7M4. What happens to the effective number of parameters, as measured by PSIS or WAIC, as a prior becomes more concentrated? Why? Perform some experiments, if you are not sure.
# the effective number of parameters will decrease as a prior become more concentrated.
# Becase pWAIC is performing as a penalty term which measures the variance in the log-likelihood for each observation, the variance will decrease when the prior becomes more concentrated and log-likelihood become more concentrated. 7M5. Provide an informal explanation of why informative priors reduce overfitting.
# the informative priors will reduce overfitting because they constrain the flexibility of the model and constrain the likelihood of having extreme value for the parameter. So that the model will learn less from a specific dataset and reduce overfitting.7M6. Provide an informal explanation of why overly informative priors result in underfitting.
# informative priors will restrict the model from learning less from the sample data, it # could result in underfitting if the informative priors is overly, as the model might learn few information from the data and thus has a very poor prediction.