The chapter began with the problem of overfitting, a universal phenomenon by which models with more parameters fit a sample better, even when the additional parameters are meaningless. Two common tools were introduced to address overfitting: regularizing priors and estimates of out-of-sample accuracy (WAIC and PSIS). Regularizing priors reduce overfitting during estimation, and WAIC and PSIS help estimate the degree of overfitting. Practical functions compare in the rethinking package were introduced to help analyze collections of models fit to the same data. If you are after causal estimates, then these tools will mislead you. So models must be designed through some other method, not selected on the basis of out-of-sample predictive accuracy. But any causal estimate will still overfit the sample. So you always have to worry about overfitting, measuring it with WAIC/PSIS and reducing it with regularization.
Place each answer inside the code chunk (grey box). The code chunks should contain a text response or a code that completes/answers the question or activity requested. Make sure to include plots if the question requests them. Problems are labeled Easy (E), Medium (M), and Hard(H).
Finally, upon completion, name your final output .html file as: YourName_ANLY505-Year-Semester.html and publish the assignment to your R Pubs account and submit the link to Canvas. Each question is worth 5 points.
7E1. State the three motivating criteria that define information entropy. Try to express each in your own words.
# Criteria (1): It is measured using a scale that is continuous, where when there are adjacent values the spacing between them should be uniform.
# Criteria (2): uncertainty is positively correlated to the amount of possibilities. When the numbers of possibilities grows, the uncertainty grows as well.
# Criteria (3): The way events are divided does not influence the additivity of independent events.7E2. Suppose a coin is weighted such that, when it is tossed and lands on a table, it comes up heads 70% of the time. What is the entropy of this coin?
p1 <- c(.7, .3)
e1 <- -sum(p1*log(p1))
e1## [1] 0.61086437E3. Suppose a four-sided die is loaded such that, when tossed onto a table, it shows “1” 20%, “2” 25%, “3” 25%, and “4” 30% of the time. What is the entropy of this die?
p2 <- c(.2, .25, .25,.3)
e2 <- -sum(p2*log(p2))
e2## [1] 1.3762277E4. Suppose another four-sided die is loaded such that it never shows “4”. The other three sides show equally often. What is the entropy of this die?
p3 <- c(1/3, 1/3, 1/3)
e3 <- -sum(p3*log(p3))
e3## [1] 1.0986127M1. Write down and compare the definitions of AIC and WAIC. Which of these criteria is most general? Which assumptions are required to transform the more general criterion into a less general one?
# AIC is the sum of the deviance of the in-sample training data (D_train) and twice the number of free parameters (2P), which makes the formula: AIC = D_train + 2p
# WAIC is defined to be the difference between the sum of average likelihood of observations and the penalty term, which makes the formula: WAIC = −2(lppd−pWAIC)
# WAIC being the most general, assuming the posterior distribution is Gaussian and the priors are flat is required to transform it to a less general one.7M2. Explain the difference between model selection and model comparison. What information is lost under model selection?
# As its name explains, model selection consists in adopting the model with the lowest information criterion value and letting go of the rest, meanwhile model comparison tests various models to see which ones is the most appropriate for predictions.
# When model selecting, getting rid of the unchosen models results in losing the information describing the relative model accuracy included in the differences among information criterion values included in the differences among information criterion values7M3. When comparing models with an information criterion, why must all models be fit to exactly the same observations? What would happen to the information criterion values, if the models were fit to different numbers of observations? Perform some experiments, if you are not sure.
# All models with information criterion must be fit to exactly the same observations to make sure they have have a proper deviance and produce an accurate view of their predictions, because the information criterion itself is built on deviance. If the number of observation fluctuates, it may result in altering the deviance and putting models in significant advantage (or disadvantage) against other which will bias the process of finding the appropriate one.7M4. What happens to the effective number of parameters, as measured by PSIS or WAIC, as a prior becomes more concentrated? Why? Perform some experiments, if you are not sure.
# When a prior becomes more concentrated, we automatically observe a decrease in the effective number of parameters, which will lead to the likelihood in the formula also becoming concentrated. As a result the variance will decreases.7M5. Provide an informal explanation of why informative priors reduce overfitting.
# By using informative priors we impose limitation to the interval where the parameters can vary, which will logically not let the model use extreme parameters and limit the overfitting of the data.7M6. Provide an informal explanation of why overly informative priors result in underfitting.
# with the same logic of the answer to the question above, by using overly informative priors we impose even stricter limitations to the interval where the parameters can vary and not being assigned high posterior probabilities, which will result in underfitting.