The chapter began with the problem of overfitting, a universal phenomenon by which models with more parameters fit a sample better, even when the additional parameters are meaningless. Two common tools were introduced to address overfitting: regularizing priors and estimates of out-of-sample accuracy (WAIC and PSIS). Regularizing priors reduce overfitting during estimation, and WAIC and PSIS help estimate the degree of overfitting. Practical functions compare in the rethinking package were introduced to help analyze collections of models fit to the same data. If you are after causal estimates, then these tools will mislead you. So models must be designed through some other method, not selected on the basis of out-of-sample predictive accuracy. But any causal estimate will still overfit the sample. So you always have to worry about overfitting, measuring it with WAIC/PSIS and reducing it with regularization.
Place each answer inside the code chunk (grey box). The code chunks should contain a text response or a code that completes/answers the question or activity requested. Problems are labeled Easy (E), Medium (M), and Hard(H).
Finally, upon completion, name your final output .html file as: YourName_ANLY505-Year-Semester.html and publish the assignment to your R Pubs account and submit the link to Canvas. Each question is worth 5 points.
7E1. State the three motivating criteria that define information entropy. Try to express each in your own words.
#Information Entropy is defined by the probability of an event given information of certain outcomes, the probability is also spaced out by the number of possible outcomes and can be added by independent events. 7E2. Suppose a coin is weighted such that, when it is tossed and lands on a table, it comes up heads 70% of the time. What is the entropy of this coin?
library(rethinking)## Loading required package: rstan## Loading required package: StanHeaders## Loading required package: ggplot2## rstan (Version 2.21.2, GitRev: 2e1f913d3ca3)## For execution on a local, multicore CPU with excess RAM we recommend calling
## options(mc.cores = parallel::detectCores()).
## To avoid recompilation of unchanged Stan programs, we recommend calling
## rstan_options(auto_write = TRUE)## Do not specify '-march=native' in 'LOCAL_CPPFLAGS' or a Makevars file## Loading required package: parallel## rethinking (Version 2.12)##
## Attaching package: 'rethinking'## The following object is masked from 'package:stats':
##
## rstudentp <- c(0.7, 1 - 0.7)
(H <- -sum(p * log(p)))## [1] 0.61086437E3. Suppose a four-sided die is loaded such that, when tossed onto a table, it shows “1” 20%, “2” 25%, “3” 25%, and “4” 30% of the time. What is the entropy of this die?
p <- c(0.20, 0.25, 0.25, 0.30)
(H <- -sum(p * log(p)))## [1] 1.3762277E4. Suppose another four-sided die is loaded such that it never shows “4”. The other three sides show equally often. What is the entropy of this die?
p <- c(1/3, 1/3, 1/3)
(H <- -sum(p * log(p)))## [1] 1.0986127M1. Write down and compare the definitions of AIC and WAIC. Which of these criteria is most general? Which assumptions are required to transform the more general criterion into a less general one?
#AIC is defined as the in-sample training deviance plus two times p where p is the number of free parameters.
#WAIC is defined as the sum of log of Pr(yi) minus the sum of the variance of the log-likelihood for observation i in the training sample all multipled by -2. WAIC is more general than AIC. To move from a more general criterion, we need to assume the posterior distribution is close to multivariate Gaussian and the priors are either flat or overwhelmed by the likelihood.7M2. Explain the difference between model selection and model comparison. What information is lost under model selection?
#The difference between model selection and model comparison is that model loses information about relative model accuracy contained in the differences among criterion values while model comparsion does not. 7M3. When comparing models with an information criterion, why must all models be fit to exactly the same observations? What would happen to the information criterion values, if the models were fit to different numbers of observations? Perform some experiments, if you are not sure.
#All models must be fit to exactly the same observation because information criteria is based on deviance which is accrued over the observations. It is a sum rather than an average. A model with more observations would usually mean a higher deviance and lower accuracy. 7M4. What happens to the effective number of parameters, as measured by PSIS or WAIC, as a prior becomes more concentrated? Why? Perform some experiments, if you are not sure.
#As prior becomes more concentrated the number of parameters decreases and the model becomes less flexible. 7M5. Provide an informal explanation of why informative priors reduce overfitting.
#Informative priors reduce overfitting by constraining the flexibility of the model, making it less likely for extreme parameter values to be assigned high posterior probability. 7M6. Provide an informal explanation of why overly informative priors result in underfitting.
#Overly informative priors result in underfitting by over constrainining the flexibility of the model which makes it less likely for the valid parameter values to be assigned high posterior probability