Open Intro Lab 2: The Normal Distribution
Anika Lewis
2023-07-23
install.packages("tidyverse")## Installing package into '/cloud/lib/x86_64-pc-linux-gnu-library/4.2'
## (as 'lib' is unspecified)install.packages("openintro")## Installing package into '/cloud/lib/x86_64-pc-linux-gnu-library/4.2'
## (as 'lib' is unspecified)Load packages
library(tidyverse)## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr 1.1.0 ✔ readr 2.1.4
## ✔ forcats 1.0.0 ✔ stringr 1.5.0
## ✔ ggplot2 3.4.2 ✔ tibble 3.1.8
## ✔ lubridate 1.9.2 ✔ tidyr 1.3.0
## ✔ purrr 1.0.1
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag() masks stats::lag()
## ℹ Use the ]8;;http://conflicted.r-lib.org/conflicted package]8;; to force all conflicts to become errorslibrary(openintro)## Loading required package: airports
## Loading required package: cherryblossom
## Loading required package: usdataThe data
Let’s take a quick peek at the first few rows of the data. Either you can use glimpse like before, or head to do this.
glimpse(fastfood)## Rows: 515
## Columns: 17
## $ restaurant <chr> "Mcdonalds", "Mcdonalds", "Mcdonalds", "Mcdonalds", "Mcdon…
## $ item <chr> "Artisan Grilled Chicken Sandwich", "Single Bacon Smokehou…
## $ calories <dbl> 380, 840, 1130, 750, 920, 540, 300, 510, 430, 770, 380, 62…
## $ cal_fat <dbl> 60, 410, 600, 280, 410, 250, 100, 210, 190, 400, 170, 300,…
## $ total_fat <dbl> 7, 45, 67, 31, 45, 28, 12, 24, 21, 45, 18, 34, 20, 34, 8, …
## $ sat_fat <dbl> 2.0, 17.0, 27.0, 10.0, 12.0, 10.0, 5.0, 4.0, 11.0, 21.0, 4…
## $ trans_fat <dbl> 0.0, 1.5, 3.0, 0.5, 0.5, 1.0, 0.5, 0.0, 1.0, 2.5, 0.0, 1.5…
## $ cholesterol <dbl> 95, 130, 220, 155, 120, 80, 40, 65, 85, 175, 40, 95, 125, …
## $ sodium <dbl> 1110, 1580, 1920, 1940, 1980, 950, 680, 1040, 1040, 1290, …
## $ total_carb <dbl> 44, 62, 63, 62, 81, 46, 33, 49, 35, 42, 38, 48, 48, 67, 31…
## $ fiber <dbl> 3, 2, 3, 2, 4, 3, 2, 3, 2, 3, 2, 3, 3, 5, 2, 2, 3, 3, 5, 2…
## $ sugar <dbl> 11, 18, 18, 18, 18, 9, 7, 6, 7, 10, 5, 11, 11, 11, 6, 3, 1…
## $ protein <dbl> 37, 46, 70, 55, 46, 25, 15, 25, 25, 51, 15, 32, 42, 33, 13…
## $ vit_a <dbl> 4, 6, 10, 6, 6, 10, 10, 0, 20, 20, 2, 10, 10, 10, 2, 4, 6,…
## $ vit_c <dbl> 20, 20, 20, 25, 20, 2, 2, 4, 4, 6, 0, 10, 20, 15, 2, 6, 15…
## $ calcium <dbl> 20, 20, 50, 20, 20, 15, 10, 2, 15, 20, 15, 35, 35, 35, 4, …
## $ salad <chr> "Other", "Other", "Other", "Other", "Other", "Other", "Oth…Exercise 1
Make a plot (or plots) to visualize the distributions of the amount of calories from fat of the options from these two restaurants. How do their centers, shapes, and spreads compare?
RESPONSE: The mcdonalds distribution appears to be skewed to the right because of its long right tail. Dairy Queen appears more normally distributed but still slightly right skewed
mcdonalds<-fastfood%>%
filter(restaurant=="Mcdonalds")
ggplot(mcdonalds,aes(cal_fat))+
geom_histogram()## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
dairyqueen<-fastfood%>%
filter(restaurant=="Dairy Queen")
ggplot(dairyqueen,aes(cal_fat))+
geom_histogram()## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
## The normal distribution
To see how accurate that description is, you can plot a normal distribution curve on top of a histogram to see how closely the data follow a normal distribution. This normal curve should have the same mean and standard deviation as the data. You’ll be focusing on calories from fat from Dairy Queen products, so let’s store them as a separate object and then calculate some statistics that will be referenced later.
dqmean<-mean(dairyqueen$cal_fat)
dqsd<-sd(dairyqueen$cal_fat)
mdmean<-mean(mcdonalds$cal_fat)
mdsd<-sd(mcdonalds$cal_fat)Next, you make a density histogram to use as the backdrop and use the lines function to overlay a normal probability curve.In a density histogram the areas of the bars add up to 1. The area of each bar can be calculated as simply the height times the width of the bar.Using a density histogram allows us to properly overlay a normal distribution curve over the histogram since the curve is a normal probability density function that also has area under the curve of 1. Frequency and density histograms both display the same exact shape; they only differ in their y-axis. You can verify this by comparing the frequency histogram you constructed earlier and the density histogram created by the commands below.
ggplot(dairyqueen,aes(cal_fat))+
geom_blank()+
geom_histogram(aes(y=..density..))+
stat_function(fun = dnorm, args = c(mean = dqmean, sd = dqsd), col = "tomato")## Warning: The dot-dot notation (`..density..`) was deprecated in ggplot2 3.4.0.
## ℹ Please use `after_stat(density)` instead.
## This warning is displayed once every 8 hours.
## Call `lifecycle::last_lifecycle_warnings()` to see where this warning was
## generated.## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
### Exercise 2 Based on the this plot, does it appear that the data
follow a nearly normal distribution?
RESPONSE: The data appears to follow a near normal distribution. The mean and sd are not exactly the same but they are close. There is a right skew
Evaluating the normal distribution
An alternative approach involves constructing a normal probability plot, also called a normal Q-Q plot for “quantile-quantile”.It’s important to note that here, instead of using x instead aes(), you need to use sample.
Notes: perfect normal distribution will be a straight line, deviations from the straight line signals deviations from the perfect normal distribution.
ggplot(dairyqueen,aes(sample=cal_fat))+
geom_line(stat="qq")
The x-axis values correspond to the quantiles of a theoretically normal
curve with mean 0 and standard deviation 1 (i.e., the standard normal
distribution). The y-axis values correspond to the quantiles of the
original unstandardized sample data. However, even if we were to
standardize the sample data values, the Q-Q plot would look identical. A
data set that is nearly normal will result in a probability plot where
the points closely follow a diagonal line. Any deviations from normality
leads to deviations of these points from that line. What do probability
plots look like for data that I know came from a normal distribution? We
can answer this by simulating data from a normal distribution using
rnorm.
sim_normdq<-rnorm(n=nrow(dairyqueen),mean=dqmean,sd=dqsd)The first argument indicates how many numbers you’d like to generate, which we specify to be the same number of menu items in the dairy_queen data set using the nrow() function. The last two arguments determine the mean and standard deviation of the normal distribution from which the simulated sample will be generated. You can take a look at the shape of our simulated data set, sim_norm, as well as its normal probability plot.
Exercise 3
Make a normal probability plot of sim_norm. Do all of the points fall on the line? How does this plot compare to the probability plot for the real data? (Since sim_norm is not a dataframe, it can be put directly into the sample argument and the data argument can be dropped.)
RESPONSE: Not all of the points fall on a diagonal line, so the qq plot is not completely normal.The upper tail in particular shows a departure from the line. It looks almost identical to the plot of the real data above.
ggplot(,aes(sample = sim_normdq)) +
geom_line(stat = "qq")
Even better than comparing the original plot to a single plot generated
from a normal distribution is to compare it to many more plots using the
following function. It shows the Q-Q plot corresponding to the original
data in the top left corner, and the Q-Q plots of 8 different simulated
normal data.
qqnormsim(sample=cal_fat,data=dairyqueen)
### Exercise 4 Does the normal probability plot for the calories from
fat look similar to the plots created for the simulated data? That is,
do the plots provide evidence that the calories from fat are nearly
normal?
RESPONSE: yes, the normal probability plot looks similar to the simulated data plots. They are all linear with an overall positive trend. As X increases, y steadily increases.They are not all perfectly straight diagonal lines and show variations in more or less the same areas (very bottom and very top)
Exercise 5
Using the same technique, determine whether or not the calories from McDonald’s menu appear to come from a normal distribution.
RESPONSE:Because the plots follow a mostly diagonal line, I agree that the calories come from a normal distribution. They vary slightly more than the Dairy Queen Cal_fat qqplots from above.Which may indicate less of a normal shape
sim_normmd<-rnorm(n=nrow(mcdonalds), mean = mdmean, sd= mdsd)
ggplot(,aes(sample=sim_normmd)) + geom_line(stat="qq")
qqnormsim(sample=cal_fat,data=mcdonalds)
## Normal Probabilities “What is the probability that a randomly chosen
Dairy Queen product has more than 600 calories from fat?” the function
pnorm() gives the area under the normal curve below a given value, q,
with a given mean and standard deviation.
1 - pnorm(q = 600, mean = dqmean, sd = dqsd)## [1] 0.01501523Exercise 6
Write out two probability questions that you would like to answer about any of the restaurants in this dataset. Calculate those probabilities using both the theoretical normal distribution as well as the empirical distribution (four probabilities in all). Which one had a closer agreement between the two methods?
RESPONSE: What is the probability that a randomly chosen product from Taco Bell has lower than 500 calories?
tacobell<-fastfood%>%
filter(restaurant=="Taco Bell")
tbmean<-mean(tacobell$calories)
tbsd<-sd(tacobell$calories)Theoretical
pnorm(q=500, mean=tbmean, sd=tbsd)## [1] 0.6200702Emperical
tacobell%>%
filter(calories<500)%>%
summarise(percent=n()/nrow(tacobell))## # A tibble: 1 × 1
## percent
## <dbl>
## 1 0.626RESPONSE: What is the probability that a randomly chosen product from Taco Bell has higher than 1000 calories?
Theoretical
1-pnorm(q=1000, mean=tbmean,sd=tbsd)## [1] 0.001272359Emperical
tacobell%>%
filter(calories>1000)%>%
summarise(percent=n()/nrow(tacobell))## # A tibble: 1 × 1
## percent
## <dbl>
## 1 0The second question had the closer agreement between the theoretical and emperical calculations
Exericse 7
Now let’s consider some of the other variables in the dataset. Out of all the different restaurants, which ones’ distribution is the closest to normal for sodium?
RESPONSE: Burger King
facet_wrap (~restaurant)
ggplot(data=fastfood,aes(sample=sodium))+
geom_line(stat="qq")+
facet_wrap(~restaurant)
Individual restaurants
arbys<-fastfood%>%
filter(restaurant=="Arbys")subway<-fastfood%>%
filter(restaurant=="Subway")chick<-fastfood%>%
filter(restaurant=="Chick Fil-A")burgerking<-fastfood%>%
filter(restaurant=="Burger King")Tacobell
ggplot(data = tacobell, aes(sample = sodium)) +
geom_line(stat = "qq")
Mcdonald’s
ggplot(data = mcdonalds, aes(sample = sodium)) +
geom_line(stat = "qq")
Burgerking
ggplot(data = burgerking, aes(sample = sodium)) +
geom_line(stat = "qq")
Chick Fil-A
ggplot(data = chick, aes(sample = sodium)) +
geom_line(stat = "qq")
Subway
ggplot(data = subway, aes(sample = sodium)) +
geom_line(stat = "qq")
Arbys
ggplot(data = arbys, aes(sample = sodium)) +
geom_line(stat = "qq")
Dairy Queen
ggplot(data = dairyqueen, aes(sample = sodium)) +
geom_line(stat = "qq")
### Exercise 8 Note that some of the normal probability plots for sodium
distributions seem to have a stepwise pattern. why do you think this
might be the case?
RESPONSE: After conducting research, I found the general consensus to be that a step-wise pattern is a result of the presence of a discrete variable.Sodium in this case is a discrete variable because they are countable, rounded and finate. For example glimpse(fastfood$sodium) returns data such as: 1110, 1580 and 1920.
Cancer Death vs Poverty by County in US
cancer_reg <- readr::read_csv("https://raw.githubusercontent.com/Arnab777as3uj/STAT6021-Cancer-Prediction-Project/master/cancer_reg.csv")## Rows: 3047 Columns: 34
## ── Column specification ────────────────────────────────────────────────────────
## Delimiter: ","
## chr (2): binnedInc, Geography
## dbl (32): avgAnnCount, avgDeathsPerYear, TARGET_deathRate, incidenceRate, me...
##
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.summary(cancer_reg)## avgAnnCount avgDeathsPerYear TARGET_deathRate incidenceRate
## Min. : 6.0 Min. : 3 Min. : 59.7 Min. : 201.3
## 1st Qu.: 76.0 1st Qu.: 28 1st Qu.:161.2 1st Qu.: 420.3
## Median : 171.0 Median : 61 Median :178.1 Median : 453.5
## Mean : 606.3 Mean : 186 Mean :178.7 Mean : 448.3
## 3rd Qu.: 518.0 3rd Qu.: 149 3rd Qu.:195.2 3rd Qu.: 480.9
## Max. :38150.0 Max. :14010 Max. :362.8 Max. :1206.9
##
## medIncome popEst2015 povertyPercent studyPerCap
## Min. : 22640 Min. : 827 Min. : 3.20 Min. : 0.00
## 1st Qu.: 38882 1st Qu.: 11684 1st Qu.:12.15 1st Qu.: 0.00
## Median : 45207 Median : 26643 Median :15.90 Median : 0.00
## Mean : 47063 Mean : 102637 Mean :16.88 Mean : 155.40
## 3rd Qu.: 52492 3rd Qu.: 68671 3rd Qu.:20.40 3rd Qu.: 83.65
## Max. :125635 Max. :10170292 Max. :47.40 Max. :9762.31
##
## binnedInc MedianAge MedianAgeMale MedianAgeFemale
## Length:3047 Min. : 22.30 Min. :22.40 Min. :22.30
## Class :character 1st Qu.: 37.70 1st Qu.:36.35 1st Qu.:39.10
## Mode :character Median : 41.00 Median :39.60 Median :42.40
## Mean : 45.27 Mean :39.57 Mean :42.15
## 3rd Qu.: 44.00 3rd Qu.:42.50 3rd Qu.:45.30
## Max. :624.00 Max. :64.70 Max. :65.70
##
## Geography AvgHouseholdSize PercentMarried PctNoHS18_24
## Length:3047 Min. :0.0221 Min. :23.10 Min. : 0.00
## Class :character 1st Qu.:2.3700 1st Qu.:47.75 1st Qu.:12.80
## Mode :character Median :2.5000 Median :52.40 Median :17.10
## Mean :2.4797 Mean :51.77 Mean :18.22
## 3rd Qu.:2.6300 3rd Qu.:56.40 3rd Qu.:22.70
## Max. :3.9700 Max. :72.50 Max. :64.10
##
## PctHS18_24 PctSomeCol18_24 PctBachDeg18_24 PctHS25_Over
## Min. : 0.0 Min. : 7.10 Min. : 0.000 Min. : 7.50
## 1st Qu.:29.2 1st Qu.:34.00 1st Qu.: 3.100 1st Qu.:30.40
## Median :34.7 Median :40.40 Median : 5.400 Median :35.30
## Mean :35.0 Mean :40.98 Mean : 6.158 Mean :34.80
## 3rd Qu.:40.7 3rd Qu.:46.40 3rd Qu.: 8.200 3rd Qu.:39.65
## Max. :72.5 Max. :79.00 Max. :51.800 Max. :54.80
## NA's :2285
## PctBachDeg25_Over PctEmployed16_Over PctUnemployed16_Over PctPrivateCoverage
## Min. : 2.50 Min. :17.60 Min. : 0.400 Min. :22.30
## 1st Qu.: 9.40 1st Qu.:48.60 1st Qu.: 5.500 1st Qu.:57.20
## Median :12.30 Median :54.50 Median : 7.600 Median :65.10
## Mean :13.28 Mean :54.15 Mean : 7.852 Mean :64.35
## 3rd Qu.:16.10 3rd Qu.:60.30 3rd Qu.: 9.700 3rd Qu.:72.10
## Max. :42.20 Max. :80.10 Max. :29.400 Max. :92.30
## NA's :152
## PctPrivateCoverageAlone PctEmpPrivCoverage PctPublicCoverage
## Min. :15.70 Min. :13.5 Min. :11.20
## 1st Qu.:41.00 1st Qu.:34.5 1st Qu.:30.90
## Median :48.70 Median :41.1 Median :36.30
## Mean :48.45 Mean :41.2 Mean :36.25
## 3rd Qu.:55.60 3rd Qu.:47.7 3rd Qu.:41.55
## Max. :78.90 Max. :70.7 Max. :65.10
## NA's :609
## PctPublicCoverageAlone PctWhite PctBlack PctAsian
## Min. : 2.60 Min. : 10.20 Min. : 0.0000 Min. : 0.0000
## 1st Qu.:14.85 1st Qu.: 77.30 1st Qu.: 0.6207 1st Qu.: 0.2542
## Median :18.80 Median : 90.06 Median : 2.2476 Median : 0.5498
## Mean :19.24 Mean : 83.65 Mean : 9.1080 Mean : 1.2540
## 3rd Qu.:23.10 3rd Qu.: 95.45 3rd Qu.:10.5097 3rd Qu.: 1.2210
## Max. :46.60 Max. :100.00 Max. :85.9478 Max. :42.6194
##
## PctOtherRace PctMarriedHouseholds BirthRate
## Min. : 0.0000 Min. :22.99 Min. : 0.000
## 1st Qu.: 0.2952 1st Qu.:47.76 1st Qu.: 4.521
## Median : 0.8262 Median :51.67 Median : 5.381
## Mean : 1.9835 Mean :51.24 Mean : 5.640
## 3rd Qu.: 2.1780 3rd Qu.:55.40 3rd Qu.: 6.494
## Max. :41.9303 Max. :78.08 Max. :21.326
## cancer_reg<-dplyr::rename(cancer_reg,"cancer_death_rate"="TARGET_deathRate")ggplot(cancer_reg, aes(x=povertyPercent, y=cancer_death_rate))+
geom_point(alpha=0.25)+
xlab("Poverty Percentage")+
ylab("Cancer Death Rate")+
ggtitle("Cancer Death vs Poverty by County in US")