In this lab, we will explore and visualize the data using the tidyverse suite of packages, and perform statistical inference using infer. The data can be found in the companion package for OpenIntro resources, openintro.
Let’s load the packages.
library(tidyverse)
library(openintro)
library(infer)
library(DATA606)
##
## Welcome to CUNY DATA606 Statistics and Probability for Data Analytics
## This package is designed to support this course. The text book used
## is OpenIntro Statistics, 4th Edition. You can read this by typing
## vignette('os4') or visit www.OpenIntro.org.
##
## The getLabs() function will return a list of the labs available.
##
## The demo(package='DATA606') will list the demos that are available.
You will be analyzing the same dataset as in the previous lab, where
you delved into a sample from the Youth Risk Behavior Surveillance
System (YRBSS) survey, which uses data from high schoolers to help
discover health patterns. The dataset is called yrbss
.
Within each category: * 4,792 reported 0 days * 925 reported 1 - 2 days * 493 reported 3 - 5 days * 311 reported 6 - 9 days * 373 reported 10 - 19 days * 298 reported 20 - 29 days * 827 reported 30 days * 4,646 reported that they did not drive and * 918 are N/A values
#View data set
view(yrbss)
#Find the count for each category in text_while_driving_30d column
<- yrbss %>%
counts count(text_while_driving_30d)
counts
## # A tibble: 9 × 2
## text_while_driving_30d n
## <chr> <int>
## 1 0 4792
## 2 1-2 925
## 3 10-19 373
## 4 20-29 298
## 5 3-5 493
## 6 30 827
## 7 6-9 311
## 8 did not drive 4646
## 9 <NA> 918
6,040 reported not wearing a helmet who has texted while driving in the past 30 days.
Remember that you can use filter
to limit the dataset to
just non-helmet wearers. Here, we will name the dataset
no_helmet
.
data('yrbss', package='openintro')
<- yrbss %>%
no_helmet filter(helmet_12m == "never")
Also, it may be easier to calculate the proportion if you create a
new variable that specifies whether the individual has texted every day
while driving over the past 30 days or not. We will call this variable
text_ind
.
<- no_helmet %>%
no_helmet mutate(text_ind = ifelse(text_while_driving_30d == "30", "yes", "no"))
#Get count of those who texted while driving everyday in the past 30 days and never wore a helmet
%>%
no_helmet count(text_ind)
## # A tibble: 3 × 2
## text_ind n
## <chr> <int>
## 1 no 6040
## 2 yes 463
## 3 <NA> 474
When summarizing the YRBSS, the Centers for Disease Control and Prevention seeks insight into the population parameters. To do this, you can answer the question, “What proportion of people in your sample reported that they have texted while driving each day for the past 30 days?” with a statistic; while the question “What proportion of people on earth have texted while driving each day for the past 30 days?” is answered with an estimate of the parameter.
The inferential tools for estimating population proportion are analogous to those used for means in the last chapter: the confidence interval and the hypothesis test.
%>%
no_helmet specify(response = text_ind, success = "yes") %>%
generate(reps = 1000, type = "bootstrap") %>%
calculate(stat = "prop") %>%
get_ci(level = 0.95)
## # A tibble: 1 × 2
## lower_ci upper_ci
## <dbl> <dbl>
## 1 0.0650 0.0775
Note that since the goal is to construct an interval estimate for a
proportion, it’s necessary to both include the success
argument within specify
, which accounts for the proportion
of non-helmet wearers than have consistently texted while driving the
past 30 days, in this example, and that stat
within
calculate
is here “prop”, signaling that you are trying to
do some sort of inference on a proportion.
#Calculate margin error
<- 6977
n <- 1.96
z <- seq(0, 1, 0.01)
p <- sqrt(p * (1 - p)/n)
se <- z * se me
infer
package, calculate confidence intervals
for two other categorical variables (you’ll need to decide which level
to call “success”, and report the associated margins of error. Interpret
the interval in context of the data. It may be helpful to create new
data sets for each of the two countries first, and then use these data
sets to construct the confidence intervals.With a 95% confidence level the proportion of students who watched TV and did not is (0.346, 0.330).
#Watched TV
<- yrbss %>%
watch_tv mutate(watched_tv = ifelse(hours_tv_per_school_day > 3, "yes", "no"))
%>%
watch_tv specify(response = watched_tv, success = "yes") %>%
generate(reps = 1000, type = "bootstrap") %>%
calculate(stat = "prop") %>%
get_ci(level = 0.95)
## # A tibble: 1 × 2
## lower_ci upper_ci
## <dbl> <dbl>
## 1 0.330 0.346
With a 95% confidence level the proportion of students who watched TV and did not is (0.324 0.309).
#Watched TV
<- yrbss %>%
training mutate(trained = ifelse(strength_training_7d > 4, "yes", "no"))
%>%
training specify(response = trained, success = "yes") %>%
generate(reps = 1000, type = "bootstrap") %>%
calculate(stat = "prop") %>%
get_ci(level = 0.95)
## # A tibble: 1 × 2
## lower_ci upper_ci
## <dbl> <dbl>
## 1 0.309 0.325
Imagine you’ve set out to survey 1000 people on two questions: are you at least 6-feet tall? and are you left-handed? Since both of these sample proportions were calculated from the same sample size, they should have the same margin of error, right? Wrong! While the margin of error does change with sample size, it is also affected by the proportion.
Think back to the formula for the standard error: \(SE = \sqrt{p(1-p)/n}\). This is then used in the formula for the margin of error for a 95% confidence interval:
\[ ME = 1.96\times SE = 1.96\times\sqrt{p(1-p)/n} \,. \] Since the population proportion \(p\) is in this \(ME\) formula, it should make sense that the margin of error is in some way dependent on the population proportion. We can visualize this relationship by creating a plot of \(ME\) vs. \(p\).
Since sample size is irrelevant to this discussion, let’s just set it to some value (\(n = 1000\)) and use this value in the following calculations:
<- 1000 n
The first step is to make a variable p
that is a
sequence from 0 to 1 with each number incremented by 0.01. You can then
create a variable of the margin of error (me
) associated
with each of these values of p
using the familiar
approximate formula (\(ME = 2 \times
SE\)).
<- seq(from = 0, to = 1, by = 0.01)
p <- 2 * sqrt(p * (1 - p)/n) me
Lastly, you can plot the two variables against each other to reveal
their relationship. To do so, we need to first put these variables in a
data frame that you can call in the ggplot
function.
<- data.frame(p = p, me = me)
dd ggplot(data = dd, aes(x = p, y = me)) +
geom_line() +
labs(x = "Population Proportion", y = "Margin of Error")
p
and
me
. Include the margin of error vs. population proportion
plot you constructed in your answer. For a given sample size, for which
value of p
is margin of error maximized?The margin error increases as the population proportion increase to 0.50. After that, the margin error drops as the population proportion increases.
We have emphasized that you must always check conditions before making inference. For inference on proportions, the sample proportion can be assumed to be nearly normal if it is based upon a random sample of independent observations and if both \(np \geq 10\) and \(n(1 - p) \geq 10\). This rule of thumb is easy enough to follow, but it makes you wonder: what’s so special about the number 10?
The short answer is: nothing. You could argue that you would be fine with 9 or that you really should be using 11. What is the “best” value for such a rule of thumb is, at least to some degree, arbitrary. However, when \(np\) and \(n(1-p)\) reaches 10 the sampling distribution is sufficiently normal to use confidence intervals and hypothesis tests that are based on that approximation.
You can investigate the interplay between \(n\) and \(p\) and the shape of the sampling distribution by using simulations. Play around with the following app to investigate how the shape, center, and spread of the distribution of \(\hat{p}\) changes as \(n\) and \(p\) changes.
Distribution appears to be normal, center is 0.1 and the spread looks to be from 0.3 to 0.17
The distribution appears to be normal, when I changed the \(p\) to 0.4 with the spread going from 0.3 to 0.5.
By changing \(n\), the distribution of \(\hat{p}\) appears to be normal, unimodal with a decrease in the spread as \(n\) increases. * * *
For some of the exercises below, you will conduct inference comparing
two proportions. In such cases, you have a response variable that is
categorical, and an explanatory variable that is also categorical, and
you are comparing the proportions of success of the response variable
across the levels of the explanatory variable. This means that when
using infer
, you need to include both variables within
specify
.
Ho: There is no difference in strength training days between students that sleep more than 10+ hours and those who do not.
Ha: There is a difference in strength training days between students that sleep more than 10+ hours and those who do not.
We are 95% confident that the proportion of those students who sleep 10+ hours per day are more likely to strength train every day of the week is between 0.221 and 0.321.
#Sleep 10+ hours per day are more likely to strength train every day of the week
<- yrbss %>%
sleep filter(school_night_hours_sleep == "10+")
<- sleep %>%
sleep mutate(training = ifelse(strength_training_7d == "7", "yes", "no"))
%>%
sleep specify(response = training, success = "yes") %>%
generate(reps = 1000, type = "bootstrap") %>%
calculate(stat = "prop") %>%
get_ci(level = 0.95)
## # A tibble: 1 × 2
## lower_ci upper_ci
## <dbl> <dbl>
## 1 0.221 0.321
A Type 1 error is known as a false positive and occurs when you reject a true null hypothesis. Based on the significance level, there is a 5% chance of there detecting a change.
9604 people would have to be sampled to enusre that we are within the guidelines.
<- 0.01
me <- 0.5
p
1.96^2 * p * (1 - p)/me^2
## [1] 9604