Getting Started

Load packages

In this lab, we will explore and visualize the data using the tidyverse suite of packages, and perform statistical inference using infer. The data can be found in the companion package for OpenIntro resources, openintro.

Let’s load the packages.

library(tidyverse)
library(openintro)
library(infer)
# Load the dataset
data('yrbss', package = 'openintro')

The data

You will be analyzing the same dataset as in the previous lab, where you delved into a sample from the Youth Risk Behavior Surveillance System (YRBSS) survey, which uses data from high schoolers to help discover health patterns. The dataset is called yrbss.

What are the counts within each category for the amount of days these students have texted while driving within the past 30 days?

** I examined the distribution of how frequently high school students reported texting while driving over the past 30 days using the YRBSS dataset.The largest group of students reported “never texted while driving” and the runner up is students who do not drive**

text_counts <- yrbss %>%
  filter(!is.na(text_while_driving_30d)) %>%
  count(text_while_driving_30d) %>%
  arrange(factor(text_while_driving_30d, levels = c("0", "1-2", "3-5", "6-9", "10-19", "20-29", "30", "did not drive")))

print(text_counts)

## # A tibble: 8 × 2
##   text_while_driving_30d     n
##   <chr>                  <int>
## 1 0                       4792
## 2 1-2                      925
## 3 3-5                      493
## 4 6-9                      311
## 5 10-19                    373
## 6 20-29                    298
## 7 30                       827
## 8 did not drive           4646

Visualization

library(ggplot2)

ggplot(text_counts, aes(x = text_while_driving_30d, y = n)) +
  geom_bar(stat = "identity", fill = "steelblue") +
  labs(title = "Texting While Driving in Past 30 Days",
       x = "Days Texted While Driving",
       y = "Count") +
  theme_minimal() +
  coord_flip()  # Flip for better readability

What is the proportion of people who have texted while driving every day in the past 30 days and never wear helmets?

The proportion of non-helmet wearers who reported texted while driving(30+ days) can rounded to 7 % or every 7 students of 100, dont wear helmets

# Load the dataset
data('yrbss', package = 'openintro')

# Create 'no_helmet' dataset (Filter students who never wore helmets)
no_helmet <- yrbss %>%
  filter(helmet_12m == "never") %>%
  mutate(text_ind = ifelse(text_while_driving_30d == "30", "yes", "no"))  # Define text_ind

# Now summarise the proportion of students who text every day while driving
no_helmet %>%
  summarise(prop_text_everyday = mean(text_ind == "yes", na.rm = TRUE))

## # A tibble: 1 × 1
##   prop_text_everyday
##                <dbl>
## 1             0.0712

Remember that you can use filter to limit the dataset to just non-helmet wearers. Here, we will name the dataset no_helmet.

data('yrbss', package='openintro')
no_helmet <- yrbss %>%
  filter(helmet_12m == "never")

Also, it may be easier to calculate the proportion if you create a new variable that specifies whether the individual has texted every day while driving over the past 30 days or not. We will call this variable text_ind.

no_helmet <- no_helmet %>%
  mutate(text_ind = ifelse(text_while_driving_30d == "30", "yes", "no"))

Inference on proportions

When summarizing the YRBSS, the Centers for Disease Control and Prevention seeks insight into the population parameters. To do this, you can answer the question, “What proportion of people in your sample reported that they have texted while driving each day for the past 30 days?” with a statistic; while the question “What proportion of people on earth have texted while driving each day for the past 30 days?” is answered with an estimate of the parameter.

The inferential tools for estimating population proportion are analogous to those used for means in the last chapter: the confidence interval and the hypothesis test.

no_helmet %>%
  drop_na(text_ind) %>% # Drop missing values
  specify(response = text_ind, success = "yes") %>%
  generate(reps = 1000, type = "bootstrap") %>%
  calculate(stat = "prop") %>%
  get_ci(level = 0.95)

## # A tibble: 1 × 2
##   lower_ci upper_ci
##      <dbl>    <dbl>
## 1   0.0652   0.0777

Note that since the goal is to construct an interval estimate for a proportion, it’s necessary to both include the success argument within specify, which accounts for the proportion of non-helmet wearers than have consistently texted while driving the past 30 days, in this example, and that stat within calculate is here “prop”, signaling that you are trying to do some sort of inference on a proportion.

What is the margin of error for the estimate of the proportion of non-helmet wearers that have texted while driving each day for the past 30 days based on this survey?

I calculated and visualized the 95% confidence interval (CI) for the proportion of non-helmet wearers who text while driving daily.This means we are 95% confident that the true proportion of non-helmet wearers who text daily falls within this range,The plot correctly displays the sample proportion as a point and the CI as an error bar

ci <- no_helmet %>%
  drop_na(text_ind) %>%
  specify(response = text_ind, success = "yes") %>%
  generate(reps = 1000, type = "bootstrap") %>%
  calculate(stat = "prop") %>%
  get_ci(level = 0.95)

margin_of_error <- (ci$upper_ci - ci$lower_ci) / 2
margin_of_error

## [1] 0.006227895

Visualization of CI

library(ggplot2)

# Convert CI into a data frame for plotting
ci_df <- data.frame(
  lower = ci$lower_ci,
  upper = ci$upper_ci,
  estimate = mean(no_helmet$text_ind == "yes", na.rm = TRUE)
)

# Plot the confidence interval
ggplot(ci_df, aes(x = estimate, y = 1)) +
  geom_point(size = 6, color = "steelblue") +  # Show the sample proportion
  geom_errorbar(aes(xmin = lower, xmax = upper), width = 0.005, color = "red", size=1) +  # CI line
  labs(title = "95% Confidence Interval for Proportion of Non-Helmet Wearers Who Text Daily",
       x = "Proportion",
       y = "") +
theme_minimal() + theme(panel.grid = element_blank())

4. Using the infer package, calculate confidence intervals for two other categorical variables (you’ll need to decide which level to call “success”, and report the associated margins of error. Interpet the interval in context of the data. It may be helpful to create new data sets for each of the two countries first, and then use these data sets to construct the confidence intervals.

Confidence Intervals: Sleep habits vs. Physical Activity, I felt like these two will have the most success and correlation, sufficient sleep is associated with higher physical activity.

# Step 1: Remove missing values from sleep and physical activity variables
yrbss_filtered <- yrbss %>%
  drop_na(school_night_hours_sleep, physically_active_7d)

# Step 2: Create a categorical variable for sleep duration
yrbss_filtered <- yrbss_filtered %>%
  mutate(sleep_category = ifelse(school_night_hours_sleep < 7, "Low Sleep", "Sufficient Sleep"))

# Step 3: Convert physically_active_7d into a categorical variable
# "Active" = 4 or more days of physical activity, "Inactive" = less than 4 days
yrbss_filtered <- yrbss_filtered %>%
  mutate(physical_activity = ifelse(physically_active_7d >= 4, "Active", "Inactive"))

# Step 4: Compute the confidence interval for physical activity among low sleepers
ci_low_sleep <- yrbss_filtered %>%
  filter(sleep_category == "Low Sleep") %>%
  specify(response = physical_activity, success = "Active") %>%
  generate(reps = 1000, type = "bootstrap") %>%
  calculate(stat = "prop") %>%
  get_ci(level = 0.95)

# Step 5: Compute the confidence interval for physical activity among sufficient sleepers
ci_sufficient_sleep <- yrbss_filtered %>%
  filter(sleep_category == "Sufficient Sleep") %>%
  specify(response = physical_activity, success = "Active") %>%
  generate(reps = 1000, type = "bootstrap") %>%
  calculate(stat = "prop") %>%
  get_ci(level = 0.95)

# Step 6: Print results
print("95% Confidence Interval for Physical Activity (Low Sleep Group):")

## [1] "95% Confidence Interval for Physical Activity (Low Sleep Group):"

print(ci_low_sleep)

## # A tibble: 1 × 2
##   lower_ci upper_ci
##      <dbl>    <dbl>
## 1    0.487    0.514

print("95% Confidence Interval for Physical Activity (Sufficient Sleep Group):")

## [1] "95% Confidence Interval for Physical Activity (Sufficient Sleep Group):"

print(ci_sufficient_sleep)

## # A tibble: 1 × 2
##   lower_ci upper_ci
##      <dbl>    <dbl>
## 1    0.592    0.616

Visualization

# Load necessary library
library(ggplot2)

# Create a data frame with CI values
ci_data <- data.frame(
  sleep_category = c("Low Sleep (<7 hrs)", "Sufficient Sleep (7+ hrs)"),
  lower_ci = c(0.487, 0.593),
  upper_ci = c(0.514, 0.615),
  mean_prop = c((0.487 + 0.514) / 2, (0.593 + 0.615) / 2)  # Calculate midpoint of CI
)

# Create the confidence interval plot
ggplot(ci_data, aes(x = sleep_category, y = mean_prop)) +
  geom_point(size = 4, color = "blue") +  # Point estimate
  geom_errorbar(aes(ymin = lower_ci, ymax = upper_ci), width = 0.005, color = "red", size=1) +  # CI bars
  labs(
    title = "95% Confidence Intervals for Physical Activity by Sleep Habits",
    x = "Sleep Category",
    y = "Proportion Physically Active"
  ) +
  theme_minimal() + theme(panel.grid = element_blank())

How does the proportion affect the margin of error?

Imagine you’ve set out to survey 1000 people on two questions: are you at least 6-feet tall? and are you left-handed? Since both of these sample proportions were calculated from the same sample size, they should have the same margin of error, right? Wrong! While the margin of error does change with sample size, it is also affected by the proportion.

Think back to the formula for the standard error: \(SE = \sqrt{p(1-p)/n}\). This is then used in the formula for the margin of error for a 95% confidence interval:

\[ ME = 1.96\times SE = 1.96\times\sqrt{p(1-p)/n} \,. \] Since the population proportion \(p\) is in this \(ME\) formula, it should make sense that the margin of error is in some way dependent on the population proportion. We can visualize this relationship by creating a plot of \(ME\) vs. \(p\).

Since sample size is irrelevant to this discussion, let’s just set it to some value (\(n = 1000\)) and use this value in the following calculations:

n <- 1000

The first step is to make a variable p that is a sequence from 0 to 1 with each number incremented by 0.01. You can then create a variable of the margin of error (me) associated with each of these values of p using the familiar approximate formula (\(ME = 2 \times SE\)).

p <- seq(from = 0, to = 1, by = 0.01)
me <- 2 * sqrt(p * (1 - p)/n)

Lastly, you can plot the two variables against each other to reveal their relationship. To do so, we need to first put these variables in a data frame that you can call in the ggplot function.

dd <- data.frame(p = p, me = me)
ggplot(data = dd, aes(x = p, y = me)) + 
  geom_line() +
  labs(x = "Population Proportion", y = "Margin of Error")

Describe the relationship between p and me. Include the margin of error vs. population proportion plot you constructed in your answer. For a given sample size, for which value of p is margin of error maximized?

The plot of margin of error vs. population proportion shows a parabolic shape, peaking at p = 0.5.The margin of error is maximized when p = 0.5.

library(ggplot2)

# Define sample size
n <- 1000  # Fixed sample size

# Create a sequence of population proportions (p)
p <- seq(from = 0, to = 1, by = 0.01)

# Calculate the margin of error for each value of p
me <- 1.96 * sqrt(p * (1 - p) / n)  # 95% confidence interval formula

# Create a data frame
dd <- data.frame(p = p, me = me)

# Plot the relationship using ggplot2
ggplot(data = dd, aes(x = p, y = me)) + 
  geom_line(color = "blue", size = 1) +  # Add line plot with styling
  labs(
    title = "Relationship Between Population Proportion and Margin of Error",
    x = "Population Proportion (p)",
    y = "Margin of Error (ME)"
  ) +
  theme_minimal()

Success-failure condition

We have emphasized that you must always check conditions before making inference. For inference on proportions, the sample proportion can be assumed to be nearly normal if it is based upon a random sample of independent observations and if both \(np \geq 10\) and \(n(1 - p) \geq 10\). This rule of thumb is easy enough to follow, but it makes you wonder: what’s so special about the number 10?

The short answer is: nothing. You could argue that you would be fine with 9 or that you really should be using 11. What is the “best” value for such a rule of thumb is, at least to some degree, arbitrary. However, when \(np\) and \(n(1-p)\) reaches 10 the sampling distribution is sufficiently normal to use confidence intervals and hypothesis tests that are based on that approximation.

You can investigate the interplay between \(n\) and \(p\) and the shape of the sampling distribution by using simulations. Play around with the following app to investigate how the shape, center, and spread of the distribution of \(\hat{p}\) changes as \(n\) and \(p\) changes.

Describe the sampling distribution of sample proportions at \(n = 300\) and \(p = 0.1\). Be sure to note the center, spread, and shape.

#At n = 300 and p = 0.1, the sampling distribution: #Center: Approximately 0.1 #Spread: Standard error formula applies. #Shape: Right-skewed because of the small proportion

# Load necessary library
library(ggplot2)

# Set parameters
n <- 300  # Sample size
p <- 0.1  # True population proportion
reps <- 5000  # Number of simulations

# Simulate sample proportions
sample_props <- replicate(reps, mean(rbinom(n, 1, p)))

# Convert to data frame
sim_data <- data.frame(p_hat = sample_props)

# Plot the sampling distribution
ggplot(sim_data, aes(x = p_hat)) +
  geom_histogram(binwidth = 0.01, fill = "steelblue", color = "black", alpha = 0.7) +
  labs(title = "Sampling Distribution of Sample Proportion",
       x = "Sample Proportion (p-hat)",
       y = "Frequency") +
  theme_minimal()

Keep \(n\) constant and change \(p\). How does the shape, center, and spread of the sampling distribution vary as \(p\) changes. You might want to adjust min and max for the \(x\)-axis for a better view of the distribution.

#As p increases, the shape moves from skewed to more symmetric. The center shifts accordingly.

# Load necessary libraries
library(ggplot2)
library(dplyr)

# Set sample size (constant)
n <- 300  

# Define different values of p to analyze
p_values <- c(0.1, 0.3, 0.5, 0.7, 0.9)  

# Simulate sampling distributions for different p-values
sim_data <- data.frame()

for (p in p_values) {
  sample_props <- replicate(5000, mean(rbinom(n, 1, p)))  # Simulating sample proportions
  temp_df <- data.frame(p_hat = sample_props, p = as.factor(p))  # Store data
  sim_data <- rbind(sim_data, temp_df)  # Combine all simulations
}

# Plot sampling distributions for different values of p
ggplot(sim_data, aes(x = p_hat, fill = p)) +
  geom_density(alpha = 0.5) +
  facet_wrap(~ p, scales = "free") +  # Separate plots for each p
  labs(title = "Effect of Changing p on the Sampling Distribution",
       x = "Sample Proportion (p-hat)",
       y = "Density") +
  theme_minimal()

Now also change \(n\). How does \(n\) appear to affect the distribution of \(\hat{p}\)?

#Increasing n: #Reduces spread (narrower confidence intervals). #Keeps center unchanged. #Makes shape more normal.

# Load necessary libraries
library(ggplot2)
library(dplyr)

# Set population proportion (constant)
p <- 0.3  

# Define different sample sizes to analyze
n_values <- c(50, 100, 300, 1000, 5000)  

# Simulate sampling distributions for different sample sizes
sim_data <- data.frame()

for (n in n_values) {
  sample_props <- replicate(5000, mean(rbinom(n, 1, p)))  # Simulating sample proportions
  temp_df <- data.frame(p_hat = sample_props, n = as.factor(n))  # Store data
  sim_data <- rbind(sim_data, temp_df)  # Combine all simulations
}

# Plot sampling distributions for different values of n
ggplot(sim_data, aes(x = p_hat, fill = n)) +
  geom_density(alpha = 0.5) +
  facet_wrap(~ n, scales = "free") +  # Separate plots for each n
  labs(title = "Effect of Increasing Sample Size (n) on the Sampling Distribution",
       x = "Sample Proportion (p-hat)",
       y = "Density") +
  theme_minimal()

More Practice

For some of the exercises below, you will conduct inference comparing two proportions. In such cases, you have a response variable that is categorical, and an explanatory variable that is also categorical, and you are comparing the proportions of success of the response variable across the levels of the explanatory variable. This means that when using infer, you need to include both variables within specify.

Is there convincing evidence that those who sleep 10+ hours per day are more likely to strength train every day of the week? As always, write out the hypotheses for any tests you conduct and outline the status of the conditions for inference. If you find a significant difference, also quantify this difference with a confidence interval.

#Hypotheses:Null (H0): No difference in strength training between groups. #Alternative (H1): 10+ hour sleepers are more likely.

# Step 1: Convert strength training into a categorical variable (Regular vs. Not Regular)
yrbss <- yrbss %>%
  mutate(strength_training = ifelse(strength_training_7d >= 4, "Regular", "Not Regular"))

# Step 2: Categorize sleep duration (Low Sleep vs. Sufficient Sleep)
yrbss <- yrbss %>%
  mutate(sleep_category = ifelse(school_night_hours_sleep < 7, "Low Sleep", "Sufficient Sleep"))

# Step 3: Compute the observed difference in proportions
obs_diff <- yrbss %>%
  filter(!is.na(sleep_category) & !is.na(strength_training)) %>%
  specify(response = strength_training, explanatory = sleep_category, success = "Regular") %>%
  calculate(stat = "diff in props", order = c("Sufficient Sleep", "Low Sleep"))

# Step 4: Perform hypothesis test using permutation
null_dist <- yrbss %>%
  filter(!is.na(sleep_category) & !is.na(strength_training)) %>%
  specify(response = strength_training, explanatory = sleep_category, success = "Regular") %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1000, type = "permute") %>%
  calculate(stat = "diff in props", order = c("Sufficient Sleep", "Low Sleep"))

# Step 5: Compute p-value using observed statistic
p_value <- null_dist %>%
  get_p_value(obs_stat = obs_diff, direction = "greater")

# Print the p-value
print(p_value)

## # A tibble: 1 × 1
##   p_value
##     <dbl>
## 1       0

Null hypotheses Visualize

# Load necessary libraries
library(ggplot2)
library(infer)
library(tidyverse)

# Load dataset
data('yrbss', package = 'openintro')

# Step 1: Convert strength training into a categorical variable
yrbss <- yrbss %>%
  mutate(strength_training = ifelse(strength_training_7d >= 4, "Regular", "Not Regular"))

# Step 2: Categorize sleep duration
yrbss <- yrbss %>%
  mutate(sleep_category = ifelse(school_night_hours_sleep < 7, "Low Sleep", "Sufficient Sleep"))

# Step 3: Compute observed difference in proportions
obs_diff <- yrbss %>%
  filter(!is.na(sleep_category) & !is.na(strength_training)) %>%
  specify(response = strength_training, explanatory = sleep_category, success = "Regular") %>%
  calculate(stat = "diff in props", order = c("Sufficient Sleep", "Low Sleep"))

# Step 4: Generate null distribution
null_dist <- yrbss %>%
  filter(!is.na(sleep_category) & !is.na(strength_training)) %>%
  specify(response = strength_training, explanatory = sleep_category, success = "Regular") %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1000, type = "permute") %>%
  calculate(stat = "diff in props", order = c("Sufficient Sleep", "Low Sleep"))

# Step 5: Plot the null distribution with observed difference
ggplot(null_dist, aes(x = stat)) +
  geom_histogram(binwidth = 0.01, fill = "blue", color = "black", alpha = 0.6) +
  geom_vline(xintercept = obs_diff$stat, color = "red", linetype = "dashed", size = 1.2) +
  labs(title = "Null Distribution vs. Observed Statistic",
       x = "Difference in Proportions (Strength Training | Sleep)",
       y = "Frequency") +
  theme_minimal()

Let’s say there has been no difference in likeliness to strength train every day of the week for those who sleep 10+ hours. What is the probablity that you could detect a change (at a significance level of 0.05) simply by chance? Hint: Review the definition of the Type 1 error.

A Type 1 error occurs if we reject H₀ when it’s true. The probability of this happening at a significance level of 0.05 is simply 5%, the red shaded areas is the type 1 error regions, while the two vertical dashed lines represents the rejection threshold.

# Load necessary library
library(ggplot2)

# Define parameters
x <- seq(-4, 4, length = 1000)  # Range of z-scores
null_dist <- dnorm(x, mean = 0, sd = 1)  # Standard normal distribution (H₀ true)
alpha <- 0.05  # Significance level

# Critical z-values for a two-tailed test at α = 0.05
z_critical <- qnorm(1 - alpha/2)

# Convert to data frame for ggplot
df <- data.frame(x = x, density = null_dist)

# Plot the null distribution
ggplot(df, aes(x = x, y = density)) +
  geom_line(color = "blue", size = 1) +  # Null distribution curve
  geom_area(data = subset(df, x >= z_critical | x <= -z_critical), 
            aes(y = density), fill = "red", alpha = 0.5) +  # Type 1 error regions
  geom_vline(xintercept = c(-z_critical, z_critical), 
             linetype = "dashed", color = "black") +  # Critical values
  labs(title = "Type 1 Error (α) in Hypothesis Testing",
       x = "Test Statistic (Z-score)",
       y = "Density") +
  theme_minimal()

Suppose you’re hired by the local government to estimate the proportion of residents that attend a religious service on a weekly basis. According to the guidelines, the estimate must have a margin of error no greater than 1% with 95% confidence. You have no idea what to expect for \(p\). How many people would you have to sample to ensure that you are within the guidelines?
Hint: Refer to your plot of the relationship between \(p\) and margin of error. This question does not require using a dataset.

We will use the formula for margin of error, Rearrange for n when ME = 0.01 and p = 0.5 (worst case):

p <- 0.5
ME <- 0.01
n <- (1.96^2 * p * (1 - p)) / (ME^2)
ceiling(n)  # Round up to nearest whole number

## [1] 9604

#Sample Size Comparison (n = 9604 vs. n = 50)

# Load necessary library
library(ggplot2)

# Define parameters
p <- 0.5  # Population proportion
n_values <- c(50, 9604)  # Sample sizes for comparison
reps <- 5000  # Number of simulations

# Simulate sample proportions for each n
sim_data <- data.frame()

for (n in n_values) {
  sample_props <- replicate(reps, mean(rbinom(n, 1, p)))  # Generate sample proportions
  temp_df <- data.frame(p_hat = sample_props, n = as.factor(n))  # Store results
  sim_data <- rbind(sim_data, temp_df)  # Combine data
}

# Plot the sampling distributions
ggplot(sim_data, aes(x = p_hat, fill = n)) +
  geom_density(alpha = 0.5) +
  facet_wrap(~ n, scales = "free") +  # Separate plots for each sample size
  labs(title = "Effect of Sample Size on Sampling Distribution",
       x = "Sample Proportion (p-hat)",
       y = "Density") +
  theme_minimal()

#Conclusion: A small sample (n = 50) leads to large fluctuations in sample proportions, while a large sample (n = 9604) provides much more precise estimates * * *

Inference for categorical data

Sheriann McLarty