Homework 3 (5%): due by next lecture on March 12th

Instructions: Start a new R markdown for the homework and call it “Yourlastname_Firstname_Homework_2”.

Copy everything below from Task 1 to Task 5. Keep the task prompt and questions, and provide your code and answer underneath.

Remember: you need all the steps for your code to work, including loading your data – otherwise it will not knit.

To generate a new code box, click on the +C sign above. Underneath your code, provide your answer to the task question.

When you are done, click on “Knit” above, then “Knit to Html”. Wait for everything to compile. If you get an error like “Execution halted”, it means there are issues with your code you must fix. Whe n all issues are fixed, it will prompt a new window. Then click on “Publish” in the top right, and then Rpubs (the first option) and follow the instructions to create your Rpubs account and get your Rpubs link for your document (i.e., html link as I provide for the tutorial).

Note: Make sure to provide both your markdown file and R pubs link. If you do not submit both, you will be penalized 2 pts. out of the 5 pts. total.

Task 1

rm(list=ls());gc()
##          used (Mb) gc trigger (Mb) limit (Mb) max used (Mb)
## Ncells 526407 28.2    1169297 62.5         NA   669445 35.8
## Vcells 969542  7.4    8388608 64.0      16384  1851679 14.2
getwd()
## [1] "/Users/vanessa/Desktop/Tutorials "
# List of packages
packages <- c("tidyverse", "infer", "fst") # add any you need here

# Install packages if they aren't installed already
new_packages <- packages[!(packages %in% installed.packages()[,"Package"])]
if(length(new_packages)) install.packages(new_packages)

# Load the packages
lapply(packages, library, character.only = TRUE)
## ── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
## ✔ dplyr     1.1.4     ✔ readr     2.1.4
## ✔ forcats   1.0.0     ✔ stringr   1.5.1
## ✔ ggplot2   3.4.4     ✔ tibble    3.2.1
## ✔ lubridate 1.9.3     ✔ tidyr     1.3.0
## ✔ purrr     1.0.2     
## ── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()
## ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
## [[1]]
##  [1] "lubridate" "forcats"   "stringr"   "dplyr"     "purrr"     "readr"    
##  [7] "tidyr"     "tibble"    "ggplot2"   "tidyverse" "stats"     "graphics" 
## [13] "grDevices" "utils"     "datasets"  "methods"   "base"     
## 
## [[2]]
##  [1] "infer"     "lubridate" "forcats"   "stringr"   "dplyr"     "purrr"    
##  [7] "readr"     "tidyr"     "tibble"    "ggplot2"   "tidyverse" "stats"    
## [13] "graphics"  "grDevices" "utils"     "datasets"  "methods"   "base"     
## 
## [[3]]
##  [1] "fst"       "infer"     "lubridate" "forcats"   "stringr"   "dplyr"    
##  [7] "purrr"     "readr"     "tidyr"     "tibble"    "ggplot2"   "tidyverse"
## [13] "stats"     "graphics"  "grDevices" "utils"     "datasets"  "methods"  
## [19] "base"
france_data <- read.fst("france_data.fst")
france_data <- read.fst("france_data.fst") %>%
  # Filter for France
  filter(cntry == "FR") %>%
  # Recoding clsprty and cleaning trstplt (will use later)
  mutate(
    lrscale = case_when(
      lrscale == 1 ~ "left",  # Recode 1 to "Yes"
      lrscale == 2 ~ "right",   # Recode 2 to "No"
      lrscale %in% c(7, 8, 9) ~ NA_character_,  # Handle other specific cases where you want to set it as NA
      TRUE ~ as.character(clsprty)  # Keep other values as-is but ensure they are characters
    ),
     # Recoding education with case_when for consistency
    educ.ba = case_when(
      essround < 5 & edulvla == 5 ~ "BA or more",
      essround >= 5 & edulvlb > 600 ~ "BA or more",
      TRUE ~ "No BA"
    )
  ) %>%
  # Ensure all NAs are handled uniformly
  mutate(
    edulvla = ifelse(edulvla %in% c(77, 88), NA_integer_, edulvla),
    edulvlb = ifelse(edulvlb %in% c(5555, 7777, 8888), NA_integer_, edulvlb),
    educ.ba = as.character(educ.ba)  # Force educ.ba to be treated as character
  )
## Not necessary, but always good to rename when doing operations so you can backtrack to clean dataset
tab_dat <- france_data

# Create a crosstable and save it into the object mytab observed
mytab <- table(tab_dat$lrscale, tab_dat$educ.ba) 

# Calculate row-wise sums of counts in table 
rsums <- rowSums(mytab)

# Calculate column-wise sums of counts in table
csums <- colSums(mytab)

# Get the total count of the table (N)
N <- sum(mytab)

# Generate table of expected proportions expected
ptab <- tcrossprod(rsums/N, csums/N)
cat("Table of Expected Proportions:\n")
## Table of Expected Proportions:
print(ptab)
##             [,1]        [,2]
## [1,] 0.084994428 0.294722859
## [2,] 0.109116823 0.378368588
## [3,] 0.001349809 0.004680537
## [4,] 0.002714132 0.009411403
## [5,] 0.007242524 0.025113850
## [6,] 0.018418362 0.063866685
# Table of expected frequencies expected 
ftab <- N * ptab
cat("Table of Expected Frequencies:\n")
## Table of Expected Frequencies:
print(ftab)
##            [,1]       [,2]
## [1,] 1310.78407 4545.21593
## [2,] 1682.79964 5835.20036
## [3,]   20.81676   72.18324
## [4,]   41.85735  145.14265
## [5,]  111.69420  387.30580
## [6,]  284.04798  984.95202
# Critical Value for Independence:
alpha <- 0.05 # significance level
c_val <- qchisq(alpha, df = 1, lower.tail = FALSE)
cat("Critical Value:", round(c_val, 3), "\n")
## Critical Value: 3.841
# Pearson's chi-squared statistic:
test_stat <- sum((mytab - ftab)^2 / ftab)
cat("Pearson's X^2:", round(test_stat, 4), "\n")
## Pearson's X^2: 224.1156
# p-value:
p_val <- pchisq(test_stat, df = 1, lower.tail = F)
cat("p-value:", round(p_val, 4), "\n")
## p-value: 0
# Using chisq.test for validation:
cat("Chi-squared test result:\n")
## Chi-squared test result:
print(chisq.test(mytab, correct = F))
## 
##  Pearson's Chi-squared test
## 
## data:  mytab
## X-squared = 224.12, df = 5, p-value < 2.2e-16

The critical value is exactly 3.841 meaning that we cannot reject the null hypothesis for these variables. The observed x-squared value is 224.12 which is much greater than the critical value. This means that there is a statistically significant association between these two variables. The calculated p-value is extremely close to O which means that we can reject the null hypothesis.These calculations suggest that there is a significant association between these two variables.

Task 2

Do Steps 1 to 3 using the infer package (as we did in the tutorial), for the variables sbbsntx (recoded to the corresponding 5 categories) as the response and domicil (recoded to urban and rural, setting 2 to urban instead of peri-urban) as the explanatory variable for the country of France.

See variable info here: https://ess.sikt.no/en/datafile/ffc43f48-e15a-4a1c-8813-47eda377c355/92?tab=1&elements=[%22874a86e2-a0f6-40b4-aef7-51c8eed98a7d/1%22])

Provide interpretations of the output (consider the variable info).

france_data <- read.fst("france_data.fst")
france_clean <- france_data %>% filter(!is.na(sbbsntx), !is.na(domicil))
  # Recoding and cleaning  (will use later)
france_clean <- france_data %>% 
  mutate(
    sbbsntx = case_when(
      sbbsntx == 1 ~ "Agree Strongly", 
      sbbsntx == 2 ~ "Agree",   # Recode 2 to "No"
      sbbsntx == 3 ~ "Neither Agree nor Disagree", 
      sbbsntx == 4 ~ "Disagree", 
      sbbsntx == 5 ~ "Disagree Strongly", 
      sbbsntx %in% c(7, 8, 9) ~ NA_character_,))

france_clean <- france_clean %>% filter(!is.na(sbbsntx), !is.na(domicil))
  # Recoding and cleaning  (will use later)
france_clean <- france_data %>% 
  
  mutate(
    domicil = case_when(
      domicil == 1 ~ "Urban",
      domicil == 2 ~ "Urban",
      domicil == 3 ~ "Rural",
      domicil == 4 ~ "Rural",
      domicil == 5 ~ "Rural", 
      domicil %in% c(7, 8, 9) ~ NA_character_,))
## Not necessary, but always good to rename when doing operations so you can backtrack to clean dataset
tab_dat <- france_clean

# Create a crosstable and save it into the object mytab observed
mytab <- table(tab_dat$sbbsntx, tab_dat$domicil) 

# Calculate row-wise sums of counts in table 
rsums <- rowSums(mytab)

# Calculate column-wise sums of counts in table
csums <- colSums(mytab)

# Get the total count of the table (N)
N <- sum(mytab)

# Generate table of expected proportions expected
ptab <- tcrossprod(rsums/N, csums/N)
cat("Table of Expected Proportions:\n")
## Table of Expected Proportions:
print(ptab)
##              [,1]        [,2]
## [1,] 0.1302584890 0.052262032
## [2,] 0.2649967673 0.106321436
## [3,] 0.1385288693 0.055580257
## [4,] 0.1102717367 0.044242990
## [5,] 0.0518621762 0.020808031
## [6,] 0.0005168988 0.000207389
## [7,] 0.0172299589 0.006912967
# Table of expected frequencies expected 
ftab <- N * ptab
cat("Table of Expected Frequencies:\n")
## Table of Expected Frequencies:
print(ftab)
##             [,1]        [,2]
## [1,]  539.530662 216.4693385
## [2,] 1097.616610 440.3833897
## [3,]  573.786577 230.2134235
## [4,]  456.745534 183.2544664
## [5,]  214.813134  86.1868662
## [6,]    2.140995   0.8590053
## [7,]   71.366490  28.6335104
# Critical Value for Independence:
alpha <- 0.05 # significance level
c_val <- qchisq(alpha, df = 1, lower.tail = FALSE)
cat("Critical Value:", round(c_val, 3), "\n")
## Critical Value: 3.841
test_stat <- france_data %>%
  specify(explanatory = domicil, # change variable name for explanatory variable
          response = sbbsntx) %>% # change variable name for outcome of interest
  hypothesize(null = "independence") %>%
  calculate(stat = "slope") # replace in between quotation marks appropriate test statistic
## Warning: Removed 14895 rows containing missing values.
## Message: The independence null hypothesis does not inform calculation of the
## observed statistic (a slope) and will be ignored.
print(test_stat$stat)
##    domicil 
## -0.0788481
null_distribution <- france_data %>%
  specify(explanatory = domicil,
          response = sbbsntx) %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1000, type = "permute") %>% # only line we are adding, use reps = 1000 as standard but change if too computationally demanding
  calculate(stat = "slope")
## Warning: Removed 14895 rows containing missing values.
p_val <- null_distribution %>% # replace name here if you assigned something other than null_distribution above
  get_pvalue(obs_stat = test_stat, direction = "two-sided") # would only replace test_stat if assigned another name in Step 1
## Warning: Please be cautious in reporting a p-value of 0. This result is an approximation
## based on the number of `reps` chosen in the `generate()` step.
## ℹ See `get_p_value()` (`?infer::get_p_value()`) for more information.
p_val
## # A tibble: 1 × 1
##   p_value
##     <dbl>
## 1       0

Task 3

Now do Step 4 (i.e., the null distribution visualization with confidence intervals) for the same variables and country as Task 2, and interpret the output.

null_distribution %>%
  visualize() +
  shade_p_value(obs_stat = test_stat, direction = "two-sided")

null_distribution
## Response: sbbsntx (numeric)
## Explanatory: domicil (numeric)
## Null Hypothesis: independence
## # A tibble: 1,000 × 2
##    replicate      stat
##        <int>     <dbl>
##  1         1  0.00516 
##  2         2  0.0273  
##  3         3 -0.0265  
##  4         4 -0.0258  
##  5         5  0.00172 
##  6         6  0.0193  
##  7         7 -0.000451
##  8         8 -0.00987 
##  9         9 -0.0160  
## 10        10  0.0117  
## # ℹ 990 more rows
conf_int <- null_distribution %>%
  get_confidence_interval(level = 0.95, type = "percentile")


null_distribution %>%
  visualize() +
  shade_p_value(obs_stat = test_stat, direction = "two-sided") +
  shade_confidence_interval(endpoints = conf_int)

null_distribution
## Response: sbbsntx (numeric)
## Explanatory: domicil (numeric)
## Null Hypothesis: independence
## # A tibble: 1,000 × 2
##    replicate      stat
##        <int>     <dbl>
##  1         1  0.00516 
##  2         2  0.0273  
##  3         3 -0.0265  
##  4         4 -0.0258  
##  5         5  0.00172 
##  6         6  0.0193  
##  7         7 -0.000451
##  8         8 -0.00987 
##  9         9 -0.0160  
## 10        10  0.0117  
## # ℹ 990 more rows

The confidence interval line is not in the null distribution section thus we can reject the null hypothesis. This indicates that these two variables are likely scientifically significant and in association with one another.

Task 4

Conduct Steps 1 to 3 using the infer package (as we did in the tutorial), for the variables ccrdprs (leaving it on the 0-10 numeric scale) as the response (or outcome) variable and lrscale (recoded as left and right, omitting “moderates” as we did in Task 1) as the explanatory variable.

Variable info here: https://ess.sikt.no/en/datafile/ffc43f48-e15a-4a1c-8813-47eda377c355/92?tab=1&elements=[%2283b08cf2-508e-49a0-9fc8-c3cc281290ec/4%22]

Provide interpretations of the output (consider the variable info).

france_clean <- france_clean %>% filter(!is.na(ccrdprs), !is.na(lrscale))
  # Recoding and cleaning  (will use later)
france_clean <- france_data %>% 
  mutate(
    ccrdprs = case_when(
      ccrdprs == 1 ~ "Not at all", 
      ccrdprs == 10 ~ "A great deal",
      ccrdprs %in% c(7, 8, 9) ~ NA_character_,))

france_clean <- france_clean %>% filter(!is.na(ccrdprs), !is.na(lrscale))
  # Recoding and cleaning  (will use later)
france_clean <- france_data %>% 
  mutate(
    lrscale = case_when(
      lrscale == 1 ~ "left",  # Recode 1 to "Yes"
      lrscale == 2 ~ "right",   # Recode 2 to "No"
      lrscale %in% c(7, 8, 9) ~ NA_character_,))
test_stat <- france_data %>%
  specify(explanatory = lrscale, # change variable name for explanatory variable
          response = ccrdprs) %>% # change variable name for outcome of interest
  hypothesize(null = "independence") %>%
  calculate(stat = "slope") # replace in between quotation marks appropriate test statistic
## Warning: Removed 14991 rows containing missing values.
## Message: The independence null hypothesis does not inform calculation of the
## observed statistic (a slope) and will be ignored.
print(test_stat$stat)
##   lrscale 
## 0.0136051
null_distribution <- france_data %>%
  specify(explanatory = lrscale,
          response = ccrdprs) %>%
  hypothesize(null = "independence") %>%
  generate(reps = 1000, type = "permute") %>% # only line we are adding, use reps = 1000 as standard but change if too computationally demanding
  calculate(stat = "slope")
## Warning: Removed 14991 rows containing missing values.
p_val <- null_distribution %>% # replace name here if you assigned something other than null_distribution above
  get_pvalue(obs_stat = test_stat, direction = "two-sided") # would only replace test_stat if assigned another name in Step 1

p_val
## # A tibble: 1 × 1
##   p_value
##     <dbl>
## 1   0.038

In France majority of people feel around 7 which is close to a great deal about what extent they personally feel responsible to reduce climate change. Very few people feel little responsibility to reduce climate change with a large percentage of the sample feeling personally responsible to reduce climate change.

Task 5

Now do Step 4 (i.e., the null distribution visualization with confidence intervals) for the same variables and country as Task 4, and interpret the output.

null_distribution %>%
  visualize() +
  shade_p_value(obs_stat = test_stat, direction = "two-sided")

null_distribution
## Response: ccrdprs (numeric)
## Explanatory: lrscale (numeric)
## Null Hypothesis: independence
## # A tibble: 1,000 × 2
##    replicate      stat
##        <int>     <dbl>
##  1         1 -0.00318 
##  2         2 -0.00231 
##  3         3  0.00687 
##  4         4 -0.00449 
##  5         5 -0.0118  
##  6         6 -0.00421 
##  7         7  0.000796
##  8         8  0.000382
##  9         9 -0.00552 
## 10        10 -0.00231 
## # ℹ 990 more rows
conf_int <- null_distribution %>%
  get_confidence_interval(level = 0.95, type = "percentile")


null_distribution %>%
  visualize() +
  shade_p_value(obs_stat = test_stat, direction = "two-sided") +
  shade_confidence_interval(endpoints = conf_int)

null_distribution
## Response: ccrdprs (numeric)
## Explanatory: lrscale (numeric)
## Null Hypothesis: independence
## # A tibble: 1,000 × 2
##    replicate      stat
##        <int>     <dbl>
##  1         1 -0.00318 
##  2         2 -0.00231 
##  3         3  0.00687 
##  4         4 -0.00449 
##  5         5 -0.0118  
##  6         6 -0.00421 
##  7         7  0.000796
##  8         8  0.000382
##  9         9 -0.00552 
## 10        10 -0.00231 
## # ℹ 990 more rows

The confidence interval falls outside of the null distribution category and thus we can reject the null hypothesis.

End.