Types of variables

Variable type helps us to determine:

1. What type of summary statistics are needed to describe the data
2. What type of visualizations best represent the data
3. What statistical methods can be applied to the data

Numeric (qualitative):

• Continuous: inifite number of values within a given range
• Discrete: specific set of numeric values that can be counted or enumerated, often counted

Categorical (qualitative):

• Ordinal: finite number of values within a given range, often measured

Categorical Data

• Is often stored as factors in R
  • factors have an important use in statistical modeling
  • Sometimes factors are undesirable, but sometimes essetial
• Common in subgroup analysis
  • Only interested in a subset of the data
  • Filter for specific levels of categorical variable

#frquency tables
table( hsb2$schtyp )

## 
##  public private 
##     168      32

#use dplyr filter to subset the 'public' data
hsb2_public <- hsb2 %>% filter( schtyp == 'public' )
head( hsb2_public )

## # A tibble: 6 x 11
##      id gender race  ses    schtyp prog        read write  math science socst
##   <int> <chr>  <chr> <fct>  <fct>  <fct>      <int> <int> <int>   <int> <int>
## 1    70 male   white low    public general       57    52    41      47    57
## 2   121 female white middle public vocational    68    59    53      63    61
## 3    86 male   white high   public general       44    33    54      58    31
## 4   141 male   white high   public vocational    63    44    47      53    56
## 5   172 male   white middle public academic      47    52    57      53    61
## 6   113 male   white middle public academic      44    52    51      63    61

The pipe operator %>%:

• take the dataframe before the operator and pass to the function following
• ex: x %>% f( y ) is effectively f( x,y )
• ex: sum( 3,4 ) is effectively 3 %>% sum( 4 )

table( hsb2_public$schtyp )

## 
##  public private 
##     168       0

The output still shows ‘private’ as an option even though it does not exist in this subset of the data.

#drop unused levels
hsb2_public$schtyp <- droplevels( hsb2_public$schtyp )
table( hsb2_public$schtyp )

## 
## public 
##    168

# Subset of emails with big numbers: email50_big
email50_big <- email50 %>%
  filter( number == 'big' )

# Glimpse the subset
#glimpse( email50_big )
table( email50_big$number )

## 
##  none small   big 
##     0     0     7

email50_big$number = droplevels( email50_big$number )
table( email50_big$number )

## 
## big 
##   7

Descretize a variable

redefining a numeric variable as a categorical variable

#descretize the average reading score feature as either above or below average
(avg_read <- mean( hsb2$read )) #the extra parentheses will tell R to print the result

## [1] 52.23

use dplyr mutate to create a new discrete variable that categorizes a record as being either above or below average reading scores

#create new variable: read_cat
hsb2 <- hsb2 %>%
  mutate( read_cat = ifelse(
    read < avg_read,       # <-- logical condition
    'below average',       # <--what to do if condition is TRUE
    'at or above average'  # <--what to do if condition is FALSE
  ))
head( hsb2 )

## # A tibble: 6 x 12
##      id gender race  ses   schtyp prog   read write  math science socst read_cat
##   <int> <chr>  <chr> <fct> <fct>  <fct> <int> <int> <int>   <int> <int> <chr>   
## 1    70 male   white low   public gene…    57    52    41      47    57 at or a…
## 2   121 female white midd… public voca…    68    59    53      63    61 at or a…
## 3    86 male   white high  public gene…    44    33    54      58    31 below a…
## 4   141 male   white high  public voca…    63    44    47      53    56 at or a…
## 5   172 male   white midd… public acad…    47    52    57      53    61 below a…
## 6   113 male   white midd… public acad…    44    52    51      63    61 below a…

# Calculate median number of characters: med_num_char
med_num_char <- median( email50$num_char )

# Create num_char_cat variable in email50
email50_fortified <- email50 %>%
  mutate(num_char_cat = ifelse(num_char < med_num_char, 'below median', "at or above median" ))
  
# Count emails in each category
email50_fortified %>%
  count(num_char_cat)

## # A tibble: 2 x 2
##   num_char_cat           n
##   <chr>              <int>
## 1 at or above median    25
## 2 below median          25

# Create number_yn column in email50
email50_fortified <- email50 %>%
  mutate(
    number_yn = case_when(
      # if number is "none", make number_yn "no"
      number == "none" ~ "no", 
      # if number is not "none", make number_yn "yes"
      number != "none" ~ "yes"  
    )
  )
  

# Visualize the distribution of number_yn
ggplot(email50_fortified, aes(x = number_yn)) +
  geom_bar()

Visualizing numerical data

ggplot2

• Modern looking hasstle-free plots (don’t have to explicitly render legends etc.)
• easy to extend code to multivariate plots
• Iterative construction (can build complex visualizations by adding relatively simple layers)

#scatterplot of math vs science scores
ggplot( data = hsb2, aes( x = science, y = math, color = prog ) ) + 
  #layers are separated by '+'
  geom_point() #having the next layer on a new line makes code readable

# Scatterplot of exclaim_mess vs. num_char
ggplot(email50, aes(x = num_char, y = exclaim_mess, color = factor(spam))) +
  geom_point()

Observational studies and experiments

Types of studies:

• Observational Study:
  • Collect data in a way that does not directly interfere with how the data arise
  • Only correlations can be inferred
• Experiments:
  • Randomly assign subjects to various treatments
  • Causation can be inferred

In an experiment, variables that migth also contribute to the outcome (confounding variables) are most likely represented equally in the study groups due to random assignment. Therefore, if there is a significant difference found between the averages of the dependent variable, then we can make causal statements.

An example of an observational study:

data( gapminder )
glimpse( gapminder )

## Rows: 1,704
## Columns: 6
## $ country   <fct> Afghanistan, Afghanistan, Afghanistan, Afghanistan, Afghani…
## $ continent <fct> Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia, Asia,…
## $ year      <int> 1952, 1957, 1962, 1967, 1972, 1977, 1982, 1987, 1992, 1997,…
## $ lifeExp   <dbl> 28.801, 30.332, 31.997, 34.020, 36.088, 38.438, 39.854, 40.…
## $ pop       <int> 8425333, 9240934, 10267083, 11537966, 13079460, 14880372, 1…
## $ gdpPercap <dbl> 779.4453, 820.8530, 853.1007, 836.1971, 739.9811, 786.1134,…

Random sampling and random assignment

Random….

• Random Sampling:
  • At selection of subjects from population
  • Helps generalizability of results
• Random Assignment:
  • At selection of subjects from population
  • helps infer causation from results

Scope of Inference

Simpson’s paradox: not considering an important variable when studying an effect

load( file="ucb_admit.RData" )
head( ucb_admit )

##      Admit Gender Dept
## 1 Admitted   Male    A
## 2 Admitted   Male    A
## 3 Admitted   Male    A
## 4 Admitted   Male    A
## 5 Admitted   Male    A
## 6 Admitted   Male    A

# Count number of male and female applicants admitted
ucb_admission_counts <- ucb_admit %>%
  count(Gender, Admit)

#calculate the proportion of males admitted overall
ucb_admission_counts %>%
  # Group by gender
  group_by(Gender) %>%
  # Create new variable
  mutate(prop = n / sum( n )) %>%
  # Filter for admitted
  filter(Admit == "Admitted")

## # A tibble: 2 x 4
## # Groups:   Gender [2]
##   Gender Admit        n  prop
##   <fct>  <fct>    <int> <dbl>
## 1 Male   Admitted  1198 0.445
## 2 Female Admitted   557 0.304

#now to find the proportions of males and females admitted by department
ucb_admission_counts <- ucb_admit %>%
  # Counts by department, then gender, then admission status
  count(Dept, Gender, Admit)

ucb_admission_counts  %>%
  # Group by department, then gender
  group_by(Dept, Gender) %>%
  # Create new variable
  mutate(prop = n/sum(n)) %>%
  # Filter for male and admitted
  filter(Gender == 'Male', Admit == "Admitted")

## # A tibble: 6 x 5
## # Groups:   Dept, Gender [6]
##   Dept  Gender Admit        n   prop
##   <chr> <fct>  <fct>    <int>  <dbl>
## 1 A     Male   Admitted   512 0.621 
## 2 B     Male   Admitted   353 0.630 
## 3 C     Male   Admitted   120 0.369 
## 4 D     Male   Admitted   138 0.331 
## 5 E     Male   Admitted    53 0.277 
## 6 F     Male   Admitted    22 0.0590

Sampling strategies

Why sample? Why not take a census of the entire population of interest?

• Conducting a census is very resource intensive
• Nearly imposible to collect data from all individuals, hence there is no guarentee of unbiased results
• Populations constantly change

Simple Random Samples: select cases randomly from the population such that each case is equally likely to be selected.
Stratefied Samples: first, divide the population into homogenious groups, or stata. Then, randomly sample within strata.
Cluster Samples: divide the population into clusters, only samplea few clusters, but sample these clusters in entirety.
Multistage Samples: add another step after cluster sampling: randomly sample observations from within randomly sampled clusters.

Sampling in R

#load the county dataframe
data( county )
glimpse( county )

## Rows: 3,142
## Columns: 15
## $ name              <chr> "Autauga County", "Baldwin County", "Barbour County…
## $ state             <fct> Alabama, Alabama, Alabama, Alabama, Alabama, Alabam…
## $ pop2000           <dbl> 43671, 140415, 29038, 20826, 51024, 11714, 21399, 1…
## $ pop2010           <dbl> 54571, 182265, 27457, 22915, 57322, 10914, 20947, 1…
## $ pop2017           <int> 55504, 212628, 25270, 22668, 58013, 10309, 19825, 1…
## $ pop_change        <dbl> 1.48, 9.19, -6.22, 0.73, 0.68, -2.28, -2.69, -1.51,…
## $ poverty           <dbl> 13.7, 11.8, 27.2, 15.2, 15.6, 28.5, 24.4, 18.6, 18.…
## $ homeownership     <dbl> 77.5, 76.7, 68.0, 82.9, 82.0, 76.9, 69.0, 70.7, 71.…
## $ multi_unit        <dbl> 7.2, 22.6, 11.1, 6.6, 3.7, 9.9, 13.7, 14.3, 8.7, 4.…
## $ unemployment_rate <dbl> 3.86, 3.99, 5.90, 4.39, 4.02, 4.93, 5.49, 4.93, 4.0…
## $ metro             <fct> yes, yes, no, yes, yes, no, no, yes, no, no, yes, n…
## $ median_edu        <fct> some_college, some_college, hs_diploma, hs_diploma,…
## $ per_capita_income <dbl> 27841.70, 27779.85, 17891.73, 20572.05, 21367.39, 1…
## $ median_hh_income  <int> 55317, 52562, 33368, 43404, 47412, 29655, 36326, 43…
## $ smoking_ban       <fct> none, none, partial, none, none, none, NA, NA, none…

set.seed( 1349 )
#remove DC (because it is not a state ?)
county_noDC <- county %>%
  filter( state != 'District of Columbia') %>%
  droplevels()

#simple random sample
county_srs <- county_noDC %>%
  sample_n( size = 150 )

( length( unique( county_srs$state ) ) ) #depending on seed, might not sample all 50 states

## [1] 43

#and would very very very unlikely sample them evenly

county_srs %>%
  group_by( state ) %>%
  count() %>%
  ggplot( aes( x = state, y = n ) ) +
  geom_bar( stat = 'identity' ) +
  theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
  ggtitle( 'Simple Sample Distribution' )

#stratefied sample
county_str <- county_noDC %>%
  group_by( state ) %>%
  sample_n( size = 3 ) %>%
  count() %>%
  ggplot( aes( x = state, y = n ) ) +
  geom_bar( stat = 'identity' ) +
  ylim( 0,5 ) +
  theme(axis.text.x = element_text(angle = 45, hjust = 1)) +
  ggtitle( 'Stratefied Sample Distribution' )
county_str

load( file="us_regions.RData" )
glimpse( us_regions )

## Rows: 51
## Columns: 2
## $ state  <fct> Connecticut, Maine, Massachusetts, New Hampshire, Rhode Island…
## $ region <fct> Northeast, Northeast, Northeast, Northeast, Northeast, Northea…

# Simple random sample: states_srs
states_srs <- us_regions %>%
  sample_n( 8 )

# Count states by region
states_srs %>%
  count(region)

##      region n
## 1   Midwest 3
## 2 Northeast 1
## 3     South 2
## 4      West 2

# Stratified sample
states_str <- us_regions %>%
  group_by(region) %>%
  sample_n(2)

# Count states by region
states_str %>%
  count()

## # A tibble: 4 x 2
## # Groups:   region [4]
##   region        n
##   <fct>     <int>
## 1 Midwest       2
## 2 Northeast     2
## 3 South         2
## 4 West          2

Principles of Experimental Design

• Control: compare treatment of interest to a control group
• Randomize: randomly assign subjects to treatment
• Replicate: collect a sufficiently large sample within a study, or replicate the entire study
• Block: account for the potential effect of confounding variables
  • Group subjects unto blocks based on variables
  • Randomize within each block to treatment groups