Social Experiments

1. Resampling
2. Permutation
3. Power and Sample Size

Resampling is a variety of methods for:

• estimating the preision of sample stats by using data subsets (jackknifing)

• drawing randomly with replacment from a set of data points (bootstraping)

Why Resampling:

• the use of obs data or of generated data to produce new hypothetical samples that mimic underlying pop the results of whihc can be analyzed

  • high cost of collecting data
  • not enough data available
  • distribution unknown

1. Monte Carlo Simulation

2. Permutation Test - shuffling and resampling the the treatment and control effects

3. Bootstraping

4. Jackknifing

Permutation Test

marathon example: - in the observed data there is not enough sample to accurately measure

Permutation is essenitally a N choose K

treatment <- c(1,2,3,4)
control <- c(5,6,7,8)
permutation <- c(treatment, control)
new_selection <- sample(permutation, 4
                        )
print(new_selection)

[1] 7 2 5 4

Using marathon times, we use the permutation method by reshuffling the actual times and assigning them to the people because we assume that there has been no impact of the training - it doesn’t make a difference if they train for 1 day or 1 month

Now that we know the average difference between those who trained and those who didn’t train, we have to see if the result is statisticall significant from zero

T-Test Formula:

\[ t = \dfrac{m - \mu}{s\sqrt{n}} \]

t <- (mean_difference - 0) / (sd(samples$mean_diff) / sqrt(nrow(samples)-1))
t.test(samples$mean_diff, alternative = "two.sided", , mu = 0)

    One Sample t-test

data:  samples$mean_diff
t = 1.1689, df = 999, p-value = 0.2427
alternative hypothesis: true mean is not equal to 0
95 percent confidence interval:
 -0.1066588  0.4209505
sample estimates:
mean of x 
0.1571458 

t

[1] 1.168363

Permutation

A permutation test gives a simple way to compute the sampling distribution for any test statistics, under the strong null hypothesis that a set of generate variants has absolutely no effect on the outcome

if the null hypothesis is true, changing the exposure would have no effect on the outcome. By randomly shuffling the exposures we can make up as many data sets as we like

if the null hypothesis is true, the suffeled data sets should look like the real data, otherwise the should look different from the real data

Steps:

1. Analyze the problem:

   -What is the hypotheses and alternative -What ditribution is the data dawn from -What losses are associated with

4. Rearrange the Observations

5. Make a Decision

Permutation Resampling Process:

• Collect Data from control and treatment

• Merge Samples to for a pseudo pop

• Sample with replacedment from pseudo pop to simulation control and treatment groups (like in marathon)

• compute the sample statistic for

install.packages('gtools')
library(gtools)
##get all permutations
##permutations(n=3,r=2,v=x,repeats.allowed=T)

permutations(n = 3, r = 2, v = c("red","green","blue"), repeats.allowed = F)

treatment <-  sample(22:67,12, replace = F)

control <- sample(42:74, 16, replace = F)


permutation_pool <- c(treatment, control)

treat_mean <- mean(treatment)
treat_mean

control_mean <- mean(control)
control_mean

test_stat <- treat_mean - control_mean
test_stat
### Step #3?????


##Step 4 permutations:

perms <- data.frame(permutation_num = 1:1000,
                    control_num_obs = 12,
                    treat_num_obs = 16,
                    mean_control= 48.58333,
                    mean_treatment = 57.1875,
                    difference_of_means =-8.604167 )

perms$mean_control <- mean(sample(permutation_pool, 16, replace = F))

perms$mean_treatment <- mean(sample(permutation_pool, 12, replace = F))

head(perms)

## based on the permutations, is the treatment significantly different from the average? stat different from total mean or control mean?