Designing, Running, and Analyzing Experiments 01

Set Working Directory

knitr::opts_knit$set(root.dir = normalizePath("C:\\Users\\Damian\\Dropbox\\Study\\Coursera\\Design of Experiments"))

Analyzing User Preferences

• Statistics: One- and two-sample tests of proportions with chi-square test, binomial test, multinomial test, G-test, Fisher’s exact test

One-sample tests of proportions

Read in a data file with 2 response categories

prefsAB <- read.csv("prefsAB.csv")
View(prefsAB)

Convert to nomial factor

prefsAB$Subject <- factor(prefsAB$Subject)
summary(prefsAB)

    Subject   Pref  
 1      : 1   A:14  
 2      : 1   B:46  
 3      : 1         
 4      : 1         
 5      : 1         
 6      : 1         
 (Other):54         

plot(prefsAB$Pref)

Pearson chi-square test

prfs <- xtabs(~Pref, data=prefsAB)
prfs

Pref
 A  B 
14 46 

chisq.test(prfs)

    Chi-squared test for given probabilities

data:  prfs
X-squared = 17.067, df = 1, p-value = 3.609e-05

The chi-squared test, the one sampled chi-square test is an asymptotic(점근성의) test. 다시 말해, p-value를 추측하는 것으로 데이터의 양이 많아질수록 나은 결과를 보인다. 주로 데이터가 1000개가 넘을 때 사용하기 좋다고 한다. 이에 대한 대안으로 the exact test(=binomial test)를 사용하도록 하자.

Binomial test

binom.test(prfs)

    Exact binomial test

data:  prfs
number of successes = 14, number of trials = 60,
p-value = 4.224e-05
alternative hypothesis: true probability of success is not equal to 0.5
95 percent confidence interval:
 0.1338373 0.3603828
sample estimates:
probability of success 
             0.2333333 

Multinomial test

Read in a data file with 3 response categories

prefsABC <- read.csv("prefsABC.csv")
View(prefsABC)

prefsABC$Subject <- factor(prefsABC$Subject)
summary(prefsABC)

prfs <- xtabs(~Pref, data=prefsABC)
chisq.test(prfs)

For multinomial test

library(XNomial)
xmulti(prfs, c(1/3, 1/3, 1/3), statName='Prob')

• Testing over the preferences we have.
• A, B, C 간에 no preference 적용
• p-value가 0.05보다 작다.
• It tells us, 차이가 있다. 근데 뭐와 뭐 간 차이인지는 모름

Post hoc binomial tests with correction for multiple comparisons (Pair wise differences)

aa <- binom.test(sum(prefsABC$Pref=="A"), nrow(prefsABC), p=1/3)
bb <- binom.test(sum(prefsABC$Pref=="B"), nrow(prefsABC), p=1/3)
cc <- binom.test(sum(prefsABC$Pref=="C"), nrow(prefsABC), p=1/3)

p.adjust(c(aa$p.value, bb$p.value, cc$p.value), method="holm")

• pref==“A”인 데이터와 그렇지 않은 데이터 간에 비교를 하는데 가설 확률이 1/3을 적용
• A, B, C 모두 적용

• adjustment

  When we’re testing statistical tests, it’s significant. It means that there’re a 1 in 20 chance that just by chance, we might think there’s a statistically significant outcome when in fact it’s just by chance. That’s what being at 0.05 means, a 1 in 20 chance for that and so if we’re doing what are multinomial comparisons if we were to do 20 of them, we’d expect 1 just by chance to become significant. So we have to adjust for that.And we do that with a Bonferroni correction. This method here is called ‘holm’, which is really the preferred one. It subsumes Bonferroini, but it’s a little bit less of a strict test. It’s a sequential test that adjusts each of p value according to just how low it is. The first p value is in this case with test multiplied by three, so increased three times. This is called Holm’s Sequential Bonferroni procedure.