Problem 1:

Repeat the above simulation for a range of values of n (at least 5 values). Create a table with a row for each n, and the power for each of the three tests as you change n (i.e., the proportion of true differences the test detected), as well as the false alarm rate for each test. A power of .8 is considered acceptably large when planning a study. Identify roughly the n needed for a power of 0.8 in each of the three tests. Discuss the relative advantages and disadvantages of the three tests with respect to power and false alarm (incorrectly rejecting the null).

Problem 2:

For the data below, compute a one-tailed t-test by hand to determine whether the average is reliably greater than 100. That is, compute the mean, standard deviation, standard error, t value, and the corresponding p value. To compute the t value for this comparison, you need to compute the mean and subtract 100, and then divide that by the standard error. Verify you did this correctly by using the t.test() function to produce the same values. Then, re-run your analysis by removing the 245 value, which might seem like it is the strongest piece of evidence for your data being greater than 100.

data1 <- c(101,103,99,92,110,105,103,102,104,106,101,101,101,102,103,101,99,104,105,102,102,103,245,103,107,101,103,108,104,101,101,102)

Mean1 <- mean(data1)
SD1 <- sd(data1)
SE1 <- SD1/sqrt(length(data1))
Tval1 <- Mean1/SE1
Tval1
## [1] 23.85868
TTest1 <-t.test(data1)
TTest1
## 
##  One Sample t-test
## 
## data:  data1
## t = 23.859, df = 31, p-value < 2.2e-16
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##   97.85331 116.14669
## sample estimates:
## mean of x 
##       107
data2 <- c(101,103,99,92,110,105,103,102,104,106,101,101,101,102,103,101,99,104,105,102,102,103,103,107,101,103,108,104,101,101,102)

Mean2 <- mean(data2)
SD2 <- sd(data2)
SE2 <- SD2/sqrt(length(data2))
Tval2 <- Mean2/SE2
Tval2
## [1] 182.4883
TTest2 <-t.test(data2)
TTest2
## 
##  One Sample t-test
## 
## data:  data2
## t = 182.49, df = 30, p-value < 2.2e-16
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  101.4007 103.6960
## sample estimates:
## mean of x 
##  102.5484

Problem 3. Wilcox test

Generate a set of 20 random normal numbers, and compare them to their sorted values with both an independent samples wilcox test, and a paired wilcox test. What value do you get for W for the independent test? Why do the p values differ?

x <- rnorm(20) y <- sort(x)

set.seed(100)
x <- rnorm(20)
y <- sort(x)

wilcox.test(x,y)
## Warning in wilcox.test.default(x, y): cannot compute exact p-value with ties
## 
##  Wilcoxon rank sum test with continuity correction
## 
## data:  x and y
## W = 200, p-value = 1
## alternative hypothesis: true location shift is not equal to 0
wilcox.test(x,y,paired = TRUE)
## Warning in wilcox.test.default(x, y, paired = TRUE): cannot compute exact p-
## value with zeroes
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  x and y
## V = 99, p-value = 0.888
## alternative hypothesis: true location shift is not equal to 0

Problem 4. Comparing t-test, wilcox, and Bayes factor t-test

Generate a set of 150 normal random numbers whose means differ by .3 standard deviation units from another set, but that are uncorrelated to the first set, as in the example below. Run one-sided tests compariing these, including standard t-test, a wilcox test, and a BayesFactor t-test. Do the results differ for the different tests? Repeat several times to determine whether there is a pattern.

library(BayesFactor) x <- rnorm(150) y <- rnorm(150) + .3

library(BayesFactor)
## Warning: package 'BayesFactor' was built under R version 3.6.2
## Loading required package: coda
## Warning: package 'coda' was built under R version 3.6.2
## Loading required package: Matrix
## ************
## Welcome to BayesFactor 0.9.12-4.2. If you have questions, please contact Richard Morey (richarddmorey@gmail.com).
## 
## Type BFManual() to open the manual.
## ************
x1 <- rnorm(150)
y1 <- rnorm(150) + .3

TTest1X<-t.test(x1)
TTest1Y<-t.test(y1)
BayesFactorTest1X<-ttestBF(x1)
WilcoxTest1X<-wilcox.test(x1)
BayesFactorTest1Y<-ttestBF(y1)
WilcoxTest1Y<-wilcox.test(y1)

x2 <- rnorm(150)
y2 <- rnorm(150) + .3

TTest2X<-t.test(x2)
TTest2Y<-t.test(y2)
BayesFactorTest2X<-ttestBF(x2)
WilcoxTest2X<-wilcox.test(x2)
BayesFactorTest2Y<-ttestBF(y2)
WilcoxTest2Y<-wilcox.test(y2)

x3 <- rnorm(150)
y3 <- rnorm(150) + .3

TTest3X<-t.test(x3)
TTest3Y<-t.test(y3)
BayesFactorTest3X<-ttestBF(x3)
WilcoxTest3X<-wilcox.test(x3)
BayesFactorTest3Y<-ttestBF(y3)
WilcoxTest3Y<-wilcox.test(y3)

TTest1X
## 
##  One Sample t-test
## 
## data:  x1
## t = -0.36492, df = 149, p-value = 0.7157
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  -0.1791175  0.1232736
## sample estimates:
##   mean of x 
## -0.02792194
BayesFactorTest1X
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 0.09708886 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTest1X
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  x1
## V = 5335, p-value = 0.5395
## alternative hypothesis: true location is not equal to 0
TTest1Y
## 
##  One Sample t-test
## 
## data:  y1
## t = 3.6128, df = 149, p-value = 0.0004135
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  0.1326106 0.4527985
## sample estimates:
## mean of x 
## 0.2927046
BayesFactorTest1Y
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 42.41651 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTest1Y
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  y1
## V = 7548, p-value = 0.0004051
## alternative hypothesis: true location is not equal to 0
TTest2X
## 
##  One Sample t-test
## 
## data:  x2
## t = -1.0137, df = 149, p-value = 0.3124
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  -0.27154910  0.08740226
## sample estimates:
##   mean of x 
## -0.09207342
BayesFactorTest2X
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 0.1503424 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTest2X
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  x2
## V = 5316, p-value = 0.5162
## alternative hypothesis: true location is not equal to 0
TTest2Y
## 
##  One Sample t-test
## 
## data:  y2
## t = 3.8057, df = 149, p-value = 0.0002059
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  0.1571886 0.4967072
## sample estimates:
## mean of x 
## 0.3269479
BayesFactorTest2Y
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 80.75484 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTest2Y
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  y2
## V = 7574, p-value = 0.0003364
## alternative hypothesis: true location is not equal to 0
TTest3X
## 
##  One Sample t-test
## 
## data:  x3
## t = 0.78158, df = 149, p-value = 0.4357
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  -0.1009243  0.2330035
## sample estimates:
##  mean of x 
## 0.06603961
BayesFactorTest3X
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 0.1226682 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTest3X
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  x3
## V = 6129, p-value = 0.3819
## alternative hypothesis: true location is not equal to 0
TTest3Y
## 
##  One Sample t-test
## 
## data:  y3
## t = 5.6167, df = 149, p-value = 9.273e-08
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  0.3163185 0.6596915
## sample estimates:
## mean of x 
##  0.488005
BayesFactorTest3Y
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 116803.6 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTest3Y
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  y3
## V = 8328, p-value = 5.728e-07
## alternative hypothesis: true location is not equal to 0

Problem 5. Robustness to transforms

For each of the data sets you created in the previous problem, exponen- tiate your samples using the exp function (e.g., exp(x) and exp(y)), and plot the distributions using hist. Then do the same three tests you did originally (t-test, wilcox test, and bayes factor test) on these two exponentiated data sets. How do this test compare to the previous test? Discuss why you see the differences and similarities with Problem 4.

E1x<-exp(x1)
E1y<-exp(y1)
E2x<-exp(x2)
E2y<-exp(y2)
E3x<-exp(x3)
E3y<-exp(y3)

hist(E1x, xlab = "First Exponential X")

hist(E1y, xlab="First Exponential Y")

hist(E2x, xlab="Second Exponential X")

hist(E2y, xlab="Second Exponential Y")

hist(E3x, xlab="Third Exponential X")

hist(E3y, xlab="Third Exponential Y")

TTestE1X<-t.test(E1x)
TTestE1Y<-t.test(E1y)
BayesFactorTestE1X<-ttestBF(E1x)
WilcoxTestE1X<-wilcox.test(E1x)
BayesFactorTestE1Y<-ttestBF(E1y)
WilcoxTestE1Y<-wilcox.test(E1y)
TTestE2X<-t.test(E2x)
TTestE2Y<-t.test(E2y)
BayesFactorTestE2X<-ttestBF(E2x)
WilcoxTestE2X<-wilcox.test(E2x)
BayesFactorTestE2Y<-ttestBF(E2y)
WilcoxTestE2Y<-wilcox.test(E2y)
TTestE3X<-t.test(E3x)
TTestE3Y<-t.test(E3y)
BayesFactorTestE3X<-ttestBF(E3x)
WilcoxTestE3X<-wilcox.test(E3x)
BayesFactorTestE3Y<-ttestBF(E3y)
WilcoxTestE3Y<-wilcox.test(E3y)

TTestE1X 
## 
##  One Sample t-test
## 
## data:  E1x
## t = 10.196, df = 149, p-value < 2.2e-16
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  1.230949 1.822803
## sample estimates:
## mean of x 
##  1.526876
BayesFactorTestE1X 
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 6.738542e+15 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTestE1X
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  E1x
## V = 11325, p-value < 2.2e-16
## alternative hypothesis: true location is not equal to 0
TTestE1Y 
## 
##  One Sample t-test
## 
## data:  E1y
## t = 9.0988, df = 149, p-value = 5.448e-16
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  1.727454 2.685920
## sample estimates:
## mean of x 
##  2.206687
BayesFactorTestE1Y 
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 1.089579e+13 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTestE1Y
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  E1y
## V = 11325, p-value < 2.2e-16
## alternative hypothesis: true location is not equal to 0
TTestE2X 
## 
##  One Sample t-test
## 
## data:  E2x
## t = 7.7018, df = 149, p-value = 1.728e-12
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  1.212466 2.049323
## sample estimates:
## mean of x 
##  1.630894
BayesFactorTestE2X 
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 4257233660 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTestE2X
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  E2x
## V = 11325, p-value < 2.2e-16
## alternative hypothesis: true location is not equal to 0
TTestE2Y
## 
##  One Sample t-test
## 
## data:  E2y
## t = 8.97, df = 149, p-value = 1.165e-15
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  1.901302 2.975647
## sample estimates:
## mean of x 
##  2.438474
BayesFactorTestE2Y
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 5.189065e+12 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTestE2Y
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  E2y
## V = 11325, p-value < 2.2e-16
## alternative hypothesis: true location is not equal to 0
TTestE3X 
## 
##  One Sample t-test
## 
## data:  E3x
## t = 10.823, df = 149, p-value < 2.2e-16
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  1.442847 2.087398
## sample estimates:
## mean of x 
##  1.765123
BayesFactorTestE3X 
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 2.836253e+17 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTestE3X
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  E3x
## V = 11325, p-value < 2.2e-16
## alternative hypothesis: true location is not equal to 0
TTestE3Y 
## 
##  One Sample t-test
## 
## data:  E3y
## t = 9.8115, df = 149, p-value < 2.2e-16
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  2.229411 3.353873
## sample estimates:
## mean of x 
##  2.791642
BayesFactorTestE3Y 
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 6.96364e+14 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
WilcoxTestE3Y
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  E3y
## V = 11325, p-value < 2.2e-16
## alternative hypothesis: true location is not equal to 0

Problem 6. Comparison to paired tests.

For the original data (not the exponentiated data), run a paired t-test, as well as a paired wilcox test, and a paired BayesFactor test (even though the data were not paired).

Discuss what the findings were and how they compare to the unpaired versions of the tests.

t.test(x1,y1,paired = TRUE)
## 
##  Paired t-test
## 
## data:  x1 and y1
## t = -2.8529, df = 149, p-value = 0.004949
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.5427029 -0.0985501
## sample estimates:
## mean of the differences 
##              -0.3206265
ttestBF(x1,y1,paired = TRUE)
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 4.452403 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
wilcox.test(x1,y1,paired = TRUE)
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  x1 and y1
## V = 4150, p-value = 0.004556
## alternative hypothesis: true location shift is not equal to 0
t.test(x2,y2,paired = TRUE)
## 
##  Paired t-test
## 
## data:  x2 and y2
## t = -3.7135, df = 149, p-value = 0.0002883
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.6419872 -0.1960554
## sample estimates:
## mean of the differences 
##              -0.4190213
ttestBF(x2,y2,paired = TRUE)
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 59.16236 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
wilcox.test(x2,y2,paired = TRUE)
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  x2 and y2
## V = 3787, p-value = 0.0004349
## alternative hypothesis: true location shift is not equal to 0
t.test(x2,y2,paired = TRUE)
## 
##  Paired t-test
## 
## data:  x2 and y2
## t = -3.7135, df = 149, p-value = 0.0002883
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  -0.6419872 -0.1960554
## sample estimates:
## mean of the differences 
##              -0.4190213
ttestBF(x2,y2,paired = TRUE)
## Bayes factor analysis
## --------------
## [1] Alt., r=0.707 : 59.16236 ±0%
## 
## Against denominator:
##   Null, mu = 0 
## ---
## Bayes factor type: BFoneSample, JZS
wilcox.test(x2,y2,paired = TRUE)
## 
##  Wilcoxon signed rank test with continuity correction
## 
## data:  x2 and y2
## V = 3787, p-value = 0.0004349
## alternative hypothesis: true location shift is not equal to 0

Problem 7: Correlations

Suppose you have a true correlation of .3 between two variables, created by mixing one uniform with a second like this:

x <- runif(100) y <- runif(100)

z <- x + .5y z1 <- z z2 <- z+10 z3 <- log(z) z4 <- z10 z5 <- z + runif(100) Compute pearson and spearman correlations between x and each of the z variables z1 through z5. Identify the statistical significance of the correlations, and report a Bayes factor for each correlation as well. Examine the pearson and spearman correlations, and discuss why for some transforms, these are unchanged, but for other transforms they are changed.

x <- runif(100)
y <- runif(100)

z <- x + .5*y
z1 <- z
z2 <- z+10
z3 <- log(z)
z4 <- z*10
z5 <- z + runif(100)

C1S<-cor(x,z1,method="spearman")
C2S<-cor(x,z2,method="spearman")
C3S<-cor(x,z3,method="spearman")
C4S<-cor(x,z4,method="spearman")
C5S<-cor(x,z5,method="spearman")

C1P<-cor(x,z1,method="pearson")
C2P<-cor(x,z2,method="pearson")
C3P<-cor(x,z3,method="pearson")
C4P<-cor(x,z4,method="pearson")
C5P<-cor(x,z5,method="pearson")

C1S
## [1] 0.8959376
C1P
## [1] 0.8989101
C2S
## [1] 0.8959376
C2P
## [1] 0.8989101
C3S
## [1] 0.8959376
C3P
## [1] 0.7666697
C4S
## [1] 0.8959376
C4P
## [1] 0.8989101
C5S
## [1] 0.7127513
C5P
## [1] 0.7341037
correlationBF(x,z1)
## Bayes factor analysis
## --------------
## [1] Alt., r=0.333 : 2.630849e+32 ±0%
## 
## Against denominator:
##   Null, rho = 0 
## ---
## Bayes factor type: BFcorrelation, Jeffreys-beta*
correlationBF(x,z2)
## Bayes factor analysis
## --------------
## [1] Alt., r=0.333 : 2.630849e+32 ±0%
## 
## Against denominator:
##   Null, rho = 0 
## ---
## Bayes factor type: BFcorrelation, Jeffreys-beta*
correlationBF(x,z3)
## Bayes factor analysis
## --------------
## [1] Alt., r=0.333 : 1.340846e+17 ±0%
## 
## Against denominator:
##   Null, rho = 0 
## ---
## Bayes factor type: BFcorrelation, Jeffreys-beta*
correlationBF(x,z4)
## Bayes factor analysis
## --------------
## [1] Alt., r=0.333 : 2.630849e+32 ±0%
## 
## Against denominator:
##   Null, rho = 0 
## ---
## Bayes factor type: BFcorrelation, Jeffreys-beta*
correlationBF(x,z5)
## Bayes factor analysis
## --------------
## [1] Alt., r=0.333 : 7.64944e+14 ±0%
## 
## Against denominator:
##   Null, rho = 0 
## ---
## Bayes factor type: BFcorrelation, Jeffreys-beta*