Main assumptions

When performing binomial test for Bernoulli experiment one can measure belief in the outcome with a little help from Bayes’ rule (http://en.wikipedia.org/wiki/Bayes'_rule). Suppose we have 5 trials for our experiment with two possible outcomes (success and failure) with equal probabilities at the beginning.

library(binom)
library(Bolstad)
library(pwr)
library(ggplot2)

First trial: 1/5

After the first successful trial if we assume the four other trials would be failed our belief can be expressed by posterior beta distribution like this.

bs=binom.bayes(x = 1,n = 5,maxit = 1000)
print(bs)
##   method x n shape1 shape2 mean       lower     upper  sig
## 1  bayes 1 5    1.5    4.5 0.25 0.001709995 0.5639827 0.05
binom.bayes.densityplot(bs,500)

Note
Compare these results with binom.test function.

binom.test(1,5,0.5)
## 
##  Exact binomial test
## 
## data:  1 and 5
## number of successes = 1, number of trials = 5, p-value = 0.375
## alternative hypothesis: true probability of success is not equal to 0.5
## 95 percent confidence interval:
##  0.005050763 0.716417936
## sample estimates:
## probability of success 
##                    0.2

So Bayes is greedy with the upper limit \(0.56\) but generous with the mean \(0.25\).

Second trial: 2/5

After two successful outcomes if we assume the three other trials would be failed our belief should be like this.

bs=binom.bayes(x = 2,n = 5,maxit = 1000)
print(bs)
##   method x n shape1 shape2      mean      lower     upper        sig
## 1  bayes 2 5    2.5    3.5 0.4166667 0.07677318 0.7681765 0.05000001
binom.bayes.densityplot(bs,500)

Note
Compare these results with binom.test function.

binom.test(2,5,0.5)
## 
##  Exact binomial test
## 
## data:  2 and 5
## number of successes = 2, number of trials = 5, p-value = 1
## alternative hypothesis: true probability of success is not equal to 0.5
## 95 percent confidence interval:
##  0.05274495 0.85336720
## sample estimates:
## probability of success 
##                    0.4

Mean probability of success \(0.42\) is close to estimate by binom.test - \(0.4\)

Third trial: 3/5

After three successful outcomes if we assume the two other trials would be failed our belief should be more distinct.

bs=binom.bayes(x = 3,n = 5,maxit = 1000)
print(bs)
##   method x n shape1 shape2      mean     lower     upper        sig
## 1  bayes 3 5    3.5    2.5 0.5833333 0.2318235 0.9232268 0.05000001
binom.bayes.densityplot(bs,500)

Note
Compare these results with binom.test function.

binom.test(3,5,0.5)
## 
##  Exact binomial test
## 
## data:  3 and 5
## number of successes = 3, number of trials = 5, p-value = 1
## alternative hypothesis: true probability of success is not equal to 0.5
## 95 percent confidence interval:
##  0.1466328 0.9472550
## sample estimates:
## probability of success 
##                    0.6

Mean probability of success \(0.58\) is close to estimate by binom.test - \(0.6\), while upper limits by both methods are getting closer too: \(0.92\), \(0.95\).

Fourth trial: 4/5

After four successful outcomes if we assume the last one would be failed our belief should be even more powerful.

bs=binom.bayes(x = 4,n = 5,maxit = 1000)
print(bs)
##   method x n shape1 shape2 mean     lower   upper  sig
## 1  bayes 4 5    4.5    1.5 0.75 0.4360173 0.99829 0.05
binom.bayes.densityplot(bs,500)

Note
Compare these results with binom.test function. Mind the tiny difference in the upper limits by two methods (\(0.998\), \(0.995\)), while Bayesian mean \(0.75\) is getting off binom.test estimate - \(0.8\).

binom.test(4,5,0.5)
## 
##  Exact binomial test
## 
## data:  4 and 5
## number of successes = 4, number of trials = 5, p-value = 0.375
## alternative hypothesis: true probability of success is not equal to 0.5
## 95 percent confidence interval:
##  0.2835821 0.9949492
## sample estimates:
## probability of success 
##                    0.8

Fifth trial: 5/5

At last after five successful outcomes our belief should be .

bs=binom.bayes(x = 5,n = 5,maxit = 1000)
print(bs)
##   method x n shape1 shape2      mean     lower upper  sig
## 1  bayes 5 5    5.5    0.5 0.9166667 0.6942544     1 0.05
binom.bayes.densityplot(bs,500)

Note
Compare these results with binom.test function. Mind the difference in the lower limits by two methods: \(0.694\), \(0.478\), while the mean by Bayes is \(0.917\) and upper limits by both methods are equal \(1.0\).

binom.test(5,5,0.5)
## 
##  Exact binomial test
## 
## data:  5 and 5
## number of successes = 5, number of trials = 5, p-value = 0.0625
## alternative hypothesis: true probability of success is not equal to 0.5
## 95 percent confidence interval:
##  0.4781762 1.0000000
## sample estimates:
## probability of success 
##                      1

Conclusions

So Bayes makes it clear with our odds for the posterior outcome but leaves some skeptics about 100% success of the whole experiment. By the way the power of our experiment with 5 trials for the mean probability of success \(0.917\) is \(60\)% only while we would get \(92\)% power for \(0.999\) probability of success under null hypothesis if necessary.

pwr.p.test(h = ES.h(p1 = 0.917, p2 = 0.5),n = 5)
## 
##      proportion power calculation for binomial distribution (arcsine transformation) 
## 
##               h = 0.9863179
##               n = 5
##       sig.level = 0.05
##           power = 0.5969847
##     alternative = two.sided
pwr.p.test(h = ES.h(p1 = 0.999, p2 = 0.5),n = 5)
## 
##      proportion power calculation for binomial distribution (arcsine transformation) 
## 
##               h = 1.50754
##               n = 5
##       sig.level = 0.05
##           power = 0.9208775
##     alternative = two.sided

Believe it or not is your choice.