2026-09-13

P-value

  • We must first set a Null Hypothesis: H0
  • We must also set up an Alternate hypothesis: Ha, the opposite of the Null.
  • If the result is less than the Significance Level (\(\alpha\)), it suggests a real effect.

Example: Is a Die fair?

Null Hypothesis \[H_0:p_1=p_2=\cdots=p_6=\frac{1}{6}\] Alternate Hypothesis \[H_a: \text{at least one } p_i \neq \frac{1}{6}\] We’ll roll a six sided die 100 and 1000 times and record how many times each face comes up.

Die Roll Simulation

set.seed(1234)
rolls <- sample(1:6, size=100, replace=TRUE)
rolls1k <- sample(1:6, size=1000, replace=TRUE)
observed <- table(rolls)
observed1k <- table(rolls1k)
observed
## rolls
##  1  2  3  4  5  6 
## 15 14 14 18 14 25
observed1k
## rolls1k
##   1   2   3   4   5   6 
## 161 152 193 152 161 181

100 rolls Plot

100 vs 1000 Rolls

Chi-Square Test

We use the Chi-Square Goodness of Fit test to see that our die is fair. It tells us whether the distribution of a single categorical variable matches a theoretical or expected distribution.

We chose \(\alpha= 0.05\) and we use that to find our Critical Value

We also need to find our degrees of freedom, which is:

\[df = n-1 = 6-1 = 5\]

Mathematically the test looks like:

\[\chi^2 = \sum \frac{(O-E)^2}{E}\]

O = observed output

E = expected output

Applying Chi Square to Data

With our example it will look something like this (16.667~ \(\frac{1}{6}*100\))

\[\chi^2 = \sum \frac{(15-16.67)^2}{16.67} + \frac{(14-16.67)^2}{16.67} + \frac{(14-16.67)^2}{16.67} + \frac{(18-16.67)^2}{16.67} + \frac{(14-16.67)^2}{16.67} + \frac{(25-16.67)^2}{16.67}\]

\(\chi^2\) ~5.72 in our 100 rolls example.

This gives us our data’s observed statistic.

Getting the P-Value

Here we run our Chi Square Test on the data we recorded earlier.

    Chi-squared test for given probabilities

data:  observed
X-squared = 5.72, df = 5, p-value = 0.3344
    Chi-squared test for given probabilities

data:  observed1k
X-squared = 8.36, df = 5, p-value = 0.1375

Visualizing the P-value

The blue region is our p-value, the probability of seeing a result this extreme or more if the die were fair. The red region is the rejection region, this is = .05.

Alternate Hypothesis

Had the observed statistic fallen on the other side of the red line we would reject H0 and have adopted the Ha.

Statistical significance does not mean practical significance, we may have just observed random variaton within the population.

However since our blue region (p-value) is greater than the red region(\(\alpha\)), our result isn’t extreme enough to reject H0.