Hypothesis Testing and Choosing a Statistical Technique
Prof. Dr. Ergül Demir
Ankara
University, Department of Educational Measurement and
Evaluation
ORCID
0000-0002-3708-8013
9 November 2026
This document has two sections. The first explains the logic of hypothesis testing and the five steps of conducting a hypothesis test. The second gives a brief classification of statistical techniques, with example research questions drawn mainly from education and psychology. Each section ends with a few short questions, and the next section opens only when all of them are answered correctly.
Section 1. The Logic and Steps of Hypothesis Testing
1.1 What is a hypothesis test?
In research we rarely have access to the whole population. We collect data from a sample and want to know whether what we see in the sample reflects something real in the population, or whether it could simply be the result of sampling error. A hypothesis test is a formal decision procedure for this question. It tells us how likely our sample result would be if there were in fact no effect, no difference or no relationship in the population.
1.2 Null and alternative hypotheses
Every hypothesis test starts with two competing statements about the population.
The null hypothesis (\(H_0\)) states that there is no effect, no difference or no relationship. For example, the mean mathematics achievement of female and male students is equal in the population (\(H_0: \mu_{female} = \mu_{male}\)).
The alternative hypothesis (\(H_1\)) states that there is an effect, a difference or a relationship. For example, the mean mathematics achievement of female and male students differs in the population (\(H_1: \mu_{female} \neq \mu_{male}\)).
The alternative hypothesis can be non-directional (two-tailed), as in the example above, or directional (one-tailed), for example students who receive feedback score higher than students who do not (\(H_1: \mu_{feedback} > \mu_{no\ feedback}\)). The direction should come from theory or previous research, not from the data at hand.
The test only asks whether the data are strong enough to reject \(H_0\). If they are, we reject \(H_0\) and support \(H_1\). If they are not, we fail to reject \(H_0\). This does not prove that \(H_0\) is true. It only means the data do not give enough evidence against it.
1.3 Supporting concepts
Significance level, alpha (\(\alpha\)). The probability of rejecting \(H_0\) when it is actually true, decided by the researcher before the analysis. In education and psychology \(\alpha\) is usually .05, and sometimes .01.
Confidence level. The complement of \(\alpha\) (\(1 - \alpha\)). With \(\alpha = .05\) the confidence level is 95%. A 95% confidence interval gives a range of plausible values for the population parameter.
Type I error. Rejecting \(H_0\) when it is true, a false positive. Its probability is \(\alpha\).
Type II error. Failing to reject \(H_0\) when it is false, a false negative. Its probability is \(\beta\).
| Reality in the population | ||
|---|---|---|
| Our decision | H0 is true | H0 is false |
| Reject H0 | Type I error false positive, probability α |
Correct decision power, probability 1 − β |
| Fail to reject H0 | Correct decision probability 1 − α |
Type II error false negative, probability β |
Two of the four outcomes of a hypothesis test are errors.
Statistical power. The probability of correctly rejecting a false \(H_0\) (\(1 - \beta\)). Power increases with larger samples, larger effects and a higher \(\alpha\).
A larger sample or a larger effect moves the two curves apart, which shrinks β and increases power.
p-value. The probability of obtaining a result at least as extreme as the observed one, assuming \(H_0\) is true. A small p-value (close to zero, in practice smaller than \(\alpha\), for example smaller than .05) means the observed result would be unusual if \(H_0\) were true.
H0 is rejected only when the test statistic falls in a shaded tail (rejection region).
Effect size. A measure of how large the effect or difference is, independent of sample size (for example Cohen’s d, eta squared or r). Statistical significance alone does not tell us whether a result is practically important.
1.4 The five steps of conducting a hypothesis test
If assumptions are violated in Step 2, a robust or nonparametric technique can be chosen in Step 3.
Step 1. Formulate the hypotheses. Start from the research question, write \(H_0\) and \(H_1\), decide whether the test is one-tailed or two-tailed, and set the significance level (and so the confidence level) before looking at the results.
Step 2. Check the assumptions. Each technique rests on assumptions about the data. Depending on the technique, these include the level of measurement of the variables, sample size, independence of observations, normality, homogeneity of variances, and linearity. If the assumptions are seriously violated, a transformation, a robust method or a nonparametric alternative may be needed.
Step 3. Choose the appropriate technique and run the analysis. The choice depends on the research question, the number and type of variables, the design (independent groups or repeated measures) and the results of the assumption checks. Section 2 gives a brief guide to this choice.
Step 4. Obtain the test statistic and make a decision. The output gives a test statistic (for example t, F, chi-square or r), its degrees of freedom and its p-value. This statistic summarises how far the sample result is from what \(H_0\) predicts. Compare the p-value with \(\alpha\), or the test statistic with the critical value. If p is less than or equal to \(\alpha\), reject \(H_0\). If p is greater than \(\alpha\), fail to reject \(H_0\).
Step 5. Report and interpret the results. Explain what the decision means in terms of the research question, and report the test statistic, degrees of freedom, p-value, effect size and, where useful, confidence intervals, following APA style. When there are several outputs to report, present them in a table rather than in the text.
1.5 Check your understanding
Answer all of the questions below correctly to open Section 2.
1. Which hypothesis is directly tested in a hypothesis test?
2. A researcher rejects H0, but in the population there is in fact no difference. Which error has occurred?
3. With α = .05, the test gives p = .032. What is the decision?
4. Which step comes directly before choosing and running the appropriate technique?
5. Mark each statement as true or false.
| True | False | |
|---|---|---|
| A non-significant result shows that H0 is true. | ||
| The significance level α is decided before the analysis. | ||
| A smaller p-value means stronger evidence against H0. | ||
| A statistically significant result is always practically important. |
Section 2. Choosing a Statistical Technique
Choosing the right technique is Step 3 of a hypothesis test. The choice depends mainly on the purpose of the analysis, the number of variables, their level of measurement, and whether the data come from independent groups or from the same participants measured more than once. Here we group the most common techniques into five branches. For a detailed guide covering more than 200 techniques, use the Statistical Technique Advisor.
The five branches are
- Comparing independent groups
- Comparing repeated measures
- Examining relationships between variables
- Classification methods
- Latent variable models
Many techniques in these branches are parametric, which means they assume normality and related conditions. When these assumptions are violated, a parametric technique is not automatically ruled out. Researchers can first consider corrections such as data transformations, or robust versions of the technique. Nonparametric alternatives are another option, especially when the data are ordinal.
2.1 Comparing independent groups
Use these techniques when different participants are in each group and you compare them on a dependent variable.
(Wilcoxon rank-sum)
Example research questions
- Independent-samples t-test, Welch t-test, or Mann-Whitney U
(Wilcoxon rank-sum) test. Do female and male students differ in
mathematics achievement?
- \(H_0: \mu_{female} = \mu_{male}\)
- \(H_1: \mu_{female} \neq \mu_{male}\)
- One-way ANOVA, Welch ANOVA, or Kruskal-Wallis H test. Does reading
comprehension differ across three teaching methods?
- \(H_0: \mu_1 = \mu_2 = \mu_3\)
- \(H_1\): at least one mean differs from the others
- Factorial ANOVA. Do teaching method and gender jointly affect
science achievement?
- Main effect of method
- \(H_0: \mu_{method1} = \mu_{method2} = \mu_{method3}\)
- \(H_1\): at least one method mean differs from the others
- Main effect of gender
- \(H_0: \mu_{female} = \mu_{male}\)
- \(H_1: \mu_{female} \neq \mu_{male}\)
- Interaction
- \(H_0\): there is no method × gender interaction
- \(H_1\): there is a method × gender interaction
- Main effect of method
- ANCOVA. Do teaching methods differ in posttest scores after
controlling for pretest scores?
- \(H_0: \mu'_1 = \mu'_2 = \mu'_3\)
- \(H_1\): at least one adjusted mean differs from the others
- MANOVA. Do school types differ in mathematics and reading
achievement together?
- \(H_0\): the mean vectors of mathematics and reading are equal across school types
- \(H_1\): at least one mean vector differs from the others
- MANCOVA. Do school types differ in mathematics and reading
achievement after controlling for socioeconomic status?
- \(H_0\): the adjusted mean vectors are equal across school types
- \(H_1\): at least one adjusted mean vector differs from the others
2.2 Comparing repeated measures
Use these techniques when the same participants are measured more than once, or when participants are matched in pairs.
Example research questions
- Paired-samples t-test, Yuen paired t-test, or Wilcoxon signed-rank
test. Do students’ anxiety scores change from before to after a
mindfulness program?
- \(H_0: \mu_{pre} = \mu_{post}\)
- \(H_1: \mu_{pre} \neq \mu_{post}\)
- Repeated measures ANOVA, robust repeated measures ANOVA, or Friedman
test. Does vocabulary knowledge change across pretest, posttest and
follow-up?
- \(H_0: \mu_{pre} = \mu_{post} = \mu_{follow\text{-}up}\)
- \(H_1\): at least one mean differs from the others
- Mixed ANOVA or robust mixed ANOVA. Does the change from pretest to
posttest differ between experimental and control groups?
- Main effect of time
- \(H_0: \mu_{pre} = \mu_{post}\)
- \(H_1: \mu_{pre} \neq \mu_{post}\)
- Main effect of group
- \(H_0: \mu_{experimental} = \mu_{control}\)
- \(H_1: \mu_{experimental} \neq \mu_{control}\)
- Interaction
- \(H_0\): there is no time × group interaction
- \(H_1\): there is a time × group interaction
- Main effect of time
- Repeated measures ANCOVA. Does students’ motivation change across
three measurements after controlling for prior achievement?
- \(H_0: \mu'_{time1} = \mu'_{time2} = \mu'_{time3}\)
- \(H_1\): at least one adjusted mean differs from the others
- Repeated measures MANOVA. Do students’ mathematics and reading
scores change from the beginning to the end of the school year?
- \(H_0\): the mean vectors of mathematics and reading are equal across time points
- \(H_1\): at least one mean vector differs from the others
- Repeated measures MANCOVA. Do students’ mathematics and reading
scores change across the school year after controlling for socioeconomic
status?
- \(H_0\): the adjusted mean vectors of mathematics and reading are equal across time points
- \(H_1\): at least one adjusted mean vector differs from the others
2.3 Examining relationships between variables
Use these techniques when you want to know whether variables are related, how strongly, and whether one variable can be predicted from others. The choice of correlation coefficient depends on the level of measurement of the variables.
| Technique | Use when | Example research question |
|---|---|---|
| Pearson correlation | Two continuous variables with a linear relationship | Is study time related to exam scores? |
| Spearman rank correlation | Two ordinal variables, or continuous variables that are not normal | Is class rank related to teacher ratings of participation? |
| Point-biserial correlation | One continuous and one truly dichotomous variable | Is gender related to self-efficacy scores? |
| Chi-square test of independence | Two categorical variables | Is school type related to students’ choice of university program? |
| Partial correlation | Relationship between two variables controlling for a third | Is reading time related to vocabulary after controlling for age? |
| Simple linear regression | Predicting a continuous outcome from one predictor | Does weekly homework time predict mathematics achievement? |
| Multiple linear regression | Predicting a continuous outcome from two or more predictors | Do motivation, self-efficacy and prior achievement predict final exam scores? |
2.4 Classification methods
Use these techniques when the aim is to assign cases to groups. Logistic regression and discriminant function analysis predict membership in groups that are already known. Cluster analysis finds groups that are not known in advance.
| Technique | Use when | Example research question |
|---|---|---|
| Binary logistic regression | Predicting a two-category outcome | Do attendance and motivation predict whether a student passes or fails the course? |
| Multinomial logistic regression | Predicting an outcome with three or more unordered categories | Do interests and achievement predict students’ choice among science, social science and arts programs? |
| Ordinal logistic regression | Predicting an outcome with three or more ordered categories | Do study habits and self-efficacy predict students’ performance level (low, medium, high)? |
| Discriminant function analysis | Finding the combination of variables that best separates known groups | Which cognitive test scores best separate students with and without learning difficulties? |
| Cluster analysis | Grouping cases with similar profiles when groups are unknown | Can students be grouped into distinct profiles based on their learning strategies? |
2.5 Latent variable models
Use these techniques when the variables of interest cannot be observed directly, such as intelligence, anxiety or motivation, and are measured through several observed items or indicators. In these models there is often no single observed dependent variable.
| Technique | Use when | Example research question |
|---|---|---|
| Exploratory factor analysis (EFA) | Discovering the underlying factor structure of a set of items | How many factors underlie the items of a new test anxiety scale? |
| Confirmatory factor analysis (CFA) | Testing a factor structure defined in advance | Does the five-factor structure of a personality inventory fit the data from Turkish university students? |
| Structural equation modeling (SEM) | Testing relationships among latent and observed variables | Does academic self-efficacy mediate the effect of teacher support on achievement? |
| Item response theory (IRT) | Modeling item responses and person ability on a common scale | How do item difficulty and discrimination differ across the items of a mathematics test? |
| Latent class or latent profile analysis | Identifying unobserved subgroups from response patterns | Are there hidden subgroups of students with different patterns of engagement? |
2.6 Check your understanding
Answer all of the questions below correctly to complete this document.
1. A researcher compares the reading scores of students taught with three different methods. Each student is taught with only one method. Which technique is appropriate?
2. The same students complete a stress scale at the beginning, middle and end of the semester, and the scores are normally distributed. Which technique is appropriate?
3. A researcher wants to know whether the relationship between two ordinal variables is significant. Which technique is appropriate?
4. A school counselor wants to predict whether a student will drop out (yes or no) from attendance, grades and family support. Which technique is appropriate?
5. For each research purpose, choose the correct branch from the drop-down list.
| Testing whether the items of a scale measure three predefined factors | |
| Comparing pretest and posttest scores of the same students | |
| Grouping students into learning profiles that are not known in advance | |
| Comparing anxiety scores of students from public and private schools |
Well done
You have completed all sections and answered all questions correctly. You can come back to this document whenever you need it. When you choose a technique for your own research, you can also use the Statistical Technique Advisor.