General Testing

If our sample is random, we know the the distribution of specific sample characteristics. Hence, if, and only if, our sample is random, we know the probability for the random occurrence of these characteristics, such as differences in means of subsamples. If the probability for a variation being random is very low (say below 5%) it seems unlikely, that our sample is random. Thus, in addition to mere random effects at least one systematic effect exists that accounts for this variation.

General process for hypothesis testing:

In the fopllowing, this process will be outlined in detail by referring to different hypotheses. We work with a small sample (n=50) from ESS8 data, namely the variables gender of respondent (gndr), self-rated health (health), highest level of education (edlv), and tolerance towards gays and lesbians (multi-item scale tolerance).

t-Test: Test if two population means are different

General test idea

If two population means do not differ, two samples A and B from this population should have two (more or less) equal means: MeanA\(\approx\)MeanB. In other words: (MeanA-MeanB)\(\approx\) 0.

If we (theoretically) drew all samples of same size than A and B, the differences (MeanA-MeanB) were symmetrically distributed with mean 0. We divide these differences by their standard error SE (i.e. the standard deviaton of a theoretical function) to normalize (standardize) the differences to the average difference. The resulting theoretical distribution is known as t-distribution.

It describes the probability for the random occurence of a given difference between two sample means under the condition that both samples were drawn from the same population, i.e. that their difference in the population is actually zero.

In other words, if we assume both samples were drawn from the same population, we can calculate the likelihood for the random occurence of our sampled difference between two means. Most differences are very close to zero. The more the differences deviate from zero towards negative or positive values, the less likely the respective sample pairs are (the less often they occur when drawn randomly from the same population).

In practice, we define the maximum error probability we are willing to accept in our analysis. This user defined threshold is called significance level \(\alpha\). Based on this we can clearly decide, if our sampled values are too unlikely to be drawn randomly. We then calculate the actual probability to draw our sample randomly from a homogeneous population were no difference between the sampled means can be observed. This probability is called the significance \(p\) of your sample. It can determined by applying the theoretical t-distribution as a “look-table” for the sample-specific t-Value.

In social sciences, significance levels of 5% or 1% are quite common. At a significance level of \(\alpha=5%\) we we are wrong in one out of every 20 test decisions on average. However, if you are building a nuclear power plant or performing heart surgery, then you should work with significantly smaller significance levels.

If the significance is greater than the significance level, there is nothing against the initial assumption that our sample was drawn randomly from a homogeneous population.

If the significance is lower than the significance level we have to question our initial premise: We made an assumption and must now acknowledge that our result (our sample) is quite unlikely under this assumption. Then we have two options: Either we doubt our own sample (for example, we may have made methodological errors in the sampling process), or we doubt the assumption that led to this result. The former is always reasonable, of course. However, if we find no errors, then we must reject the basic assumption that our sample comes randomly from a homogeneous population. Conversely, we must acknowledge that ou sample was not drawn radomly from a homogeneous population, i.e. that the differences in our sample means are not random, that there is a significant differnce between the groups in the population, that group membership has a significant effect on the variable in question.

The initial assumption that everything we observe in our sample is just random is called the null-hypothesis (H0). In most cases we are interested in the rejection of H0, because we want to show that our observations are not random.

Note, that H0 can not be proven but only be rejected. In other words, the fact that H0 cannot be rejected does not necessarily mean that H0 is true. If judiciary sends me to jail because of murder they’re sufficiently certain, that I am guilty. However, if they don’t jail me for want of evidence, it doesn’t mean that I am innocent. They simply are not save enough to jail me although I might still be a murder. Same with H0: If we can’t reject it, it may still be false, i.e. the effect we’re after might still exist.

Example: Does tolerance towards gays and lesbians differ between gender?

# create survey design
essDesign <- svydesign(id=~1, data=essData, weights=~essData$w)

# weighted sample means of tolerance by gender
svyby(~tolerance, ~gndr, essDesign, svymean, na.rm=T)
##          gndr tolerance        se
## Male     Male  3.588056 0.4195371
## Female Female  3.134684 0.4194148

We observe a clear difference in tolerance levels between male (3.588) and female (3.135) in our sample, indicating that female are less tolerant than male. The difference in means is 3.13 - 3.59 = -.453. Does this difference deviate sufficiently far from zero to be able to say that it is not random? The t-Test provides an answer:

# test, if tolerance towards gay and lesbians differs between gender
svyttest(tolerance~gndr, essDesign)
## 
##  Design-based t-test
## 
## data:  tolerance ~ gndr
## t = -0.76425, df = 48, p-value = 0.4485
## alternative hypothesis: true difference in mean is not equal to 0
## 95 percent confidence interval:
##  -1.6461384  0.7393936
## sample estimates:
## difference in mean 
##         -0.4533724

Although we observe a clear differences in tolerance levels between male and female in our sample, this difference is quite likely to occur randomly (\(p\)=.449). If we test on a significance level of \(\alpha\) =5%=.05 we see that \(p > \alpha\), indicating that there is no significant difference in tolerance between gender in the population.

Degrees of freedom

Note that the t-distribution for our test depends on the size of our sample (n=50). More precisely, the shape of the theoretical distribution depends on the number of degrees of freedom (df=50-2=48) of our sample. What does this mean?

Imagine a sample of n=3, e.g. tolerance levels of three individuals, namely 3, 4, and 5. This results in a sample mean of 4 and a sample standard deviation of 0.81 (square root of 2/3). We can freely choose who we include in our survey, so all three sample values are free, i.e. we had three degrees of freedom when collecting our sample data.

Now, if we draw conclusions to a population we must take all parameters into account that are used in our argumentation. As we know that our sample mean is 4 and we freely draw two answers, say 3 and 4, the third answer must be 5 in order to match the constraint that the average of all three answers is 4. If we have a sample of three with a mean of 4, we can freely chose two sample data. Estmating the population mean, thus, decreases the number of freely vaying sample values by one. Same is true for the second parameter: When estimating the standard error we make use of the sample standard deviation. In a sample of three with a given mean and standard error, we can freely choose one element.

In general, degrees of freedom are equal to the sample size minus number of estimated parameters.

ANOVA: Test if three or more population means are different

General test idea