Reviewing Last Quarter

Choosing the right technique

Techniques & when to use them
Purpose Technique Notes
compare 2 means (quantitative/ numeric) T-test
compare 2 or more categories (qualitative/ counts) chi square test
compare continuous variable ANOVA analysis

Pearson’s Chi Square Test

  • Used for qualitative (categorical) comparisons.
  • You are testing: are these different categories independent = not related at all? or are they dependent = there is some connection between them?
  • For example, compare the numbers for clicked/ not clicked on 3 different websites.
  • You can also use this to test your model– you compare your expected results (based on your model) to your actual results and see if they are connected

More info: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3900058/

Assumptions:
  • Cell should be counts or frequencies, not percentages
  • The categories or levels are mutually exclusive
  • Each subject may only contribute to one cell, a customer visits many times to the store, then Chi Square cannot be used * The groups have to be independent of each other, they cannot be paired
  • There are two variables, both nominal but can be ordinal
  • Total cases should be the number of cells times 5, or each cell should have expected count larger than 5
  • Number of cells is not limited but usually not larger than 20
Chi Square Distributions:
  • The hypothesis is that probabilities for matching categories are ALL equal
  • Null hypothesis: \(H_0: \mu_1 = \mu_2 = \mu_3\) for each category
  • Alternative hypothesis: \(H_A\) : some of the means are different
  • We have two variables, say smoking and lung cancer, we answer the question of if they have an association or not
  • Can also be framed as: are they independent?
Process (manual– this is simpler with R)

  • Collect your data, and summarize it in a two-way table. These numbers represent the observed cell counts.

  • Set up your null hypothesis, Ho: Variables are independent; and the alternative hypothesis, Ha: Variables are dependent.

  • Calculate the expected cell counts under the assumption of independence.

  • Check the conditions of the Chi-square test before proceeding; each expected cell count must be greater than or equal to five.

  • Figure the Chi-square test statistic. This statistic finds the observed cell count minus the expected cell count, squares the difference, and divides it by the expected cell count. Do these steps for each cell, and then add them all up.

  • Look up your test statistic on the Chi-square table (Table A-3 in the appendix) and find the p-value (or one that’s close).

  • If your result is less than your predetermined cutoff (the α level), usually 0.05, reject Ho and conclude dependence of the two variables.

Example of a two-way table
  • You can calculate different conditional probablities based on the table– see calculations next to the table

example with table

The Math:
  • Sum of sqaures and degrees of freedom
  • \(\chi^2 = \sum ij (obs-expected)^2/(expected)\) where i and j are the rows (category) and columns (samples)
  • Expected is calculated from the totals for all samples for each category
  • Calculated expected values should be greater than 5
  • Degrees of freedom is (row – 1)(col – 1) = (I– 1)(J – 1)
Graph of Distribution:

image of graph

Running the Test In R Code:
datamatrix = matrix(c(4700,3500,4300, 3300, 1500,1000), nrow = 3, ncol = 2, byrow = TRUE, dimnames=list(c("treatment1", "treatment2", "control"), c("died", "alive")))

datamatrix
##            died alive
## treatment1 4700  3500
## treatment2 4300  3300
## control    1500  1000
chisq.test(datamatrix)
## 
##  Pearson's Chi-squared test
## 
## data:  datamatrix
## X-squared = 9.0245, df = 2, p-value = 0.01097
print("Conclusion: at least one is different because the p-value is < 5%.")
## [1] "Conclusion: at least one is different because the p-value is < 5%."

Bonferroni Adjustment– use for everything

  • Not related to chi square test
  • It is a way of adjusting the confidence level (alpha)– where the standard would be 5%, in reality we need to make it lower
  • Goal is to reduce type 1 error
  • remember: Type 1 error= disprove H0 when H0 is true= stop production when everything is fine = 0.05 = 5/100 times if you do this test you will get a false alarm
Why do we need this?

Why Bonferroni? It’s designed to avoid overanalyzing data until you find a connection.

Suppose a researcher wants to find out what variable is related to sales of bedroom slippers. He collects data on everything he can think of, including the size of people’s feet, the frequency with which they go out to get the paper in their slippers, and their favorite colors.

Not finding anything significant, he goes on to examine marital status, age, and income. Still coming up short, he goes out on a limb and looks at hair color, whether or not the subjects have seen a circus, and where they like to sit on an airplane (aisle or window, sir?).

Then wouldn’t you know, he strikes gold. Turns out that, according to his data, people who sit on the aisles on planes are more likely to buy bedroom slippers than those who sit by the window or in the middle of a row.

What’s wrong with this picture? Too many tests. Each time the researcher examines one variable and conducts a test on it, he chooses an α level at which to conduct the test. (Recall that the α level is the amount of chance you’re willing to take of rejecting the null hypothesis and making a false alarm.)

As the number of tests increases, the α’s pile up. Suppose α is chosen to be 0.05. The researcher then has a 5 percent chance of being wrong in finding a significant conclusion, just by chance. So if he does 100 tests, each with a 5 percent chance of an error, on average 5 of those 100 tests will result in a statistically significant result, just by chance.

However, researchers who don’t know that (or who know and go ahead regardless) find results that they claim are significant even though they’re really bogus.

Rumsey, Deborah J.. Statistics II for Dummies (pp. 185-186). Wiley. Kindle Edition.

The math: How much lower should we set alpha = confidence level?
  • new \(\alpha\) = original \(\alpha\) / n
  • original \(\alpha\) = without correction (usually 5 or 1%)
  • n = The total number of comparisons or tests being performed = Combination: if we have 3 categories = 3 hypothesis tests = choose(3,2)
  • why the 2? because each category has 2 options: yes or no