A key step in the data science pipeline is hypothesis testing, also known as statistical testing. Statistical tests are mathematical procedures used to determine whether there are significant differences between groups. They provide statistical evidence to reduce errors and support accurate inferences.

Providing evidence

• By using statistical tests, we can gather evidence to support or reject null hypotheses, leading to a deeper understanding of a particular subject or phenomenon.

Reducing errors

• Statistical tests help minimize errors in drawing conclusions from data by identifying issues in data collection or sampling that could result in inaccurate conclusions.

Making Inferences

• By using statistical tests, we can infer characteristics of a population from sample data, enabling us to make predictions and draw assumptions.

One important test used to test statistics is called the Pearson’s Chi-squared test. \[ \chi^2 = \sum \frac{(O - E)^2}{E} \]

Where:

• \(\chi^2\) = Chi-squared statistic
  
• \(O\) = Observed frequency
  
• \(E\) = Expected frequency
  
• \(\Sigma\) = Summation symbol, meaning you sum over all categories

The chi-squared test answers the question: Are these two groups independent of each other?

The expected frequency is calculated assuming that the distribution of responses is the same across both groups.

\[ E = \frac{(Row\ Total \times Column\ Total)}{Grand\ Total} \]

Where:

• Row Total = Total frequency for each row
• Column Total = Total frequency for each category
• Grand Total = Total frequency of all observations across all cells

The p-value measures the probability of observing results as extreme as, or more extreme than, the ones in your data, assuming the null hypothesis is true.

• Low p-value: Strong evidence against the null hypothesis, suggesting the result is statistically significant.
• High p-value: Weak evidence against the null hypothesis, meaning you fail to reject it.

conclusion <- function(p_value){
  alpha= 0.05
  if (p_value < alpha){
    print("We have enough evidence to reject the null hypothesis")
  }else{
    print("We fail to reject the null hypothesis")
  }
}

Note: All the data in this study are fictional.

# Sample data
data <- matrix(c(30, 20, 50, 40), nrow = 2,
               dimnames = list(Gender = c("Male", "Female"), 
                               Preference = c("Like", "Dislike")))

# Chi-squared test results
result1 <- chisq.test(data)

# Convert to data frame for ggplot
df1 <- as.data.frame(as.table(data))

# Bar plot with ggplot
ggplot(df1, aes(x = Gender, y = Freq, fill = Preference)) +
  geom_bar(stat = "identity", position = "dodge") +
  labs(title = "Preference by Gender", y = "Frequency") +
  theme_minimal() +
  theme(plot.title = element_text(hjust = 0.5)) +
  ylim(0, max(df1$Freq) + 10) # Adds extra space above bars

Hypotheses:

• H₀ (Null): There is no association between gender and preference.
• H₁ (Alternative): There is an association between gender and preference.

Result of test:

• X-squared = 0.10954
• p-value = 0.7407

Conclusion: “We fail to reject the null hypothesis”

Note: All the data in this study are fictional.

Hypotheses:

• H₀ (Null): The observed distribution matches the expected distribution.
• H₁ (Alternative): The observed distribution does not match the expected distribution.

Result of test:

• X-squared = 14
• p-value = 0.0009119

Conclusion: “We have enough evidence to reject the null hypothesis”

Note: All the data in this study are fictional.

Hypotheses:

• H₀ (Null): The distribution of responses is the same across both groups.
• H₁ (Alternative): The distribution of responses differs between the groups.

Result of test:

• X-squared = 0.043403
• p-value = 0.835

Conclusion: “We fail to reject the null hypothesis”

These results highlight the importance of hypothesis testing in statistical analysis. While some tests confirmed a lack of association, the goodness-of-fit test revealed a significant deviation, emphasizing that statistical conclusions vary based on the nature of the hypothesis being tested. This study reinforces the necessity of carefully selecting and interpreting statistical tests to ensure valid and meaningful insights.