STA320 Final Exam Team 4

1 Research Question & Response Variable

What is your research question and your response variable? Give a detailed answer. Do the factors exam format, practice frequency, feedback timing, and class time structure improve student exam scores? We want to improve student exam performance, grade, in an introductory statistics course and are testing if these factors, and the possible interactions between them, do that. Higher exam scores can lead to student success, reflects well on the university and the professor, and can give insight as to how to properly assess students.

2 Factors, Levels, and Design Choice

List your factors and their levels. What is your design choice? Discuss any constraints or restrictions on randomization. Report your design table which includes run randomization. Discuss how you should employ randomization and blocking in this experiment. Our factors are exam format, practice frequency, feedback timing, and class time structure. Each factor has two levels that we will define as “Low” and “High.” This design is a 2^4 full factorial design, has a total of 16 groups, and will be replicated 3 times. We will use a blocking variable called “major” to control for the possible confounding variable that each major/college could perform differently on exams. The blocking variable has three levels, “Science and Math,” “Education,” and “Arts and Humanities.” However, blocking restricts complete randomization. To adjust for this, we will employ randomization within each block. The students in each block will be randomly assigned to one of the 16 groups to ensure that there is equal variation within each block.

3 Analysis Results

Write a comprehensive paragraph on the results of your statistical analysis. Use APA style. Refer to the interaction plots if necessary.

The experiment used a full 2^4 factorial ANOVA design to examine the effects of four two‑level factors (format, frequency, time, and structure) on exam score. This design produced 16 unique treatment combinations. A three-level blocking variable was implemented to control for variation between majors. Each treatment combination received equal replication, resulting in a balanced design suitable for estimating both main effects and interactions. A factorial ANOVA was conducted to evaluate the effects of the four factors and their interactions on the response. The results showed statistical significance for Format: F(1,32)=81.257, p<0.001, Frequency: F(1,32)=11.551, p=0.001829, Format:Frequency: F(1,32)=64.76, p>0.001, Format:Time: F(1,32)=9.469, p=0.00426, and Format:Time:Structure: F(1,32)=188.817, p<0.001. The significant two-way interactions can also be seen in the interaction plots. The lines within the plots are not parallel, they intersect. This statistical significance tells us a lot about the practical significance. We see that format and frequency alone and when combined with other factors showed statistical significance, indicating that these factors are the driving forces of improving student exam scores. On the other hand, time and structure alone are not significantly impacting student exam score, but when combined with format, they show significance in improving scores.

4 Power

Calculate the post-hoc power for this design. Use the effect size from your output. The post-hoc power using the design parameters and using the effect size f=0.0825 is 8.592 percent. This design yields very low power which is concerning. This indicates that there is a low chance (8.592 percent) of finding a real, true effect if it is present. Having low power can inflate the effect size, displaying that there is a important practical effect, when there actually is not (a false positive). Knowing this, we would want to increase sample size for the next experiment to increase our power.

The four-way interaction effect size is eta²=0.1935526. Degrees of freedom for the power calculation is df=1.

5 Residual Analysis

Are your assumptions met? Use the plots to support your argument. The assumptions of normality and equal variances are met. The histogram shows an approximate bell shaped curve, indicating normality. The versus fit plot displays an even spread between blocks and shows no patterns, indicating equal variances.

6 Factorial Regularities

Do the results of your factorial experiment display sparsity, heredity, and hierarchy? Support your answer with your results.

The results of our factorial experiment demonstrates sparsity. Only 5 effects are significant out of 15, meaning only 33 percent are actually significant. This experiment also shows heredity. All of the significant interactions include one parent factor that is significant. The parent factor “format” is highly significant and seen in every significant interaction. There is a weak indication of hierarchy in this design. The hierarchy principle states that main effects tend to be larger than two-way interactions, two-way interactions tend to be larger than three-way interactions, and so on. In this design, 2 main effects were significant, 2 two-way interactions were significant, and 1 three-way interaction was significant. While this is not a huge deviation from the principle, it is worth noting. This is especially important to recognize because two main effects are not statistically significant.

7 Limitations & What You’d Do Next

Discuss issues you see with this design. Do you have issues with Confounding effects? Are there design weaknesses? Give follow up experiment ideas. The main issue we see with this design is the small sample size. Using GPower, we calculated that with a medium effect size, alpha=0.05, and 0.80 power, the sample size needed would be 128. This would fix the low power issue and increase our effect size. We intended to account for the issues of confounding effects by creating a blocking variable. This would allow us to determine if it was the factors that truly effected the exam score without the effect of the nuisance variable “major.” If this experiment was done again, we suggest rerunning it without a blocking variable and instead running a completely randomized design. The blocking variable we used was not statistically significant (p=0.29894), so it is unnecessary to create. With these improvements, the experimenter may be able to find the factors that best improve student exam scores in the course.