STAT 7220 - Module 3 Group Porject
Deron Khelawan
2026-09-26
- 1 Research Question 1
- 2 1. Does temperature setting affect mean perplexity at all, or could these differences just be run-to-run noise?
- 3 Research Question 2
- 4 Research Question 3
- 5 p-value = 0.0549, which is greater than alpha of 0.05, therefore, we fail to reject the null hypothesis. As a result of the close p-value, we can assume potential validity of the equal variance assumption once more data is collected
- 6 Concluding Paragraph
- 7 Recommendation
1 Research Question 1
2 1. Does temperature setting affect mean perplexity at all, or could these differences just be run-to-run noise?
#Load the data#
class.dat <- read.csv("~/Downloads/Stat7220_Class_Dataset.csv")
dat.3 <- subset(class.dat, design_block == "anova_rcbd" & task == "factual_qa"
)
str(dat.3); head(dat.3); dim(dat.3); table(dat.3$temperature)## 'data.frame': 20 obs. of 17 variables:
## $ run_id : int 43 44 45 46 63 64 65 66 83 84 ...
## $ design_block : chr "anova_rcbd" "anova_rcbd" "anova_rcbd" "anova_rcbd" ...
## $ rep_idx : int 0 1 2 3 0 1 2 3 0 1 ...
## $ seed : int 957224230 37729121 626094248 1522976423 106291693 251961167 122700152 1561262825 1865297715 1660505496 ...
## $ prompt_id : chr NA NA NA NA ...
## $ judge_id : chr NA NA NA NA ...
## $ temperature : num 0.1 0.1 0.1 0.1 0.3 0.3 0.3 0.3 0.5 0.5 ...
## $ top_p : num 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 0.9 ...
## $ top_k : int 50 50 50 50 50 50 50 50 50 50 ...
## $ repetition_penalty: num 1.2 1.2 1.2 1.2 1.2 1.2 1.2 1.2 1.2 1.2 ...
## $ task : chr "factual_qa" "factual_qa" "factual_qa" "factual_qa" ...
## $ prompt_length : chr "short" "short" "short" "short" ...
## $ perplexity : num 13 13.2 13.1 13.2 15.1 ...
## $ distinct2 : num 0.458 0.363 0.423 0.452 0.563 ...
## $ coherence : num 1 0.885 0.951 0.966 0.913 ...
## $ output_tokens : int 118 109 93 125 133 137 111 115 117 113 ...
## $ latency_ms : num 2105 1951 1659 2267 2403 ...## run_id design_block rep_idx seed prompt_id judge_id temperature top_p
## 43 43 anova_rcbd 0 957224230 <NA> <NA> 0.1 0.9
## 44 44 anova_rcbd 1 37729121 <NA> <NA> 0.1 0.9
## 45 45 anova_rcbd 2 626094248 <NA> <NA> 0.1 0.9
## 46 46 anova_rcbd 3 1522976423 <NA> <NA> 0.1 0.9
## 63 63 anova_rcbd 0 106291693 <NA> <NA> 0.3 0.9
## 64 64 anova_rcbd 1 251961167 <NA> <NA> 0.3 0.9
## top_k repetition_penalty task prompt_length perplexity distinct2
## 43 50 1.2 factual_qa short 12.979 0.4582
## 44 50 1.2 factual_qa short 13.235 0.3634
## 45 50 1.2 factual_qa short 13.086 0.4227
## 46 50 1.2 factual_qa short 13.192 0.4518
## 63 50 1.2 factual_qa short 15.064 0.5627
## 64 50 1.2 factual_qa short 17.208 0.5693
## coherence output_tokens latency_ms
## 43 1.0000 118 2105.1
## 44 0.8848 109 1951.4
## 45 0.9514 93 1659.0
## 46 0.9660 125 2267.3
## 63 0.9129 133 2402.8
## 64 0.9865 137 2527.0## [1] 20 17##
## 0.1 0.3 0.5 0.7 0.9
## 4 4 4 4 4#A summary table (n, mean, sd of perplexity) for each of the 5 temperature levels#
summary <- aggregate(perplexity~temperature, data = dat.3,
FUN = function(x) c(n = length (x),
mean = mean(x),
sd = sd(x)))
summary## temperature perplexity.n perplexity.mean perplexity.sd
## 1 0.1 4.0000000 13.1230000 0.1146153
## 2 0.3 4.0000000 16.2672500 1.1520820
## 3 0.5 4.0000000 18.9765000 0.6147316
## 4 0.7 4.0000000 22.1812500 0.9809490
## 5 0.9 4.0000000 24.7757500 0.9521318#temperature perplexity.n perplexity.mean perplexity.sd #1 0.1 4.0000000 13.1230000 0.1146153 #2 0.3 4.0000000 16.2672500 1.1520820 #3 0.5 4.0000000 18.9765000 0.6147316 #4 0.7 4.0000000 22.1812500 0.9809490 #5 0.9 4.0000000 24.7757500 0.9521318#
#visual of the relationship between temperature and perplexity#
#Box plot of perplexity by temperature#
boxplot(perplexity~as.factor(temperature), data=dat.3)
mod3<-lm(perplexity~as.factor(temperature), data=dat.3)#Anova Table#
anova(mod3);## Analysis of Variance Table
##
## Response: perplexity
## Df Sum Sq Mean Sq F value Pr(>F)
## as.factor(temperature) 4 341.87 85.466 119.13 3.562e-11 ***
## Residuals 15 10.76 0.717
## ---
## Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1After running the ANOVA test, the p value is < 0.05, therefore, we reject the null hypothesis. This tells us that each temperature grouping has a different average and because of how small the p value is we can assume that the difference is statistically significant and not just noise.
3 Research Question 2
If it matters, which specific temperature level brings the lowest perplexity (i.e., the most predictable, fluent output)? Is that lowest value significantly different from the others?
tapply(dat.3$perplexity,dat.3$temperature,function(x) c(mean(x), sd(x)))## $`0.1`
## [1] 13.1230000 0.1146153
##
## $`0.3`
## [1] 16.267250 1.152082
##
## $`0.5`
## [1] 18.9765000 0.6147316
##
## $`0.7`
## [1] 22.181250 0.980949
##
## $`0.9`
## [1] 24.7757500 0.95213180.1 13.1230000 0.1146153
0.3 16.267250 1.152082
0.5 18.9765000 0.6147316
0.7 22.181250 0.980949
0.9 24.7757500 0.9521318
The variances for each perplexity-temperature combination is different, this aligns with the boxplots that were produced earlier. Additionally, the lowest temperature group (0.1) has the lowest perplexiity level.
pairwise.t.test(dat.3$perplexity,as.factor(dat.3$temperature),p.adjust.method="none")##
## Pairwise comparisons using t tests with pooled SD
##
## data: dat.3$perplexity and as.factor(dat.3$temperature)
##
## 0.1 0.3 0.5 0.7
## 0.3 9.8e-05 - - -
## 0.5 6.8e-08 0.00040 - -
## 0.7 1.7e-10 5.9e-08 8.1e-05 -
## 0.9 4.7e-12 4.2e-10 7.6e-08 0.00059
##
## P value adjustment method: none0.1 0.3 0.5 0.7 0.3 9.8e-05 - - -
0.5 6.8e-08 0.00040 - -
0.7 1.7e-10 5.9e-08 8.1e-05 -
0.9 4.7e-12 4.2e-10 7.6e-08 0.00059
P value adjustment method: none
After running a pairwise t test, we were able to see that all pairs of temperature levels are significantly different because their p values < 0.05, this lines up with the boxplot.#
We should be cautious with p value adjustment method: none because it increases the likelihood of falsely concluding significance. #
TukeyHSD(aov(mod3))## Tukey multiple comparisons of means
## 95% family-wise confidence level
##
## Fit: aov(formula = mod3)
##
## $`as.factor(temperature)`
## diff lwr upr p adj
## 0.3-0.1 3.14425 1.2948074 4.993693 0.0007965
## 0.5-0.1 5.85350 4.0040574 7.702943 0.0000006
## 0.7-0.1 9.05825 7.2088074 10.907693 0.0000000
## 0.9-0.1 11.65275 9.8033074 13.502193 0.0000000
## 0.5-0.3 2.70925 0.8598074 4.558693 0.0031595
## 0.7-0.3 5.91400 4.0645574 7.763443 0.0000005
## 0.9-0.3 8.50850 6.6590574 10.357943 0.0000000
## 0.7-0.5 3.20475 1.3553074 5.054193 0.0006602
## 0.9-0.5 5.79925 3.9498074 7.648693 0.0000007
## 0.9-0.7 2.59450 0.7450574 4.443943 0.00457314 Research Question 3
- How much of the total variability is explained by temperature vs. random error?
Analysis of Variance Table
Response: perplexity Df Sum Sq Mean Sq F value Pr(>F)
as.factor(temperature) 4 341.87 85.466 119.13 3.562e-11 *** Residuals 15
10.76 0.717
— Signif. codes: 0 ‘’ 0.001 ‘’ 0.01 ‘’ 0.05
‘.’ 0.1 ‘ ’ 1
Temperature: 341.87 Residual: 10.76
SSTreatment: 341.87 SSTotal: 341.87 + 10.76 = 352.63
R^2 : SSTreatment/SSTotal = 341.87/352.63 = 0.9694864
Therefore, we can say that the ANOVA test can explain 96.9% of the variations in the data.
4.1 Research Question 4
Can we trust the test?
Bartlett Test Statistic and p-value, with interpretation
bartlett.test(perplexity~as.factor(temperature), data=dat.3)##
## Bartlett test of homogeneity of variances
##
## data: perplexity by as.factor(temperature)
## Bartlett's K-squared = 9.2608, df = 4, p-value = 0.0549Bartlett test of homogeneity of variances
data: perplexity by as.factor(temperature) Bartlett’s K-squared = 9.2608, df = 4, p-value = 0.0549
5 p-value = 0.0549, which is greater than alpha of 0.05, therefore, we fail to reject the null hypothesis. As a result of the close p-value, we can assume potential validity of the equal variance assumption once more data is collected
#A normal probability plot of the residuals#
’’’{r} res3<- resid(mod3); fit3<-mod3$fitted par(mfrow=c(1,3)) plot(c(1:20),res3) qqnorm(res3);qqline(res3);plot(fit3,res3) ```
The distribution of the residuals between 0 and 5 are clustered together, however, as it increases, the distribution becomes more random. The QQ plot shows that most points are distributed along the line of normality with a few outliers on both ends. The third plot shows closer variation below 14 with a more random distribution as the observations increase.
The QQ Plots show no direct pattern vs run order, so therefore, it supports independence#
The ANOVA test provided a p value that was less than 0.05 and as a result the null hypothesis is rejected. After running the QQ Plots, we were able to match it against the ANOVA assumptions, in this case, independence, equal variance and normality were all met. Additionally, the differences between temperature groups are too significant for there to be any explanation for potential error. #
6 Concluding Paragraph
After conducting the research, we can conclude that temperature has an impact on output quality, as measured by perplexity. An ANOVA test was conducted and the p value was less than 0.05 which means that we reject the null hypothesis. Additionally, the ANOVA test, provided an r^2 of 0.9694864 which tells us that temperature has an extremely high impact on any variability in observations.
Did your results match what you expected before the analysis: Yes, I expected temperature to have a significant impact on output quality. Which step took the most troubleshooting: The analyses take the most troubleshooting because sometimes it is hard to draw results from some of the graphs If you were deploying a production system, would you pick one “safe” temperature, or does your analysis suggest something more nuanced: If I were deploying a production system, I would pick more specific temperatures because while temperature does have a direct impact on the output, limiting it to a “safe” temperature would also limit the output results. It is important to have a wide range to see the true range of temperature impact.
7 Recommendation
For anyone deploying this model, it is recommended to be specific with temperature levels as there is a high correlation between temperature and output quality. Additionally, depending on the output that is desired, there should be specific attention paid to the temperature levels being utilized.