This is a presentation on the topic of Hypothesis Testing

In the following slides, we will answer the question

• “Do ChatGPT and Claude explain sorting algorithms in the same amount of time?”

The goal is to use hypothesis testing to see whether there’s a statistically significant difference between ChatGPT and Claude when it comes to answering sorting algorithm prompts.

• Explain how:
  • Bubble Sort works
  • Merge Sort works
  • Quick Sort works
  • Heap Sort works
  • Counting Sort works
  • Insertion Sort works
  • Selection Sort works
  • Radix sort works
  • Bucket sort works
  • Shell sort works

Both ChatGPT and Claude will receive the same exact prompt. I will time how long it takes for them to finish their responses (ms).

• “Explain how x sorting algorithm works”

##         Algorithm ChatGPT_Time_ms Claude_Time_ms
## 1     Bubble Sort           10130          11950
## 2      Merge Sort           18780          13090
## 3      Quick Sort           18510          13400
## 4       Heap Sort           16820          18680
## 5   Counting Sort           13720          18590
## 6  Insertion Sort           18220          16440
## 7  Selection Sort           14340          15260
## 8      Radix Sort           22480          21380
## 9     Bucket Sort           23070          22480
## 10     Shell Sort           23380          26540

The Null Hypothesis \[H_0 : \mu_{ChatGPT} = \mu_{Claude}\]

• There is NO significant difference in mean response times between ChatGPT and Claude

The Alternative Hypothesis \[H_1 : \mu_{ChatGPT} \neq \mu_{Claude}\]

• There exists a significant difference in mean response times

What kind of test will we do?

• Since ChatGPT and Claude were given the exact same prompts to answer, a paired samples t-test is the option to go with

The paired differences are defined as: \[d_i = X_{ChatGPT,i} - X_{Claude,i}\] where \(d_i\) is the difference in response time for the \(i^{th}\) algorithm

The sample mean difference is: \[\bar{d} = \frac{1}{n} \sum_{i=1}^{n} d_i\]

The standard deviation of the differences is: \[s_d = \sqrt{\frac{\sum_{i=1}^{n} (d_i - \bar{d})^2}{n - 1}}\]

The test statistic is: \[t = \frac{\bar{d}}{s_d / \sqrt{n}}\]

Degrees of freedom: \[df = n - 1\]

Decision rule (two-tailed test, \(\alpha = 0.05\)): \[\text{Reject } H_0 \text{ if } |t| > t_{\alpha/2,\,df}\]

The previous formulas are used to find the p-value.

Depending on the p-value, we can reject or fail to reject our Null Hypothesis

I will use the buil in t.test function in RStudio to produce the p-value based on the data I gathered:

## 
##  Paired t-test
## 
## data:  sorting_data$ChatGPT_Time_ms and sorting_data$Claude_Time_ms
## t = 0.15237, df = 9, p-value = 0.8823
## alternative hypothesis: true mean difference is not equal to 0
## 95 percent confidence interval:
##  -2270.861  2598.861
## sample estimates:
## mean difference 
##             164

Running the built in t.test function, we see our p-value was 0.882.

We use the following to interpret the p-value:

• p value <= 0.05 –> Reject the Null Hypothesis

• p-value >= 0.05 –> FAIL to reject the Null Hypothesis

Our p-value is 0.882, which is >= 0.05 Therefore, our data shows that there is no signifcant statistical difference between the response times from ChatGPT vs. Claude

library(plotly)

comparing <- plot_ly( data = sorting_data , 
                      x = ~Algorithm ,
                      y = ~ChatGPT_Time_ms, 
                      type = 'bar' , 
                      name = 'ChatGPT', 
                      marker=list(color = 'red'))

comparing <- add_trace(
  comparing, 
  y = ~Claude_Time_ms , 
  name = 'Claude', 
  marker = list(color = 'blue')
)

comparing <- layout(
  comparing, 
  title = 'Comparison of Response Times between ChatGPT and Claude' , 
  xaxis = list(title = "Sorting Algorithm"),
  yaxis = list(title = "Response Time (ms)"),
  barmode ='group'
)