library(readr)
library(dplyr)
library(irr)

Overview

In this activity, our group members individually will first coded prompts in csv files and then merged into a shared codebook. Then we measured agreement using Cohen’s Kappa as we had 2 people on assingment.

The codebook:

Definition: asks for the meaning of a concept
Explanation: asks for a process, mechanism, or why/how something works
Differentiate Concepts: compares or contrasts two related ideas
Examples: asks for examples, cases, or real-world instances
Simplification: asks for easier wording, summarizing, or a simpler explanation
Test Knowledge: asks for quiz questions, flashcards, or another form of assessment

Read in your coded CSV file

Before running this R Markdown, our group filled in the CSV file so that:

rater1 contains the first group member’s codes : Uditi Madan
rater2 contains the second group member’s codes : Julia Gomez

df1 <- read.csv("genai_prompt_intent_uditi_coding.csv")
df2 <- read.csv("genai_prompt_intent_group_coding.csv")
combined_df <- data.frame(
  prompt_id = df1$prompt_id,
  prompt    = df1$prompt,
  rater1    = df1$rater1,
  rater2    = df2$rater2
)

# Combining the 2 CSV Files in one
write.csv(combined_df,"genai_prompt_intent_combined_coding.csv",row.names = FALSE)

# Assigning name "codes" for the merged csv file for easier in code use
codes <- read_csv("genai_prompt_intent_combined_coding.csv", show_col_types = FALSE)

Inspect the coding columns

codes %>% select(prompt_id, prompt, rater1, rater2)

## # A tibble: 40 × 4
##    prompt_id prompt                                                rater1 rater2
##        <dbl> <chr>                                                 <chr>  <chr> 
##  1         1 What is photosynthesis?                               Defin… Defin…
##  2         2 Can you define cellular respiration in simple terms?  Simpl… Simpl…
##  3         3 What is chlorophyll?                                  Defin… Defin…
##  4         4 Define ATP for a high school student.                 Test … Simpl…
##  5         5 Explain how photosynthesis works.                     Expla… Expla…
##  6         6 How does cellular respiration make energy for the ce… Expla… Expla…
##  7         7 Can you walk me through the stages of photosynthesis? Expla… Expla…
##  8         8 Why do plants need sunlight to make food?             Expla… Expla…
##  9         9 How are photosynthesis and cellular respiration conn… Expla… Diffe…
## 10        10 What is the difference between photosynthesis and ce… Diffe… Diffe…
## # ℹ 30 more rows

Agreement for groups of 2: Cohen’s Kappa

codes_2r <- codes %>%
  select(rater1, rater2)

kappa2(codes_2r)

##  Cohen's Kappa for 2 Raters (Weights: unweighted)
## 
##  Subjects = 40 
##    Raters = 2 
##     Kappa = 0.757 
## 
##         z = 10.6 
##   p-value = 0

Interpretation Guide

A common rough interpretation scale is:

below 0.20 = slight agreement
0.21 to 0.40 = fair agreement
0.41 to 0.60 = moderate agreement
0.61 to 0.80 = substantial agreement
0.81 to 1.00 = almost perfect agreement

Reflection Questions

Did your group use Cohen’s Kappa or Fleiss’ Kappa? Why?

ANS: Cohen’s Kappa as we have only 2 people in our group (also makes the calculation way easier)

What was your group’s kappa value?

ANS: Kappa = 0.757

Based on the interpretation scale above, how would you describe your level of agreement?

ANS: Substantial agreement

Which prompts created the most disagreement?

ANS: Prompts mentioning show the nature of high schoolers,“What should I absolutely know for a high school quiz on these topics?” and “Define ATP for a high school student.”, showed main differences which tells that maybe different levels of education rises a different sense of opinions. Also some testing questions of “Make me a short study guide on photosynthesis and respiration” and “Give me a simple chart comparing photosynthesis and respiration.” were not in agreement.

Which pairs of codes were easiest to confuse?

ANS: Testing Knowledge and Differentiate Concepts

What did this activity teach you about the importance of having a clear codebook?