Definition:

• A/B Testing is a usability testing method for comparing two versions of a user interface (UI).
  • Measures which version performs better on specific metrics
  • Examples: time-on-task, errors, or user engagement
• Used to determine whether design changes improve user experience or performance.

Image used under license from DG-Studio / stock.adobe.com

A/B testing is an example of an experiment:

• Two conditions (Version A vs. Version B)
• Compare outcomes between groups

Uses hypothesis testing:

• Determines if differences are statistically significant or due to chance

Involves independent and dependent variables:

• Independent: UI version (A vs. B)
• Dependent: user metrics (task completion rates, time on task, number of errors)

Requires data collection and analysis:

• Collect results from users
• Analyze differences using statistical methods

Relies on random assignment:

• Users are split into groups to reduce bias

We compare conversion rates between two versions where:

• \(p_A\), \(p_B\) = conversion rates of Version A and B

We begin by assuming no difference between the two versions:

\[ H_0: p_A = p_B \quad \text{(null hypothesis: no difference in performance)} \]

We test whether a difference exists:

\[ H_1: p_A \ne p_B \quad \text{(alternative hypothesis: performance differs)} \]

Decision rule (how we determine significance):

\[ p < \alpha \Rightarrow \text{Reject } H_0 \]

• \(p\) = probability result occurred by chance (p-value)
• \(\alpha\) = significance level (commonly 0.05)

A small p-value indicates the observed difference is unlikely due to chance.

Effectiveness:

Measures how accurately users complete tasks.

\[ \text{Effectiveness} = \frac{\text{Number of tasks completed successfully}}{\text{Total number of tasks attempted}} \times 100\% \]

Time-Based Efficiency:

Measures how efficiently users complete tasks over time.

\[ \text{Efficiency} = \frac{\sum_{j=1}^{R} \sum_{i=1}^{N} \frac{n_{ij}}{t_{ij}}}{NR} \]

• \(N\) = number of tasks
• \(R\) = number of users
• \(n_{ij}\) = 1 if task completed successfully, 0 otherwise
• \(t_{ij}\) = time taken to complete task

To better understand how A/B testing works, we will generate example data to demonstrate statistical analysis.

• Version A and Version B represent two UI designs.
• Each user attempts tasks and their outcomes are recorded as:
  • Task success (1 = success, 0 = failure)
  • Time taken to complete the task

The following R code generates example data for analysis:

# Create groups representing versions A and B (10 of each)
group = c(rep("A", 10), rep("B", 10))
# Maintain the randomly produced values below over each run of code
set.seed(123)
# Create 20 random task success values (0 and 1)
success = sample(c(0, 1), size = 20, replace = TRUE)
# Create 20 random time spent on task values (in seconds)
time = sample(c(5:25), size = 20, replace = TRUE)
# Combine into dataset
data = data.frame(group, success, time)

This interactive plot compares task completion time across UI versions.

Each box shows the distribution of completion times for users:

• X-axis: UI version (A vs. B)
• Y-axis: time to complete task
• Color: success (completed (1) vs. not completed (0))

This helps compare both efficiency and effectiveness between versions.