Probability Distribution
Week 11
INSTITUT TEKNOLOGI SAINS BANDUNG
Name : Dhefio Alim Muzakki
Student ID : 52250014
Major : Data Science
Lecturer : Mr. Bakti Siregar, M.Sc., CDS.
library(tidyverse)
library(readr)
library(ggplot2)
library(dplyr)
library(ggridges)
library(knitr)
library(DT)1 Problem Description
An e-commerce platform wants to estimate the average number
of daily transactions per user after launching a new
feature.
The population standard deviation is known from historical data.
Given:
\[ \sigma = 3.2 \]
\[ n = 100 \]
\[ \bar{x} = 12.6 \]
1.1 Statistical Method Selection
Because: - The population standard deviation \(\sigma\) is known - The objective is to estimate the population mean - The sample size is large (\(n \ge 30\))
The appropriate method is the:
\[ \boxed{\text{Z-Confidence Interval for the Population Mean}} \]
This is justified by the Central Limit Theorem.
1.2 Confidence Interval Formula
The general Z-confidence interval for the mean is:
\[ \bar{x} \pm z_{\alpha/2}\left(\frac{\sigma}{\sqrt{n}}\right) \]
Where:
\(\bar{x}\) = sample mean
\(z_{\alpha/2}\) = critical value from standard normal distribution
\(\sigma\) = population standard deviatio
\(n\) = sample size
1.3 Step-by-Step Numerical Calculation
Step 1: Standard Error
\[ SE = \frac{\sigma}{\sqrt{n}} = \frac{3.2}{\sqrt{100}} = \frac{3.2}{10} = 0.32 \]
Step 2: Z-Critical Values
| Confidence Level | \(\alpha\) | \(z_{\alpha/2}\) |
|---|---|---|
| 90% | 0.10 | 1.645 |
| 95% | 0.05 | 1.960 |
| 99% | 0.01 | 2.576 |
Step 3: Margin of Error
\[ ME = z_{\alpha/2} \times SE \]
90%: \[ ME = 1.645 \times 0.32 = 0.526 \]
95%: \[ ME = 1.960 \times 0.32 = 0.627 \]
99%: \[ ME = 2.576 \times 0.32 = 0.824 \]
Step 4: Confidence Intervals
\[ CI = (\bar{x} - ME,\; \bar{x} + ME) \]
90% CI: \[ (12.6 - 0.526,\; 12.6 + 0.526) = (12.074,\; 13.126) \]
95% CI: \[ (11.973,\; 13.227) \]
99% CI: \[ (11.776,\; 13.424) \]
Setup and Calculations
# Given values
sigma <- 3.2
n <- 100
xbar <- 12.6
# Standard error
SE <- sigma / sqrt(n)
# Confidence levels
conf_levels <- c(0.90, 0.95, 0.99)
# Z-values
z_values <- qnorm((1 + conf_levels) / 2)
# Compute confidence intervals
lower <- xbar - z_values * SE
upper <- xbar + z_values * SE
# Combine results
ci_table <- data.frame(
Confidence_Level = conf_levels,
Z_Value = round(z_values, 3),
Lower_Bound = round(lower, 3),
Upper_Bound = round(upper, 3)
)
ci_tableVisualization
# 1. Load libraries
library(plotly)
library(dplyr)
# 2. Set Parameters
mu <- 12.6
se <- 0.32
x <- seq(mu - 4*se, mu + 4*se, length.out = 500)
y <- dnorm(x, mu, se)
df <- data.frame(x = x, y = y)
# 3. Create the Interactive Plot
p <- plot_ly(df, x = ~x, y = ~y, type = 'scatter', mode = 'lines',
line = list(color = 'black', width = 2), name = 'Normal Curve') %>%
# Add 99% CI Layer
add_polygons(x = c(mu - 2.576*se, seq(mu - 2.576*se, mu + 2.576*se, length.out = 100), mu + 2.576*se),
y = c(0, dnorm(seq(mu - 2.576*se, mu + 2.576*se, length.out = 100), mu, se), 0),
fillcolor = 'rgba(222, 235, 247, 0.6)', line = list(color = 'transparent'),
name = "99% CI (11.78 - 13.42)") %>%
# Add 95% CI Layer
add_polygons(x = c(mu - 1.96*se, seq(mu - 1.96*se, mu + 1.96*se, length.out = 100), mu + 1.96*se),
y = c(0, dnorm(seq(mu - 1.96*se, mu + 1.96*se, length.out = 100), mu, se), 0),
fillcolor = 'rgba(158, 202, 225, 0.7)', line = list(color = 'transparent'),
name = "95% CI (11.97 - 13.23)") %>%
# Add 90% CI Layer
add_polygons(x = c(mu - 1.645*se, seq(mu - 1.645*se, mu + 1.645*se, length.out = 100), mu + 1.645*se),
y = c(0, dnorm(seq(mu - 1.645*se, mu + 1.645*se, length.out = 100), mu, se), 0),
fillcolor = 'rgba(66, 146, 198, 0.8)', line = list(color = 'transparent'),
name = "90% CI (12.07 - 13.13)") %>%
# Layout and Professional Styling
layout(title = "Interactive Transaction Confidence Mountain",
xaxis = list(title = "Average Daily Transactions per User"),
yaxis = list(title = "Density", showticklabels = FALSE),
hovermode = "x unified",
legend = list(orientation = 'h', x = 0.1, y = -0.2))
pinterpretation
This visualization utilizes a layered probability density approach. The darkest central area represents our most likely range (90%), while the lighter “wings” illustrate the extra space required to be 95% or 99% certain of our results.
When presenting this “mountain” to stakeholders, we use the following three-pillar framework:
The Margin of Error as Risk Measurement The “base” of the mountain represents the total statistical risk. As we move from a 90% to a 99% confidence level, we are effectively stating: > “To reduce the chance of being wrong from 10% down to 1%, we must accept a wider range of possible outcomes.”
For the e-commerce platform, the 99% range \([11.78, 13.42]\) is considered the “Safe Zone” for financial planning and infrastructure scaling.
Point Estimate vs. Interval Estimate The peak of the mountain (\(\bar{x} = 12.6\)) is our “best guess” (Point Estimate). However, in professional analytics, we prioritize the Interval because it accounts for Sampling Error.
\[SE = \frac{\sigma}{\sqrt{n}}\]
The interval acknowledges the natural fluctuation that occurs because we analyzed a sample of \(n = 100\) users rather than the entire population of millions.
Assessing Operational Thresholds We use the mountain to test if we have met specific business KPIs:
- Target of 13.0: Since 13.0 falls inside the 90% dark blue region, we can report that we are 90% confident the target is reachable.
- Certainty Gap: We are not 99% certain that 13.0 is the true average, as the 99% interval extends significantly lower to 11.78. This indicates that while the results are promising, there is still a small statistical risk of performing below the target.
2 Problem Description
A UX Research team analyzes task completion time
(minutes) for a new mobile application.
Data collected from 12 users:
\[ 8.4,\; 7.9,\; 9.1,\; 8.7,\; 8.2,\; 9.0,\; 7.8,\; 8.5,\; 8.9,\; 8.1,\; 8.6,\; 8.3 \]
2.1 Appropriate Statistical Method
Because: - The population standard deviation \(\sigma\) is unknown - The sample size is small (\(n = 12\)) - We are estimating the population mean
The appropriate method is the:
\[ \boxed{\text{t-Confidence Interval for the Population Mean}} \]
This method uses the Student’s t-distribution, which accounts for additional uncertainty from estimating \(\sigma\).
2.2 Confidence Interval Formula (σ Unknown)
\[ \bar{x} \pm t_{\alpha/2,\,n-1}\left(\frac{s}{\sqrt{n}}\right) \]
Where: - \(\bar{x}\) = sample
mean
- \(s\) = sample standard
deviation
- \(t_{\alpha/2,\,n-1}\) = critical
value from t-distribution
- \(n\) = sample size
2.3 Step-by-Step Manual Calculation
Sample Mean
\[ \bar{x} = \frac{1}{n}\sum_{i=1}^{n}x_i \]
\[ \bar{x} = \frac{8.4 + 7.9 + \cdots + 8.3}{12} = 8.458 \]
Sample Standard Deviation
\[ s = \sqrt{\frac{\sum (x_i - \bar{x})^2}{n-1}} \]
\[ s = 0.408 \]
Standard Error
\[ SE = \frac{s}{\sqrt{n}} = \frac{0.408}{\sqrt{12}} = 0.118 \]
t-Critical Values (df = 11)
| Confidence Level | \(\alpha\) | \(t_{\alpha/2,11}\) |
|---|---|---|
| 90% | 0.10 | 1.796 |
| 95% | 0.05 | 2.201 |
| 99% | 0.01 | 3.106 |
Margin of Error
\[ ME = t_{\alpha/2,11} \times SE \]
90%: \[ ME = 1.796 \times 0.118 = 0.212 \]
95%: \[ ME = 2.201 \times 0.118 = 0.260 \]
99%: \[ ME = 3.106 \times 0.118 = 0.366 \]
Confidence Intervals
\[ CI = (\bar{x} - ME,\; \bar{x} + ME) \]
90% CI: \[ (8.246,\; 8.670) \]
95% CI: \[ (8.198,\; 8.718) \]
99% CI: \[ (8.092,\; 8.824) \]
library(plotly)
# 1. Calculated Parameters
mu <- 8.458
se <- 0.118
df_value <- 11 # Degrees of freedom (n-1)
# Generate distribution data
x <- seq(mu - 4*se, mu + 4*se, length.out = 500)
y <- dt((x - mu)/se, df = df_value) / se # t-distribution density
df_plot <- data.frame(x = x, y = y)
# 2. Build Interactive Plot
p <- plot_ly(df_plot, x = ~x, y = ~y, type = 'scatter', mode = 'lines',
line = list(color = '#252525', width = 2.5), name = 't-Distribution (df=11)') %>%
# 99% CI Layer
add_polygons(x = c(mu - 3.106*se, seq(mu - 3.106*se, mu + 3.106*se, length.out = 100), mu + 3.106*se),
y = c(0, (dt((seq(mu - 3.106*se, mu + 3.106*se, length.out = 100) - mu)/se, df = df_value)/se), 0),
fillcolor = 'rgba(239, 243, 255, 0.6)', line = list(color = 'transparent'),
name = "99% CI (8.09 - 8.82)") %>%
# 95% CI Layer
add_polygons(x = c(mu - 2.201*se, seq(mu - 2.201*se, mu + 2.201*se, length.out = 100), mu + 2.201*se),
y = c(0, (dt((seq(mu - 2.201*se, mu + 2.201*se, length.out = 100) - mu)/se, df = df_value)/se), 0),
fillcolor = 'rgba(189, 215, 231, 0.7)', line = list(color = 'transparent'),
name = "95% CI (8.20 - 8.72)") %>%
# 90% CI Layer
add_polygons(x = c(mu - 1.796*se, seq(mu - 1.796*se, mu + 1.796*se, length.out = 100), mu + 1.796*se),
y = c(0, (dt((seq(mu - 1.796*se, mu + 1.796*se, length.out = 100) - mu)/se, df = df_value)/se), 0),
fillcolor = 'rgba(107, 174, 214, 0.8)', line = list(color = 'transparent'),
name = "90% CI (8.25 - 8.67)") %>%
# Labels and Styling
layout(title = list(text = "<b>Task Completion Time: Confidence Mountain</b>", y = 0.95),
xaxis = list(title = "Time in Minutes", gridcolor = '#f0f0f0'),
yaxis = list(title = "Density", showticklabels = FALSE, gridcolor = '#f0f0f0'),
hovermode = "x unified",
paper_bgcolor = 'white', plot_bgcolor = 'white',
legend = list(orientation = 'h', x = 0.1, y = -0.2))
plibrary(plotly)
# 1. Data Setup
case2_data <- data.frame(
Level = c("90%", "95%", "99%"),
Mean = c(8.458, 8.458, 8.458),
Lower = c(8.246, 8.198, 8.092),
Upper = c(8.670, 8.718, 8.824),
Color = c('rgba(107, 174, 214, 1)', 'rgba(66, 146, 198, 1)', 'rgba(33, 113, 181, 1)')
)
# 2. Build Plot
p <- plot_ly(case2_data) %>%
# Add the horizontal error bars (Intervals)
add_segments(x = ~Lower, xend = ~Upper, y = ~Level, yend = ~Level,
line = list(color = '#737373', width = 6), name = "Interval Range") %>%
# Add the specific bound markers
add_markers(x = ~Lower, y = ~Level, marker = list(symbol = "line-ns-open", size = 15, color = "black"),
showlegend = FALSE) %>%
add_markers(x = ~Upper, y = ~Level, marker = list(symbol = "line-ns-open", size = 15, color = "black"),
showlegend = FALSE) %>%
# Add the Sample Mean point
add_markers(x = ~Mean, y = ~Level, marker = list(color = 'red', size = 12, symbol = "diamond"),
name = "Sample Mean (8.458)") %>%
# Add text labels for the numbers
add_annotations(x = ~Lower, y = ~Level, text = ~Lower, showarrow = FALSE, yshift = 15) %>%
add_annotations(x = ~Upper, y = ~Level, text = ~Upper, showarrow = FALSE, yshift = 15) %>%
# Styling
layout(title = "<b>Case 2: Comparison of Task Time Intervals</b>",
xaxis = list(title = "Completion Time (Minutes)", range = c(7.9, 9.0)),
yaxis = list(title = "Confidence Level"),
margin = list(l = 100),
showlegend = FALSE)
pUX Research Interpretation: Task Completion Volatility
This visualization applies a layered Student’s t-distribution approach. Because our sample size is small (\(n = 12\)), the “mountain” has heavier tails, reflecting the higher uncertainty inherent in small-scale usability testing.
When presenting these task-time metrics to Product Managers or Designers, we utilize the following framework:
Accounting for Small Sample Uncertainty The width of this mountain’s base is driven by the t-critical value. Because we do not know the population standard deviation (\(\sigma\)), we use the sample standard deviation (\(s = 0.408\)). > “With only 12 users, our ‘mountain’ must be wider to ensure the true average task time is captured, especially at the 99% certainty level.”
For the design team, the 99% range \([8.09, 8.82]\) represents the Expected Performance Envelope.
**Evaluating the “Efficiency Threshold In UX research, we often have a maximum acceptable time (e.g., a ”Redline” of 8.0 minutes). * Target Analysis:** Our entire 99% confidence interval sits above 8.0 minutes. * Insight: We can state with high statistical confidence that the average user is currently slower than the 8-minute threshold. Even the”best-case” lower bound (8.09) does not cross into the target zone.
Reliability of the Mean The peak (\(\bar{x} = 8.46\) minutes) is the average time observed in this study. However, the Margin of Error (\(\pm 0.26\) at 95% confidence) warns stakeholders that individual user experiences vary.
\[\bar{x} \pm t_{\alpha/2, n-1} \left( \frac{s}{\sqrt{n}} \right)\]
The interval \([8.20, 8.72]\) provides a more realistic expectation for future users than the single point estimate of 8.46.
What Controls the “Mountain” Width?
In UX research, the width of the confidence interval (the distance between the “feet” of the mountain) is governed by two primary levers. Understanding these is critical for deciding how many users to test in future studies.
The Impact of Sample Size (\(n\)) There is an inverse relationship between sample size and interval width.
- The Logic: As \(n\) increases, the Standard Error (\(SE = \frac{s}{\sqrt{n}}\)) shrinks. A larger sample provides more “evidence,” which reduces uncertainty.
- In Case 2: With only \(n = 12\) users, our mountain is relatively wide. To cut the width of this mountain in half, we would need to quadruple our sample size to \(n = 48\).
The Impact of Confidence Level There is a direct relationship between the confidence level and interval width.
- The Certainty Tax: To be more certain (moving from 95% to 99%), you must “pay” by accepting a wider, less precise range.
- The Multiplier: This is controlled by the \(t\)-score. For \(df = 11\), the multiplier jumps from \(2.201\) (for 95%) to \(3.106\) (for 99%), pushing the walls of the mountain significantly outward.
3 Problem Description
A data science team runs an A/B test on a new Call-To-Action (CTA) button design.
The experiment yields:
\[ n = 400 \quad \text{(total users)} \]
\[ x = 156 \quad \text{(users who clicked the CTA)} \]
The objective is to estimate the true click-through rate (CTR).
Appropriate Statistical Method
Because: - The parameter of interest is a population proportion - Sample size is large - Both conditions are satisfied: \[ np \ge 10 \quad \text{and} \quad n(1-p) \ge 10 \]
The appropriate method is the:
\[ \boxed{\text{Z-Confidence Interval for a Population Proportion}} \]
Sample Proportion Calculation
The sample proportion is:
\[ \hat{p} = \frac{x}{n} \]
\[ \hat{p} = \frac{156}{400} = 0.39 \]
Thus, the observed click-through rate is 39%.
Confidence Interval Formula for a Proportion
\[ \hat{p} \pm z_{\alpha/2} \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}} \]
Where: - \(\hat{p}\) = sample
proportion
- \(z_{\alpha/2}\) = critical value
from standard normal distribution
- \(n\) = sample size
3.1 Step-by-Step Manual Calculation
Standard Error
\[ SE = \sqrt{\frac{0.39(1 - 0.39)}{400}} \]
\[ SE = \sqrt{\frac{0.2379}{400}} = 0.0244 \]
Z-Critical Values
| Confidence Level | \(\alpha\) | \(z_{\alpha/2}\) |
|---|---|---|
| 90% | 0.10 | 1.645 |
| 95% | 0.05 | 1.960 |
| 99% | 0.01 | 2.576 |
3.2 Margin of Error
\[ ME = z_{\alpha/2} \times SE \]
90%: \[ ME = 1.645 \times 0.0244 = 0.040 \]
95%: \[ ME = 1.960 \times 0.0244 = 0.048 \]
99%: \[ ME = 2.576 \times 0.0244 = 0.063 \]
Confidence Intervals
\[ CI = (\hat{p} - ME,\; \hat{p} + ME) \]
90% CI: \[ (0.350,\; 0.430) \]
95% CI: \[ (0.342,\; 0.438) \]
99% CI: \[ (0.327,\; 0.453) \]
3.3 Calculation (Verification)
x <- 156
n <- 400
p_hat <- x / n
SE <- sqrt(p_hat * (1 - p_hat) / n)
z <- c(1.645, 1.96, 2.576)
lower <- p_hat - z * SE
upper <- p_hat + z * SE
data.frame(
Confidence_Level = c("90%", "95%", "99%"),
Lower_Bound = round(lower, 3),
Upper_Bound = round(upper, 3)
)library(plotly)
# 1. Data Setup from Case 3
case3_data <- data.frame(
Level = c("90%", "95%", "99%"),
P_hat = c(0.39, 0.39, 0.39),
Lower = c(0.350, 0.342, 0.327),
Upper = c(0.430, 0.438, 0.453)
)
# 2. Build Plot
p <- plot_ly(case3_data) %>%
# Add the horizontal lines (Intervals)
add_segments(x = ~Lower, xend = ~Upper, y = ~Level, yend = ~Level,
line = list(color = '#e7298a', width = 6), name = "Interval Range") %>%
# Add specific markers for the bounds
add_markers(x = ~Lower, y = ~Level, marker = list(symbol = "line-ns-open", size = 15, color = "black"),
showlegend = FALSE) %>%
add_markers(x = ~Upper, y = ~Level, marker = list(symbol = "line-ns-open", size = 15, color = "black"),
showlegend = FALSE) %>%
# Add the Sample Proportion point
add_markers(x = ~P_hat, y = ~Level, marker = list(color = 'black', size = 12, symbol = "x"),
name = "Observed CTR (39%)") %>%
# Add text labels for the numbers (as percentages)
add_annotations(x = ~Lower, y = ~Level, text = paste0(round(case3_data$Lower*100, 1), "%"),
showarrow = FALSE, yshift = 20) %>%
add_annotations(x = ~Upper, y = ~Level, text = paste0(round(case3_data$Upper*100, 1), "%"),
showarrow = FALSE, yshift = 20) %>%
# Layout
layout(title = "<b>Case 3: Comparison of CTR Confidence Intervals</b>",
xaxis = list(title = "Click-Through Rate (%)", tickformat = ".0%", range = c(0.30, 0.50)),
yaxis = list(title = "Confidence Level"),
margin = list(l = 100),
showlegend = FALSE)
pProfessional A/B Testing Framework
Precision and “The Base” of the Mountain The Standard Error for this proportion is \(0.0244\) (\(2.44\%\)). * Insight: Even with 400 users, there is still a “swing” in the data. The 99% interval shows that the true CTR could be as low as 32.7% or as high as 45.3%. * Risk Management: If the current “Control” version of the app has a CTR of 30%, we can see that our entire 99% mountain sits above that line. This is a very strong signal of a “winning” test.
How Confidence Levels Drive Product Decisions Choosing a confidence level is not just a math choice; it is a business risk choice:
- 90% Confidence (The “Agile” Threshold): Used when the cost of being wrong is low. If we launch a button color change that doesn’t actually work, the business doesn’t lose much. We accept a 10% risk of a “False Positive” to move faster.
- 95% Confidence (The “Industry Standard”): The gold standard for most product experiments. It balances the need for speed with a strict requirement that we only ship features that have a clear, repeatable impact.
- 99% Confidence (The “High-Stakes” Threshold): Used when a change is expensive or risky (e.g., changing the checkout flow or pricing). We demand a 99% confidence level because we cannot afford to be wrong; the “Mountain” must be wide enough to ensure that even the worst-case scenario is acceptable.
4 Problem Description
Two data teams measure API latency (milliseconds) under different conditions.
Team A (σ Known) \[ n = 36,\quad \bar{x} = 210,\quad \sigma = 24 \]
Team B (σ Unknown) \[ n = 36,\quad \bar{x} = 210,\quad s = 24 \]
The objective is to compare confidence interval precision.
4.1 Statistical Method Identification
Team A - Population standard deviation is known - Uses the Z-Confidence Interval
\[ \boxed{\text{Z-Interval for the Mean}} \]
Team B - Population standard deviation is unknown - Uses the t-Confidence Interval
\[ \boxed{\text{t-Interval for the Mean}} \]
4.2 Confidence Interval Formulas
Z-Confidence Interval (Team A)
\[ \bar{x} \pm z_{\alpha/2} \left(\frac{\sigma}{\sqrt{n}}\right) \]
t-Confidence Interval (Team B)
\[ \bar{x} \pm t_{\alpha/2,\,n-1} \left(\frac{s}{\sqrt{n}}\right) \]
4.3 Step-by-Step Calculations
Standard Error (Both Teams)
\[ SE = \frac{24}{\sqrt{36}} = \frac{24}{6} = 4 \]
Critical Values
| Confidence | \(z_{\alpha/2}\) | \(t_{\alpha/2,35}\) |
|---|---|---|
| 90% | 1.645 | 1.690 |
| 95% | 1.960 | 2.030 |
| 99% | 2.576 | 2.724 |
Margin of Error
\[ ME = \text{Critical Value} \times SE \]
Team A (Z):
- 90%: \(1.645 \times 4 = 6.58\)
- 95%: \(1.960 \times 4 = 7.84\)
- 99%: \(2.576 \times 4 = 10.30\)
Team B (t):
- 90%: \(1.690 \times 4 = 6.76\)
- 95%: \(2.030 \times 4 = 8.12\)
- 99%: \(2.724 \times 4 = 10.90\)
Confidence Intervals
\[ CI = (\bar{x} - ME,\; \bar{x} + ME) \]
| Team | Confidence | Lower | Upper |
|---|---|---|---|
| A | 90% | 203.42 | 216.58 |
| A | 95% | 202.16 | 217.84 |
| A | 99% | 199.70 | 220.30 |
| B | 90% | 203.24 | 216.76 |
| B | 95% | 201.88 | 218.12 |
| B | 99% | 199.10 | 220.90 |
4.4 Calculation
xbar <- 210
SE <- 4
z <- c(1.645, 1.96, 2.576)
t <- c(1.69, 2.03, 2.724)
lower_A <- xbar - z * SE
upper_A <- xbar + z * SE
lower_B <- xbar - t * SE
upper_B <- xbar + t * SE
data.frame(
Team = rep(c("A (Z)", "B (t)"), each = 3),
Confidence = rep(c("90%", "95%", "99%"), 2),
Lower = round(c(lower_A, lower_B), 2),
Upper = round(c(upper_A, upper_B), 2)
)library(plotly)
# Data Setup
case4_data <- data.frame(
Team = rep(c("Team A (Z)", "Team B (t)"), each = 3),
Level = rep(c("90%", "95%", "99%"), 2),
Lower = c(203.42, 202.16, 199.70, 203.24, 201.88, 199.10),
Upper = c(216.58, 217.84, 220.30, 216.76, 218.12, 220.90),
Mean = 210
)
# Create Grouping for Y-axis
case4_data$Group <- paste(case4_data$Team, case4_data$Level)
p <- plot_ly(case4_data) %>%
# Add horizontal segments
add_segments(x = ~Lower, xend = ~Upper, y = ~Group, yend = ~Group,
color = ~Team, colors = c("#1f77b4", "#ff7f0e"),
line = list(width = 8), name = ~Team) %>%
# Add markers for bounds
add_markers(x = ~Lower, y = ~Group, marker = list(symbol = "line-ns-open", size = 12, color = "black"),
showlegend = FALSE) %>%
add_markers(x = ~Upper, y = ~Group, marker = list(symbol = "line-ns-open", size = 12, color = "black"),
showlegend = FALSE) %>%
# Add Point for Mean
add_markers(x = ~Mean, y = ~Group, marker = list(symbol = "diamond", size = 10, color = "pink"),
name = "Mean (210)") %>%
# Layout
layout(title = "<b>Latency Analysis: Z-Interval (Known σ) vs. t-Interval (Unknown σ)</b>",
xaxis = list(title = "API Latency (ms)"),
yaxis = list(title = "", categoryorder = "array",
categoryarray = rev(case4_data$Group)),
margin = list(l = 150),
hovermode = "y unified")
plibrary(plotly)
# 1. Setup Parameters
mu <- 210
se <- 4
df_val <- 35
x <- seq(mu - 4.5*se, mu + 4.5*se, length.out = 500)
# 2. Team A (Z-Dist) Function
p1 <- plot_ly(x = x) %>%
add_lines(y = dnorm(x, mu, se), name = "Z-Distribution", line = list(color = '#1f77b4'), legendgroup = "Z") %>%
# 99% Layer
add_polygons(x = c(mu - 2.576*se, seq(mu - 2.576*se, mu + 2.576*se, length.out = 100), mu + 2.576*se),
y = c(0, dnorm(seq(mu - 2.576*se, mu + 2.576*se, length.out = 100), mu, se), 0),
fillcolor = 'rgba(31, 119, 180, 0.2)', line = list(color = 'transparent'), name = "99% CI (Z)") %>%
# 95% Layer
add_polygons(x = c(mu - 1.96*se, seq(mu - 1.96*se, mu + 1.96*se, length.out = 100), mu + 1.96*se),
y = c(0, dnorm(seq(mu - 1.96*se, mu + 1.96*se, length.out = 100), mu, se), 0),
fillcolor = 'rgba(31, 119, 180, 0.4)', line = list(color = 'transparent'), name = "95% CI (Z)") %>%
# 90% Layer
add_polygons(x = c(mu - 1.645*se, seq(mu - 1.645*se, mu + 1.645*se, length.out = 100), mu + 1.645*se),
y = c(0, dnorm(seq(mu - 1.645*se, mu + 1.645*se, length.out = 100), mu, se), 0),
fillcolor = 'rgba(31, 119, 180, 0.6)', line = list(color = 'transparent'), name = "90% CI (Z)") %>%
add_segments(x = mu, xend = mu, y = 0, yend = 0.1, line = list(color = "red", dash = "dash"), name = "Mean")
# 3. Team B (t-Dist) Function
p2 <- plot_ly(x = x) %>%
add_lines(y = dt((x - mu)/se, df = df_val)/se, name = "t-Distribution", line = list(color = '#ff7f0e'), legendgroup = "t") %>%
# 99% Layer
add_polygons(x = c(mu - 2.724*se, seq(mu - 2.724*se, mu + 2.724*se, length.out = 100), mu + 2.724*se),
y = c(0, (dt((seq(mu - 2.724*se, mu + 2.724*se, length.out = 100) - mu)/se, df = df_val)/se), 0),
fillcolor = 'rgba(255, 127, 14, 0.2)', line = list(color = 'transparent'), name = "99% CI (t)") %>%
# 95% Layer
add_polygons(x = c(mu - 2.03*se, seq(mu - 2.03*se, mu + 2.03*se, length.out = 100), mu + 2.03*se),
y = c(0, (dt((seq(mu - 2.03*se, mu + 2.03*se, length.out = 100) - mu)/se, df = df_val)/se), 0),
fillcolor = 'rgba(255, 127, 14, 0.4)', line = list(color = 'transparent'), name = "95% CI (t)") %>%
# 90% Layer
add_polygons(x = c(mu - 1.69*se, seq(mu - 1.69*se, mu + 1.69*se, length.out = 100), mu + 1.69*se),
y = c(0, (dt((seq(mu - 1.69*se, mu + 1.69*se, length.out = 100) - mu)/se, df = df_val)/se), 0),
fillcolor = 'rgba(255, 127, 14, 0.6)', line = list(color = 'transparent'), name = "90% CI (t)") %>%
add_segments(x = mu, xend = mu, y = 0, yend = 0.1, line = list(color = "red", dash = "dash"), showlegend = FALSE)
# 4. Final Subplot Layout
p <- subplot(p1, p2, nrows = 1, margin = 0.05, shareY = TRUE, titleX = TRUE) %>%
layout(title = "<b>Multi-Level Precision: Team A (Z) vs. Team B (t)</b>",
legend = list(orientation = 'h', y = -0.2),
margin = list(t = 80, b = 100),
xaxis = list(title = "Latency (Z-Known)"),
xaxis2 = list(title = "Latency (t-Unknown)"))
pThe “Information Penalty” (Z vs. t)
In this case, we compare the results of two teams measuring API latency. While their raw data looks identical, the underlying assumptions about their knowledge of the population change the precision of their estimates.
Statistical Method Identification
Team A: Z-Confidence Interval * Condition: Population standard deviation \(\sigma\) is known. * Formula: \(\bar{x} \pm z_{\alpha/2} \left(\frac{\sigma}{\sqrt{n}}\right)\)
Team B: t-Confidence Interval * Condition: Population standard deviation \(\sigma\) is unknown (\(s\) is used as an estimate). * Formula: \(\bar{x} \pm t_{\alpha/2, n-1} \left(\frac{s}{\sqrt{n}}\right)\)
Why the Interval Widths Differ
Despite both teams having a Standard Error (\(SE\)) of \(4\), Team B’s intervals are consistently wider.
- The Uncertainty of \(s\): When we estimate \(\sigma\) using the sample standard deviation \(s\), we introduce additional sampling error. To compensate, the Student’s t-distribution features “heavier tails” than the Standard Normal distribution.
- The Critical Value Penalty: To maintain the same
level of confidence, the critical value for \(t\) (\(t_{\alpha/2, 35}\)) is mathematically
larger than \(z_{\alpha/2}\).
- Example (95%): \(z = 1.960\) vs. \(t = 2.030\).
- Convergence: As \(n\) increases, the t-distribution converges to the Z-distribution. At \(n=36\), the “penalty” for using \(t\) is small but statistically significant for high-precision engineering.
Strategic Recommendation for the SaaS Company
Your next step involves the Lower Bound analysis for the premium feature. In a “success-threshold” scenario:
- The Problem: The company needs \(\ge 70\%\) adoption.
- The Data: \(\hat{p} = \frac{185}{250} = 0.74\) (\(74\%\)).
- The Decision: Even though \(74\% > 70\%\), management must check the 95% Lower Bound. If the lower bound is above \(70\%\), the feature launch is a confirmed success. If it dips below \(70\%\), there is still a statistical risk that the true adoption rate is failing to meet the target.
5 Problem Description
A SaaS company wants to ensure that at least 70% of weekly active users utilize a premium feature.
From the experiment:
\[ n = 250 \quad \text{(total users)} \]
\[ x = 185 \quad \text{(active premium users)} \]
Management is only interested in the lower bound of the confidence interval.
5.1 Type of Confidence Interval and Test
Because: - The parameter of interest is a population proportion - Sample size is large - Only the minimum acceptable performance is of interest
The appropriate method is:
\[ \boxed{\text{One-Sided (Lower) Z-Confidence Interval for a Proportion}} \]
This approach is commonly used in SLA guarantees, compliance checks, and product thresholds.
5.2 Sample Proportion Calculation
\[ \hat{p} = \frac{x}{n} \]
\[ \hat{p} = \frac{185}{250} = 0.74 \]
The observed premium feature usage rate is 74%.
5.3 One-Sided Confidence Interval Formula (Lower Bound)
For a one-sided lower confidence interval:
\[ \text{Lower Bound} = \hat{p} - z_{\alpha} \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}} \]
Where: - \(z_{\alpha}\) is the critical value for a one-tailed test - \(\alpha = 1 - \text{Confidence Level}\)
5.4 Step-by-Step Manual Calculation
Standard Error
\[ SE = \sqrt{\frac{0.74(1 - 0.74)}{250}} \]
\[ SE = \sqrt{\frac{0.1924}{250}} = 0.0277 \]
Z-Critical Values (One-Sided)
| Confidence Level | \(\alpha\) | \(z_{\alpha}\) |
|---|---|---|
| 90% | 0.10 | 1.282 |
| 95% | 0.05 | 1.645 |
| 99% | 0.01 | 2.326 |
Lower Confidence Bounds
\[ \text{Lower Bound} = \hat{p} - z_{\alpha} \times SE \]
90% Lower CI: \[ 0.74 - 1.282(0.0277) = 0.705 \]
95% Lower CI: \[ 0.74 - 1.645(0.0277) = 0.694 \]
99% Lower CI: \[ 0.74 - 2.326(0.0277) = 0.676 \]
5.5 SaaS Analysis (One-Sided Bounds)
In this scenario, we move from estimating a range to confirming a threshold. Management needs to know the “worst-case scenario” (the lower bound) to ensure the 70% KPI is met.
Statistical Strategy: One-Sided Z-Interval Since we are dealing with a large sample proportion (\(n=250\), \(np \ge 10\)) and only care about the minimum performance, we use the One-Sided Lower Z-Confidence Interval.
\[\text{Lower Bound} = \hat{p} - z_{\alpha} \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}\]
Executive Summary of Lower Bounds Based on our calculations (\(SE = 0.0277\)):
| Confidence Level | \(z_{\alpha}\) (One-Tail) | Lower Bound | Statistically \(> 70\%\)? |
|---|---|---|---|
| 90% | 1.282 | 70.5% | YES |
| 95% | 1.645 | 69.4% | NO |
| 99% | 2.326 | 67.6% | NO |
Visualization: The “Success Threshold”
library(plotly)
# 1. Setup Parameters
p_hat <- 0.74
se <- 0.0277
target <- 0.70
x <- seq(0.64, 0.84, length.out = 500)
y <- dnorm(x, p_hat, se)
# 2. Build One-Sided Mountain
p <- plot_ly(x = x, y = y, type = 'scatter', mode = 'lines',
line = list(color = '#2ca02c', width = 3), name = "Likelihood Distribution") %>%
# Shading the "Safe Zone" for 90% (One-sided)
add_polygons(x = c(0.705, seq(0.705, 0.84, length.out = 100), 0.84),
y = c(0, dnorm(seq(0.705, 0.84, length.out = 100), p_hat, se), 0),
fillcolor = 'rgba(44, 160, 44, 0.4)', line = list(color = 'transparent'),
name = "90% Lower Zone") %>%
# Target Line (70%)
add_segments(x = target, xend = target, y = 0, yend = max(y),
line = list(color = "red", width = 2, dash = "dot"), name = "70% Target") %>%
# Point Estimate (74%)
add_segments(x = p_hat, xend = p_hat, y = 0, yend = max(y),
line = list(color = "black", width = 1), name = "Observed (74%)") %>%
# Layout
layout(title = "<b>SaaS Feature Adoption: Lower Bound Analysis</b>",
xaxis = list(title = "Adoption Rate (%)", tickformat = ".0%"),
yaxis = list(title = "Density", showticklabels = FALSE),
annotations = list(
list(x = 0.70, y = 10, text = "Target 70%", showarrow = T, arrowhead = 2),
list(x = 0.74, y = 14, text = "Current 74%", showarrow = F)
))
plibrary(plotly)
# Data Setup
lower_bounds <- data.frame(
Confidence = c("90%", "95%", "99%"),
Bound = c(0.705, 0.694, 0.676),
Status = c("Pass", "Fail", "Fail")
)
p <- plot_ly(lower_bounds) %>%
# 1. Add the "Success Area" (70% and above)
add_polygons(x = c(0.70, 0.80, 0.80, 0.70), y = c(-0.5, -0.5, 2.5, 2.5),
fillcolor = 'rgba(0, 255, 0, 0.1)', line = list(color = 'transparent'),
name = "Success Zone (>70%)", showlegend = TRUE) %>%
# 2. Add the Lower Bound Markers
add_segments(x = ~Bound, xend = 0.74, y = ~Confidence, yend = ~Confidence,
line = list(color = '#737373', width = 4), name = "Distance to Mean") %>%
add_markers(x = ~Bound, y = ~Confidence,
marker = list(symbol = "line-ns-open", size = 20, color = ~Status,
colors = c("Pass" = "green", "Fail" = "red")),
name = "Lower Bound") %>%
# 3. Add Target Line
add_segments(x = 0.70, xend = 0.70, y = -0.5, yend = 2.5,
line = list(color = "red", dash = "dash", width = 3), name = "70% Target") %>%
# Layout
layout(title = "<b>Critical Lower Bound vs. 70% Target</b>",
xaxis = list(title = "Adoption Rate (%)", tickformat = ".1%", range = c(0.65, 0.76)),
yaxis = list(title = "Confidence Level"),
showlegend = TRUE)
pIs the 70% target satisfied? The answer depends entirely on the company’s Risk Tolerance:
- At 90% Confidence: YES. The lower bound is \(70.5\%\). We are \(90\%\) certain that the true adoption rate is at least \(70.5\%\), which clears the target.
- At 95% Confidence: NO. The lower bound drops to \(69.4\%\). While the average is \(74\%\), there is more than a \(5\%\) statistical chance that the true adoption is actually slightly below the target.
- Final Recommendation: * If this is a minor
feature, the \(90\%\) result
is likely enough to claim success.
- If this is a critical premium gate required for investors, we should increase the sample size (\(n\)) to shrink the \(SE\) and push the lower bound safely above \(70\%\).