1.1 Identify the appropriate statistical test and justify your choice.

Appropriate Statistical Test

Z-interval (Confidence Interval for the Mean)

Justification

• The objective of the analysis is to estimate the population mean of daily transactions per user based on a single random sample.

• The population standard deviation (\(\sigma = 3.2\)) is assumed to be known.

• The sample size is sufficiently large (\(n = 100 \ge 30\)), allowing the sampling distribution of the sample mean to be approximated by a normal distribution according to the Central Limit Theorem.

1.2 Confidence Interval Computation

The confidence interval for the population mean when \(\sigma\) is known is given by:

a. 90% Confidence Interval

Critical values:

Calculation:

Conclusion

With a 90% confidence level, the true population mean of daily transactions per user is estimated to lie between 12.07 and 13.13 transactions per day

b. 95% Confidence Interval

Critical values

Calculation

Conclusion

With a 95% confidence level, the population mean of daily transactions per user is estimated to be between 11.97 and 13.23 transactions per day.

c. 99% Confidence Interval

Critical values

Calculation

Conclusion

With a 99% confidence level, the true population mean is estimated to fall within the interval (11.78, 13.42). The wider interval reflects a higher confidence level at the cost of lower precision.

1.3 Comparison of 90%, 95%, and 99% Confidence Intervals

This section was created to compare the width and position of three confidence intervals (90%, 95%, and 99%) of the average number of daily transactions per user. Through this visualization, differences in confidence levels can be observed directly, particularly that the higher the confidence level, the wider the resulting estimation interval.

ci_data <- data.frame(
  Confidence_Level = factor(
    c("90%", "95%", "99%"),
    levels = c("90%", "95%", "99%")
  ),
  Lower = c(12.07, 11.97, 11.78),
  Upper = c(13.13, 13.23, 13.42),
  Mean  = c(12.6, 12.6, 12.6)
)

ggplot(ci_data, aes(
  y = Confidence_Level,
  x = Mean,
  color = Confidence_Level
)) +
  geom_errorbar(
    aes(xmin = Lower, xmax = Upper),
    orientation = "y",
    linewidth = 1
  ) +
  geom_point(size = 3) +
  labs(
    title = "Comparison of Confidence Intervals",
    subtitle = "Mean Daily Transactions per User",
    x = "Average Number of Daily Transactions",
    y = "Confidence Level",
    color = "Confidence Interval"
  ) +
  theme_minimal(base_size = 14) +
  theme(
    plot.title = element_text(face = "bold", hjust = 0.5),
    legend.position = "right"
  )

Interpretation of Interval Comparison

Based on a comparison of the three confidence intervals (90%, 95%, and 99%) that I have obtained, it can be concluded that the higher the confidence level used, the wider the resulting estimation interval. Although all intervals have the same mean (sample mean of 12.6), the 99% confidence interval covers the widest range of values to ensure that the population mean is truly within that interval. This shows a trade-off between confidence level and estimation precision, where a higher confidence level produces a wider but still statistically valid interval.

1.4 Business analytics interpretation

From a business analytics perspective, the estimated average number of daily transactions per user after the launch of the new feature is consistently above 12 transactions per day across all confidence levels examined (90%, 95%, and 99%). This provides strong statistical evidence that the feature is associated with a stable and meaningful level of user engagement.

The confidence intervals serve not merely as descriptive summaries, but as risk-aware decision tools. Even under the most conservative scenario (99% confidence level), the lower bound of the interval remains close to 12 transactions per user. This indicates that, with very high confidence, the true average performance of the feature does not fall substantially below the observed mean.

For business decision-making, this has clear implications. A narrower interval at the 90% and 95% confidence levels offers greater precision and is suitable for operational planning and short-term optimization, such as forecasting daily system load or setting performance benchmarks. Meanwhile, the wider 99% confidence interval provides a more cautious estimate, which is appropriate for strategic decisions involving higher risk tolerance, such as long-term investment or scaling the feature to a larger user base.

Overall, the results suggest that the new feature delivers a reliable improvement in transaction activity, and the consistency of the confidence intervals across multiple confidence levels strengthens managerial confidence in the stability of this outcome. Rather than relying on a single point estimate, the use of confidence intervals enables the business to balance precision, uncertainty, and risk in a statistically principled manner. This statistical evidence supports the decision to maintain or further optimize the feature, as the observed performance is both stable and economically meaningful.

2.1 Identify the Appropriate Statistical Test

Selected Statistical Method

t-Confidence Interval for the Mean (One-Sample t-Interval)

*Justification **

• The analysis aims to estimate the population mean based on a single sample.

• The population standard deviation (σ is unknown).

• The sample size is relatively small (n = 12 < 30).

• Under these conditions, the t-distribution is more appropriate than the normal distribution, assuming the data are approximately normally distributed.

• Therefore, a one-sample t-confidence interval is the appropriate statistical method.

It is assumed that the task completion time is approximately normally distributed, which is reasonable given the controlled UX testing environment and the absence of extreme outliers.

2.2 Compute the Confidence Intervals for

Data and Basic Statistics

a. 90% Confidence Intervals

Significance Level

\[ \alpha = 1 - 0.90 = 0.10 \]

\[ \frac{\alpha}{2} = 0.05 \] Critical Value from the t-Distribution

\[ t_{0.05,11} \approx 1.796 \]

Margin of Error (MOE)

\[ \text{MOE} = t_{0.05,11} \times \frac{s}{\sqrt{n}} \]

\[ \text{MOE} = 1.796 \times \frac{0.421}{\sqrt{12}} \]

\[ \text{MOE} \approx 0.218 \]

Confidence Interval Estimation

\[ \bar{x} \pm \text{MOE} \]

\[ 8.458 \pm 0.218 \]

\[ \boxed{\text{CI}_{90\%} = (8.240,\; 8.676)} \]

b. 95% Confidence Intervals

Significance Level

\[ \alpha = 1 - 0.95 = 0.05 \]

\[ \frac{\alpha}{2} = 0.025 \]

Critical Value from the t-Distribution

\[ t_{0.025,11} \approx 2.201 \]

Margin of Error (MOE)

\[ \text{MOE} = t_{0.025,11} \times \frac{s}{\sqrt{n}} \]

\[ \text{MOE} = 2.201 \times \frac{0.421}{\sqrt{12}} \]

\[ \text{MOE} \approx 0.267 \]

Confidence Interval Estimation

\[ \bar{x} \pm \text{MOE} \]

\[ 8.458 \pm 0.267 \]

\[ \boxed{\text{CI}_{95\%} = (8.191,\; 8.725)} \]

c. 99% Confidence Intervals

Significance Level

\[ \alpha = 1 - 0.99 = 0.01 \]

\[ \frac{\alpha}{2} = 0.005 \]

Critical Value from the t-Distribution

\[ t_{0.005,11} \approx 3.106 \]

Margin of Error (MOE)

\[ \text{MOE} = t_{0.005,11} \times \frac{s}{\sqrt{n}} \]

\[ \text{MOE} = 3.106 \times \frac{0.421}{\sqrt{12}} \]

\[ \text{MOE} \approx 0.377 \]

Confidence Interval Estimation

\[ \bar{x} \pm \text{MOE} \]

\[ 8.458 \pm 0.377 \]

\[ \boxed{\text{CI}_{99\%} = (8.081,\; 8.835)} \]

2.3 Comparison of 90%, 95%, and 99% Confidence Intervals

This section presents a visual comparison of the 90%, 95%, and 99% confidence intervals for the mean task completion time. The visualization is designed to highlight how increasing the confidence level affects the width of the interval while keeping the sample mean fixed. By examining these intervals side by side, the trade-off between statistical reliability and estimation precision becomes explicit, providing a clear basis for informed decision making in UX performance evaluation.

library(ggplot2)

# Confidence interval results (from your calculations)
ci_data <- data.frame(
  Confidence_Level = factor(
    c("90%", "95%", "99%"),
    levels = c("90%", "95%", "99%")
  ),
  Lower = c(8.240, 8.191, 8.081),
  Upper = c(8.676, 8.725, 8.835),
  Mean  = c(8.458, 8.458, 8.458)
)

# Plot
ggplot(ci_data, aes(
  y = Confidence_Level,
  x = Mean,
  color = Confidence_Level
)) +
  
  # Confidence interval lines
  geom_errorbar(
    aes(xmin = Lower, xmax = Upper),
    linewidth = 1.2,
    width = 0.15
  ) +
  
  # Mean point
  geom_point(size = 3) +
  
  # Vertical line for sample mean
  geom_vline(
    xintercept = 8.458,
    linetype = "dashed",
    linewidth = 1,
    color = "black"
  ) +
  
  # Annotation
  annotate(
    "text",
    x = 8.458,
    y = 3.4,
    label = expression(bar(x) == 8.458),
    hjust = -0.1
  ) +
  
  labs(
    title = "Comparison of t-Confidence Intervals",
    subtitle = "Mean Task Completion Time (σ Unknown, n = 12)",
    x = "Task Completion Time (minutes)",
    y = "Confidence Level",
    color = "Confidence Level"
  ) +
  
  theme_minimal(base_size = 14) +
  theme(
    plot.title = element_text(face = "bold", hjust = 0.5),
    legend.position = "right"
  )

Conclusion

The visualization shows that increasing the confidence level from 90% to 99% widens the confidence interval while the sample mean remains unchanged. This indicates a clear trade-off between statistical certainty and estimation precision.

In a UX analytics context, the 95% confidence interval provides the most practical balance sufficiently reliable for decision making while still precise enough to inform design and performance improvements.

2.4 Influence of Sample Size and Confidence Level on the Width of Confidence Intervals

The width of a confidence interval is influenced by both the sample size and the selected confidence level. In this study, the sample size is fixed at \(n = 12\), meaning that variability in interval width is not driven by changes in the number of observations. Generally, larger sample sizes reduce the standard error, leading to narrower confidence intervals and more precise estimates of the population mean.

Regarding confidence level, as it increases from \(90\%\) to \(95\%\) and further to \(99\%\), the corresponding critical value from the t-distribution becomes larger. This results in progressively wider confidence intervals, reflecting a statistical trade-off between precision and reliability. Higher confidence levels ensure a greater likelihood that the true population mean is captured within the interval, but at the cost of reduced precision. Overall, this analysis demonstrates that uncertainty in the estimated mean task completion time is primarily driven by the chosen confidence level rather than sampling variability. Consequently, selecting an appropriate confidence level becomes a strategic decision that should align with the risk tolerance and decision context of UX evaluation.

3.1 Compute the sample proportion \({\hat{p}}\)

The sample proportion represents the proportion of users in the experiment who clicked the Call To Action (CTA) button. It is calculated as the ratio of the number of users who clicked the CTA to the total number of users in the experiment.

\[\boldsymbol{\hat{p}} = \frac{x}{n} = \frac{156}{400} = 0.39\] Thus, the estimated sample proportion of users who clicked the CTA button is 0.39, meaning that approximately 39% of users in the experiment interacted with the CTA.

3.2 Compute Confidence Intervals for the proportion at:

Standard Error of the Proportion \[ SE(\hat{p}) = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}} = \sqrt{\frac{0.39(1-0.39)}{400}} = \sqrt{0.00059475} \approx 0.0244 \] a. 90% Confidence Interval \[ \begin{aligned} \alpha &= 0.10 \\ z_{1-\alpha/2} &= z_{0.95} = 1.645 \end{aligned} \] \[ \text{MOE} = 1.645 \times 0.0244 \approx 0.040 \]

\[ \boldsymbol{\hat{p}} \pm \text{MOE} = 0.39 \pm 0.040 \]

\[ \boxed{\text{CI}_{90\%} = (0.350,\; 0.430)} \]

b. 95% Confidence Interval*

\[ \begin{aligned} \alpha &= 0.05 \\ z_{1-\alpha/2} &= z_{0.975} = 1.96 \end{aligned} \]

\[ \text{MOE} = 1.96 \times 0.0244 \approx 0.048 \]

\[ \boldsymbol{\hat{p}} \pm \text{MOE} = 0.39 \pm 0.048 \]

\[ \boxed{\text{CI}_{95\%} = (0.342,\; 0.438)} \]

c. 99% Confidence Interval

\[ \begin{aligned} \alpha &= 0.01 \\ z_{1-\alpha/2} &= z_{0.995} = 2.576 \end{aligned} \]

\[ \text{MOE} = 2.576 \times 0.0244 \approx 0.063 \]

\[ \boldsymbol{\hat{p}} \pm \text{MOE} = 0.39 \pm 0.063 \]

\[ \boxed{\text{CI}_{99\%} = (0.327,\; 0.453)} \] Interpretation

As the confidence level increases from 90% to 99%, the confidence interval becomes wider. This reflects the trade off between precision and reliability higher confidence ensures a greater likelihood that the true population proportion is captured, but results in less precise estimates. Visualize and compare the three intervals. Explain how confidence level affects decisionmaking in product experiments.

3.3 Visualization and Comparison of Confidence Intervals

This visualization is used to compare the width and position of the 90%, 95%, and 99% confidence intervals for the sample proportion of users who clicked the CTA. The comparison highlights how increasing confidence levels result in wider intervals while maintaining the same point estimate.

3.4 Effect of Confidence Level on Decision Making in Product Experiments

In product experiments such as A/B testing, the confidence level determines the degree of statistical certainty required before making a decision. Formally, the confidence level is defined as \(1 - \alpha\), where \(\alpha\) represents the probability of committing a Type I error (incorrectly concluding that an effect exists).

As the confidence level increases (which corresponds to a smaller value of 𝛼 α), the critical value from the sampling distribution also increases. A larger critical value directly leads to a wider confidence interval, reflecting greater statistical assurance at the cost of reduced precision.

The relationship between the confidence level and the width of the confidence interval can be expressed as follows:

\[ \text{Confidence Interval Width} \propto z_{1-\alpha/2} \times \text{Standard Error} \]

For small sample sizes, where the population standard deviation is unknown, the 𝑡 t-distribution is used instead of the normal distribution. In this case, the relationship becomes:

\[ \text{Confidence Interval Width} \propto t_{1-\alpha/2,\,df} \times \text{Standard Error} \]

This theoretical relationship is clearly illustrated in the confidence interval visualization, where the \(99\%\) interval is the widest, followed by \(95\%\), and then \(90\%\). although all intervals are centered at the same sample proportion \(\hat{p}\), their widths differ due solely to the chosen confidence level.

From a decision making perspective, a \(90\%\) confidence level \((\alpha = 0.10)\) produces narrower intervals, enabling faster and more decisive product iterations, but with a higher risk of false positives. A \(95\%\) confidence level \((\alpha = 0.05)\) offers a balance between reliability and precision, making it the most commonly adopted standard in industry experiments. A \(99\%\) confidence level \((\alpha = 0.01)\) results in very wide intervals, reducing the risk of incorrect decisions but potentially delaying actionable conclusions when effect sizes are small.

Therefore, higher confidence levels increase statistical reliability but reduce decisiveness. In product analytics, selecting an appropriate confidence level is a strategic choice that must align with business risk tolerance, experiment cost, and potential impact.

Why the Visualization Matters

The visualization reinforces this concept empirically: even though the point estimate \(\hat{\theta}\) remains constant, increasing confidence levels expand the interval range, signaling greater uncertainty in pinpointing the true effect. This visual comparison helps stakeholders understand why higher statistical rigor may slow down decision making in practice.

4.1 Identify the Statistical Test

Appropriate Statistical Test

Team A : Z-Interval (Confidence Interval for the Mean)

Team B : t-Interval (Confidence Interval for the Mean)

Justification

Team A (Z-Interval)

• The objective is to estimate the population mean API latency based on a single sample.

• The population standard deviation is known, \(\sigma = 24\).

• The sample size is sufficiently large \((n = 36 \ge 30)\).

• By the Central Limit Theorem, the sampling distribution of \(\bar{x}\) can be approximated by a normal distribution.

Team B (t-Interval)

• The objective remains estimating the population mean.

• The population standard deviation is unknown and is estimated using the sample standard deviation \(s = 24\).

• This additional uncertainty requires the use of the t-distribution with degrees of freedom \(df = n - 1\).

4.2 Confidence Interval Computation

General Formulas

Z-Interval (Team A): \[ \bar{x} \pm z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}} \] t-Interval (Team B): \[ \bar{x} \pm t_{1-\alpha/2,\,df}\frac{s}{\sqrt{n}} \] Team A — Confidence Intervals (σ Known)

\[ n = 36, \quad \bar{x} = 210, \quad \sigma = 24 \]

Standard error: \[ \frac{\sigma}{\sqrt{n}} = \frac{24}{\sqrt{36}} = 4 \]

a. 90% Confidence Interval \[ \begin{aligned} \alpha &= 0.10 \\ z_{1-\alpha/2} &= z_{0.95} = 1.645 \end{aligned} \]

\[ \begin{aligned} \bar{x} \pm z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}} &= 210 \pm 1.645 \times \frac{24}{\sqrt{36}} \\ &= 210 \pm 1.645 \times 4 \\ &= 210 \pm 6.58 \\ &= (203.42,\;216.58) \end{aligned} \]

b. 95% Confidence Interval \[ \begin{aligned} \alpha &= 0.05 \\ z_{1-\alpha/2} &= z_{0.975} = 1.96 \end{aligned} \]

\[ \begin{aligned} \bar{x} \pm z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}} &= 210 \pm 1.96 \times \frac{24}{\sqrt{36}} \\ &= 210 \pm 1.96 \times 4 \\ &= 210 \pm 7.84 \\ &= (202.16,\;217.84) \end{aligned} \]

c. 99% Confidence Interval \[ \begin{aligned} \alpha &= 0.01 \\ z_{1-\alpha/2} &= z_{0.995} = 2.576 \end{aligned} \]

\[ \begin{aligned} \bar{x} \pm z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}} &= 210 \pm 2.576 \times \frac{24}{\sqrt{36}} \\ &= 210 \pm 2.576 \times 4 \\ &= 210 \pm 10.304 \\ &= (199.70,\;220.30) \end{aligned} \]

Team B — Confidence Interval for the Mean (σ Unknown)

\[ n = 36,\quad \bar{x} = 210,\quad s = 24,\quad df = 35 \]

Standard error: \[ \frac{s}{\sqrt{n}} = \frac{24}{\sqrt{36}} = 4 \]

a. 90% Confidence Interval \[ \begin{aligned} t_{0.05,\,35} &\approx 1.690 \\ \\ 210 \pm 1.690 \times 4 &= 210 \pm 6.76 \\ &= (203.24,\;216.76) \end{aligned} \]

b. 95% Confidence Interval \[ \begin{aligned} t_{0.025,\,35} &\approx 2.030 \\ \\ 210 \pm t_{0.025,\,35}\times 4 &= 210 \pm 2.030 \times 4 \\ &= 210 \pm 8.12 \\ &= (201.88,\;218.12) \end{aligned} \]

c. 99% Confidence Interval \[ \begin{aligned} t_{0.005,\,35} &\approx 2.724 \\ \\ 210 \pm t_{0.005,\,35}\times 4 &= 210 \pm 2.724 \times 4 \\ &= 210 \pm 10.896 \\ &= (199.10,\;220.90) \end{aligned} \]

4.3 Visual Comparison of Confidence Intervals: Z-Test vs t-Test

This visualization compares the confidence intervals at 90%, 95%, and 99% for Team A (Z-interval) and Team B (t-interval) to highlight differences in precision when the population standard deviation is known versus unknown.

library(ggplot2)

# Data CI dari perhitungan manual Case Study 4
ci_data <- data.frame(
  Team = rep(c("Team A (Z-Interval)", "Team B (t-Interval)"), each = 3),
  Confidence_Level = factor(
    rep(c("90%", "95%", "99%"), times = 2),
    levels = c("90%", "95%", "99%")
  ),
  Lower = c(
    203.42, 202.16, 199.70,   # Team A (Z)
    203.24, 201.88, 199.10    # Team B (t)
  ),
  Upper = c(
    216.58, 217.84, 220.30,   # Team A (Z)
    216.76, 218.12, 220.90    # Team B (t)
  ),
  Mean = 210
)

ggplot(ci_data, aes(
  y = Confidence_Level,
  x = Mean,
  color = Team
)) +
  
  # Confidence interval bars
  geom_errorbar(
    aes(xmin = Lower, xmax = Upper),
    width = 0.18,
    linewidth = 1.1,
    position = position_dodge(width = 0.6)
  ) +
  
  # Mean point
  geom_point(
    size = 3,
    position = position_dodge(width = 0.6)
  ) +
  
  # Sample mean (x-bar)
  geom_vline(
    xintercept = 210,
    linetype = "dashed",
    linewidth = 1.2,
    color = "black"
  ) +
  
  annotate(
    "text",
    x = 210,
    y = 3.6,
    label = expression(bar(x) == 210),
    hjust = -0.1
  ) +
  
  labs(
    title = "Visual Comparison of Confidence Intervals",
    subtitle = "Z-Interval (σ known) vs t-Interval (σ unknown)",
    x = "API Latency (milliseconds)",
    y = "Confidence Level",
    color = "Statistical Method"
  ) +
  
  theme_minimal(base_size = 14) +
theme(
  plot.title = element_text(hjust = 0.5, face = "bold"),
  plot.subtitle = element_text(hjust = 0.5),
  legend.position = "bottom"
)

Insight

The visualization demonstrates a clear difference in interval precision between Team A (Z-test) and Team B (t-test) across all confidence levels (90%, 95%, and 99%). Although both teams share the same sample size and sample mean, Team B’s confidence intervals are consistently wider. This occurs because the t-distribution incorporates additional uncertainty arising from estimating the population standard deviation using the sample standard deviation. As the confidence level increases, this difference becomes more pronounced, especially at the 99% level, where the heavier tails of the t-distribution significantly expand the interval width. This comparison confirms a fundamental statistical principle: when population parameters are known, estimation is more precise; when they are estimated from data, confidence intervals must be wider to maintain the same level of statistical reliability. In practical analytics and engineering contexts, this implies that teams with incomplete population information must accept reduced precision in exchange for valid inference.

4.4 Interval Width Comparison

90% Confidence Level

• Z-interval width: 13.16 ms
• t-interval width: 13.52 ms

95% Confidence Level

• Z-interval width: 15.68 ms
• t-interval width: 16.24 ms

99% Confidence Level

• Z-interval width: 20.60 ms
• t-interval width: 21.80 ms

4.5 Explanation of Why the Interval Widths Differ, Even with Similar Data

Although Team A and Team B use the same sample size \((n = 36)\) and identical sample means \((\bar{x} = 210)\), the widths of their confidence intervals differ due to differences in assumptions about population variability and the resulting sampling distributions.

For Team A, the population standard deviation is known \((\sigma = 24)\), allowing the use of the Z-distribution. The confidence interval for the mean is given by

\[ \bar{x} \pm z_{1-\alpha/2}\frac{\sigma}{\sqrt{n}} \]

Because \(\sigma\) is fixed and known, the standard error \(\frac{\sigma}{\sqrt{n}}\) is constant, and the critical values \(z_{1-\alpha/2}\) are relatively smaller.

In contrast, Team B does not know the population standard deviation and must estimate it using the sample standard deviation \(s = 24\). This introduces additional uncertainty, requiring the use of the t-distribution with degrees of freedom \(df = n - 1\). The corresponding confidence interval is

\[ \bar{x} \pm t_{1-\alpha/2,\,df}\frac{s}{\sqrt{n}} \]

Since the t-distribution has heavier tails than the normal distribution, the critical values \(t_{1-\alpha/2,\,df}\) are larger than the corresponding \(z\)-values for the same confidence level. Consequently, even with similar data, the confidence intervals for Team B are wider.

This difference becomes increasingly pronounced as the confidence level rises from 90% to 95% and 99%, because higher confidence levels require larger critical values. The visualization clearly reflects this theoretical relationship: although all intervals are centered at the same mean, Team B’s intervals expand more rapidly than Team A’s as confidence increases.

Overall, this comparison illustrates a fundamental principle of statistical inference: when population parameters are estimated rather than known, confidence intervals must widen to maintain the stated level of confidence.

5.1 Identify the type of Confidence Interval and the appropriate test.

Type of Confidence Interval

The appropriate interval is a one-sided confidence interval for a population proportion.

This type of interval is suitable when the objective is to ensure that the true population proportion is at least a certain target value, rather than estimating both lower and upper bounds.

Appropriate Statistical Test

The appropriate statistical method is a One Sample Z-Test for a Proportion (Z-based confidence interval).

Justification

\[ \hat{p} = \frac{x}{n} \] where \(x\) is the number of users with the desired outcome and \(n\) is the total number of users in the sample.

The Z-test for a proportion is appropriate because:

• The data represent binary outcomes (success/failure).

• The sample size is sufficiently large.

• The normal approximation conditions are satisfied:

\[ n\hat{p} \ge 10 \quad \text{and} \quad n(1-\hat{p}) \ge 10 \] Thus, the sampling distribution of \(\hat{p}\) can be approximated by a normal distribution.

Sample Proportion \[ \begin{aligned} n &= 250 \\ x &= 185 \\ \hat{p} &= \frac{x}{n} = \frac{185}{250} = 0.74 \end{aligned} \] One Sided Lower Confidence Interval (Lower Bound Only)

For a one sided lower confidence interval, the formula is: \[ \hat{p} - z_{1-\alpha}\sqrt{\dfrac{\hat{p}(1-\hat{p})}{n}} \]

Standard Error

\[ \sqrt{\frac{\hat{p}(1-\hat{p})}{n}} = \sqrt{\frac{0.74(1-0.74)}{250}} = \sqrt{0.0007696} \approx 0.0277 \]

5.2 One Sided Lower Confidence Intervals

a. 90% Confidence Level

\[ \alpha = 0.10, \quad z_{1-\alpha} = z_{0.90} = 1.282 \]

\[ \text{Lower CI}_{90\%} = 0.74 - 1.282(0.0277) \approx 0.704 \]

b. 95% Confidence Level

\[ \alpha = 0.05, \quad z_{1-\alpha} = z_{0.95} = 1.645 \]

\[ \text{Lower CI}_{95\%} = 0.74 - 1.645(0.0277) \approx 0.694 \]

c. 99% Confidence Level

\[ \alpha = 0.01, \quad z_{1-\alpha} = z_{0.99} = 2.33 \]

\[ \text{Lower CI}_{99\%} = 0.74 - 2.33(0.0277) \approx 0.675 \]

Final Summary (Lower Bounds Only)

\[ \begin{aligned} \text{Lower CI}_{90\%} &\approx 0.704 \\ \text{Lower CI}_{95\%} &\approx 0.694 \\ \text{Lower CI}_{99\%} &\approx 0.675 \end{aligned} \]

Interpretation

• At 90% confidence, the minimum guaranteed proportion of active premium users is 70.4%.

• At 95% confidence, the minimum guaranteed proportion drops to 69.4%.

• At 99% confidence, the minimum guaranteed proportion further decreases to 67.5%.

As the confidence level increases, the lower bound becomes more conservative because the interval widens to provide stronger statistical assurance.

5.3 Visualization

To better understand how the choice of confidence level affects the minimum guaranteed performance, the following visualization presents the one sided lower confidence bounds of the estimated proportion at 90%, 95%, and 99% confidence levels. This comparison highlights how increasing statistical confidence leads to more conservative lower bound estimates, which is particularly important in decision making contexts where ensuring a minimum performance threshold is critical.

# Data
lower_ci <- data.frame(
  Confidence_Level = factor(
    c("90%", "95%", "99%"),
    levels = c("90%", "95%", "99%")
  ),
  Lower_Bound = c(0.704, 0.694, 0.675)
)

p_hat  <- 0.74
target <- 0.70

ggplot(lower_ci, aes(y = Confidence_Level)) +

  # One-sided 
  geom_segment(
    aes(
      x = Lower_Bound,
      xend = p_hat,
      yend = Confidence_Level,
      color = Confidence_Level
    ),
    linewidth = 1.4
  ) +

  # Lower bound points
  geom_point(
    aes(x = Lower_Bound, color = Confidence_Level),
    size = 4
  ) +

  # Target line (70%)
  geom_vline(
    xintercept = target,
    linetype = "dashed",
    linewidth = 1.2,
    color = "red"
  ) +

  # Sample proportion line (p-hat)
  geom_vline(
    xintercept = p_hat,
    linetype = "dashed",
    linewidth = 1.2,
    color = "black"
  ) +

  # Annotations
  annotate(
    "text",
    x = target,
    y = 3.4,
    label = "Target = 70%",
    color = "red",
    hjust = -0.1
  ) +

  annotate(
    "text",
    x = p_hat,
    y = 3.4,
    label = expression(hat(p) == 0.74),
    hjust = -0.1
  ) +

  labs(
    title = "One-Sided Lower Confidence Intervals for Premium Feature Usage",
    subtitle = "Lower Bounds Compared Across Confidence Levels",
    x = "Proportion of Active Premium Users",
    y = "Confidence Level",
    color = "Confidence Level"
  ) +

  coord_cartesian(xlim = c(0.65, 0.76)) +
  theme_minimal(base_size = 14) +
  theme(
    plot.title = element_text(face = "bold", hjust = 0.5),
    legend.position = "right"
  )

Interpretation of the Visualization

The visualization clearly demonstrates that as the confidence level increases from 90% to 99%, the lower confidence bound decreases, even though the underlying sample estimate remains unchanged. This occurs because higher confidence levels require stronger statistical assurance, which results in wider intervals and more conservative lower bound estimates.

From a practical perspective, this means that:

• The 90% confidence level provides a higher lower bound, allowing for more optimistic but riskier conclusions.

• The 95% confidence level offers a balanced trade off between reliability and decision making precision.

• The 99% confidence level yields the most conservative estimate, ensuring a high level of certainty at the cost of reduced decisiveness.

Therefore, this visualization supports the conclusion that higher statistical confidence enhances reliability but tightens decision making flexibility. By explicitly comparing lower bounds, stakeholders can better align experimental conclusions with business risk tolerance and performance guarantees.

5.4 Determining Whether the 70% Target Is Statistically Satisfied

To determine whether the business target of 70% is statistically met, we calculated the lower bound of the one-sided confidence interval for the proportion of active users who utilize premium features. The following table presents a comparison of the calculated lower bounds at confidence levels of 90%, 95%, and 99% against the set target threshold. This analysis is crucial because it highlights how the choice of confidence level affects the company’s final decision and statistical risk tolerance.

Translated with DeepL.com (free version)

The business target is considered statistically satisfied if the one sided lower confidence bound of the population proportion is at least 70%, that is:

\[ \begin{aligned} \text{Lower Bound}_{90\%} &= 0.704 \ge 0.70 \\ \text{Lower Bound}_{95\%} &= 0.694 < 0.70 \\ \text{Lower Bound}_{99\%} &= 0.675 < 0.70 \end{aligned} \]

The 70% minimum target is only statistically supported at the 90% confidence level, but fails to meet the requirement at the 95% and 99% confidence levels. Therefore, the claim that at least 70% of users utilize the premium feature cannot be made with high statistical confidence.