Study Cases
Confidence Interval~ Week 13
Cahaya Medina Semidang
NIM: 52250053
Data Science Undergraduate at ITSB
1 Case Study 1
Confidence Interval for Mean, \(\sigma\) Known: An e-commerce platform wants to estimate the average number of daily transactions per user after launching a new feature. Based on large-scale historical data, the population standard deviation is known.
\[ \begin{eqnarray*} \sigma &=& 3.2 \quad \text{(population standard deviation)} \\ n &=& 100 \quad \text{(sample size)} \\ \bar{x} &=& 12.6 \quad \text{(sample mean)} \end{eqnarray*} \]
Tasks
- Identify the appropriate statistical test and justify your choice.
- Compute the Confidence Intervals for:
- \(90\%\)
- \(95\%\)
- \(99\%\)
- Create a comparison visualization of the three confidence intervals.
- Interpret the results in a business analytics context.
1.1 Answer number 1
1.1.1 Identify the Appropriate Statistical Test
When a population sample mean and the population standard deviation is known, or when the sample size is large (typically \(n \ge 30\)), we can use the normal \(z\) distribution to construct a Confidence Interval.
In the question it is known that the sample mean (\(n\)) is \(100\) and population standard deviation (\(\sigma\)) is \(3.2\) and also the sample size is \(100\) which is (\(>30\)). So, based on the Central Limit Theorem (which was previously studied), the distribution of the sample mean approaches a normal distribution, so the use of the \(z\) distribution is valid.
1.2 Answer number 2
1.2.1 Confidence Intervals
To calculate the Confidence Interval, we use the \(z\) distribution. Here is the formula:
\[CI = \bar{x} \pm z_{\alpha/2}\left(\frac{\sigma}{\sqrt{n}}\right)\] Where:
- \(\bar{x}\) = Sample mean
- \(z_{\alpha/2}\) = Critical Value (it can be found in the normal table)
- \(\alpha = 1 - CL\) (Confidence Level)
- \(\frac{\sigma}{\sqrt{n}}\) = Standard error
- \(\sigma\) = Population Standard Deviation
- \(n\) = Sample size
Summary data:
Sample mean
\[\bar{x} = 12.6 \ \text {transaction/user}\]Population Standard Deviation
\[\sigma = 3.2 \ \text {transaction}\]Sample size
\[n = 100\]
Standard Error (SE) \[\begin{array}{rl} SE = \frac{\sigma}{\sqrt{n}} = \frac{3.2}{\sqrt{100}} = \frac{3.2}{10} = 0.32 \end{array}\]
Critical value \(z\) for \(90\%\) CI
Significance level
\[CL = 90\% \\ \alpha = 1 - 0,9\\ \alpha = 0.1\\ \frac{\alpha}{2} = 0.05\]Standard normal distribution table
\[z_{0.05} = 1.645\]Significance level
\[CL = 95\% \\ \alpha = 1 - 0,95\\ \alpha = 0.05\\ \frac{\alpha}{2} = 0.025\]Standard normal distribution table
\[z_{0.025} = 1.960\]Significance level
\[CL = 99\% \\ \alpha = 1 - 0,99\\ \alpha = 0.01\\ \frac{\alpha}{2} = 0.005\]Standard normal distribution table
\[z_{0.005} = 2.576\]Confidence Interval \(90\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha/2} \times SE \\ & = 1.645 \times 0.32 \\ & = 0.5264 \\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{90\%} & = \bar{x} \pm ME \\ & = 12.6 \pm 0.5264 \\ & = 12.0736,\: 13.1264\\ & \approx (12.07,\; 13.13) \\ \end{array}\]
This means that we are \(90\%\) confident that the true average number of daily transaction per user lies between \(12.07\) and \(13.13\) transaction. This interval is relatively narrow, reflecing less uncertainty compared to higher confidence levels.
Confidence Interval \(95\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha/2} \times SE \\ & = 1.960 \times 0.32 \\ & = 0.6272 \\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{95\%} & = \bar{x} \pm ME \\ & = 12.6 \pm 0.6272 \\ & = 11.9728,\: 13.2272\\ & \approx (11.97,\; 13.23) \\ \end{array}\]
This means that we are \(95\%\) confident that the true average number of daily transaction per user lies between \(11.97\) and \(13.23\) transaction. Compared to the \(90\%\) CI, this interval is slightly wider, reflecting the increased leve of confidence.
Confidence Interval \(99\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha/2} \times SE \\ & = 2.576 \times 0.32 \\ & = 0.8243 \\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{99\%} & = \bar{x} \pm ME \\ & = 12.6 \pm 0.8243 \\ & = 11.7757,\: 13.4243\\ & \approx (11.78,\; 13.42) \\ \end{array}\]
This means that we are \(90\%\) confident that the true average number of daily transaction per user lies between \(11.78\) and \(13.42\) transaction. This interval is wider than both the \(90\%\) and \(95\%\) CI, reflecting the trade-of between precision and confidence.
1.3 Answer number 3
The visualization represents the sampling distribution of the mean number of daily transactions per user following the launch of the new feature.The vertical dashed line at the center corresponds to the sample mean (12.6 transactions). The shaded regions represent confidence intervals at different confidence levels.At the 90% confidence level, the interval is relatively narrow, indicating high precision but lower certainty. As the confidence level increases to 95% and 99%, the intervals widen, reflecting greater statistical caution.Importantly, all confidence intervals are centered around the same mean value, indicating consistency and stability in user transaction behavior.
1.4 Answer number 4
1.4.1 Interpret
Based on analysis, the average number of daily transactions per user after the new feature launch is 12.6 transactions. However, rather than relying on a single point estimate, we evaluated a range of plausible values using confidence intervals to reflect statistical uncertainty.
At a 95% confidence level, we are statistically confident that the true average number of transactions per user lies between 11.97 and 13.23.
Even under a very conservative scenario (99% confidence level), the average is still expected to be no lower than 11.78 transactions per user.
Regardless of the confidence level used, the results consistently indicate that users are completing approximately 12 to 13 transactions per day, which reflects strong engagement with the platform after the feature launch.
Business Impact and Strategic Implications
Evidence of Successful Feature Adoption
The relatively high average transaction count suggests that the new feature is not only being adopted, but is also actively influencing user behavior. Users are interacting more frequently with the platform, which is a strong indicator of feature relevance and usability.From a business perspective, this supports the hypothesis that the feature:
Encourages repeat interactions
Increases transaction frequency
Enhances overall platform stickiness
Stability and Predictability of User Behavior
The confidence intervals are fairly narrow, especially considering the scale of the user base. This indicates:
Low variability in user transaction behavior
Consistent usage patterns across users
Reduced risk of unexpected drops in activity
Risk Assessment and Decision Confidence
Using different confidence levels allows us to quantify risk tolerance in decision-making:
At 90–95% confidence, management can make operational decisions with high certainty while maintaining reasonably tight performance estimates.
At 99% confidence, even under the most conservative assumptions, performance remains strong.
This provides a safety margin for decisions such as:
Scaling infrastructure
Increasing marketing spend
Expanding feature availability to new user segments
Implications for Revenue and KPI Forecasting
Transaction frequency is often directly linked to:
Gross merchandise value (GMV)
Transaction-based fees
User lifetime value (LTV)
Given that the lower bounds of the confidence intervals remain high, revenue forecasts based on these transaction rates can be considered robust, even when accounting for uncertainty.
In practical terms:
Even if actual performance trends toward the lower end of the interval, the platform is still operating at a healthy transaction level.The statistical analysis shows that the new feature has resulted in a stable and consistently high level of user transaction activity. With strong confidence, the average number of daily transactions per user is estimated to be around 12 to 13. The narrow confidence intervals indicate predictable user behavior, reducing operational risk and supporting continued investment in the feature.
2 Case Study 2
Confidence Interval for Mean, \(\sigma\) Unknown: A UX Research team analyzes task completion time (in minutes) for a new mobile application. The data are collected from 12 users:
\[ 8.4,\; 7.9,\; 9.1,\; 8.7,\; 8.2,\; 9.0,\; 7.8,\; 8.5,\; 8.9,\; 8.1,\; 8.6,\; 8.3 \]
Tasks:
- Identify the appropriate statistical test and explain why.
- Compute the Confidence Intervals for:
- \(90\%\)
- \(95\%\)
- \(99\%\)
- Visualize the three intervals on a single plot.
- Explain how sample size and confidence level influence the interval width.
2.1 Answer number 1
2.1.1 Identify the Appropriate Statistical Test
When a population sample mean and the population standard deviation is unknown, or when the sample size is small (typically \(n < 30\)), we can use \(t\) distribution to construct a Confidence Interval.
In the question it is known that the sample size (\(n\)) is \(12\) which is \(< 30\) and population standard deviation (\(\sigma\)) is unknown. Thus, the use of the t-distribution in conditions where the sample size is small and the population standard deviation \((\sigma)\) is unknown is the more appropriate choice, because it accounts for the additional variability from the estimation of the sample standard deviation and provides an interval that better reflects statistical reality.
2.2 Answer number 2
2.2.1 Confidence Intervals
To calculate the Confidence Interval, we use the \(t\) distribution. Here is the formula:
\[CI =\bar{x} \,\pm\, t_{\alpha/2,\, df} \left( \frac{s}{\sqrt{n}} \right)\]
Where
- \(t_{\alpha/2,\, df}\) = critical \(t\) value from the \(t\) distributon
- \(\alpha = 1 - CL\) (Confidence Level)
- \(df = n - 1\) = Degrees of freedom
- \(\frac{s}{\sqrt{n}}\) = Standard Error
- \(n\) = Sample size
- \(s\) = Sample standard deviation
Summary data:
Sample mean \[\begin{array}{rl} \bar{x} &= \frac{1}{n} \sum_{i=1}^{n} x_i \\[2mm] &= \frac{1}{12} (x_1 + x_2 + \cdots + x_{12}) \\[1mm] &= \frac{1}{12} (8.4+7.9+9.1+8.7+8.2+9.0+7.8+8.5+8.9+8.1+8.6+8.3) \\[1mm] &= \frac{101.5}{12} \\[1mm] &= 8.4583 \ \text{minutes} \end{array}\]
Sample Standard Deviation \(s\)
\[\begin{array}{rl} s & = \sqrt{\frac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n-1}} \\ & = \sqrt{\frac{(8.4-8.4583)^2 + (7.9-8.4583)^2 + \dots + (8.3-8.4583)^2}{12-1}} \\ & = \sqrt{\frac{1.8892}{11}} \\ & = \sqrt {0.171745} \\ & \approx 0.4144\ \text{minutes} \end{array}\]
Sample Size
\[n = 12\]
Standard Error (SE)
\[\begin{array}{rl} SE & = \frac{s}{\sqrt{n}} \\ & = \frac{0.4144}{\sqrt{12}} \\ & \approx 0.1196 \end{array}\]
Degrees of Freedom \(df\)
\[df = n - 1 \\ df = 12 - 1 \\ df = 11\]
Critical value \(z\) for \(90\%\) CI
Significance level
\[CL = 90\% \\ \alpha = 1 - 0,9\\ \alpha = 0.1\\ \frac{\alpha}{2} = 0.05\]From the \(t\) table:
\[t_{0.05},_{11} = 1.796\]Significance level
\[CL = 95\% \\ \alpha = 1 - 0,95\\ \alpha = 0.05\\ \frac{\alpha}{2} = 0.025\]From the \(t\) table:
\[t_{0.025},_{11} = 2.201\]Significance level
\[CL = 99\% \\ \alpha = 1 - 0,99\\ \alpha = 0.01\\ \frac{\alpha}{2} = 0.005\]From the \(t\) table:
\[t_{0.005},_{11} = 3.106\]Confidence Interval \(90\%\)
- Margin Error (ME)
\[\begin{array}{rl} ME & = t_{\alpha/2,\,df} \times SE \\ & = 1.796 \times 0.1196 \\ & \approx 0.2148 \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{90\%} & = \bar{x} \pm ME \\ & = 8.4583 \pm 0.2148 \\ & \approx (8.24,\; 8.67) \end{array}\]
This means that we are \(90\%\) confident that the true average task completion time between \(8.24\) and \(8.67\) minutes.
Confidence Interval \(95\%\)
- Margin Error (ME)
\[\begin{array}{rl} ME & = t_{\alpha/2,\,df} \times SE \\ & = 2.201 \times 0.1196 \\ & \approx 0.2632 \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{95\%} & = \bar{x} \pm ME \\ & = 8.4583 \pm 0.2632 \\ & \approx (8.20,\; 8.72) \end{array}\]
This means that we are \(95\%\) confident that the true average task completion time between \(8.20\) and \(8.72\) minutes.
Confidence Interval \(99\%\)
- Margin Error (ME)
\[\begin{array}{rl} ME & = t_{\alpha/2,\,df} \times SE \\ & = 3.106 \times 0.1196 \\ & \approx 0.3715 \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{99\%} & = \bar{x} \pm ME \\ & = 8.4583 \pm 0.3715 \\ & \approx (8.09,\; 8.83) \end{array}\]
This means that we are \(99\%\) confident that the true average task completion time between \(8.09\) and \(8.83\) minutes.
2.3 Answer number 3
This chart illustrates the estimated average task completion time, which is centered at approximately 8.5 minutes, along with the associated uncertainty expressed through confidence intervals.The dashed vertical line in the center represents the sample mean, while the shaded areas and their boundary lines indicate the range of values within which the true population mean is statistically plausible.The 90% confidence interval is the narrowest, providing the most precise estimate but with lower certainty. The 95% confidence interval offers the best balance between precision and reliability and is therefore the most appropriate level for decision-making. The 99% confidence interval is the widest, reflecting a highly conservative estimate designed to achieve near-complete confidence.Importantly, although the confidence level increases, the estimated mean remains stable. What changes is the width of the uncertainty, not the central estimate itself. Overall, the visualization demonstrates that task completion time is consistent and remains within an acceptable range, even under the most conservative assumptions.
2.4 Answer number 4
2.4.1 Factors Influences the Width of a Confidence Interval
The width of a confidence interval is influenced by several key factors:
Larger sample sizes lead to narrower confidence intervals. This is because a larger sample provides more information about the population, reducing the uncertainty in the estimate. As the sample size increases, the standard deviation of the sampling distribution decreases, which in turn narrows the confidence interval.
The chosen confidence level directly affects the width of the confidence interval. A higher confidence level (e.g., 99% compared to 95%) results in a wider interval, as it requires a larger margin of error to ensure that the true population parameter is captured within the interval. For example, if a 95% confidence interval is (0.2, 0.4), increasing the confidence level to 99% might yield an interval like (0.15, 0.45).
The standard deviation of the sample data also plays a crucial role. A larger standard deviation indicates greater variability among the data points, which results in a wider confidence interval. This is because more variability means that the estimate is less precise.
The width of the confidence interval can be expressed as:
\[Width = Upper \ Limit \ - \ Lower \ Limit\]
The margin of error is a function of the standard deviation and the critical value associated with the chosen confidence level. Therefore, any increase in the margin of error will widen the confidence interval.
2.4.2 How Sample Size and Confidence Level Influence the Interval Width
Sample Size \((n)\)
- Larger Simple Size \((n
\uparrow)\): Leads to a narrower confidence
interval
Why: More data reduces uncertainty and the standard error (the variabilitty of sample means), giving a more precise estimate of the population.
- Larger Simple Size \((n
\downarrow)\): Leads to a wider confidence
interval
Why: Less data increases uncertainty, resulting in a broader range to capture the true value.
Confidence Level
- HIgher Confidence Level\((e.g., 99\%
\uparrow)\):Results in a wider interval.
Why: To be more confident \((e.g., 99\% \ vs \ 95\%)\) that the interval contains the true population parameter, you need a larger buffer (margin of error), making the interval broader.
- Lower Confidence Level\((e.g., 90\%
\downarrow)\): Results in a narrower interval.
Why: You’re willing to accept a smaller margin of error because you don’t need to be as certain, leading to a tighter, more precise, but less reliable, range.
3 Case Study 3
Confidence Interval for a Proportion, A/B Testing: A data science team runs an A/B test on a new Call-To-Action (CTA) button design. The experiment yields:
\[ \begin{eqnarray*} n &=& 400 \quad \text{(total users)} \\ x &=& 156 \quad \text{(users who clicked the CTA)} \end{eqnarray*} \]
Tasks:
- Compute the sample proportion \(\hat{p}\).
- Compute Confidence Intervals for the proportion at:
- \(90\%\)
- \(95\%\)
- \(99\%\)
- Visualize and compare the three intervals.
- Explain how confidence level affects decision-making in product experiments.
3.1 Answer number 1
3.1.1 Sample Proportion \(\hat{p}\)
Formula \[\hat{p} = \frac{x}{n}\] Where
- \(x\) = Number of successes
- \(n\) = Sample size
Compute
- \(x\) = \(156\)
- \(n\) = \(400\) \[\hat{p} = \frac{x}{n} = \frac{156}{400} = 0.39\]
3.2 Answer number 2
3.2.1 Confidence Interval
To compute confidence intervals from population proportion, we use the formula:
\[\begin{array}{rl} CI &= \hat{p} \,\pm\, z_{\alpha/2} \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}} \end{array}\]
Where:
- \(\hat{p}\) = Sample proportion
- \(z_{\alpha/2}\) = Critical Value (it can be found in the normal table)
- \(\alpha = 1 - CL\) (Confidence Level)
- \(\sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}\) = Standard Error
- \(1- \hat{p}\) = \(\hat{q}\)
- \(\sqrt{n}\) = Sample size
Sample size
\[n = 400\]Number of successes
\[x = 156\]Sample proportion
\[\begin{array}{rl} \hat {p} &= \frac {x}{n} \\ & = \frac {156}{400} \\ & = 0.39 \end{array}\]
Standard Error (SE)
\[\begin{array}{rl} SE &= \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}} = \sqrt{\frac{0.39(1 - 0.39)}{400}} \\ & = \sqrt{\frac{0.39 \times 0.61}{400}} = \sqrt{\frac{0.2379}{400}} \\ & = \sqrt {0.00059475} \approx 0.02439 \end{array}\]Critical value \(z\) for \(90\%\) CI
Significance level
\[CL = 90\% \\ \alpha = 1 - 0,9\\ \alpha = 0.1\\ \frac{\alpha}{2} = 0.05\]Standard normal distribution table
\[z_{0.05} = 1.645\]Significance level
\[CL = 95\% \\ \alpha = 1 - 0,95\\ \alpha = 0.05\\ \frac{\alpha}{2} = 0.025\]Standard normal distribution table
\[z_{0.025} = 1.960\]Significance level
\[CL = 99\% \\ \alpha = 1 - 0,99\\ \alpha = 0.01\\ \frac{\alpha}{2} = 0.005\]Standard normal distribution table
\[z_{0.005} = 2.576\]Confidence Interval \(90\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha/2} \times SE \\ & = 1.645 \times 0.02439 \\ & = 0.04012 \\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{90\%} & = \hat{p} \pm ME \\ & = 0.39 \pm 0.04012 \\ & \approx (0.3499,\: 0.4301)\\ \end{array}\]
This means that we are \(90\%\) confident that the true average CTA between \(34,99\%\) and \(43,01\%\).
Confidence Interval \(95\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha/2} \times SE \\ & = 1.960 \times 0.02439\\ & = 0.04780 \\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{95\%} & = \hat{p} \pm ME \\ & = 0.39 \pm 0.04780 \\ & \approx (0.3422,\; 0.4378) \\ \end{array}\]
This means that we are \(95\%\) confident that the true average CTA between \(34,22\%\) and \(43,78\%\).
Confidence Interval \(99\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha/2} \times SE \\ & = 2.576 \times 0.02439 \\ & = 0.06282 \\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{99\%} & = \hat{p} \pm ME \\ & = 0.39 \pm 0.06282 \\ & \approx (0.3272,\; 0.4528) \\ \end{array}\]
This means that we are \(99\%\) confident that the true average CTA between \(32,72\%\) and \(45,28\%\). This interval is wider than both the \(90\%\) and \(95\%\) CI, reflecting the trade-of between precision and confidence.
3.3 Answer number 3
The 90% confidence interval is the shortest line, indicating the most precise estimate but with a higher tolerance for uncertainty. The 95% confidence interval has moderate width and represents the standard level of confidence typically used for product and business decisions. The 99% confidence interval is the longest line, reflecting a highly conservative estimate that prioritizes certainty over precision.The black point in the center marks the estimated click-through rate of 39%, while the differing lengths of the intervals visually highlight the trade-off between precision and confidence. As the confidence level increases, the interval widens, showing that greater certainty requires accepting more uncertainty in the estimated range..
3.4 Answer number 4
3.4.1 How Confidence Level Affects Decision-making in Product Experiments
In product experimentation, the confidence level directly influences decision-making by balancing the need for certainty in results against the desire for precision and speed. A higher confidence level increases certainty but widens the range of possible outcomes (confidence interval), while a lower level narrows the range but raises the risk of making a wrong decision.
Balancing Risk (Type I vs. Type II Errors):
Higher Confidence Levels (e.g., 99%) are used for high-stakes decisions where a false positive (concluding an effect exists when it doesn’t) would be costly or harmful, such as in medical research or a major, expensive product overhaul. This minimizes the risk of implementing a change that has no real impact but comes at the cost of potentially missing a true positive (a false negative or Type II error).
Lower Confidence Levels (e.g., 90%) may be acceptable for early-stage, exploratory experiments or low-risk decisions (e.g., A/B testing a minor UI change). This approach allows teams to iterate and make decisions more quickly, even if it means accepting a higher chance of a false positive..
Precision of Estimates: The confidence level determines the width of the confidence interval.
A higher confidence level yields a wider interval (less precision) because it needs a larger range to be more certain that the true population value is captured. For example, being 99% confident the lift is between 1% and 10% is less precise than being 90% confident it is between 4% and 7%.
A lower confidence level results in a narrower interval (more precision). Decision-makers must weigh whether the increased certainty of a wider interval is more valuable than the more specific estimate of a narrower one.
Statistical Significance:
The confidence level sets the threshold for statistical significance (e.g., a 95% confidence level corresponds to a 5% significance level or p-value of 0.05). If the confidence interval for an effect (like a lift in conversion rate) does not include the “null value” (e.g., zero difference), the result is considered statistically significant, giving a clear signal for action.
Resource Management:
Higher confidence levels require larger sample sizes to maintain a useful interval width, which can increase the time and cost of running an experiment. Decision-makers may choose a lower confidence level to gather quick insights when time or data is limited.
Ultimately, decision-makers must select a confidence level that aligns with their specific goals, the risk tolerance of the organization, and the practical implications of being wrong.
4 Case Study 4
Precision Comparison (Z-Test vs t-Test): Two data teams measure API latency (in milliseconds) under different conditions.
\[\begin{eqnarray*} \text{Team A:} \\ n &=& 36 \quad \text{(sample size)} \\ \bar{x} &=& 210 \quad \text{(sample mean)} \\ \sigma &=& 24 \quad \text{(known population standard deviation)} \\[6pt] \text{Team B:} \\ n &=& 36 \quad \text{(sample size)} \\ \bar{x} &=& 210 \quad \text{(sample mean)} \\ s &=& 24 \quad \text{(sample standard deviation)} \end{eqnarray*}\]
Tasks
- Identify the statistical test used by each team.
- Compute Confidence Intervals for 90%, 95%, and 99%.
- Create a visualization comparing all intervals.
- Explain why the interval widths differ, even with similar data.
4.1 Answer number 1
4.1.1 Identify the Statistical Test
For team A, the appropriate statistical test to use is the z-test. The reason is that the standard deviation \((\sigma)\) is already known with certainly, which is 24. In statistics, when we know the actual population parameters \((\sigma)\), the z-test is the most accurate method regardless of whether the sample size is small or large.
For team B, the appropriate statistical test to use is the t-test. The reason is that the standard deviation \((\sigma)\) is unknown. Use of the t-distribution in conditions where the population standard deviation \((\sigma)\) is unknown is the more appropriate choice, because it accounts for the additional variability from the estimation of the sample standard deviation and provides an interval that better reflects statistical reality.
4.2 Answer number 2
4.2.1 Confidence Interval
Sample mean
\[\bar{x} = 210\]Population Standard Deviation
\[\sigma= 24\]Sample size
\[n = 36\]Standard Error (SE) \[\begin{array}{rl} SE = \frac{\sigma}{\sqrt{n}} = \frac{24}{\sqrt{36}} = \frac{24}{6} = 4 \end{array}\]
Critical value \(z\) for \(90\%\) CI
Significance level
\[CL = 90\% \\ \alpha = 1 - 0,9\\ \alpha = 0.1\\ \frac{\alpha}{2} = 0.05\]Standard normal distribution table
\[z_{0.05} = 1.645\]Significance level
\[CL = 95\% \\ \alpha = 1 - 0,95\\ \alpha = 0.05\\ \frac{\alpha}{2} = 0.025\]Standard normal distribution table
\[z_{0.025} = 1.960\]Significance level
\[CL = 99\% \\ \alpha = 1 - 0,99\\ \alpha = 0.01\\ \frac{\alpha}{2} = 0.005\]Standard normal distribution table
\[z_{0.005} = 2.576\]Confidence Interval \(90\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha/2} \times SE \\ & = 1.645 \times 4 \\ & = 6.58\\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{90\%} & = \bar{x} \pm ME \\ & = 210 \pm 6.582 \\ & = 203.42,\: 216.58\\ \end{array}\]
This means that we are \(90\%\) confident that the true average API latency between \(203.42\) and \(216.58\) ms.
Confidence Interval \(95\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha/2} \times SE \\ & = 1.960 \times 4 \\ & = 7.84 \\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{95\%} & = \bar{x} \pm ME \\ & = 210 \pm 7.84 \\ & = 202.16,\: 217.84\\ \end{array}\]
This means that we are \(95\%\) confident that the true average API latency between \(202.16\) and \(217.84\) ms.
Confidence Interval \(99\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha/2} \times SE \\ & = 2.576 \times 4 \\ & = 10.304 \\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{99\%} & = \bar{x} \pm ME \\ & = 210 \pm 10.304 \\ & = 199.696,\: 220.304\\ & \approx (199.70,\; 220.30) \\ \end{array}\]
This means that we are \(99\%\) confident that the true average API latency between \(199.70\) and \(220.30\) ms. This interval is wider than both the \(90\%\) and \(95\%\) CI, reflecting the trade-of between precision and confidence.
Sample
- Sample size
\[n = 36\]
- Sample mean
\[\bar {x} = 210\]
Sample standard Deviation
\[s = 24\]Standard Error (SE)
\[\begin{array}{rl} SE & = \frac{s}{\sqrt{n}} \\ & = \frac{24}{\sqrt{36}} \\ & = 6 \end{array}\]
Degress of Freedom
\[df = n \ - 1 \\ df = 36 \ - 1 \\ df = 35\]Critical value \(z\) for \(90\%\) CI
Significance level
\[CL = 90\% \\ \alpha = 1 - 0,9\\ \alpha = 0.1\\ \frac{\alpha}{2} = 0.05\]From the \(t\) table:
\[t_{0.05},_{35} = 1.690\]Significance level
\[CL = 95\% \\ \alpha = 1 - 0,95\\ \alpha = 0.05\\ \frac{\alpha}{2} = 0.025\]From the \(t\) table:
\[t_{0.025},_{35} = 2.030\]Significance level
\[CL = 99\% \\ \alpha = 1 - 0,99\\ \alpha = 0.01\\ \frac{\alpha}{2} = 0.005\]From the \(t\) table:
\[t_{0.005},_{35} = 2.724\]Confidence Interval \(90\%\)
- Margin Error (ME)
\[\begin{array}{rl} ME & = t_{\alpha/2,\,df} \times SE \\ & = 1.690 \times 4 \\ & = 6.76 \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{90\%} & = \bar{x} \pm ME \\ & = 210 \pm 6.76 \\ & = (203.24,\; 216.76) \end{array}\]
This means that we are \(90\%\) confident that the true average API latency between \(203.24\) and \(216.76\) ms.
Confidence Interval \(95\%\)
- Margin Error (ME)
\[\begin{array}{rl} ME & = t_{\alpha/2,\,df} \times SE \\ & = 2.030 \times 4 \\ & = 8.12 \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{95\%} & = \bar{x} \pm ME \\ & = 210 \pm 8.12 \\ & = (201.88,\; 218.12) \end{array}\]
This means that we are \(95\%\) confident that the true average API latency between \(201.88\) and \(218.12\) ms.
Confidence Interval \(99\%\)
- Margin Error (ME)
\[\begin{array}{rl} ME & = t_{\alpha/2,\,df} \times SE \\ & = 2.724 \times 4 \\ & = 10.896 \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{99\%} & = \bar{x} \pm ME \\ & = 210 \pm 10.896 \\ & = 199.104,\; 220.896 \\ & \approx (199.10,\; 220.90) \end{array}\]
This means that we are \(99\%\) confident that the true average API latency between \(199.10\) and \(220.90\) ms.
4.3 Answer number 3
This visualization aligns Z-test and t-test confidence intervals by confidence level.For each confidence level, the Z-test interval appears slightly narrower, while the t-test interval is slightly wider, even though both teams report identical sample size, mean, and variability.This difference occurs because the t-test accounts for uncertainty in estimating the population standard deviation, whereas the Z-test assumes it is known with certainty.As confidence increases from 90% to 99%, both intervals widen, illustrating the trade-off between precision and certainty. .
4.4 Answer number 4
4.4.1 Why the Interval Widths Differ, Even With Similar Data.
Interval widths can differ between z-tests and t-tests, even with similar data, primarily due to the different distributions they rely on to account for varying levels of certainty about the population’s characteristics, specifically the population standard deviation.
Z-Test
A z-test is used when the population standard deviation \((\sigma )\) is known, or the sample size is very large (generally > 30). Because the population variability is known, the z-test uses the normal distribution, which has fixed critical values for a given confidence level. This fixed variability results in consistent interval widths for a given dataset size and confidence level.
T-Test
A t-test is used when the population standard deviation is unknown and must be estimated from the sample data. This introduces an extra layer of uncertainty. To account for this, the t-test uses the t-distribution, which has heavier tails and is more spread out than the normal distribution, especially with small sample sizes.
The Key Reasons for The Difference in Interval Widths are:
Uncertainty
The t-distribution is designed to be more conservative and wider than the z-distribution to reflect the added uncertainty of estimating the population standard deviation.
Degrees of Freedom
The shape of the t-distribution changes based on the degrees of freedom (which are related to the sample size). For smaller samples, the t-distribution is much wider than the normal distribution, leading to wider confidence intervals. As the sample size increases, the t-distribution approaches the normal distribution, and the interval widths become similar.
Critical Values
For the same confidence level (e.g., 95%), the critical value from the t-distribution will always be larger than the z-critical value \((e.g., \ t_{crit}>z_{crit}\). This larger multiplier directly results in a wider confidence interval for the t-test compared to the z-test.
5 Case Study 5
One-Sided Confidence Interval: A Software as a Service (SaaS) company wants to ensure that at least 70% of weekly active users utilize a premium feature.
From the experiment:
\[ \begin{eqnarray*} n &=& 250 \quad \text{(total users)} \\ x &=& 185 \quad \text{(active premium users)} \end{eqnarray*} \]
Management is only interested in the lower bound of the estimate.
Tasks:
- Identify the type of Confidence Interval and the appropriate test.
- Compute the one-sided lower Confidence Interval at:
- \(90\%\)
- \(95\%\)
- \(99\%\)
- Visualize the lower bounds for all confidence levels.
- Determine whether the 70% target is statistically satisfied.
5.1 Answer number 1
5.1.1 Identify Type of CI and Appropriate Test
\[\begin{array}{rl} CI_{lower} = \hat{p} \,- z_{\alpha/2} \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}} \end{array}\]
We use the lower confidence interval because the company is only interested in the lower bound.
\[\hat{p} = \frac{x}{n} \]
use the z-test for proportions because the z-test for proportions is used to estimate the actual proportion across the population based on sample data.
5.2 Answer number 2
5.2.1 Confidence Interval
Summary DataSample size
\[n = 250\]Number of successes
\[x = 185\]Sample proportion
\[\begin{array}{rl} \hat {p} &= \frac {x}{n} \\ & = \frac {185}{250} \\ & = 0.74 \end{array}\]
Standard Error (SE)
\[\begin{array}{rl} SE &= \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}} = \sqrt{\frac{0.74(1 - 0.74)}{250}} \\ & = \sqrt{\frac{0.74 \times 0.26}{400}} = \sqrt{\frac{0.1924}{400}} \\ & = \sqrt {0.0007696} \approx 0.02774 \end{array}\]Critical value \(z\) for \(90\%\) one-sided CI
Significance level
\[CL = 90\% \\ \alpha = 1 - 0,9\\ \alpha = 0.1\\\]Standard normal distribution table
\[z_{0.1} = 1.282\]Significance level
\[CL = 95\% \\ \alpha = 1 - 0,95\\ \alpha = 0.05\\\]Standard normal distribution table
\[z_{0.05} = 1.645\]Significance level
\[CL = 99\% \\ \alpha = 1 - 0,99\\ \alpha = 0.01\\\]Standard normal distribution table
\[z_{0.01} = 2.326\]One-Sided Confidence Interval \(90\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha} \times SE \\ & = 1.282 \times 0.02774 \\ & = 0.03556 \\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{lower} & = \hat{p} \ - ME \\ & = 0.74 \ - 0.03556 \\ & = 0.70444\\ \end{array}\]
This means that we are \(90\%\) confident that the true proportion users utilize the premium feature is \(70,44\%\).
One-Sided Confidence Interval \(95\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha} \times SE \\ & = 1.645 \times 0.02774 \\ & = 0.04563 \\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{lower} & = \hat{p} \ - ME \\ & = 0.74 \ - 0.04563 \\ & = 0.69437\\ \end{array}\]
This means that we are \(95\%\) confident that the true proportion users utilize the premium feature is \(69,43\%\).
One-Sided Confidence Interval \(99\%\)
- Margin of Error (ME)
\[\begin{array}{rl} ME & = z_{\alpha} \times SE \\ & = 2.326 \times 0.02774 \\ & = 0.06453 \\ \end{array}\]
- Confidence Interval
\[\begin{array}{rl} CI_{lower} & = \hat{p} \ - ME \\ & = 0.74 \ - 0.06453 \\ & = 0.67547\\ \end{array}\]
This means that we are \(99\%\) confident that the true proportion users utilize the premium feature is \(67,54\%\).
5.3 Answer number 3
This chart displays the lower bounds of the estimated proportion of weekly active users who use the premium feature.At the 90% confidence level, the lower bound lies above the 70% target, indicating that the company can be reasonably confident that at least 70% of users are engaging with the premium feature.However, at the 95% and 99% confidence levels, the lower bounds fall below the 70% threshold, which means that under more stringent confidence requirements, the target cannot be statistically guaranteed.Visually, this demonstrates that as the confidence level increases, the lower bound becomes more conservative. Achieving higher statistical confidence requires accepting a wider margin of uncertainty, which directly affects whether the business target can be claimed as met.
5.4 Answer number 4
5.4.1 Determine whether the 70% target is statistically satisfied.
| Confidence_Level | Lower_Bound | Status | Interpret |
|---|---|---|---|
| 90% | 70,44% | Fulfilled | The lower bound of the estimated proportion is 70.4%, which is above the 70% target. |
| 95% | 69,43% | Unfulfilled | the lower bound falls below 70%, meaning the target cannot be guaranteed with higher statistical confidence. |
| 99% | 67,54 | Unfulfilled | the lower bound falls below 70%, meaning the target cannot be guaranteed with higher statistical confidence. |
Statement
Based on the one-sided lower confidence interval analysis, the minimum proportion of weekly active users utilizing the premium feature exceeds 70% only at the 90% confidence level. At higher confidence levels (95% and 99%), the lower bound falls below the target threshold, indicating insufficient statistical evidence to confirm the 70% adoption target.
References
[1] Emmanuel, Joshua. (2016, July 26). Confidence Interval for a population mean - σ known.
[2] Emmanuel, Joshua. (2022, March 9). Confidence Interval for a population mean - t distribution.
[3] Emmanuel, Joshua. (2022, July 11). Confidence Interval for a population proportion.
[5] Chauhan, Amit. (2022). Different Between Z-test and T-test with Confidence Interval.