Statistics plays an important role in monitoring modern computer systems.
Engineers collect measurements such as:
Statistical techniques can help determine when system behavior is normal and when an observation may indicate a problem.
In this presentation, we will use statistics to detect abnormally slow server responses.
Suppose server response time is represented by the random variable \(X\).
The sample mean is:
\[ \bar{x} = \frac{1}{n} \sum_{i=1}^{n} x_i \]
The sample standard deviation is:
\[ s = \sqrt{ \frac{ \sum_{i=1}^{n}(x_i-\bar{x})^2 }{ n-1 } } \]
The mean describes the typical response time, while the standard deviation describes how much response times normally vary.
A z-score measures how far an observation is from the mean in units of standard deviation:
\[ z_i = \frac{x_i-\bar{x}}{s} \]
A commonly used statistical rule marks observations beyond approximately three standard deviations as unusual:
\[ |z_i| > 3 \]
Equivalently, the monitoring limits can be written as:
\[ UCL = \bar{x} + 3s \]
\[ LCL = \bar{x} - 3s \]
Values outside these limits can be investigated as possible system anomalies.
For this example, we created a simulated dataset containing 200 server requests.
Each request contains:
Most requests behave normally, but several artificial latency spikes were introduced to represent possible performance problems.
## request_id response_time_ms cpu_usage memory_usage ## 1 1 136.5 24.8 59.1 ## 2 2 113.2 35.3 44.6 ## 3 3 124.4 45.9 50.9 ## 4 4 127.6 54.1 51.4 ## 5 5 124.9 25.7 47.8 ## 6 6 118.7 25.3 44.8
The dashed horizontal line represents the three-standard-deviation upper limit.
Points above this threshold are statistically unusual and may deserve investigation.
Most requests are concentrated around the normal operating range.
The unusually large response times appear in the right tail of the distribution.
mean_latency <- mean(server_data$response_time_ms)
sd_latency <- sd(server_data$response_time_ms)
server_data <- server_data %>%
mutate(
z_score =
(response_time_ms - mean_latency) /
sd_latency,
anomaly = abs(z_score) > 3
)
server_data %>%
filter(anomaly)
The program calculates a z-score for every request and flags observations more than three standard deviations from the mean.
This interactive plot shows three system measurements simultaneously.
Rotate the graph and hover over individual requests to inspect their measurements.
The average response time was approximately:
\[ \bar{x} = 124.57 \text{ ms} \]
The standard deviation was approximately:
\[ s = 27.31 \text{ ms} \]
The three-standard-deviation upper limit was:
\[ UCL = 206.5 \text{ ms} \]
The detector identified 8 anomalous requests.
## request_id response_time_ms z_score cpu_usage memory_usage ## 1 15 238.4 4.17 80.4 63.9 ## 2 42 225.7 3.70 80.0 67.5 ## 3 67 259.0 4.92 90.2 77.4 ## 4 91 261.7 5.02 79.1 70.4 ## 5 123 236.5 4.10 76.2 70.5 ## 6 145 245.0 4.41 83.6 71.3 ## 7 171 222.2 3.58 83.8 62.0 ## 8 188 266.7 5.20 91.1 65.3
Statistical anomaly detection can be incorporated into automated monitoring systems.
Possible applications include:
Instead of waiting for a complete system failure, engineers can investigate statistically unusual behavior earlier.
The three-standard-deviation method is useful, but it is not perfect.
A system may naturally change over time because of:
If the normal operating distribution changes, fixed statistical thresholds may generate false alarms or fail to detect new problems.
More advanced monitoring systems may therefore use moving statistics, machine learning, or adaptive anomaly-detection methods.
Statistical monitoring allows engineers to detect unusual system behavior before it necessarily becomes a system failure.