We can never observe the world as it is. Every observation consists of the “true” state of the world plus various forms of noise. For example, when observing the location of planets in the night sky, there is a true location of the planet, but it is obscured by the noise of measurement error.
Noise is dangerous for people building models of the world. We want the model's estimates to get close to the true state of the world, but the only way to assess models is by how close they are to our noisy observations.
In the diagram below, the solid line represents the discrepancy of the model with the observation, but it is just an apparent discrepancy. We actually want the model to be close to the actual location of the planet. That true discrepancy is represented by the dashed line. We want to assess the model by the dashed line but can only observe the solid line. (You can see the R code I used to generate the diagram, but that is not the focus of this discussion.)
x <- c(0.5, 2, 2.5)
y <- c(0.5, 2.2, 1.8)
type_col <- c("M", "A", "O")
label_col <- c("Model estimate", "Mars true location", "Mars observation")
data <- data.frame(x, y, type_col, label_col)
library(ggplot2)
makePlot <- function(data) {
ggplot(data, aes(x, y, label = label_col)) + geom_point() + geom_text(hjust = -0.1,
vjust = 0.6) + coord_cartesian(xlim = c(0, 3.5), ylim = c(0, 3.5)) +
geom_line(data = subset(data, type_col %in% c("M", "A")), linetype = "dashed") +
geom_line(data = subset(data, type_col %in% c("M", "O")))
}
makePlot(data)
When an initial, rough model is created, the model is typically so far from either the true value or the observation that the effect of observational noise on the model is minimal. Both the dashed and solid lines are long, as in the diagram above. Any attempt at shortening the solid line will also shorten the dashed line.
The modeler fiddles with the model to reduce the discrepancies from observations. Below is an example of an improvement on the prior diagram. Notice that both the solid and dashed lines are shorter.
data2 <- data
data2[1, 1] <- 1.5
data2[1, 2] <- 1.2
makePlot(data2)
However, as the model estimate gets closer to the two points, the danger should be getting clearer. The model cannot forever shorten both lines. At some point, the model will get closer to the observation but farther from the actual location. At that point, the model is “overfitting.” Put another way, it is fitting the observational noise rather than the true world.
data3 <- data
data3[1, 1] <- 2.3
data3[1, 2] <- 1.9
makePlot(data3)
With these diagreams, we can give an intuitive definition of overfitting:
DEFINITION: A fitting method is overfitting (a.k.a fitting noise) when it increases the length of the dashed line (the actual discrepancy) while it is decreasing the length of the solid line (the apparent discrepancy).
In other words, a researcher's efforts to improve a model's fit to the data are not just subject to diminishing returns– they can make the true fit worse (even though the apparent fit looks better).
If you get a big enough sample for your modeling technique, the observational noise will cancel out and preven this problem.
Given a style of analysis, statistical textbooks provides formulas for telling you the minimum sample size. However, in practice it is very difficult to use the right formula for every analysis. When statisticians review published papers in other disciplines, they find that insufficient sample sizes are all too common.
Instead of trying to guess if your sample size is enough, you can get a better estimate of how close your fit was to the “true” value with resampling, which re-use your sample. Resampling is also known as boostrapping or cross-validation or back-testing. Fit your model on one set of data, then assess it on a totally different set of data (and then rinse and repeat). That is, in one set of data you tweak the model to shorten the solid lines. Then in the other set of data, you measure how long that data's solid lines are. Since the 2nd set of data has different noise, those solid lines are a better judge of the model's quality. (If only we could measure the dashed lines!)
Below is a variant of the prior diagrams to illustrate how resampling helps. Suppose we meet astronomer 2, who also made an observation of Mars at the same time and place. Now when we have our very well-fit model, we can judge its performance by the observations from astronomer 2. The fitting process shortened the distance to the initial observations, the solid line. But we assess the model with the model's distance to “Mars observation 2,” which came from astronomer 2, the dotted line.
We would really like to know the distance to the true location of mars (dashed line), but we don't know the true location. The point is that the model's distance from observation 2 (dotted line) is a better estimate of the model's distance from the true value (dashed line) than its distance to observation 1 (solid line) because the solid line is what we were trying to minimize in the fitting step.
x_o2 <- c(x, 2.2)
y_o2 <- c(y, 3)
type_col_o2 <- c("M", "A", "O", "O2")
label_col_o2 <- c("Model estimate", "Mars true location", "Mars observation",
"Mars observation 2")
data_o2 <- data.frame(x = x_o2, y = y_o2, type_col = type_col_o2, label_col = label_col_o2)
data_o2[1, 1] <- 2.3
data_o2[1, 2] <- 1.9
p_o2 <- makePlot(data_o2)
p_o2 + geom_line(data = subset(data_o2, type_col %in% c("M", "O2")), linetype = "dotted")
Of course, the best final model of Mars would try to minimize the distance to both observation 1 and 2. There is no good reason to exclude data from the final fit! However, when we are trying to assess how good our model and fitting methods are– compared to a different model (e.g. with more variables) and possibly different fitting methods– resampling is a valuable tool. It is a form of “model selection” or “meta-modeling.”