Areas of Application

Machine learning has found productive applications across many areas of the social sciences.

Requirements of Application

Not every research situation calls for machine learning. Some requirements and considerations:

• Sample size: ML generally benefits from larger samples. With very small samples (e.g., \(n < 100\)), classical models may outperform.
• Number of features: The comparative advantage of ML grows with the number of predictive features (especially \(P > 10\)).
• Research goal: ML excels at prediction; causality still requires careful design and theory.
• Interpretability needs: Results must be communicable. Black-box models require additional tools.

ML may be an ‘overkill’ for small, tightly controlled experiments with few variables—classical regression may suffice.

Dataset Quality & Data Preparation

The quality of ML results is fundamentally bounded by the quality of the data.

Common pre-processing steps:

In R, the recipes package from the tidymodels ecosystem handles most of these steps in a reproducible pipeline.

Best-Case Examples in the Social Sciences

In each case, the research design has to be theoretically informed. ML identifies patterns on the basis of a researchers design, but does it does not interpret them. Both design and interpretation have to work together.

Definition

The design of algorithms that learn from data and are capable of improvement as new experiences emerge. Specifically, the algorithm “is said to learn from experience \(E\) with respect to some task \(T\) and performance measure \(P\), if its performance at task \(T\), as measured by \(P\), improves with experience \(E\)” (Mitchell, 1997).

Three core concepts:

• Experience \(E\): data
• Task \(T\): e.g., regression, classification, clustering
• Performance \(P\): a metric that quantifies how well the task is accomplished

A continuum of ML algorithms

Main uses of ML in applied social sciences

Three major uses have been identified in the literature (Grimmer et al., 2021):

1. Automation

Tedious, time-consuming, error-prone coding tasks. Example: scaling thousands of open-ended survey responses.

2. Prediction

Predicting future or out-of-sample behavior. Example: forecasting electoral turnout at the precinct level.

3. Induction — theory generation

Discovering previously unseen or unconsidered patterns; developing new theoretical hypotheses from the data.

Controversy around inductive uses remains, particularly in relation to the Humean critique of induction. Past experience alone does not guarantee future accuracy.

ML Jargon

The ML community uses slightly different terminology from classical statistics.

The Generic Learning Problem

An algorithm learns the following model

\[ y_i = \phi \left(\boldsymbol{x}_i, \boldsymbol{\theta} \right) + \varepsilon_i \]

where:

• \(i\) is the instance
• \(y\) is the label (a class or a numerical score)
• \(\phi(.)\) is an unknown function to be learnt from data
• \(\boldsymbol{x}\) is the vector of predictive features
• \(\boldsymbol{\theta}\) contains the parameters (weights and tuning parameters)
• \(\varepsilon\) is irreducible error

The algorithm then generates predictions

\[ \psi_i = f \left( \boldsymbol{x}_i^*, \boldsymbol{q} \right) \]

where \(\boldsymbol{x}^*\) is the subset of relevant features selected during training, and \(\boldsymbol{q}\) contains the optimal parameter values.

The Loss Function

In statistics, loss equals error. For at least one instance, \(\psi_i \neq y_i\). The loss function \(L(\psi_i, y_i)\) quantifies the total error across all instances.

The learning process selects \(f(.)\) and \(\boldsymbol{q}\) such that loss is minimized.

Performance Metrics

Performance metrics \(P(\psi_i, y_i)\) measure how well the model accounts for the data.

For regression tasks, three metrics are standard (computed on the test set):

\[\text{MAE} = \frac{1}{n_2} \sum_{i=1}^{n_2} |\psi_i - y_i|\]

\[\text{RMSE} = \sqrt{\frac{1}{n_2} \sum_{i=1}^{n_2} (\psi_i - y_i)^2}\]

\[R^2 = r_{\psi,y}^2\]

• MAE and RMSE: smaller is better (they measure error)
• \(R^2\): larger is better (it measures explained variance)

Our goal: achieve best possible predictive performance, but not at the cost of interpretability.

A Division of Labor

Two distinct processes must be kept strictly separate:

Training

I.e. learning the functional form, parameters, and relevant features from data.

Evaluation

I.e. assessing the performance of your model This requires data that the model has never seen.

Cardinal rule: Never train and evaluate on the same data.

Measuring performance on training data produces re-substitution error, i.e. an optimistic estimate of true performance. The model has simply memorized the training data.

The Split Sample Approach

The most fundamental approach to separating training from evaluation:

1. If necessary, randomize the \(n\) instances.
2. Set aside a fraction \(p\) for training \(\Rightarrow\) training set of size \(n_1 = p \cdot n\).
3. Set aside the remaining fraction \(1 - p\) for evaluation \(\Rightarrow\) test set of size \(n_2 = n - n_1\).

\[ \underbrace{n_1}_{\text{training}} + \underbrace{n_2}_{\text{test}} = n \]

The Dog Days Are Over

Imagine you want to teach a computer to guess how much a dog costs to buy, based on facts you know about it.

Features:

• How big it is
• How old it is
• How big its kennel is
• What breed it is

Train/test split:

• We split our collection of dogs into two piles
• 60% go into the training pile — i.e. the examples we show the computer
• 40% go into the test pile — i.e. dogs hidden from the computer completely, and are used to test it later
• We check that both piles have a similar spread of prices, so neither pile is biased.

Pre-processing:

• Most dogs are priced normally, but a few rare breeds are astronomically expensive
• A computer fixates on outliers — like a champion pedigree worth fifty thousand euros among hundreds of mixed breeds at a few hundred
• The log transformation compresses the scale so the rare pedigree does not dominate everything the computer learns

Or Have They Just Begun?

Training:

• We fit a linear regression — the simplest possible model
• It draws a straight line through the data: for every extra kilogram of body weight, the price goes up by roughly this much
• We inspect the coefficients to see which features matter most
• We check \(R^2\) to see how much of the variation in price we can explain
• We check RMSE to see on average how many euros wrong our predictions are on the dogs from the hidden test pile

Interpretation:

• We draw a partial dependence plot for body size
• This asks: if we could magically change just the size of every dog and leave everything else the same, how would predicted price change?
• It isolates one feature’s contribution — holding breed, age, and kennel size constant, how does size alone affect price?

Overview

A supervised ML workflow follows a consistent cycle:

Data Preparation → Training → Evaluation → Improvement → Deployment
                        ↑                       ↓
                        └───────────────────────┘
                              (tuning loop)

Each stage has specific objectives and tools.

The tidymodels Ecosystem

tidymodels is a collection of R packages built around a consistent grammar for ML.

Workflows are critical, as they prevent the cardinal error of applying test-set information during training.