Statchat - Machine Learning (ML) from the stats side

• Demystify the area for those outside ML
• Fly over some common methods which are indicative
• Moan a little

Image processing - the poster boy of ML:

https://www.dropbox.com/s/bmmrhtyn88hnnar/CNN_classifier.mp4?dl=0

https://www.youtube.com/watch?v=qrzQ_AB1DZk

Predictive models on satellite data:

https://www.dropbox.com/s/7ufi99upvon81zi/VMS.mp4?dl=0

https://globalfishingwatch.org/map/

The term was coined in the 50s - naively/tellingly: Machine + Learning

• Machines - could be various things (cars, robots, an app), but ultimately now computers or things controlled by them
• Learning - teaching said machines to perform some task. They “learn” by exposure to data.

Basically algorithms informed by data. This sounds familiar…

Why not ask Wikipedia? (Dr Paxton - are you here?):

Machine learning (ML) is the scientific study of algorithms and statistical models that computer systems use to perform a specific task without using explicit instructions, relying on patterns and inference instead. … Machine learning algorithms build a mathematical model based on sample data, known as “training data”, in order to make predictions or decisions without being explicitly programmed to perform the task

Sounds very familiar…

Machine learning is frequently statistical modelling, with some disregard for inference, focussed on predictive performance.

\[ y = f(x_1, ..., x_p; \theta_1, ..., \theta_k) + e \]

\( y \) is the target (response), \( x \) are the features (covariates), \( f \) signal with parameters \( \theta \), \( e \) noise

In stats:

• We'd usually have a formal model for the signal \( f \) and for the noise \( e \)
• Estimation of \( \hat{f} \) usually built from the distribution of \( e \) (e.g. the other ML: max likelihood)
• Similarly for ML, but focus on accurate \( \hat{f} \), and the distribution of \( e \) often roughly considered

This is the core of what we're doing, by whatever means necessary to get a good estimate

Usually a supervised problem (we have observations of both \( y \) and a set of \( x \) to estimate \( f \))

• Assume a form for \( f \), estimate the best parameters using the data
• Most statistical models would be claimed under ML banner
• Philosophically, tend more towards flexible classes of model for \( f \), rather than classical stats modelling with a fairly rigid model structure

• Choose class for \( f \): polynomial regression, neural networks, boosted classification trees etc
• Estimate best parameters:
  • define an objective/loss function e.g. \( \sum_i (y_i-\hat{y}_i)^2 \)
  • find parameters to minimise this for a specific \( f \) e.g. gradient search over \( \theta_1, ..., \theta_k \)
  • repeat altering the complexity of \( f \) to minimise generalisation error e.g. the loss function on a validation set, \( k \)-fold cross-validation etc
• Appropriate model complexity becomes something to estimate too

This all sounds very familar…

Building complex models from simple blocks

• Trees
• Boosted trees
• Random forests
• Neural networks

• These are a bit rubbish by themselves
• However, using lots of such models as building blocks, we get better predictions
• This is akin to model averaging where we might combine predictions from competing models, weighted by AIC.
• This is ensemble modelling in ML parlance, it is commonplace

\[ \hat{y} = \hat{\beta}_0 + \hat{\beta}_1z_1 + \hat{\beta}_2z_2 + \hat{\beta}_3z_3 \]

where

\[ \begin{align*} z_1 &= \tanh( \hat{\alpha}_4 + \hat{\alpha}_5x_1 + \hat{\alpha}_6x_2)\\ z_2 &= \tanh( \hat{\alpha}_7 + \hat{\alpha}_8x_1 + \hat{\alpha}_9x_2)\\ z_3 &= \tanh( \hat{\alpha}_{10} + \hat{\alpha}_{11}x_1 + \hat{\alpha}_{12}x_2) \end{align*} \]

• So NNs are a complex thing built from lots of simple components.

If you want to get to grips with NNs as a newbie - this is highly recommended:

https://playground.tensorflow.org/

I'll now play with it….

Recent years have seen a NN renaissance, largely in image processing:

• So called Deep-learning: add lots of layers to your NN and you're now doing deep-learning.
• Relatedly, Convolutional Neural Networks (CNNs)

• There has been a recent upsurge in NN popularity. Mainly due to great performance in image classification problems. The NNs in question can be massively complicated:

  • 2012 imageNet classification - NN with \( 1\times 10^9 \) parameters
  • Used 1000 computers - 16,000 cores to fit

• Hence the term deep-learning - very deep/complex NNs

• The distinction between shallow and deep learning is a bit vague - but the extremes are clearly different

  • See Schmidhuber, J. (2015) Deep learning in neural networks: An overview. Neural Networks Volume 61, Pages 85-117.

• MNIST data consists of 60000 training images, 28x28 pixels each, clean and centered
• Our basic NN:
  • Each input node is a pixel (784)
  • The output nodes are classes of image i.e. numbers 0-9
  • Even with a few hidden nodes and layers, a lot of parameters

This is clearly naive - the pixels are arranged spatially! Which is not reflected in the architecture.

Basic idea is simple:

• Don't fully connect all inputs to all first layers of NN as though they are independent
• Filter inputs in various ways similar to photo-shop filters
• Motivated by real vision - group process blocks of inputs based on proximity (like scanning with your eye)
• Do some spatial averaging
• Amongst other things, this reduces the number of linkages

CNNs have many features we're familiar with from other NNs, but also:

• kernels (/convolutions); convolution layers
• pooling layers
• fully connected layers
• strides & receptive fields

Typically we're talking images - if colour, think 3 values per pixel (RGB).

• So we have an input volume, say \( n\times n \) pixels by 3 channels (RGB)
• We'll have 1000s of these for training with labels
• Note each channel is a matrix
• Key are the kernels, which are 'small' windows (e.g. 4x4 pixel/matrix) that pass over the lerger image/matrix, filtering the values
• This is the convolving in the name - a set of matrix operations

• These are familiar to us i.e. inputs feed through hidden layers with full connectivity
• they tend to be later in the CNN, when the complexity has been reduced by convolving and pooling.

See Krizhevsky et al (2012) - they describe their CNN for the imagenet problem:

• Got relatively great results at that point in time
• Predicting 1000 classes from 1.2 million training images
• 60 million parameters, 650,000 neurons
• ReLU activation functions
• 5 convolutional layers + max-pooling layers
• 3 fully connected layers
• Softmax output function (1000 class)

See Krizhevsky et al (2012) - they describe their CNN for the imagenet problem:

• Substantial time spent specifying architecture
• Results were sensitive to architecture choices
• Required fitting with GPUs - two here directed at different parts of the NN
• Weight decay, momentum parameters and dropout
• 6 days to fit

Tricky! Later CNNs made this look small (see previous)

Transfer learning is very popular now.

• CNNs can be huge and take a looong time to fit.
• So, take the first bits of an extant CNN which has various convolutions and pooling with parameters already determined.
• Attach your bit to the end, so only fit the final layers for your specific problem.

• Fiddle your data - “feature engineering”, data wrangling & munging
• Focus on predictive accuracy, usually assessed by data unseen to the model fitting process
• Be relatively agnostic about the function for the signal - choose broad classes that permit lots of complexity
• Choose loss-functions/objectives that suit response type e.g. squared-error loss for numeric problems
• Tune model complexity as part of the process - again focussing on data unseen to the modelling process
• Try/compare lots of competing types - winning is clearly defined (lowest generalisation error)
• [try “ensembling” your models when in trouble]

• Python has become the go-to tool for much ML
• The Keras library is commonly used, often accessing Tensorflow
• [NB you can access all this through R too]
• Lots and lots of material/examples/etc of vastly varying quality around (e.g. see stock-market prediction using RNN-LSTM, Yuk!)
• End result is much ML is very “same-y”, everybody is throwing these at problems

I've talked about ML models - this is usually a small part to make something useful:

• Data acquisition/management: video streams, APIs, web-scraping
• Delivery: apps & GUIs

• A clear sensible objective measured direction - generalisation error/future predictive error using data unseen to the modelling process
• Super-flexible model classes that can capture complexity without fine-scale management e.g. interactions are a pain in vanilla regressions
• Lots and lots of cool tools, practicioners & activity (CS folk are very engaged)
• Computational efficiency is well-covered

You might consider the ML “approach” (and/or somewhat non-statistical models) if:

• Prediction is key, over inference
• You're not particularly interested in interpreting pieces of the model
• You don't have strong views about the underlying function for the signal
• You're dealing with image/video classification problems (definitely)