Deep learning, concepts and frameworks: Find your way through the jungle (Part 2)
Sigrid Keydana, Trivadis
02/06/2018
Depth alone is not enough
How ever deep we make our network, if we just chain layers of matrix multiplication one after another, all we get is a linear combination of the inputs.
How can we solve non-linear problems with neural networks?
The classic: sigmoid activation function
sigmoid <- function(x) 1/(1 + exp(-x))
x <- seq(-10,10, by = 0.01)
plot(x, sigmoid(x), type = "l", xlab = "", ylab = "")
The current default, kind of: ReLU
relu <- function(x){
x[x<0] <- 0
x
}
x <- seq(-10,10, by = 0.01)
plot(x, relu(x), type = "l", xlab = "", ylab = "")
Mostly for LSTMs: tanh
x <- seq(-10,10, by = 0.01)
plot(x, tanh(x), type = "l", xlab = "", ylab = "")
Going spatial: Convnets
LeNet: First successful application of convolutional neural networks by Yann LeCun, Yoshua Bengio et al.
The convolution operation (1)
The convolution operation (2)
Tasks in computer vision
 |
| Source: [24] |
Classification
In classification, the required output is a probability for each class.
 |
| Source: [24] |
Localization
In localization, the network needs to identify the position of an object in an image.
 |
| Source: [25] |
Object Detection
In object detection (a.k.a. image recognition), the network has to classify and localize multiple objects in an image.
 |
| Source: [26] |
Segmentation
In segmentation, the network needs to predict a class value for each input pixel.
 |
| Source: [27] |
Semantic vs. Instance Segmentation
Semantic segmentation segments by class, instance segmentation by class instance.
 |
| Source: [28] |
Until now, all we've seen are static snapshots
How do we handle sequences?
- language: words, sentences, paragraphs…
- all kinds of serial information: sensor data, stock prices…
When Peter came home from work, dinner again wasn't ready. What the heck had Alicia been doing all day?
Frustrated, ___ opened another bottle of beer.
I need to remember: Introducing hidden state
 |  |
| Sources: [29], [31] |
Basic RNN closeup
The basic RNN at every step combines new input and existing state.
 |
| Source: [29] |
Remembering is not enough
Sometimes we also need to forget!
Two kinds of state: the LSTM "conveyor belt"
The LSTM (Long Short Term Memory) architecture adds an additional state layer, the cell state:
 |
| Source: [29] |
LSTM cell state and the three gates
The LSTM cell state is protected by three gates, the forget, input, and output gates:
 |  |  |
| Source: [29] |
RNN Example: Machine Translation
In translation, we have two sets of sequential data, one on the source and one on the target side!
Enter: sequence-to-sequence models
 |
| Source: [30] |
TensorFlow and the dataflow programming paradigm
 |
| Source: [32] |
TensorFlow demos
- A simple graph for function optimization: function_optimizer_tf.ipynb
- Classifying digits using low-level TensorFlow: tf_raw_mnist.ipynb
- Classifying digits using the estimators API: tf_estimators_mnist.ipynb
Keras: more than just a high-level API for TensorFlow
- Uses one of TensorFlow, CNTK or Theano (discontinued) for low-level operations
- Also available as a high-level API for TensorFlow (tf.keras)
- Like TensorFlow, also available from R
Demo:
- Classifying digits with Keras: keras_mnist.ipynb
Flexibility first: PyTorch
In PyTorch, computation graphs are dynamic.Thus:
- Computations may be different on different training iterations
- Tensor values can be inspected directly
- Debugging works just as in “normal software”
Demo:
- Classifying digits with PyTorch: pytorch_mnist.ipynb
Questions?
Thanks a lot for your attention!!
Sources
[1] https://cacm.acm.org/news/221108-artificial-intelligence-pioneer-says-we-need-to-start-over
[2] Wikipedia
[3] https://crude2refined.wordpress.com/2015/08/14/my-inns-big-data-conference-review-part-1/
[4] MIT 6.S094 Deep Learning for Self-Driving Cars Lecture Slides
[5] Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
[6] Esteva et al. Dermatologist-level classification of skin cancer with deep neural networks
[7] Wikipedia. AlphaGo versus Lee Sedol
[8] Yoshihara et al. Leveraging temporal properties of news events for stock market prediction
Sources
Sources
[18] https://ajolicoeur.wordpress.com/cats/
[20] https://deepmind.com/research/alphago/alphago-china/
[21] https://deepmind.com/blog/alphago-zero-learning-scratch/a/a>
[22] http://ruder.io/optimizing-gradient-descent/
[23] https://colah.github.io/posts/2015-08-Backprop/
[24] Stanford CS231n Convolutional Neural Networks Lecture Notes
[25] Erhan et al. Scalable Object Detection using Deep Neural Networks
[26] ImageNet Large Scale Visual Recognition Challenge 2014 (ILSVRC2014)
[27] Long et al. Fully Convolutional Networks for Semantic Segmentation
Sources
[28] Silberman et al. Instance Segmentation of Indoor Scenes using a Coverage Loss
[29] Chris Olah. Understanding LSTM Networks
[30] Tensorflow seq2seq tutorial
[31] Goodfellow et al. 2016, Deep Learning
[32] TensorFlow Documentation: Graphs and Sessions
[33] LeCun et al., Gradient-based learning applied to document recognition.