Improvement techniques in neural network training

Batch gradient descent

In this type of gradient descent, update of the parameter values are only done after all of the samples have been analyzed in preparing the cost function.

While this type of learning allows for inclusion of all of the training data and certainly maximizes the potential for properly approaching global minima, it can be computationally expensive. This happens when the dataset is too big to fit into memory and constant retrieval from slower storage space must be made.

Mini-batch gradient descent

The process of gradient descent can be hastened if the training sample set can be broken up into parts, called mini-batches. During an epoch, gradient descent can take place after each mini-batch so that at the end of the epoch, a lot of progress can potentially be made. The process of forward propagation and backpropagation takes place during each mini-batch process. An epoch still refers to completing all of the mini-batches.

The term batch refers to the complete training set. Note, though, that in code, the argument batch_size = is used. This actually refers to the mini-batch sample size.

As a rule of thumb, the whole dataset can be used when it is relatively small and will not penalize the overall time taken to converge to a minimum value for the cost function. For bigger datasets, a power of \(2\) such as \(16,32,64,128,256,512\) is useful as mini-batch size. In most cases this works well with parallel graphics card memory architecture and optimizes its use. Irrespective of the size used, it is important that it fits within physical memory allocation.

Stochastic gradient descent

The extreme form of a mini-batch size is \(1\). Every sample in the training dataset is its own mini-batch. The gradient descent that results is called stochastic gradient descent (SGD) and can vary quite considerably for each sample. The search for a minimum is therefor usually termed noisy.

SGD is able to explore a larger hypothesis space (different parts of the multidimensional cost function) and find global minima this way. It also requires less memory usage as only a single sample is required at each turn.

It is not always clear, however, if and how learning is progressing. Usually, the cost is only shown at the end of each epochs and it may take many epochs to confirm progress. It can also be computationally expensive to require many updates for each epoch.

Gradient descent with momentum

Gradient descent with momentum uses the concept of an exponentially weighted moving average to increase the rate of gradient descent. Instead of updating the parameters during backpropagation with the usual learning rate times the partial derivative of the specific parameter, note is kept of these partial derivatives. An exponentially weighted moving average (EWMA) is calculated which is then instead multiplied by the learning rate.

An EWMA considers sequential values and calculates a moving average along each of the values such that there is an exponential decay in how previous values in the sequence contributes to the current average.

The code chunk below creates data point values along the \(x\)-axis and then calculates the sine of each of these values for the \(y\)-axis, but adds some random noise and the integer \(1\) to each value.

x = seq(from = 0, to = 2*pi, by = pi/180)
y = sin(x) + rnorm(length(x), mean = 0, sd = 0.1) + 1

Figure 1 below shows the sine curve and the data with noise.

f1 <- plot_ly(x = x,
             y = y,
             name = "data",
             type = "scatter",
             mode = "markers") %>% 
  add_trace(x = x,
            y = sin(x) + 1,
            name = "sine",
            type = "scatter",
            mode = "lines") %>% 
  layout(title = "Data point values along sine curve",
         xaxis = list(title = "Input values"),
         yaxis = list(title = "Output values"))
f1

For a given coefficient, \(\beta\), a moving average, \(v_i\), over each of the output values \(y_i\), is given in equation (1).

\[v_i = \beta \times v_{i-1} + \left( 1 - \beta \right) \times y_i \tag{1}\]

The code chunk below uses a for loop over the output values. Figure 2 shows the EWMA for \(\beta = 0.9\).

N <- length(x)
beta <- 0.9
v <- vector(length = N)
for (i in 2:N){
  v[i] <- (beta * v[i - 1]) + ((1 - beta) * y[i])
}
f1 <- f1 %>% add_trace(x = x,
                       y = v,
                       name = "EWMA",
                       type = "scatter",
                       mode = "lines")
f1

Note the initial start at zero and the time taken to catch up. This is usually of no consequence in deep neural network training as training will occur over many epochs.

Expansion of equation (1) for a specific value for \(\beta\) and some \(i \in \mathbb{N}\) shows that the number of previous data points over which the average is computed is approximately given in equation (2).

\[\approx \frac{1}{1 - \beta^i}\tag{2}\]

The effect of an EWMA is that gradient descent orthogonal (in higher dimensional space) to the idealized direction to be taken are averaged out over time, but those in the correct direction become additive, hence the term momentum, i.e. gradient descent builds up momentum in the right direction.

Equation (3) below shows that instead of the partial derivative of the weight being stored, it is updated as an exponential moving average for some coefficient \(\beta \in \left[ 0,1 \right]\).

\[V_{\partial W_{i}} = \beta_v V_{\partial W_{i-1}} + \left( 1 - \beta_v \right) \partial W_{i} \tag{3}\]

This exponential moving average update of the derivative is then used to update the weight as shown in equation (4).

\[W_{i} = W_{i-1} \times \alpha V_{\partial W_{i-1}} \tag{4}\]

Root mean square propagation

Root mean square propagation (RMSprop) also attempts to speed up gradient descent. It differs from momentum by squaring the value of the derivative in each iteration. The change in equations (3) and (4) are reflected in equation (5).

\[S_{\partial W_{i}} = \beta_s S_{\partial W_{i-1}} + \left( 1 - \beta_s \right) \partial W_{i}^2 \\ W_{i} = W_{i-1} \times \alpha \frac{\partial W_{i-1}}{\sqrt{S_{\partial W_{i-1}}}} \tag{5}\]

Combining momentum and root mean square propagation

One of the most widely used optimization algorithms for gradient descent combines momentum and RMSprop into adapative moment estimation (ADAM).

One addition to the combination to these two algorithms is the correction of bias that exists at the start of exponentially weighted moving average calculations. For any iteration, \(t\), in this calculation the current average is simply divided as shown in equation (6).

\[\rho_{\partial W}^{\text{corrected}} = \frac{\rho_{\partial W}}{1 - \beta^t}\tag{6}\]

Here \(\rho\) refers to either \(V\) as for momentum or \(S\) as for RMSprop. This corrects for small values of \(t\), but for larger values of \(t\) the denominator approaches \(1\) and makes very little difference.

The parameter update is shown in equation (7).

\[W_{i+1} = W_{i} \times \alpha \frac{V_{\partial W_i}^{\text{corrected}}}{\sqrt{S_{\partial W_i}^{\text{corrected}}}} \tag{7}\]

Note that ADAM requires the setting of hyperparameters \(\alpha\), \(\beta_v\), and \(\beta_s\). Typical values include \(\beta_v = 0.9\) and \(\beta_s = 0.999\).

Learning rate decay

A constant learning rate can lead to slow learning requiring many epochs to minimize the cost function. On the contrary, it may also lead to overshoot once the global minimum is reached. The obvious solution may exist in an initial larger learning rate, which decreases as the values of the parameters converge to the minimum Equation (8) shows how to decrease the value of the learning rate, \(\alpha\), at a decay rate, \(\eta\), over each epoch, \(\mathscr{E}\).

\[\alpha_{\mathscr{H+1}} = \frac{1}{1 + \eta \left(\mathscr{H+1}\right)} \alpha_{\mathscr{H}}\tag{8}\]

The code below demonstrates this process. The learning rate starts at \(0.01\), and at each update stage, it decreases by \(\eta = 0.1\).

alpha <- c(0.01)
eta <- 0.1

for (i in 2:20){
  alpha = append(alpha, 1 / (1 + eta * i) * alpha[i - 1])
}

f2 <- plot_ly(x = 1:20,
             y = alpha,
             name = "data",
             type = "scatter",
             mode = "markers") %>% 
  layout(title = "Decreasing decay rate",
         xaxis = list(title = "Batch"),
         yaxis = list(title = "Learning rate"))
f2

The decay rate, \(\eta\), is another hyperparameter that must be set.

Note that there are a number of decay types such as exponential decay and staircase decay. More sophisticated methods look at updating the learning rate differently for every parameter.