DATA425-25S1-Assignment#1
Task #1: The Impact of a composite activation function on a binary Network:
A binary classification task with the following threshold:
\[ f(x) = \begin{cases} class "1", & \text{if } f(x) \geq 0.5 \\ class "0", & \text{if } f(x) < 0.5 \end{cases} \]
The activation function is a composite function as below:
\[f(x) = g(k(x)) = g(\max(0, x))\]
where:
\[g(z) \text{ is the sigmoid function: } g(z) = \frac{1}{1 + e^{-z}}\]
\[k(x) \text{ is the ReLU function: } k(x) = \max(0, x)\]
Before analysing the impact of neural network, both sigmoid and relu functions are plotted.
import numpy as np
import matplotlib.pyplot as plt
# ReLU function
def relu(x):
return np.maximum(0, x)
# Sigmoid function
def sigmoid(x):
return 1 / (1 + np.exp(-x))
# the combined function f(x) = g(k(x)) = sigmoid(ReLU(x))
def f(x):
return sigmoid(relu(x))
# x values
x = np.linspace(-5, 5, 100)
# Compute function values
relu_values = relu(x)
sigmoid_values = sigmoid(x)
fx_values = f(x) # Sigmoid applied after ReLU
# Plot
plt.figure(figsize=(8, 5))
plt.plot(x, relu_values, label="ReLU: $k(x) = \max(0, x)$", color="blue")
plt.plot(x, sigmoid_values, label="Sigmoid: $g(x) = \dfrac{1}{1 + e^{-x}}$", color="red")
plt.plot(x, fx_values, label=r"$f(x) = g(k(x)) = g(\max(0, x))$", color="green")
plt.axhline(0, color="black", linewidth=0.5, linestyle="--")
plt.axvline(0, color="black", linewidth=0.5, linestyle="--")
plt.legend()
plt.xlabel("x")
plt.ylabel("f(x)")
plt.title(r"ReLU, Sigmoid, and $f(x) = g(k(x)) = g(\max(0, x))$")
plt.grid()
# Show the plot
plt.show()
As shown in the above graph, a sigmoid function is a symmetric function, with both positive and negative input, generating a binary output by setting a threshold (0.5 here). The Relu function generates a linear output for all positive inputs but offsets all negative inputs to 0.
Impact of Applying a ReLU function before the sigmoid function: outputs of the ReLU function are input to the sigmoid. Since the outputs of the ReLU function are all positive, the left half of the sigmoid function, where initially generating the outputs \(< 0.5\) is completely eliminated. On the other hand, they are indicative of the second class (“0”). This results in biasing the model towards the first class (“1”) only and is incapable of confidently predicting class “0”. This activation function does not allow to have a balanced decision boundary as it is skewed towards the first class.
Task #2: Analysing a simple perceptron-like network
- Draw a conceptual diagram: This figure shows a network with two inputs and connected by a sigmoid function to make a binary output.
\[ h_1 = g(W_{1,1} \cdot x_1 + W_{1,2} \cdot x_2 + b_1) \]
where, g is a sigmoid function.
\[ [W_{1,1} W_{1,2}]×\begin{bmatrix} x_1 \\ x_2 \end{bmatrix}+[b1]=[a_1]\]
2. What type of learning task could be solved with this neural network? According to the lectures, this kind of simple perceptron can be used to design a network for binary classification to distinguish between two classes. The simple perceptron can be used for logistic models. The sigmoid activation can be interpreted as a probability score to ensure the outputs are switching between two classes.
3. Converting to a Multi-Class Classification function
Indeed one of the modifications is to use multiple outputs (neurons), instead of one output. In this case, instead of the sigmoid function, the softmax needs to be used, ensuring that the sum of outputs equals 1 (class probabilities). Furthermore, the loss function should also be changed into categorical cross-entropy instead of binary cross-entropy.
4. Demonstrate the use of forward propagation for this neural network \[ h_1 = \frac{1}{1 + e^{-(W_{1,1} \cdot x_1 + W_{1,2} \cdot x_2 + b_1)}} \]
Example 1: x1= 0.6, x2=0.5 w1=0.5 w2=-0.4 b=0.2 sum=(0.5×0.6)+(−0.4×0.5)+0.2 sum=0.3−0.2+0.2=0.3
\[ h_1 = \frac{1}{1 + e^{-0.3}} \]
\[ h_1 = \frac{1}{1 + 0.7408}=0.5744 \] This data belongs to the first class (1) as it is greater than 0.5.
Example 2: x1= 1, x2=2 w1=0.5 w2=-0.5 b=-3 sum=(0.5×1)+(−0.5×2)+(−3) sum=-3.5
\[ h_1 = \frac{1}{1 + e^{3.5}} \]
\[ h_1 = \frac{1}{1 + 33.12}=0.029 \] This data belongs to the second class (0) as it is less than 0.5.
Task #3: A multi-layer (linear) network with identity activation function
A) Conceptual diagram:
Here is the NN structure:
- Two input layer (\(x_1\) and \(x_2\))
- L-1 Hidden layers, with two nodes each
- One output layer: \(\hat y\)
- Activation function: identity function which is purely linear.
B.
The multi-layer network with the identity activation function mentioned in this question is actually a generalised form of a simple network with one hidden layer and one node, as explained in the week #2 lecture.
Activation function is $ g_i(a_i)=a_i $ for each hidden layer. Thereby, the formula for the hidden layer is:
\[ H_i=W_iH_{i-1}+B_i\]
where, \[W= \begin{bmatrix} w_{11} &w_{12} \\ w_{21} & w_{22} \end{bmatrix}\]
All biases are zero, \(B_i=0\), so the general form of the h for each layer is: \[H_i=W_iH_{i-1}\]
Note that \(H_0=\begin{bmatrix} x_1 \\ x_2 \end{bmatrix}\) and the output function is: \[\hat y=W_iH_{L-1}\]
Effects of each weight initializations: 1. \[W_i= \begin{bmatrix} 0 &0 \\ 0 & 0 \end{bmatrix}\] \[W_L= \begin{bmatrix} 0 &0 \\ \end{bmatrix}\] Substituting the output function, \(\hat y=0\)
- \[W_i= \begin{bmatrix}1.5 &0 \\ 0 & 1.5 \end{bmatrix}\] \[W_L= \begin{bmatrix} 1.5 &0 \\ \end{bmatrix}\]
the first bit: \[H_1=W_1X\] , \[ \begin{bmatrix}1.5 &0 \\ 0 & 1.5 \end{bmatrix}\times\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}=\begin{bmatrix}1.5x_1 \\ 1.5x_2 \end{bmatrix}=1.5\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}=1.5X\]
The second bit: \[ \begin{bmatrix}1.5 &0 \\ 0 & 1.5 \end{bmatrix}\times1.5\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}=1.5\begin{bmatrix}1.5x_1 \\ 1.5x_2 \end{bmatrix}=1.5^2\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}=1.5^2X\] Therefore, \[H_{100-1}=1.5^{100-1}X\] \[\hat y=W_L\times H_{L-1}=1.5^{100-1}\begin{bmatrix}1.5&0 \end{bmatrix}\begin{bmatrix}x_1 \\ x_2 \end{bmatrix}=1.5^{100}x_1=4x10^{17}x_1\]
- The third one can also be calculated with the same way:
\[\hat y=0.5^{100}x_1=7.8\times10^{-31}x_1\]
Task #4: Exploring the NN performance using SGD compared to batch gradient descent
To explore the neural network performance using both batch and stochastic gradient descent, I applied the changes step by step and examined the results.
STEP #0 The input function is generated based on 20000 data points. The initial data points were split into 20% and 80% test and training datasets.
STEP #1 The Neural Network descined in Lab #2, has three layers with two input and two output and a single hidden layer. In this assignment, a three-node hidden layer with the tanh and softmax activation functions is used. In the first run, the following hyperparameters are applied:
- Learning rate (epsilon)=0.01
- Regularisation strength=0.01
- Number of itterations=20000
To make sure I can get a steady result from the batch gradient descent function with the increased data points, I probed the loss values and plotted against the iteration for both 2000 and 20000 datasets. Here are the results:
Fig#1
As shown in the above picture, using the same hyperparameters, the loss values which were steady in the small number of data points, oscillated while increasing the number of data points. The possible solution could be:
- decreasing the learning rate, to have small steps in the batch gradient
- Replacing tanh activation function by ReLU
- Increasing the regularisation strength. Without enough regularization, weight updates might be unstable.
These three solution will be tested later. The most significant stability improvement can be seen by replacing tanh by ReLU activation function of the hidden layer.
STEP #2
Two models (functions) for batch gradient descent and SGD are designed. In each model, loss values, accuracy, and training time are computed.
** Note: The SGD (stochastic gradient descent) function was coded to run each data point which computationally was so expensive, it took about 40 min to run 20000 iterations but only for 2000 data points. Increasing the data points massively increased the training time which seemed to be impossible to run using my equipment. Whereas using a mini-batch of 32 (16, 32, 64 are common according to available literature), it took less time (10 min approx) to run 20000 iterations for 20000 data points. Therefore, after the initial assessment of the code, a mini-batch of 32 was considered for the rest of the SGD analysis compared to batch gradient descent.
The below image shows how the loss values massively decrease using SGD compared to batch gradient descent.
Fig#2
STEP #3 Loss and accuracy were called and plotted vs iteration. Here is a table showing how SGD/BGD time and accuracy trends change when the learning rate is changing.
Below is a table comparing the training time and accuracy level between a couple of parameter adjustments.
As shown in this table, using 20000 datapoints, the loss/accuracy trends of BGD (batch) are oscillating over 10000 iterations. Increasing the learning rate to 0.1 even caused the oscillation between a few numbers and infinity showing how unstable the network is. Changing the regularisation strength, improved the batch training response a bit, but not completely (see an improved response in Fig #5).
Fig#3
Fig#4
Fig #3,4 obtained from 10000 iterations (display every #500 it). The steady pattern of the SGD loss/accuracy can be seen compared to the oscilating pattern of BGD.
The instability of BGD could be due to one of the reasons mentioned previously. As shown in Fig #5, increasing regularisation strength (from 0.01 to 0.1) could make it slightly less unstable but it is not enough. Applying momentum or replacing the activation function may be more helpful.
STEP 4 The decision boundary plot is also presented when each of the training approaches is applied to the test dataset. As shown in the following plots, SGD has a more accurate decision boundary when it applies to a small dataset of tests (4000 dp) comaped to BGD.
In the final step, I replaced the tanh function with ReLU to see if I could get a more stable response for the batch gradient when 20000 data points are used. Interestingly, replacing the ReLU function could significantly improve the stability of the model with batch gradient descent. I had to decrease the learning rate to 0.001 to be able to achieve this stability. The below graph compares the loss values for batch gradient descent and stochastic gradient descent using the ReLU function. It is noteworthy that the loss values are very high for BGD compared to a negligible loss for SGD function.
In conclusion, this simple neural network model with SGD provides stable and more accurate results, particularly for larger datasets. While it may take a bit longer to train the model, the improved accuracy is well worth the additional time.