Neural Network Demonstration and Deep Learning Explanation

Introduction

In this document, we’ll explore the basics of neural networks and deep learning. We’ll visualize a simple neural network with 3 input nodes, a hidden layer with 3 nodes, and an output layer with 2 nodes. We will also demonstrate a basic feedforward pass and a backpropagation process to illustrate the core concepts.

Neural Networks and Deep Learning

Deep learning is a subset of machine learning that involves neural networks with multiple layers, known as deep neural networks. Neural networks are composed of neurons (or nodes) organized in layers:

1. Input Layer: Receives the input data.
2. Hidden Layer: Performs computations and feature transformations.
3. Output Layer: Produces the final prediction.

# Install and load required libraries
if (!require('DiagrammeR')) install.packages('DiagrammeR', dependencies=TRUE)
library(DiagrammeR)

# Function to draw the neural network graph
draw_neural_network <- function() {
  grViz("
  digraph G {
    rankdir=LR;
    
    node [shape=circle, fontname=Arial, fontsize=10];
    
    # Input Layer
    subgraph cluster_input {
      label = \"Input Layer\";
      I1 [label=\"Input 1\"];
      I2 [label=\"Input 2\"];
      I3 [label=\"Input 3\"];
      color=none;
    }
    
    # Hidden Layer
    subgraph cluster_hidden {
      label = \"Hidden Layer\";
      H1 [label=\"Hidden 1\"];
      H2 [label=\"Hidden 2\"];
      H3 [label=\"Hidden 3\"];
      color=none;
    }
    
    # Output Layer
    subgraph cluster_output {
      label = \"Output Layer\";
      O1 [label=\"Output 1\"];
      O2 [label=\"Output 2\"];
      color=none;
    }
    
    # Connect Input Layer to Hidden Layer
    I1 -> H1;
    I1 -> H2;
    I1 -> H3;
    I2 -> H1;
    I2 -> H2;
    I2 -> H3;
    I3 -> H1;
    I3 -> H2;
    I3 -> H3;
    
    # Connect Hidden Layer to Output Layer
    H1 -> O1;
    H1 -> O2;
    H2 -> O1;
    H2 -> O2;
    H3 -> O1;
    H3 -> O2;
  }
  ")
}

# Draw the neural network graph
draw_neural_network()

The Sigmoid Activation Function and Its Derivative

Sigmoid Activation Function The sigmoid activation function is a widely-used nonlinear activation function in neural networks, especially for binary classification problems. It maps the input values to the (0, 1) range, which is particularly useful for models where the output is expected to be a probability.

Mathematically, the sigmoid function σ(x) is defined as:

\[σ(x) = \frac{1}{1 + e^{-x}}\]

Here, ( x ) can be any real-valued number, and ( e ) is the base of the natural logarithm (approximately 2.71828). The function outputs values between 0 and 1, making it suitable for producing probabilities.

The sigmoid function has a characteristic “S” shaped curve (hence the name “sigmoid”), which smoothly transitions from 0 to 1.

Derivative of the Sigmoid Function The derivative of the sigmoid function, often referred to as ( σ’(x) ), plays a crucial role in backpropagation. It is used to compute the gradient, which helps in updating the weights during training.

The derivative of the sigmoid function is given by:

\[[ σ'(x) = σ(x) \cdot (1 - σ(x)) ]\]

This derivative indicates how steeply the sigmoid curve is changing at any point ( x ). For values of ( x ) far from 0, the sigmoid function’s slope is small, which can lead to the “vanishing gradient” problem. However, near ( x = 0 ), the slope is maximal, which is beneficial for learning.

# Define sigmoid activation function
sigmoid <- function(x) {
  1 / (1 + exp(-x))
}

# Define the derivative of the sigmoid function
sigmoid_derivative <- function(x) {
  x * (1 - x)
}

Initialize Input Features

The input layer of our neural network consists of 3 nodes. We will initialize these input features with random numbers to simulate real-world data.

# Set seed for reproducibility
set.seed(42)

# Initialize input features with random numbers
inputs <- matrix(runif(3, min = 0, max = 1), nrow = 1)

# Print the initialized input features
print("Initialized input features (3 nodes):")

[1] "Initialized input features (3 nodes):"

print(inputs)

         [,1]      [,2]      [,3]
[1,] 0.914806 0.9370754 0.2861395

Initialize Weights

Define the weights between the input layer and the hidden layer, as well as the weights between the hidden layer and the output layer. These weights are initialized with random values.

1. Input to Hidden Layer Weights: We define a matrix of weights connecting the input layer (with 3 nodes) to the hidden layer (with 4 nodes).
2. Hidden to Output Layer Weights: We define another matrix of weights connecting the hidden layer (with 4 nodes) to the output layer (with 2 nodes).

Input to Hidden Layer Weights:

\[[ W_{\text{input_hidden}} = \begin{bmatrix} w_{1,1} & w_{1,2} & w_{1,3} & w_{1,4} \\ w_{2,1} & w_{2,2} & w_{2,3} & w_{2,4} \\ w_{3,1} & w_{3,2} & w_{3,3} & w_{3,4} \end{bmatrix} ]\]

Hidden to Output Layer Weights:

\[[ W_{\text{hidden_output}} = \begin{bmatrix} v_{1,1} & v_{1,2} \\ v_{2,1} & v_{2,2} \\ v_{3,1} & v_{3,2} \\ v_{4,1} & v_{4,2} \end{bmatrix} ]\]

# Weights between input layer and hidden layer (3x4)
weights_input_hidden <- matrix(c(
  0.4, 0.1, 0.2, 0.3,
  0.7, 0.5, 0.3, 0.2,
  0.9, 0.6, 0.8, 0.4
), nrow = 3, byrow = TRUE)
print("Weights between input and hidden layer:")
print(weights_input_hidden)

# Weights between hidden layer and output layer (4x2)
weights_hidden_output <- matrix(c(
  0.2, 0.6,
  0.3, 0.5,
  0.7, 0.8,
  0.1, 0.4
), nrow = 4, byrow = TRUE)
print("Weights between hidden and output layer:")
print(weights_hidden_output)

Feedforward Process

The feedforward process in a neural network involves passing the input data through the various layers of the network to produce the output. Here, we describe the steps involved in the feedforward process for our neural network with 3 input nodes, a hidden layer with 3 nodes, and an output layer with 2 nodes.

Step 1: Compute Hidden Layer Inputs

First, we compute the input to the hidden layer by multiplying the input features with the weights connecting the input layer to the hidden layer. Let’s denote the input features as \(\mathbf{X}\) and the weights between the input and hidden layers as \(\mathbf{W_{\text{input\_hidden}}}\).

Mathematically, this can be represented as:

\[ \mathbf{H_{\text{input}}} = \mathbf{X} \times \mathbf{W_{\text{input\_hidden}}} \]

where: - \(\mathbf{X}\) is a 1x3 matrix (input features). - \(\mathbf{W_{\text{input\_hidden}}}\) is a 3x4 matrix (weights between input and hidden layers). - \(\mathbf{H_{\text{input}}}\) will be a 1x4 matrix (input to hidden layer).

# Feedforward process
hidden_input <- inputs %*% weights_input_hidden
hidden_output <- sigmoid(hidden_input)
print("Output of hidden layer (after activation):")

[1] "Output of hidden layer (after activation):"

print(hidden_output)

          [,1]      [,2]      [,3]      [,4]
[1,] 0.9890131 0.9478464 0.9608343 0.8698915

Feedforward Process: Output Layer Computation

In this section, we will compute the input to the output layer and then apply the sigmoid activation function to obtain the final output of the neural network.

Step 3: Compute Input to Output Layer

The inputs to the output layer are computed by performing matrix multiplication between the outputs from the hidden layer and the weights connecting the hidden layer to the output layer.

Mathematically, this is represented as:

\[ \mathbf{Y_{\text{input}}} = \mathbf{H_{\text{output}}} \times \mathbf{W_{\text{hidden\_output}}} \]

where: - \(\mathbf{H_{\text{output}}}\) is the 1x4 matrix of outputs from the hidden layer after the activation function. - \(\mathbf{W_{\text{hidden\_output}}}\) is the 4x2 matrix of weights between the hidden layer and the output layer. - \(\mathbf{Y_{\text{input}}}\) is the resulting 1x2 matrix of inputs to the output layer nodes.

# Compute input to output layer
final_input <- hidden_output %*% weights_hidden_output
final_output <- sigmoid(final_input)
print("Output of final layer (after activation):")

[1] "Output of final layer (after activation):"

print(final_output)

          [,1]      [,2]
[1,] 0.7758649 0.8987994

Assume Expected Output

In order to evaluate the performance of our neural network, we need to compare its predictions to the actual expected output. This process is essential during training, as it provides the basis for calculating errors and subsequently updating the weights through backpropagation.

Defining the Expected Output

For this demonstration, let’s assume the expected output of our neural network. The expected output serves as the ground truth against which we will measure the network’s performance.

The following R code defines a simple expected output matrix. In this example, let’s assume the expected output for our single input example is \([0, 1]\).

# Assume expected output
expected_output <- matrix(c(0, 1), nrow = 1)

print("Assumed expected output:")

[1] "Assumed expected output:"

print(expected_output)

     [,1] [,2]
[1,]    0    1

Calculate Error of the Output

In this step, we calculate the error of the output of the neural network. This error represents the difference between the predicted output generated by the neural network and the expected (ground truth) output. Measuring this error is a critical part of the training process, as it provides the information needed to adjust the network’s weights to improve accuracy.

Error Calculation

The error is computed as the difference between the expected output and the actual output produced by the neural network. Mathematically, this can be represented as:

\[ \text{Error} = \mathbf{Y_{\text{expected}}} - \mathbf{Y_{\text{output}}} \]

where: - \(\mathbf{Y_{\text{expected}}}\) is the matrix of expected (ground truth) outputs. - \(\mathbf{Y_{\text{output}}}\) is the matrix of actual outputs produced by the neural network.

# Calculate error of the output
output_error <- expected_output - final_output
print("Output error:")

[1] "Output error:"

print(output_error)

           [,1]      [,2]
[1,] -0.7758649 0.1012006

Backpropagation Process

In the backpropagation process, we calculate the error and update the weights to minimize this error. This process is essential for training the neural network.

Calculating Delta for the Output Layer

The first step in backpropagation is to compute the delta (gradient) for the output layer. The delta represents the error term adjusted by the derivative of the activation function. For the sigmoid activation function, the derivative is:

\[ σ'(x) = σ(x) \cdot (1 - σ(x)) \]

The delta for the output layer can be computed as:

\[ \delta_{\text{output}} = \text{output\_error} \cdot σ'(\mathbf{Y_{\text{output}}}) \]

where: - \(\text{output\_error}\) is the error of the output computed earlier. - \(σ'(\mathbf{Y_{\text{output}}})\) is the derivative of the sigmoid function applied to the output layer values.

# Define the derivative of the sigmoid function
sigmoid_derivative <- function(x) {
  x * (1 - x)
}

# Calculate the delta for the output layer
d_output <- output_error * sigmoid_derivative(final_output)
print("Delta for the output layer:")

[1] "Delta for the output layer:"

print(d_output)

           [,1]        [,2]
[1,] -0.1349218 0.009205111

Update Weights: Hidden to Output Layer

After calculating the deltas for the output layer in the backpropagation process, we proceed to update the weights connecting the hidden layer to the output layer. The weight update process aims to minimize the error in the network by adjusting the weights based on the calculated deltas.

Weight Update Process

Using the computed deltas (\(\delta_{\text{output}}\)), we adjust the weights between the hidden and output layers. The weight update rule for gradient descent is as follows:

\[ \text{New Weight} = \text{Old Weight} + \eta \cdot (\mathbf{H_{\text{output}}}^T \cdot \delta_{\text{output}}) \]

where: - \(\eta\) is the learning rate, controlling the step size of weight updates. - \(\mathbf{H_{\text{output}}}^T\) is the transpose of the hidden layer output matrix. - \(\delta_{\text{output}}\) is the delta for the output layer.

# Define the learning rate for weight updates
learning_rate <- 0.5

# Update the weights between hidden and output layers
weights_hidden_output <- weights_hidden_output + t(hidden_output) %*% d_output * learning_rate

print("Updated weights between hidden and output layer:")

[1] "Updated weights between hidden and output layer:"

print(weights_hidden_output)

           [,1]      [,2]
[1,] 0.13328030 0.6045520
[2,] 0.23605744 0.5043625
[3,] 0.63518127 0.8044223
[4,] 0.04131635 0.4040037

Conclusion: Interpretation of Output Error

In the neural network context, the “Output error:” matrix conveys the disparities between the predicted output values generated by the network and the actual expected output values. Understanding and analyzing these output errors are fundamental to evaluating the network’s performance and guiding the training process towards optimization.

Analysis of Output Error Values

• Negative Error (Column 1):
  • The negative value of approximately -0.7758649 suggests an underestimation by the neural network in predicting the corresponding target or class. The larger the magnitude of the negative error, the more significant the underestimation relative to the actual expected output.
• Positive Error (Column 2):
  • The positive value of around 0.1012006 reveals an overestimation in the network’s predictions for the corresponding target or class. The magnitude of the positive error indicates the degree of overestimation relative to the ground truth.

Insight for Training and Optimization

• Error Minimization:
  • Minimizing the output error is the main objective during training. Techniques like backpropagation and weight adjustments reduce these errors, aligning the network’s predictions more closely with the expected outputs and improving its overall accuracy.
• Guiding Network Improvement:
  • Analyzing and monitoring the output error values provides insights into the network’s performance, enabling iterative adjustments to enhance prediction quality. Understanding the error patterns helps in identifying areas for refinement and optimizing the neural network’s learning process.

In conclusion, the interpretation of output error values serves as a metric in assessing the neural network’s predictive performance and refining its accuracy. By minimizing these errors through targeted optimization strategies, the network’s predictive capabilities become more reliable and precise predictions.