### Neural Network Architecture Overview

Deep Neural Networks (DNN) leverage various architectures for training, with one of the simplest and most fundamental being the Feedforward Neural Network (FNN). Figure 2-1 illustrates our simple, three-layer FNN.

#### Input Layer:

The first layer receives the initial data, consisting of parameters X1, X2, and X3. Each neuron in the input layer passes these data parameters to the next hidden layer.

#### Hidden Layer:

The neurons in the hidden layer calculate a weighted sum of the input data, which is then passed through an activation function. In our example, we are using the Rectified Linear Unit (ReLU) activation function. These calculations produce activation values for neurons. The activation value is modified input data value received from the input layer and published to upper layer.

#### Output Layer:

Neurons in this layer calculate the weighted sum in the same manner as neurons in the hidden layer, but the result of the activation function is the final output.

The process described above is known as the Forwarding pass operation. Once the forward pass process is completed, the result is passed through a loss function, where the received value is compared to the expected value. The difference between these two values triggers the backpropagation process. The Loss calculation is the initial phase of Backpropagation process. During backpropagation, the network fine-tunes the weight values , neuron by neuron, from the output layer through the hidden layers. The neurons in the input layer do not participate in the backpropagation process because they do not have weight values to be adjusted.

After the backpropagation process, a new iteration of the forward pass begins from the first hidden layer. This loop continues until the received and expected values are close enough to expected value, indicating that the training is complete.

**Figure 2-1:**

*Deep Neural Network Basic Structure and Operations.*

#### Forwarding Pass

*n*where

*n*is the number of neurons in the previous layer. In this example, with two input neurons, this gives a variance of 1 (=2/2). The weights are then drawn from a normal distribution ~

*N(0,√variance),*which in this case is ~N(0,1). Basically, this means that the randomly generated weight values are centered around zero with a standard deviation of one.

Z3 = (X1 ⋅ W31) + (X2 ⋅ W32) + b3Given:Z3 = (1 ⋅ 0.5) + (0 ⋅ 0.5) + 0Z3 = 0.5 + 0 + 0Z3 = 0.5

ReLU (Z3) = ReLU (0.5) = 0.5

Z4 = (X1 ⋅ W41) + (X2 ⋅ W42) + b4Given:Z4 = (1 ⋅ 1) + (0 ⋅1) + 0.5Z4 = 1 + 0 + 0.5Z4 = 1.5

ReLU (Z4) = ReLU (1.5) = 1.5

**Figure 2-2:**

*Forwarding Pass on Hidden Layer.*

#### Backpropagation process

Loss (L) = (H3 x W53 + H4 x W54 + b5 – Ye)2Given:L = (0.5 x 1 + 1.5 x 1 + 0.5 - 2)2L = (0.5 + 1.5 + 0.5 - 2)2L = 0.52L= 0.25

**Figure 2-3:**

*Forwarding Pass on Output Layer.*

*η*(the learning rate) controls the step size during weight updates in the backpropagation process, balancing the speed of convergence with the stability of training. In our example, we are using a learning rate of 1/100 = 0.01. The term hyper-parameters refers to parameters that affect the final result.

**Gradient Calculation:**

∂L= 2W53 x (Yr – Ye)∂W53

Given

= 2 x 0.5 x (2.5 - 2)= 1 x 0.5= 0.5

**New weight value calculation.**

W53 (new) = W53(old) – η x ∂L/∂W53Given:W53 (new) = 1–0.01 x 0.5W53 (new) = 0.995

**Figure 2-4:**

*Backpropagation - Gradient Calculation and New Weight Value Computation.*

**Figure 2-5:**

*Backpropagation - Gradient Calculation and New Bias Computation.*

__W53__+ H4 x

__W54__+ b5 – Ye)2

__∂L__=

__∂L__x

__∂H3__

## No comments:

## Post a Comment