Backpropagation is one of the most important concepts in neural networks, however, it is challenging for learners to understand its concept because it is the most notation heavy part. Luckily, when the layers of notation are pealed back, the simplicity of backprop is revealed. Backprop is just a very simple process to tell us which parameters to change in a neural network.

For the remainder of the article, by marrying explanation, notation, and interactive tools, the aim is to get a understanding of the foundations. Note that throughout the article there will be highlighted words that will give extra explanation on mouse over.

The goal in a neural network, or any optimization problem for that matter, is to lower and minimize whatever loss function we define. For this article, we will be performing regression and using mean squared error loss MSE.

Before hitting the calculus, tune the weight and bias on one neuron by dragging the slider. By observing changes in the loss, try to make it reach 0. When you feel like you've lowered the loss enough (or need some help), press the Click to reveal the graph button below.

Reflection

The intuition and logic from the exercise above, is the foundation for what backpropagation and optimization aims to achieve.

When you start to change the weight, you observe where the loss moves, if it moves up, you move the weight the other direction to lower the loss. This, without the math, is the main principle behind gauging the rate of change and optimizing a neural network. Let's apply this method of thinking!

First to get the instantaneous rate of change(derivative), we get our point of interest and another point that is infinitely close and calculate slope, this can be visualized as a tangent line at the point of interest for a function with one variable like $f(x)$.

In the context of our one neuron neural network, we can compose the whole network as a nested function

$\text{loss}(\text{neuron}(x,w,b),y)$

Since want to tune the parameters weight $w$ and bias $b$, we want to know how each parameters will affect the loss, or in other words, the partial derivative of loss with respect to each parameter, also called the gradient. By computing the gradient of loss, we effectively have gauged how changing each parameter will affect the loss: the direction of steepest ascent. But we want to lower loss, therefore we use the opposite direction to get the direction of steepest descent, to perform gradient descent.

This step is can be visualized by graphing out a loss function. Let's graph the Mean Squared error loss function with same data that will show up later (stay tuned). The contour plot below is the result.

This plot shows the different losses represented by colors descending down from the greens, to the blues, to the purples, down to a loss of 0 in white. It also shows the weights represented by the x axis and biases represented by the y axis. For example if you have a weight=-1 and bias=5 the point of (-1, 5) on the plot, we see that the loss is 0.0 (white on the contour); that would be the place we want to eventually get to. You could also think of it as a physical hole where the colors represent depth. You might already see how we could take small steps down the hole till we reach the bottom where loss is minimized.

The idea of gradient descent for optimization is below starting with 1)

And thats all there is to gradient descent! 1) Start with a point, get the 2) steepest ascent, 3) flip it to reflect the steepest descent, then 4) take a step in that direction. After doing this enough times, we will reach the minimum loss possible.

The question then becomes, how can we calculate the gradient? To answer that, we need a bit of calculus to calculate all the derivatives with respect to the loss. Mainly you will need to use the chain rule from calculus because of the nature of the function we've composed.

Below is a color coded example of the chain rule. Start by sliding the slider and notice how the output is the input to the next function and so forth. Then read the explanation below.

Suppose we wanted to see how blue was affected by pink.
First lets start at blue, then

1. observe how blue was affected by orange
2. observe how orange was affected by green
3. observe how green was affected by pink

By chaining these observations together, we get how how blue was affected by pink.

This logic applied to our one neuron neural network looks like

$\frac{ \partial \text{loss}}{\partial w} = \frac{ \partial \text{loss}}{\partial \text{neuron}} \frac{ \partial \text{neuron}}{\partial w}$

$\frac{ \partial \text{loss}}{\partial b} = \frac{ \partial \text{loss}}{\partial \text{neuron}} \frac{ \partial \text{neuron}}{\partial b}$

These chains can be broken up into more intermediate derivatives all the way down to their primitives (basis of automatic differentiation). The main takeaway is that we first observe how the loss output was affected by the neuron output $\frac{ \partial \text{loss}}{\partial \text{neuron}}$, then we observe how the neuron output was affected by each parameter $\frac{ \partial \text{neuron}}{\partial \text{parameter}}$ and can chain these to observe how the loss was affected by the parameter $\frac{ \partial \text{loss}}{\partial \text{parameter}}$. Notice that we compute these derivatives going backward (where the term backpropagation comes from), which has the added benefit of reusing values computed in the forward propagation (for more on this check out reverse mode automatic differentiation).

Let's go through a concrete example of forward propagation, then an emphasis on backward propagation. The training example will be ($x = 2.1$, $y = 4$), the weight will be $w = 1$, and the bias will be $b = 0$. $\hat{y}$ represents the neuron output and predicted value, and $\text{loss}$ represents squared error loss.

Forward Overview

Click here to see descriptions of the diagram above

Forward Computation

Now we can go backward and compute partial derivatives with the chain rule to get the $\nabla \text{loss}$

Backward Overview

Click here to see descriptions of the diagram above

Backward Computation

$\frac{\partial \text{loss}}{\partial w} = -7.98$

$\frac{\partial \text{loss}}{\partial b} = -3.8$

Then, we update the parameters with opposite gradient to descend loss, in this case learning rate is $\text{lr} = 0.01$

$w := w - \text{lr} \cdot \frac{\partial \text{loss}}{\partial w} = (1) - (0.01) \cdot (-7.98) = 1.0798$

$b := b - \text{lr} \cdot \frac{\partial \text{loss}}{\partial b} = (0) - (0.01) \cdot (-3.8) = 0.038$

To see how well our tuned parameters do, let's do one more forward pass

$\text{loss} = ((1.0798)(2.1) + 0.038) - 4)^2 = 2.87$

Total loss decrease of $3.61 - 2.87 = 0.74$. Loss went down!

To see the what we just did (forward, backward, update) on more data and on the entire batch as opposed to a single training example, press to start the training process. Watch how as the line gets better fit, the loss decreases.

To fit more interesting data that is non-linear (e.g. sine wave or quadratic), we need to add complexity to vary output to make sure we are not constrained to only linear outputs. We can do this by adding more neurons per layer, more layers, and adding non-linearities (activation functions) to the outputs. If you think of our entire neural network as a function, then by adding more neurons and more layers we are creating a more nested function. Not only does this create more parameters that we can tune to vary the output, it also maintains the property of differentiability: important so we can compute the gradient. And by adding non-linear activation functions with points of deactivation, certain neurons may have no effect on the output while others may become more activated, contributing to outputs that don't have to follow linear constraints. We will be using the ReLU activation function in the hidden layers.

Above is an example of a neural network with one input, three hidden layers with eight neurons each, and one output neuron. The output of each neuron is fed into the neurons of the next layer and so forth (like a nested function). Each link represents a weight and a corresponding input into the respective neuron: notice how the more neurons we add, the more connections there are and the more parameters we can tune to get our desired output.

1. Forward propagation resulting in an output and loss
2. Backward propagation using the chain rule to compute the gradient
3. Descend the loss by performing gradient descent

The process doesn't change from the single neuron example! Since the network is deeper, we have to calculate more derivatives going backwards and have to tune more parameters with gradient descent, but the logic stays the same.

A great way to visualize backpropagation in a large network is with vertical arrows representing which direction we need to nudge the neuron output in order to lower loss: $-\frac{\partial \text{loss}}{\partial \text{activation} }$. In the neural network below, you will be able to visualize all phases of a single epoch with an emphasis on backpropagation. Read the instructions below to get a quick start!

1. Press to start training
2. Then press Click to animate epoch # to see forward propagation, backward propagation, and update animation at the epoch #
3. To go back to fitting mode click Go back to fitting

Click on to reveal extra descriptions