Making Backpropagation, Autograd, MNIST Classifier from scratch in Python

from freepik

Backpropagation (backward propagation of errors) — is a widely used algorithm in training feedforward networks. It computes the gradient of the loss function with respect to the weights of the network. The main idea of it is to break big functions in small parts and use partial derivatives to get function derivative with using the Chain Rule.

So training the model is basically solving this equation:

And because solving this can be a very hard task, here comes backpropagation and gradient descent(updating weights by a small amount based on the gradient to move in the way of loss minimization).

All this is based on a simple differentiation chain rule:

For example, we have a simple function with 2 nodes(operation):


Let’s say we have 3 variables x=-2, y=5, z=-4, the result will be f=-12, and our target for training is -13. Simple loss=f-target=1.

Now we need to propagate our error(1) back to the x,y,z. To do this we will need to calculate partial derivatives for every function(operation):


And our chain is:

Let’s build a simple Autograd example

Here we will build a simple automatic differentiation example from the above formulas.

Now let’s see what will happens

Checking with PyTorch Autograd

As you can see we got the same gradients as in our simple Autograd solution

Let’s build Neural Networks and solve 2 simple tasks

For simplicity, we will not use Autograd here and will solve a derivative for each layer.

Running simple point interpolation

Classifying MNIST with our custom network

Now let’s solve some classification task on MNIST dataset. We will use some PyTorch utils to make it simpler

And that’s all. Don’t forget that you can run this code in Google Colab


Become a Patron and support our community to make more interesting articles & tutorials

Want to become an expert in Computer Vision & Object Detection?

Subscribe on New Practical Course




Machine Learning and Computer Vision Researcher. Founder LearnML.Today

Love podcasts or audiobooks? Learn on the go with our new app.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Andrey Nikishaev

Andrey Nikishaev

Machine Learning and Computer Vision Researcher. Founder LearnML.Today

More from Medium

Multi-Class Image Classification with TensorFlow

Deblurring an Image using inverse filtering

Forward and Backward propagation of Max Pooling Layer in Convolutional Neural Networks

Neural Network From Scratch in Python pt-5 (softmax function) + code