Are you working hard on your Machine Learning code but are ignoring your Gradient Descent algorithm? Your training times are likely suffering!

While machine learning is used to solve some amazing problems it’s fascinating to learn how they narrow in on the solutions. For us humans you might think of this related to any problem, say throwing a ball into a basketball ring.

You throw the ball and it might sail over the ring missing it. Try again and this time there’s not enough power so it falls short. So on and so forth. Eventually you throw the ball and after learning you score a point. What you just did was learn to complete a task (getting the ball in the ring) by trying and adjusting your next attempt accordingly.

Gradient Descent Task

You got closer and closer to completing the task as you narrowed in on what exactly it was you had to do. This is described as a “learning algorithm” in the machine world. There are many learning algorithms for many different tasks.

Gradient Descent is one of the biggest ones and getting your head around it is very important.

Gradient Descent

Essentially gradient descent is a mathematical way of describing what any logical person might do.

You try and complete the task. Then you test to see how close you were to achieving the task. Finally you try again but this time with your settings slightly changed towards the target. Then you repeat.

So if you threw the basketball and it missed to the left, you’d change your throw to be more towards the right. Easy!

However just like with many other things in Machine Learning, it gets much more complicated! An excellent source of information to further explain this and many other forms of Gradient Descent is Sebastian Ruder’s Overview post.

In it he goes through all the most popular versions and not only explains how they work, but why they are good and bad. His hugely detailed piece culminates in a fantastic GIF of all the various versions competing in a race.

Gradient Descent Race

Gradient Descent Race. Source:


Here you can see all the various versions starting off and going through their various tests. Eventually they all find their way to the star which represents the lowest error rate but at different speeds.

The speed at which each of the versions reaches the star translates to faster network training times. So as you can imagine everyone wants to find the quickest way to find the lowest error rate and reduce training time.

If you’re just beginning with Machine Learning, Deep Learning or Neural Nets it’s an excellent read. Even if you’ve been doing Machine Learning for a while it’s still a great resource so you know exactly which version to use for various cases. You might even learn of a faster version you’ve been neglecting!

For the newer readers... if you’re interested in learning more about being mortgage free in under 10 years, automatically and without cutting back on the things you love... You’ll probably like How To Pay Off Your Mortgage Early, Go From No Idea To Mortgage Free In Under 10 Years.

The benefits include: 1) How to pay off your mortgage faster than 99% of people with one hour a month of work 2) How to get rid of your debt and have the freedom to spend money on the things you love, guilt free 3) Clear outline of how to setup your expenses, mortgage and general finance 4) How offset accounts work and how to get the same result without being gouged by the big banks 5) How to cut through the crap and focus on the things that truly matter when taking down a mortgage 6) How to adjust the strategy so it works for you, even if you have kids, even if you only have one income 7) How to do all of these things and maintain a normal social life (and never be cheap).


Alex Shoolman

How To Pay Off A Mortgage Early – What if you could be mortgage free in under 10 years, automatically and without cutting back on the things you love? – Super Special! – 52% off until the end of November!

Editor in Chief at Alex Shoolman – Learn how to improve your life with technology, see where it’s going in the future and how you can take advantage of it.

Editor in Chief at Mutilate The Mortgage – We help people go from “no idea” to mortgage free in under 10 years.