{"id":259,"date":"2017-09-07T00:01:35","date_gmt":"2017-09-06T14:01:35","guid":{"rendered":"https:\/\/www.alexshoolman.com\/blog\/?p=259"},"modified":"2024-08-08T10:28:06","modified_gmt":"2024-08-08T00:28:06","slug":"what-gradient-descent-algorithms-should-i-use","status":"publish","type":"post","link":"https:\/\/www.alexshoolman.com\/blog\/2017\/09\/07\/what-gradient-descent-algorithms-should-i-use\/","title":{"rendered":"What Gradient Descent Algorithms Should I Use?"},"content":{"rendered":"

Are you working hard on your Machine Learning code but are ignoring your Gradient Descent algorithm? Your training times are likely suffering!<\/p>\n

While machine learning is used to solve some amazing problems it’s fascinating to learn how they narrow in on the solutions. For us humans you might think of this related to any problem, say throwing a ball into a basketball ring.<\/p>\n

You throw the ball and it might sail over the ring missing it. Try again and this time there’s not enough power so it falls short. So on and so forth. Eventually you throw the ball and after learning<\/strong><\/em> you score a point. What you just did was learn to complete a task (getting the ball in the ring) by trying and adjusting your next attempt accordingly.<\/p>\n

\"Gradient<\/p>\n

You got closer and closer to completing the task as you narrowed in on what exactly it was you had to do. This is described as a “learning algorithm” in the machine world. There are\u00a0many\u00a0<\/strong><\/em>learning algorithms for many different tasks.<\/p>\n

Gradient Descent is one of the biggest ones and getting your head around it is very important.<\/p>\n

Gradient Descent<\/h3>\n

Essentially gradient descent is a mathematical way of describing what any logical person might do.<\/p>\n

You try and complete the task. Then you test to see how close you were to achieving the task. Finally you try again but this time with your settings slightly changed towards the target. Then you repeat.<\/p>\n

So if you threw the basketball and it missed to the left, you’d change your throw to be more towards the right. Easy!<\/p>\n

However just like with many other things in Machine Learning, it gets\u00a0much more complicated!<\/strong><\/em> An excellent source of information to further explain this and many other forms of Gradient Descent is\u00a0Sebastian Ruder’s Overview post<\/a>.<\/p>\n

In it he goes through all the most popular<\/strong><\/em> versions and not only explains how they work, but why they are good and bad. His hugely detailed piece culminates in a fantastic GIF of all the various versions competing in a race.<\/p>\n

\"Gradient

Gradient Descent Race. Source: ruder.io<\/p><\/div>\n

Here you can see all the various versions starting off and going through their various tests. Eventually they all find their way to the star which represents the lowest error rate but at different speeds.<\/p>\n

The speed at which each of the versions reaches the star translates to faster network training times. So as you can imagine everyone wants to find the quickest way to find the lowest error rate and reduce training time.<\/p>\n

If you’re just beginning with Machine Learning, Deep Learning or Neural Nets it’s an excellent read. Even if you’ve been doing Machine Learning for a while it’s still a great resource so you know exactly which version to use for various cases. You might even learn of a faster version you’ve been neglecting!<\/p>\n","protected":false},"excerpt":{"rendered":"

Are you working hard on your Machine Learning code but are ignoring your Gradient Descent algorithm? Your training times are likely suffering! While machine learning is used to solve some amazing problems it’s fascinating to learn how they narrow in on the solutions. For us humans you might think of […]<\/p>\n","protected":false},"author":1,"featured_media":860,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[40],"tags":[25],"class_list":["post-259","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai-robotics","tag-machine-learning"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"https:\/\/www.alexshoolman.com\/blog\/wp-content\/uploads\/2030\/08\/Mountains-Compressed.jpg","jetpack_shortlink":"https:\/\/wp.me\/p92j6e-4b","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/www.alexshoolman.com\/blog\/wp-json\/wp\/v2\/posts\/259","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.alexshoolman.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.alexshoolman.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.alexshoolman.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.alexshoolman.com\/blog\/wp-json\/wp\/v2\/comments?post=259"}],"version-history":[{"count":6,"href":"https:\/\/www.alexshoolman.com\/blog\/wp-json\/wp\/v2\/posts\/259\/revisions"}],"predecessor-version":[{"id":4882,"href":"https:\/\/www.alexshoolman.com\/blog\/wp-json\/wp\/v2\/posts\/259\/revisions\/4882"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.alexshoolman.com\/blog\/wp-json\/wp\/v2\/media\/860"}],"wp:attachment":[{"href":"https:\/\/www.alexshoolman.com\/blog\/wp-json\/wp\/v2\/media?parent=259"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.alexshoolman.com\/blog\/wp-json\/wp\/v2\/categories?post=259"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.alexshoolman.com\/blog\/wp-json\/wp\/v2\/tags?post=259"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}