优化算法动画演示Alec Radford's animations for optimization algorithms

Posted

tags:

篇首语:本文由小常识网(cha138.com)小编为大家整理,主要介绍了优化算法动画演示Alec Radford's animations for optimization algorithms相关的知识,希望对你有一定的参考价值。

Alec Radford has created some great animations comparing optimization algorithms SGDMomentumNAGAdagradAdadelta,RMSprop (unfortunately no Adam) on low dimensional problems. Also check out his presentation on RNNs.

"Noisy moons: This is logistic regression on noisy moons dataset from sklearn which shows the smoothing effects of momentum based techniques (which also results in over shooting and correction). The error surface is visualized as an average over the whole dataset empirically, but the trajectories show the dynamics of minibatches on noisy data. The bottom chart is an accuracy plot."
技术分享
"Beale‘s function: Due to the large initial gradient, velocity based techniques shoot off and bounce around - adagrad almost goes unstable for the same reason. Algos that scale gradients/step sizes like adadelta and RMSProp proceed more like accelerated SGD and handle large gradients with more stability."
技术分享
"Long valley: Algos without scaling based on gradient information really struggle to break symmetry here - SGD gets no where and Nesterov Accelerated Gradient / Momentum exhibits oscillations until they build up velocity in the optimization direction. Algos that scale step size based on the gradient quickly break symmetry and begin descent."
技术分享
"Saddle point: Behavior around a saddle point. NAG/Momentum again like to explore around, almost taking a different path. Adadelta/Adagrad/RMSProp proceed like accelerated SGD."
技术分享
 
from: http://www.denizyuret.com/2015/03/alec-radfords-animations-for.html

以上是关于优化算法动画演示Alec Radford's animations for optimization algorithms的主要内容,如果未能解决你的问题,请参考以下文章

十大排序算法(程序员必会)

zookeeperraft 共识算法 动画演示 网站

卷积算法动画演示

用Turtle库写一个冒泡排序动画演示

动画演示广度优先算法寻找最短路径

动画演示深度优先算法搜寻逃出迷宫的路径