Optimizers

There are many optimizers:

  1. Gradient Descent
  2. Stochastic Gradient Descent (SGD)
  3. Mini Batch SGD
  4. Stochastic Gradient Descent with Momentum
  5. Nesterov Accelerated Gradient (NAG)
  6. AdaGrad
  7. RMSProp
  8. AdaDelta
  9. Adam
  10. AdaMax
  11. NADAM
  12. AMSGrad

References

  1. https://www.ruder.io/optimizing-gradient-descent
  2. https://www.scaler.com/topics/deep-learning/optimizers-in-deep-learning/
  3. https://medium.com/mlearning-ai/optimizers-in-deep-learning-7bf81fed78a0
  4. https://towardsdatascience.com/optimizers-for-training-neural-network-59450d71caf6