Deep Neural Networks (DNNs) are thought to be probably the most efficient instruments for locating patterns in giant datasets by coaching. On the core of the coaching issues, we have now complicated loss landscapes and the coaching of a DNN boils all the way down to optimizing the loss because the variety of iterations will increase. Just a few of essentially the most generally used optimizers are Stochastic Gradient Descent, RMSProp (Root Imply Sq. Propagation), Adam (Adaptive Second Estimation) and so forth.
Not too long ago (September 2024), researchers from Apple (and EPFL) proposed a brand new optimizer, AdEMAMix¹, which they present to work higher and sooner than AdamW optimizer for language modeling and picture classification duties.
On this publish, I’ll go into element concerning the mathematical ideas behind this optimizer and talk about some very fascinating outcomes offered on this paper. Matters that can be lined on this publish are:
- Assessment of Adam Optimizer
- Exponential Shifting Common (EMA) in Adam.
- The Primary Concept Behind AdEMAMix: Combination of two EMAs.
- The Exponential Decay Fee Scheduler in AdEMAMix.