Skip to content

#Tag

Continuous optimisation

#continuous-optimisation · 24 entries. Projects, essays and notes, newest first.

02-02-222Learning-Rate SchedulesNoteAdvanced optimisers: stochastic and adaptive 02-02-223Momentum (SGD with momentum)NoteAdvanced optimisers: stochastic and adaptive 02-02-233Learning Rate (step size)NoteGradient descent 02-02-235Saddle Points in Loss LandscapesNoteGradient descent 02-02-237Floating-Point PrecisionNoteNumerical stability methods 02-02-238Gradient ClippingNoteNumerical stability methods 02-02-240Mixed Precision TrainingNoteNumerical stability methods 02-02-242Stable SoftmaxNoteNumerical stability methods 02-02-219Adam Optimizer (first moment)NoteAdvanced optimisers: stochastic and adaptive 02-02-220Adam Optimizer (parameter update)NoteAdvanced optimisers: stochastic and adaptive 02-02-221Adam Optimizer (second moment)NoteAdvanced optimisers: stochastic and adaptive 02-02-224Stochastic Gradient Descent (SGD)NoteAdvanced optimisers: stochastic and adaptive 02-02-225Conjugate dual mappingNoteConstrained optimisation and Lagrange multipliers 02-02-226Lagrangian FormulationNoteConstrained optimisation and Lagrange multipliers 02-02-227Saddle-Point Min-Max dualityNoteConstrained optimisation and Lagrange multipliers 02-02-228Convex Conjugate (Legendre-Fenchel transformation)NoteConvexity 02-02-229Convex Function (Definition)NoteConvexity 02-02-230Convex SetNoteConvexity 02-02-231Operations preserving convexityNoteConvexity 02-02-232Gradient Descent with Momentum (Update Rule)NoteGradient descent 02-02-234ObjectiveNoteGradient descent 02-02-236Update RuleNoteGradient descent 02-02-239Log-Sum-Exp Trick (prevents underflowoverflow)NoteNumerical stability methods 02-02-241Newton-Raphson MethodNoteNumerical stability methods