Skip to content

#Tag

Calculus and optimisation

#calculus · 121 entries. Projects, essays and notes, newest first. Page 1 of 6.

02-02-222Learning-Rate SchedulesNoteAdvanced optimisers: stochastic and adaptive 02-02-223Momentum (SGD with momentum)NoteAdvanced optimisers: stochastic and adaptive 02-02-233Learning Rate (step size)NoteGradient descent 02-02-235Saddle Points in Loss LandscapesNoteGradient descent 02-02-237Floating-Point PrecisionNoteNumerical stability methods 02-02-238Gradient ClippingNoteNumerical stability methods 02-02-240Mixed Precision TrainingNoteNumerical stability methods 02-02-242Stable SoftmaxNoteNumerical stability methods 02-02-303Directional DerivativeNoteDifferential operators: gradients, Jacobians, Hessians 02-02-309Gradient Flow (vanishing and exploding gradients)NoteMatrix calculus identities and automatic differentiation 02-02-219Adam Optimizer (first moment)NoteAdvanced optimisers: stochastic and adaptive 02-02-220Adam Optimizer (parameter update)NoteAdvanced optimisers: stochastic and adaptive 02-02-221Adam Optimizer (second moment)NoteAdvanced optimisers: stochastic and adaptive 02-02-224Stochastic Gradient Descent (SGD)NoteAdvanced optimisers: stochastic and adaptive 02-02-225Conjugate dual mappingNoteConstrained optimisation and Lagrange multipliers 02-02-226Lagrangian FormulationNoteConstrained optimisation and Lagrange multipliers 02-02-227Saddle-Point Min-Max dualityNoteConstrained optimisation and Lagrange multipliers 02-02-228Convex Conjugate (Legendre-Fenchel transformation)NoteConvexity 02-02-229Convex Function (Definition)NoteConvexity 02-02-230Convex SetNoteConvexity 02-02-231Operations preserving convexityNoteConvexity 02-02-232Gradient Descent with Momentum (Update Rule)NoteGradient descent 02-02-234ObjectiveNoteGradient descent 02-02-236Update RuleNoteGradient descent