Skip to content

02-02Notes

A working library of the mathematics behind the analysis.

524 notes in 7 areas, from notation to machine learning. Each takes one formula: what it says, a small worked example, and code that runs. Revised in place. Page 10 of 22.

IndexAll notes

02-02-217Vector representationLinear algebra and geometryVector space axioms and coordinates 02-02-218Zero VectorLinear algebra and geometryVector space axioms and coordinates 02-02-219Adam Optimizer (first moment)Calculus and optimisationAdvanced optimisers: stochastic and adaptive 02-02-220Adam Optimizer (parameter update)Calculus and optimisationAdvanced optimisers: stochastic and adaptive 02-02-221Adam Optimizer (second moment)Calculus and optimisationAdvanced optimisers: stochastic and adaptive 02-02-222Learning-Rate SchedulesCalculus and optimisationAdvanced optimisers: stochastic and adaptive 02-02-223Momentum (SGD with momentum)Calculus and optimisationAdvanced optimisers: stochastic and adaptive 02-02-224Stochastic Gradient Descent (SGD)Calculus and optimisationAdvanced optimisers: stochastic and adaptive 02-02-225Conjugate dual mappingCalculus and optimisationConstrained optimisation and Lagrange multipliers 02-02-226Lagrangian FormulationCalculus and optimisationConstrained optimisation and Lagrange multipliers 02-02-227Saddle-Point Min-Max dualityCalculus and optimisationConstrained optimisation and Lagrange multipliers 02-02-228Convex Conjugate (Legendre-Fenchel transformation)Calculus and optimisationConvexity 02-02-229Convex Function (Definition)Calculus and optimisationConvexity 02-02-230Convex SetCalculus and optimisationConvexity 02-02-231Operations preserving convexityCalculus and optimisationConvexity 02-02-232Gradient Descent with Momentum (Update Rule)Calculus and optimisationGradient descent 02-02-233Learning Rate (step size)Calculus and optimisationGradient descent 02-02-234ObjectiveCalculus and optimisationGradient descent 02-02-235Saddle Points in Loss LandscapesCalculus and optimisationGradient descent 02-02-236Update RuleCalculus and optimisationGradient descent 02-02-237Floating-Point PrecisionCalculus and optimisationNumerical stability methods 02-02-238Gradient ClippingCalculus and optimisationNumerical stability methods 02-02-239Log-Sum-Exp Trick (prevents underflowoverflow)Calculus and optimisationNumerical stability methods 02-02-240Mixed Precision TrainingCalculus and optimisationNumerical stability methods