Skip to content

Explanation

Explanation is the discussion that deepens understanding. A reader arrives wanting to know why — why a method works, why it was designed this way, why one approach is preferred over another.

Explanation is read rather than consulted, and it is the one section where the prose may take a position and argue for it.

What belongs here

  • Derivations. The chain rule applied through a network, worked out in full.
  • The reasoning behind a design: why bias correction matters early in training, why initialisation variance depends on fan-in.
  • Comparisons that argue, as opposed to tabulate: what momentum buys over plain gradient descent, and what it costs.
  • Context and history: what problem a method was introduced to solve, and what it replaced.
  • Connections between topics that no single implementation owns.

An explanation is free to be discursive. It is the right place for a caveat, an alternative view, or an admission that a practice is convention rather than consequence.

What does not belong here

  • Instructions. The moment a page tells a reader what to type to achieve their own goal, it is a how-to guide.
  • A lesson for a newcomer. That is a tutorial. An explanation may assume a reader who already knows the vocabulary.
  • The definitive table of defaults. That is reference. An explanation may discuss why 0.9 is a reasonable momentum coefficient without becoming the page a reader consults to find that number.
  • Restated API documentation. Link to the generated reference instead.

Contents

  • Forward and backward propagation — the full derivation from the L-layer document, including the recursive formula for hidden layers.
  • Matrix calculus for neural networks — the chain rule in matrix form, and why vectorisation changes the complexity.
  • The optimization landscape — why optimization is difficult, and what gradient descent variants trade against each other.
  • Exponential weighted averages — the shared mechanism underneath momentum, RMSprop, and Adam, and why bias correction is needed.
  • Adaptive methods — what per-parameter learning rates buy, and their costs.
  • Why initialisation matters — vanishing and exploding gradients, symmetry breaking, and the variance argument behind each scheme.
  • Regularisation theory — the bias-variance tradeoff, and how L1, L2, dropout, and batch normalisation each act on it.
  • How gradient checking works — the finite-difference approximation and why it is a debugging tool rather than a training one.
  • The hyperparameter tuning landscape — why some hyperparameters matter far more than others, and why random search beats grid search.
  • Recurrent neural networks — why a long-range gradient vanishes or explodes in an RNN, tied to the recurrent weight's spectral radius, and what gradient clipping does and does not fix.
  • LSTM and GRU — the constant error carousel, what the forget gate does to the cell-state derivative, and why a gate is what buys a longer memory.
  • Language modeling and sampling — what a language model estimates, why perplexity is the loss exponentiated, and what temperature is doing to the distribution it samples from.
  • Word embeddings — how a discrete symbol becomes a vector, why linear structure appears in the geometry, and what the analogy arithmetic is actually computing.
  • Word2Vec and GloVe — what the training objectives actually estimate, why two such different algorithms recover the same geometry, and what negative sampling buys.
  • Debiasing word embeddings — why bias lives in the geometry, what projecting a direction out removes, and why the structure survives the projection.
  • Contextual embeddings — what changes when a word's vector depends on its sentence, why pooling choices change the answer, and what the retrieval benchmarks measure.
  • Modern sequence models — what attention buys and what it costs, and how state-space models and gated linear recurrences recover linear-time inference.