Skip to content

Reference

Reference describes the machinery. A reader arrives knowing what they want and needing a precise detail: an equation, a shape, a default, a signature.

Reference is consulted, not read. Its organisation should mirror the thing it describes rather than any path through it.

What belongs here

  • Equations, stated rather than derived. The update rule as it is applied. The derivation that produces it is an explanation.
  • Tables: shapes and dimensions, default values, comparisons between methods on stated criteria.
  • API documentation, generated from the docstrings so that the published reference and the installed package cannot disagree.
  • Anything a reader would return to repeatedly for one line at a time.

Accuracy is the only virtue that matters here. A reference page is austere on purpose: it makes no argument, teaches nothing, and takes no position on what a reader should do.

What does not belong here

  • Why a default is what it is. That is an explanation. Reference states that beta2 defaults to 0.999; the reason that value works belongs elsewhere.
  • A recommendation. "Use Adam unless you have a reason not to" is a how-to or an explanation. Reference lists what each optimizer does and lets the reader choose.
  • A worked example that teaches. A short illustrative snippet is fine; a guided sequence is a tutorial.
  • Hand-written API documentation. The API pages are generated. A hand-copied signature is the first thing in a repository to go stale, and the docstring is already required by the house style.

Contents

  • Network shapes and dimensions — the forward and backward dimension tables from the L-layer derivation, and its dimensional-analysis verifications.
  • Recurrent neural networks — the RNN recurrence and its forward/backward shapes, with the backpropagation-through-time gradient equations in lookup form.
  • LSTM and GRU — the gate equations, cell-state recurrence, and the minGRU/minLSTM linear recurrence forms, with the parameter shapes each one expects.
  • Language modeling and sampling — the per-timestep loss and gradient, the sampling procedure, and what temperature does to the distribution.
  • Word embeddings — the embedding matrix and its lookup identity, cosine similarity, the analogy arithmetic, and nearest-neighbour ranking.
  • Word2Vec and GloVe — the SGNS / CBOW objectives, the GloVe weighted-least-squares form and its loss, and the hyperparameters each method exposes.
  • Debiasing word embeddings — the gender-direction estimator, the neutralize and equalize equations, the WEAT statistic with its effect size and permutation test, and the fixture tolerances.
  • Contextual embeddings — the type-versus-token distinction, the encoder / pooling / instruction / normalisation contract, the retrieval regimes, and the method-family comparison with its benchmark taxonomy.
  • Modern Sequence Models — attention, state-space models, and the gated linear recurrences that compete with them, with their complexity per layer.
  • Activation functions — each function with its derivative.
  • Parameter update rule — the gradient-descent update equations for an L-layer network.
  • Optimizer update rules — the update equation and default hyperparameters for each method, plus the comparison table.
  • Learning rate schedules — each schedule's formula and parameters.
  • Initialisation schemes — the variance each scheme uses and when it applies.
  • Regularisation methods — equations, implementations, and framework examples for regularisation techniques.
  • Gradient checking — result tolerances and links to the generated implementation reference.
  • Hyperparameter search methods — the method comparison table and the documented defaults.
  • Deep learning resources — curated books, papers, courses, frameworks, and supplementary reading.

API reference

Generated from the package docstrings, one page per subpackage: