Word embeddings

Static word vectors: the embedding matrix and lookup identity, the geometry of word vectors (cosine similarity, nearest neighbors, the analogy property), and a from-scratch PCA projection for visualization.

dlhub.embeddings.word_embeddings

Word Embeddings -- Representation, Properties, and Projection

A from-scratch, teaching reference for static word embeddings: the embedding matrix and the lookup identity, the geometry of word vectors (cosine similarity, nearest neighbors, and the analogy / parallelogram property), and a from-scratch PCA projection to two dimensions for visualization.

This module treats the vectors as given -- a small, hand-planted toy matrix (see :func:make_fixture) whose every answer is exact and checkable by eye. How embeddings are learned (skip-gram, CBOW, negative sampling, GloVe, fastText) is a separate topic, and so are bias geometry and contextual embeddings. This is the foundational head: what a word vector is and what you can read off it.

Notation follows Andrew Ng's Deep Learning Specialization, Course 5, Week 2. The embedding matrix E has shape (d, V) -- d features (rows) by V vocabulary (columns). For a word with index i, its one-hot column o_w in {0, 1}^V selects a column::

e_w = E o_w = E[:, i]    in R^d

which is the module's spine: an embedding layer is a bias-free linear layer whose input is one-hot, so in practice one gathers the column by index rather than materializing the matrix product. (PyTorch stores the transpose, weight of shape (V, d), so there a word vector is a row, weight[i, :]; the idiom port documents that convention.)

References

• Mikolov, T., Yih, W., & Zweig, G. (2013). Linguistic Regularities in Continuous Space Word Representations. NAACL-HLT, 746-751. (The analogy / parallelogram property.)
• Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. https://arxiv.org/abs/1301.3781
• Pennington, J., Socher, R., & Manning, C. (2014). GloVe: Global Vectors for Word Representation. EMNLP. https://nlp.stanford.edu/pubs/glove.pdf
• Levy, O., Goldberg, Y., & Dagan, I. (2015). Improving Distributional Similarity with Lessons Learned from Word Embeddings. TACL, 3, 211-225. (3CosMul.)
• Ng, A. (2018). Deep Learning Specialization, Course 5 (Sequence Models), Week 2. https://www.coursera.org/specializations/deep-learning

Author

Deep Learning Reference Hub

License

MIT

Notes

• Deterministic PCA sign. The SVD fixes each principal direction only up to sign, so two runs could return mirror-image coordinates. :func:pca_project_2d removes the ambiguity with a fixed convention -- flip each component so its largest-magnitude entry is positive -- which makes the projection byte-for-byte reproducible. The reconstruction is invariant to the flip (a component and its coordinate column flip together), so pinning the sign changes nothing else.
• The toy matrix is rank-2 by construction. Only two of the four coordinates carry signal (royalty, gender); the other two are zero. PCA to 2-D is therefore loss-less -- it recovers the semantic plane exactly -- which is why the reconstruction and distance-preservation checks hold to 1e-10 rather than only approximately.
• Cosine zero-guard. Cosine similarity is undefined for a zero vector (its norm is zero). Rather than return a nan that would silently poison a ranking, :func:cosine_similarity returns 0.0 -- "no similarity" -- when either argument is the zero vector. The fixture has no zero vectors; the guard is defensive.

PCAProjection

Bases: NamedTuple

The result of projecting an embedding matrix to two dimensions.

Attributes:

Name	Type	Description
coords	ndarray	The 2-D coordinates, shape (V, 2) -- one row per word, X_c W[:, :2].
components	ndarray	The top-2 principal directions, shape (2, d), sign-fixed for determinism (each row's largest-magnitude entry is made positive).
mean	ndarray	The mean word vector removed before projecting, shape (d,).
singular_values	ndarray	All singular values of the centered data, descending, shape (min(V, d),). The first two belong to components; their squares are the variances explained by PC1 and PC2.

Source code in src/dlhub/embeddings/word_embeddings.py

class PCAProjection(NamedTuple):
    """
    The result of projecting an embedding matrix to two dimensions.

    Attributes
    ----------
    coords : np.ndarray
        The 2-D coordinates, shape ``(V, 2)`` -- one row per word, ``X_c
        W[:, :2]``.
    components : np.ndarray
        The top-2 principal directions, shape ``(2, d)``, sign-fixed for
        determinism (each row's largest-magnitude entry is made positive).
    mean : np.ndarray
        The mean word vector removed before projecting, shape ``(d,)``.
    singular_values : np.ndarray
        All singular values of the centered data, descending, shape
        ``(min(V, d),)``. The first two belong to ``components``; their squares
        are the variances explained by PC1 and PC2.
    """

    coords: np.ndarray
    components: np.ndarray
    mean: np.ndarray
    singular_values: np.ndarray

make_fixture

make_fixture() -> dict

Build the planted toy embedding matrix every test and document shares.

The matrix is hand-written, not trained: d = 4 features by V = 6 words, with only two coordinates carrying signal -- coordinate 0 is royalty, coordinate 1 is gender (+1 masculine, -1 feminine), and coordinates 2-3 are zero. Every similarity, neighbor, and analogy is therefore exact and checkable by reading the numbers, and the data lives in a 2-D plane inside R^4 so the PCA projection is loss-less.

The vocabulary is stored sorted, which fixes the column order of E and the word_to_ix mapping.

Returns:

Type	Description
dict	{"E", "vocab", "word_to_ix", "dims"} where E is the (d, V) float64 embedding matrix (one word per column), vocab is the sorted word list, word_to_ix maps each word to its column index, and dims records {"d", "V"}.

Source code in src/dlhub/embeddings/word_embeddings.py

def make_fixture() -> dict:
    """
    Build the planted toy embedding matrix every test and document shares.

    The matrix is hand-written, not trained: ``d = 4`` features by ``V = 6``
    words, with only two coordinates carrying signal -- coordinate 0 is
    *royalty*, coordinate 1 is *gender* (``+1`` masculine, ``-1`` feminine), and
    coordinates 2-3 are zero. Every similarity, neighbor, and analogy is
    therefore exact and checkable by reading the numbers, and the data lives in a
    2-D plane inside ``R^4`` so the PCA projection is loss-less.

    The vocabulary is stored sorted, which fixes the column order of ``E`` and
    the ``word_to_ix`` mapping.

    Returns
    -------
    dict
        ``{"E", "vocab", "word_to_ix", "dims"}`` where ``E`` is the ``(d, V)``
        ``float64`` embedding matrix (one word per column), ``vocab`` is the
        sorted word list, ``word_to_ix`` maps each word to its column index, and
        ``dims`` records ``{"d", "V"}``.
    """
    vocab = ["king", "man", "prince", "princess", "queen", "woman"]

    # Each word's vector as (royalty, gender, 0, 0) -- the dossier's planted axes.
    vectors = {
        "king": [2.0, 1.0, 0.0, 0.0],
        "man": [0.0, 1.0, 0.0, 0.0],
        "prince": [1.0, 1.0, 0.0, 0.0],
        "princess": [1.0, -1.0, 0.0, 0.0],
        "queen": [2.0, -1.0, 0.0, 0.0],
        "woman": [0.0, -1.0, 0.0, 0.0],
    }

    # E is (d x V): stack the word vectors as rows, then transpose so each word
    # is a column -- Ng's (d x V) convention.
    E = np.array([vectors[word] for word in vocab], dtype=np.float64).T
    word_to_ix = {word: i for i, word in enumerate(vocab)}
    dims = {"d": E.shape[0], "V": E.shape[1]}
    return {"E": E, "vocab": vocab, "word_to_ix": word_to_ix, "dims": dims}

one_hot

one_hot(index: int, size: int) -> np.ndarray

Build the one-hot column vector o_w for a word index.

Parameters:

Name	Type	Description	Default
index	int	The word's position i in the vocabulary, 0 <= index < size.	required
size	int	The vocabulary size V -- the length of the returned vector.	required

Returns:

Type	Description
ndarray	A (size,) float64 vector, 1.0 at index and 0.0 elsewhere.

Raises:

Type	Description
ValueError	If size < 1 or index is outside [0, size).

Source code in src/dlhub/embeddings/word_embeddings.py

def one_hot(index: int, size: int) -> np.ndarray:
    """
    Build the one-hot column vector ``o_w`` for a word index.

    Parameters
    ----------
    index : int
        The word's position ``i`` in the vocabulary, ``0 <= index < size``.
    size : int
        The vocabulary size ``V`` -- the length of the returned vector.

    Returns
    -------
    np.ndarray
        A ``(size,)`` ``float64`` vector, ``1.0`` at ``index`` and ``0.0``
        elsewhere.

    Raises
    ------
    ValueError
        If ``size < 1`` or ``index`` is outside ``[0, size)``.
    """
    if size < 1:
        raise ValueError(f"size must be >= 1; got {size}")
    if not 0 <= index < size:
        raise ValueError(f"index must be in [0, {size}); got {index}")

    o = np.zeros(size, dtype=np.float64)
    o[index] = 1.0
    return o

lookup

lookup(E: ndarray, index: int) -> np.ndarray

Select a word's embedding by index -- the gather form of E o_w.

This returns the same vector as E @ one_hot(index, V) but by indexing the column directly, which is what every real embedding layer does: the one-hot matrix product is the definition, the column gather is the implementation.

Parameters:

Name	Type	Description	Default
E	ndarray	The embedding matrix, shape (d, V).	required
index	int	The word's column index i.	required

Returns:

Type	Description
ndarray	The embedding e_w = E[:, index], shape (d,).

Source code in src/dlhub/embeddings/word_embeddings.py

def lookup(E: np.ndarray, index: int) -> np.ndarray:
    """
    Select a word's embedding by index -- the gather form of ``E o_w``.

    This returns the same vector as ``E @ one_hot(index, V)`` but by indexing the
    column directly, which is what every real embedding layer does: the one-hot
    matrix product is the *definition*, the column gather is the *implementation*.

    Parameters
    ----------
    E : np.ndarray
        The embedding matrix, shape ``(d, V)``.
    index : int
        The word's column index ``i``.

    Returns
    -------
    np.ndarray
        The embedding ``e_w = E[:, index]``, shape ``(d,)``.
    """
    return E[:, index]

cosine_similarity

cosine_similarity(u: ndarray, v: ndarray) -> float

Cosine similarity between two vectors.

Computes (u . v) / (||u|| ||v||), the cosine of the angle between u and v. It is +1 for identical directions, 0 for orthogonal vectors, and -1 for opposite directions, independent of magnitude.

Parameters:

Name	Type	Description	Default
u	ndarray	First vector, shape (d,).	required
v	ndarray	Second vector, shape (d,).	required

Returns:

Type	Description
float	The cosine similarity, or 0.0 if either vector is the zero vector (for which the cosine is undefined -- see the module Notes).

Source code in src/dlhub/embeddings/word_embeddings.py

def cosine_similarity(u: np.ndarray, v: np.ndarray) -> float:
    """
    Cosine similarity between two vectors.

    Computes ``(u . v) / (||u|| ||v||)``, the cosine of the angle between ``u``
    and ``v``. It is ``+1`` for identical directions, ``0`` for orthogonal
    vectors, and ``-1`` for opposite directions, independent of magnitude.

    Parameters
    ----------
    u : np.ndarray
        First vector, shape ``(d,)``.
    v : np.ndarray
        Second vector, shape ``(d,)``.

    Returns
    -------
    float
        The cosine similarity, or ``0.0`` if either vector is the zero vector
        (for which the cosine is undefined -- see the module Notes).
    """
    norm_u = np.linalg.norm(u)
    norm_v = np.linalg.norm(v)
    if norm_u == 0.0 or norm_v == 0.0:
        return 0.0
    return float(np.dot(u, v) / (norm_u * norm_v))

nearest_neighbors

nearest_neighbors(E: ndarray, query: int, k: int = 1) -> list[tuple[int, float]]

Rank the other words by cosine similarity to a query word.

Scores every column against E[:, query] with :func:cosine_similarity, sorts by similarity descending, and returns the top k. The query itself is excluded, and ties are broken by ascending index, so the ranking is deterministic.

Parameters:

Name	Type	Description	Default
E	ndarray	The embedding matrix, shape (d, V).	required
query	int	Column index of the query word.	required
k	int	How many neighbors to return. Clipped to V - 1 (every word except the query). Defaults to 1 -- the single nearest neighbor.	1

Returns:

Type	Description
list[tuple[int, float]]	(index, cosine) pairs, most similar first, length min(k, V - 1).

Raises:

Type	Description
ValueError	If query is outside [0, V) or k < 1.

Source code in src/dlhub/embeddings/word_embeddings.py

def nearest_neighbors(E: np.ndarray, query: int, k: int = 1) -> list[tuple[int, float]]:
    """
    Rank the other words by cosine similarity to a query word.

    Scores every column against ``E[:, query]`` with :func:`cosine_similarity`,
    sorts by similarity descending, and returns the top ``k``. The query itself
    is excluded, and ties are broken by ascending index, so the ranking is
    deterministic.

    Parameters
    ----------
    E : np.ndarray
        The embedding matrix, shape ``(d, V)``.
    query : int
        Column index of the query word.
    k : int, optional
        How many neighbors to return. Clipped to ``V - 1`` (every word except the
        query). Defaults to ``1`` -- the single nearest neighbor.

    Returns
    -------
    list[tuple[int, float]]
        ``(index, cosine)`` pairs, most similar first, length ``min(k, V - 1)``.

    Raises
    ------
    ValueError
        If ``query`` is outside ``[0, V)`` or ``k < 1``.
    """
    V = E.shape[1]
    if not 0 <= query < V:
        raise ValueError(f"query must be in [0, {V}); got {query}")
    if k < 1:
        raise ValueError(f"k must be >= 1; got {k}")

    q = E[:, query]
    scored = [(j, cosine_similarity(q, E[:, j])) for j in range(V) if j != query]

    # Descending similarity; ascending index breaks ties -> deterministic order.
    scored.sort(key=lambda pair: (-pair[1], pair[0]))
    return scored[:k]

analogy

analogy(E: ndarray, a: int, b: int, c: int) -> int

Solve "a is to b as c is to ?" by 3CosAdd.

Forms the target t = e_b - e_a + e_c and returns the vocabulary index whose embedding is most cosine-similar to t. The three input words are excluded from the candidates -- without this, king - man + woman returns king -- and ties are broken by ascending index.

The multiplicative variant 3CosMul (Levy, Goldberg & Dagan, 2015) is more robust on real corpora but is not implemented here; on this planted fixture the additive form already recovers the exact answer.

Parameters:

Name	Type	Description	Default
E	ndarray	The embedding matrix, shape (d, V).	required
a	int	Column index of the first word (subtracted).	required
b	int	Column index of the second word (added).	required
c	int	Column index of the third word (added).	required

Returns:

Type	Description
int	The column index of the analogy's answer.

Raises:

Type	Description
ValueError	If any of a, b, c is outside [0, V), or if excluding them leaves no candidate words.

Source code in src/dlhub/embeddings/word_embeddings.py

def analogy(E: np.ndarray, a: int, b: int, c: int) -> int:
    """
    Solve "a is to b as c is to ?" by 3CosAdd.

    Forms the target ``t = e_b - e_a + e_c`` and returns the vocabulary index
    whose embedding is most cosine-similar to ``t``. The three input words are
    excluded from the candidates -- without this, ``king - man + woman`` returns
    ``king`` -- and ties are broken by ascending index.

    The multiplicative variant 3CosMul (Levy, Goldberg & Dagan, 2015) is more
    robust on real corpora but is not implemented here; on this planted fixture
    the additive form already recovers the exact answer.

    Parameters
    ----------
    E : np.ndarray
        The embedding matrix, shape ``(d, V)``.
    a : int
        Column index of the first word (subtracted).
    b : int
        Column index of the second word (added).
    c : int
        Column index of the third word (added).

    Returns
    -------
    int
        The column index of the analogy's answer.

    Raises
    ------
    ValueError
        If any of ``a``, ``b``, ``c`` is outside ``[0, V)``, or if excluding them
        leaves no candidate words.
    """
    V = E.shape[1]
    for name, idx in (("a", a), ("b", b), ("c", c)):
        if not 0 <= idx < V:
            raise ValueError(f"{name} must be in [0, {V}); got {idx}")

    # The parallelogram target: start at c, add the b - a displacement.
    target = lookup(E, b) - lookup(E, a) + lookup(E, c)

    excluded = {a, b, c}
    best_index = -1
    best_score = -np.inf
    for j in range(V):
        if j in excluded:
            continue
        score = cosine_similarity(E[:, j], target)
        # Strict ">" keeps the first (lowest) index when two scores tie.
        if score > best_score:
            best_score = score
            best_index = j

    if best_index == -1:
        raise ValueError("no candidate words remain after excluding a, b, c")
    return best_index

pca_project_2d

pca_project_2d(E: ndarray) -> PCAProjection

Project word vectors to two dimensions with PCA, from scratch via SVD.

Lays the words out as rows X = E.T (V x d), removes the mean word vector to get X_c, and takes the SVD X_c = U S W^T. The first two rows of W^T are the directions of greatest variance; the 2-D coordinates are the centered data projected onto them, X_c W[:, :2].

Parameters:

Name	Type	Description	Default
E	ndarray	The embedding matrix, shape (d, V).	required

Returns:

Type	Description
PCAProjection	The coords (V, 2) together with the components, mean, and singular_values needed to reconstruct the data and read off variance.

Raises:

Type	Description
ValueError	If E is not a 2-D array.

Notes

The SVD fixes each principal direction only up to sign. To make the output reproducible, each component is flipped so its largest-magnitude entry is positive; the corresponding coordinate column flips with it, so the reconstruction is unchanged.

Source code in src/dlhub/embeddings/word_embeddings.py

def pca_project_2d(E: np.ndarray) -> PCAProjection:
    """
    Project word vectors to two dimensions with PCA, from scratch via SVD.

    Lays the words out as rows ``X = E.T`` (``V x d``), removes the mean word
    vector to get ``X_c``, and takes the SVD ``X_c = U S W^T``. The first two rows
    of ``W^T`` are the directions of greatest variance; the 2-D coordinates are
    the centered data projected onto them, ``X_c W[:, :2]``.

    Parameters
    ----------
    E : np.ndarray
        The embedding matrix, shape ``(d, V)``.

    Returns
    -------
    PCAProjection
        The ``coords`` ``(V, 2)`` together with the ``components``, ``mean``, and
        ``singular_values`` needed to reconstruct the data and read off variance.

    Raises
    ------
    ValueError
        If ``E`` is not a 2-D array.

    Notes
    -----
    The SVD fixes each principal direction only up to sign. To make the output
    reproducible, each component is flipped so its largest-magnitude entry is
    positive; the corresponding coordinate column flips with it, so the
    reconstruction is unchanged.
    """
    if E.ndim != 2:
        raise ValueError(f"E must be 2-D (d, V); got ndim {E.ndim}")

    X = E.T  # one word per row, shape (V, d)
    mean = X.mean(axis=0)  # the mean word vector, shape (d,)
    X_c = X - mean  # center the rows on the mean word vector

    # Economy SVD: U is (V, r), S is (r,), Wt is (r, d) with r = min(V, d). We
    # project via the right singular vectors (the components) below, so the left
    # vectors U -- the word scores -- are not needed here.
    U, S, Wt = np.linalg.svd(X_c, full_matrices=False)

    components = Wt[:2].copy()  # top-2 principal directions, shape (2, d)
    # Sign convention for determinism: make each component's largest-magnitude
    # entry positive.
    for row in range(components.shape[0]):
        pivot = np.argmax(np.abs(components[row]))
        if components[row, pivot] < 0:
            components[row] = -components[row]

    coords = X_c @ components.T  # project onto the two components, shape (V, 2)
    return PCAProjection(
        coords=coords, components=components, mean=mean, singular_values=S
    )

main

main() -> None

Run the fixture end to end and print its known answers.

Source code in src/dlhub/embeddings/word_embeddings.py

def main() -> None:
    """Run the fixture end to end and print its known answers."""
    fixture = make_fixture()
    E, vocab, word_to_ix = fixture["E"], fixture["vocab"], fixture["word_to_ix"]
    V = E.shape[1]

    print(f"E has shape {E.shape} (d x V); vocabulary: {vocab}")

    identities = all(np.array_equal(E @ one_hot(i, V), lookup(E, i)) for i in range(V))
    print(f"E @ one_hot(i) == lookup(E, i) for every i: {identities}")

    king, queen = word_to_ix["king"], word_to_ix["queen"]
    man, woman = word_to_ix["man"], word_to_ix["woman"]
    print(f"cos(king, queen) = {cosine_similarity(E[:, king], E[:, queen]):.4f}")
    print(f"cos(man, woman)  = {cosine_similarity(E[:, man], E[:, woman]):.4f}")

    print("nearest neighbors (top-1, self excluded):")
    for i, word in enumerate(vocab):
        j, score = nearest_neighbors(E, i, k=1)[0]
        print(f"  {word:8s} -> {vocab[j]:8s} (cos = {score:.4f})")

    answer = analogy(E, a=man, b=woman, c=king)
    print(f"man : woman :: king : {vocab[answer]}")

    projection = pca_project_2d(E)
    print(
        f"PCA coords shape {projection.coords.shape}; "
        f"singular values {np.round(projection.singular_values, 4)}"
    )

Learned word vectors: skip-gram with negative sampling (SGNS), CBOW, the smoothed-unigram negative sampler, and GloVe's weighted least-squares objective, each loss paired with its exact gradient, trained from scratch on a toy corpus.

dlhub.embeddings.word2vec_and_glove

Word2Vec and GloVe -- Learning Word Vectors from a Corpus

A from-scratch, teaching reference for how static word embeddings are learned. The sibling :mod:dlhub.embeddings.word_embeddings treats the vectors as given and reads geometry off them; this module produces those vectors by optimising an objective over a toy corpus. It implements three objectives, each with its loss and its exact gradient:

• Skip-gram with negative sampling (SGNS) -- predict context from centre, with K negatives drawn from the smoothed unigram sampler.
• CBOW -- the same tables and sampler with the roles reversed: average the context, predict the centre.
• GloVe -- a weighted least-squares fit to the log co-occurrence counts.

The gradients are the payload. Each is written term by term in the notation the reference card uses, and :mod:tests.test_word2vec_and_glove checks every one against a finite-difference approximation, because a flipped sign in a teaching module teaches confidently.

Notation follows the dossier, which departs from the head topic deliberately. The head's embedding matrix E is (d, V) (a word is a column). Here the two learned tables are stored row-wise, (V, d), because every objective indexes a word's vector by row and takes inner products -- gathering columns from (d, V) would be all transpose noise:

• W -- the centre (input) table, shape (V, d); W[w] is centre word w's vector.
• C -- the context (output) table, shape (V, d); C[c] is context word c's vector.

The head's matrix is what this topic produces: E is the learned W (the head's (d, V) is this module's W transposed).

References

• Mikolov, T., Chen, K., Corrado, G., & Dean, J. (2013). Efficient Estimation of Word Representations in Vector Space. https://arxiv.org/abs/1301.3781
• Mikolov, T., Sutskever, I., Chen, K., Corrado, G., & Dean, J. (2013). Distributed Representations of Words and Phrases and their Compositionality. NeurIPS. https://arxiv.org/abs/1310.4546 (Negative sampling, unigram^0.75.)
• Pennington, J., Socher, R., & Manning, C. (2014). GloVe: Global Vectors for Word Representation. EMNLP. https://nlp.stanford.edu/pubs/glove.pdf
• Levy, O., & Goldberg, Y. (2014). Neural Word Embedding as Implicit Matrix Factorization. NeurIPS. (SGNS factorises shifted PMI.)
• Rong, X. (2014). word2vec Parameter Learning Explained. https://arxiv.org/abs/1411.2738 (The step-by-step gradient derivation.)

Author

Deep Learning Reference Hub

License

MIT

Notes

• Row-major tables. W and C are (V, d) -- a word's vector is a row -- which is the opposite of the head's (d, V) E. The convention is chosen so the objectives read as W[w] @ C[c] rather than E[:, w] @ E[:, c]; the relationship is E == W.T after training.
• Stable sigmoid. :func:sigmoid multiplies through by exp(-|z|) so no intermediate exp overflows: for z >= 0 it is 1 / (1 + e^-z) and for z < 0 it is e^z / (1 + e^z), which are equal to the textbook form but never evaluate exp of a large positive number.
• Stable log-sigmoid in the loss. The losses need log sigma(z). Evaluating log(sigmoid(z)) underflows to -inf once sigmoid(z) rounds to 0, so the losses use log sigma(z) = -softplus(-z) with softplus(x) = logaddexp(0, x) -- the same value, computed without the underflow. This is why the loss code calls :func:numpy.logaddexp rather than spelling the formula with log and sigmoid directly.
• Windows do not cross sentence boundaries. The corpus is a list of sentences; :func:training_pairs runs on one sentence at a time, so no context window straddles two sentences. On the planted fixture every sentence is drawn from a single semantic family, so this is exactly what makes the co-occurrence counts within-family and zero across families.
• One generator drives the run. :func:train_sgns seeds a single numpy.random.default_rng(seed) and draws from it in a fixed order -- W then C for initialisation, then the per-pair negatives in pair order -- so a run is reproducible to the bit. The negatives for each pair are drawn once and reused every epoch, which fixes the objective and makes the training loss a clean descent curve (classic word2vec resamples every epoch; a teaching module that asserts a monotone loss on a toy corpus fixes the sample instead).
• GloVe zero-count guard. log X_ij is defined only for X_ij > 0. :func:glove_loss and :func:glove_gradients sum over observed entries only; a zero count contributes no term rather than log 0. GloVe's final vector is the sum of the two tables, W[i] + Wtilde[i].

make_fixture

make_fixture() -> dict

Build the planted toy corpus every test and document shares.

The corpus is a literal list of 12 sentences x 3 tokens = 36 tokens, three sentences from each of four semantic families (pets, food, actions, places) of four words each. It is hand-written, never downloaded, so every frequency and co-occurrence count is known by construction and the sampler's distribution is closed-form.

Returns:

Type	Description
dict	{"corpus", "vocab", "word_to_ix", "sentences", "families", "dims"} where corpus is the list of 12 word-lists, vocab is the sorted vocabulary (V = 16), word_to_ix maps each word to its index, sentences is corpus mapped to indices, families is the word-family grouping, and dims records {"V"}.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def make_fixture() -> dict:
    """
    Build the planted toy corpus every test and document shares.

    The corpus is a literal list of **12 sentences x 3 tokens = 36 tokens**,
    three sentences from each of four semantic families (``pets``, ``food``,
    ``actions``, ``places``) of four words each. It is hand-written, never
    downloaded, so every frequency and co-occurrence count is known by
    construction and the sampler's distribution is closed-form.

    Returns
    -------
    dict
        ``{"corpus", "vocab", "word_to_ix", "sentences", "families", "dims"}``
        where ``corpus`` is the list of 12 word-lists, ``vocab`` is the sorted
        vocabulary (``V = 16``), ``word_to_ix`` maps each word to its index,
        ``sentences`` is ``corpus`` mapped to indices, ``families`` is the
        word-family grouping, and ``dims`` records ``{"V"}``.
    """
    corpus = [
        sentence
        for words in _FAMILIES.values()
        for sentence in _family_sentences(words)
    ]
    vocab, word_to_ix = build_vocabulary(corpus)
    sentences = [[word_to_ix[word] for word in sentence] for sentence in corpus]
    return {
        "corpus": corpus,
        "vocab": vocab,
        "word_to_ix": word_to_ix,
        "sentences": sentences,
        "families": {name: list(words) for name, words in _FAMILIES.items()},
        "dims": {"V": len(vocab)},
    }

build_vocabulary

build_vocabulary(corpus: Sequence[Sequence[str]]) -> tuple[list[str], dict[str, int]]

Build the vocabulary and the word -> index map from a corpus.

The vocabulary is the sorted list of unique tokens, so the index of a word is fixed by the corpus alone and does not depend on the order words were first seen. Sorting is what makes word_to_ix reproducible across runs.

Parameters:

Name	Type	Description	Default
corpus	Sequence[Sequence[str]]	The corpus as a sequence of sentences, each a sequence of word strings.	required

Returns:

Type	Description
tuple[list[str], dict[str, int]]	(vocab, word_to_ix) -- the sorted vocabulary and the map from each word to its position in it.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def build_vocabulary(
    corpus: Sequence[Sequence[str]],
) -> tuple[list[str], dict[str, int]]:
    """
    Build the vocabulary and the ``word -> index`` map from a corpus.

    The vocabulary is the sorted list of unique tokens, so the index of a word is
    fixed by the corpus alone and does not depend on the order words were first
    seen. Sorting is what makes ``word_to_ix`` reproducible across runs.

    Parameters
    ----------
    corpus : Sequence[Sequence[str]]
        The corpus as a sequence of sentences, each a sequence of word strings.

    Returns
    -------
    tuple[list[str], dict[str, int]]
        ``(vocab, word_to_ix)`` -- the sorted vocabulary and the map from each
        word to its position in it.
    """
    tokens = {word for sentence in corpus for word in sentence}
    vocab = sorted(tokens)
    word_to_ix = {word: i for i, word in enumerate(vocab)}
    return vocab, word_to_ix

training_pairs

training_pairs(sentence: Sequence[int], window: int = 2) -> list[tuple[int, int]]

Extract (centre, context) index pairs from one sentence.

For every position i in the sentence, the centre token t_i is paired with each context token t_{i+j} for offset j in {-window, ..., -1, 1, ..., window} that lands inside the sentence. Pairs are emitted by centre position, then by offset ascending, which fixes the order in which stochastic descent sees them. The window is applied to a single sentence so it never crosses a sentence boundary.

Parameters:

Name	Type	Description	Default
sentence	Sequence[int]	One sentence as a sequence of token indices.	required
window	int	The half-width m of the context window.	2

Returns:

Type	Description
list[tuple[int, int]]	The (centre, context) index pairs in deterministic order.

Raises:

Type	Description
ValueError	If window < 1.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def training_pairs(sentence: Sequence[int], window: int = 2) -> list[tuple[int, int]]:
    """
    Extract ``(centre, context)`` index pairs from one sentence.

    For every position ``i`` in the sentence, the centre token ``t_i`` is paired
    with each context token ``t_{i+j}`` for offset ``j`` in
    ``{-window, ..., -1, 1, ..., window}`` that lands inside the sentence. Pairs
    are emitted **by centre position, then by offset ascending**, which fixes the
    order in which stochastic descent sees them. The window is applied to a
    single sentence so it never crosses a sentence boundary.

    Parameters
    ----------
    sentence : Sequence[int]
        One sentence as a sequence of token indices.
    window : int, default=2
        The half-width ``m`` of the context window.

    Returns
    -------
    list[tuple[int, int]]
        The ``(centre, context)`` index pairs in deterministic order.

    Raises
    ------
    ValueError
        If ``window < 1``.
    """
    if window < 1:
        raise ValueError(f"window must be >= 1; got {window}")

    length = len(sentence)
    pairs: list[tuple[int, int]] = []
    for i in range(length):
        for j in range(-window, window + 1):
            if j == 0:
                continue
            k = i + j
            if 0 <= k < length:
                pairs.append((sentence[i], sentence[k]))
    return pairs

cooccurrence_matrix

cooccurrence_matrix(sentences: Sequence[Sequence[int]], vocab_size: int, window: int = 2) -> np.ndarray

Count co-occurrences into the (V, V) matrix GloVe fits.

X[i, j] is the number of times word j falls inside word i's context window, summed over the corpus. It reuses :func:training_pairs, so the window rule is identical to the one SGNS and CBOW see, and because the window stays inside a sentence the counts never cross a semantic family.

Parameters:

Name	Type	Description	Default
sentences	Sequence[Sequence[int]]	The corpus as sentences of token indices.	required
vocab_size	int	The vocabulary size V -- the matrix is (V, V).	required
window	int	The half-width of the context window.	2

Returns:

Type	Description
ndarray	The (V, V) integer co-occurrence matrix X.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def cooccurrence_matrix(
    sentences: Sequence[Sequence[int]], vocab_size: int, window: int = 2
) -> np.ndarray:
    """
    Count co-occurrences into the ``(V, V)`` matrix GloVe fits.

    ``X[i, j]`` is the number of times word ``j`` falls inside word ``i``'s
    context window, summed over the corpus. It reuses :func:`training_pairs`, so
    the window rule is identical to the one SGNS and CBOW see, and because the
    window stays inside a sentence the counts never cross a semantic family.

    Parameters
    ----------
    sentences : Sequence[Sequence[int]]
        The corpus as sentences of token indices.
    vocab_size : int
        The vocabulary size ``V`` -- the matrix is ``(V, V)``.
    window : int, default=2
        The half-width of the context window.

    Returns
    -------
    np.ndarray
        The ``(V, V)`` integer co-occurrence matrix ``X``.
    """
    X = np.zeros((vocab_size, vocab_size), dtype=np.int64)
    for sentence in sentences:
        for centre, context in training_pairs(sentence, window):
            X[centre, context] += 1
    return X

unigram_probabilities

unigram_probabilities(sentences: Sequence[Sequence[int]], vocab_size: int, exponent: float = 0.75) -> np.ndarray

Build the smoothed unigram distribution the negative sampler draws from.

Each word's probability is its corpus frequency raised to exponent and normalised::

P(w) = f(w)^exponent / sum_w' f(w')^exponent

The default exponent = 3/4 is the Word2Vec choice. Raising a sub-unit power pulls the distribution toward uniform, so rare words are drawn as negatives more often than their raw frequency -- which is the point: negatives drawn in proportion to raw frequency would be dominated by stop-words and carry little signal.

Parameters:

Name	Type	Description	Default
sentences	Sequence[Sequence[int]]	The corpus as sentences of token indices.	required
vocab_size	int	The vocabulary size V -- the length of the returned vector.	required
exponent	float	The smoothing exponent applied to each frequency.	0.75

Returns:

Type	Description
ndarray	A (V,) float64 probability vector that sums to 1.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def unigram_probabilities(
    sentences: Sequence[Sequence[int]], vocab_size: int, exponent: float = 0.75
) -> np.ndarray:
    """
    Build the smoothed unigram distribution the negative sampler draws from.

    Each word's probability is its corpus frequency raised to ``exponent`` and
    normalised::

        P(w) = f(w)^exponent / sum_w' f(w')^exponent

    The default ``exponent = 3/4`` is the Word2Vec choice. Raising a sub-unit
    power pulls the distribution toward uniform, so **rare words are drawn as
    negatives more often than their raw frequency** -- which is the point:
    negatives drawn in proportion to raw frequency would be dominated by
    stop-words and carry little signal.

    Parameters
    ----------
    sentences : Sequence[Sequence[int]]
        The corpus as sentences of token indices.
    vocab_size : int
        The vocabulary size ``V`` -- the length of the returned vector.
    exponent : float, default=0.75
        The smoothing exponent applied to each frequency.

    Returns
    -------
    np.ndarray
        A ``(V,)`` ``float64`` probability vector that sums to 1.
    """
    counts = np.zeros(vocab_size, dtype=np.float64)
    for sentence in sentences:
        for token in sentence:
            counts[token] += 1.0

    smoothed = counts**exponent
    return smoothed / smoothed.sum()

sample_negatives

sample_negatives(rng: Generator, probabilities: ndarray, k: int) -> np.ndarray

Draw k negative word indices i.i.d. from the unigram sampler.

This is rng.choice(V, size=k, p=probabilities) -- a thin wrapper that names the operation and keeps every draw on the single generator the training run threads through, so the whole run stays reproducible.

Parameters:

Name	Type	Description	Default
rng	Generator	The run's generator; advancing it here is what keeps the draw order deterministic.	required
probabilities	ndarray	The (V,) sampling distribution from :func:unigram_probabilities.	required
k	int	How many negatives to draw.	required

Returns:

Type	Description
ndarray	A (k,) array of word indices.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def sample_negatives(
    rng: np.random.Generator, probabilities: np.ndarray, k: int
) -> np.ndarray:
    """
    Draw ``k`` negative word indices i.i.d. from the unigram sampler.

    This is ``rng.choice(V, size=k, p=probabilities)`` -- a thin wrapper that
    names the operation and keeps every draw on the single generator the training
    run threads through, so the whole run stays reproducible.

    Parameters
    ----------
    rng : np.random.Generator
        The run's generator; advancing it here is what keeps the draw order
        deterministic.
    probabilities : np.ndarray
        The ``(V,)`` sampling distribution from :func:`unigram_probabilities`.
    k : int
        How many negatives to draw.

    Returns
    -------
    np.ndarray
        A ``(k,)`` array of word indices.
    """
    vocab_size = probabilities.shape[0]
    return rng.choice(vocab_size, size=k, p=probabilities)

sigmoid

sigmoid(z: float | ndarray) -> float | np.ndarray

The logistic sigmoid 1 / (1 + e^-z), computed without overflow.

The textbook spelling evaluates exp(-z), which overflows for large negative z. Multiplying through by exp(-|z|) -- which never exceeds 1 -- gives the identical value from a bounded exponential: 1 / (1 + e^-z) for z >= 0 and e^z / (1 + e^z) for z < 0.

Parameters:

Name	Type	Description	Default
z	float or ndarray	The pre-activation score (a scalar inner product, or an array of them).	required

Returns:

Type	Description
float or ndarray	sigma(z) in (0, 1), a float for scalar input.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def sigmoid(z: float | np.ndarray) -> float | np.ndarray:
    """
    The logistic sigmoid ``1 / (1 + e^-z)``, computed without overflow.

    The textbook spelling evaluates ``exp(-z)``, which overflows for large
    negative ``z``. Multiplying through by ``exp(-|z|)`` -- which never exceeds
    ``1`` -- gives the identical value from a bounded exponential: ``1 / (1 +
    e^-z)`` for ``z >= 0`` and ``e^z / (1 + e^z)`` for ``z < 0``.

    Parameters
    ----------
    z : float or np.ndarray
        The pre-activation score (a scalar inner product, or an array of them).

    Returns
    -------
    float or np.ndarray
        ``sigma(z)`` in ``(0, 1)``, a ``float`` for scalar input.
    """
    z = np.asarray(z, dtype=np.float64)
    exp_neg_abs = np.exp(-np.abs(z))
    result = np.where(
        z >= 0.0, 1.0 / (1.0 + exp_neg_abs), exp_neg_abs / (1.0 + exp_neg_abs)
    )
    return float(result) if result.ndim == 0 else result

sgns_loss

sgns_loss(W: ndarray, C: ndarray, centre: int, context: int, negatives: Sequence[int]) -> float

Skip-gram negative-sampling loss for one (centre, context) pair.

With centre vector v = W[centre], positive context u = C[context] and K negatives u_k = C[n_k]::

l = -log sigma(v . u) - sum_k log sigma(-v . u_k)

The positive term rewards a high score for the true context; each negative term rewards a low score for a sampled non-context. The sigma(-z) spelling of the negative term is the numerically stable one (see the module Notes), and both terms are evaluated through :func:_log_sigmoid.

Parameters:

Name	Type	Description	Default
W	ndarray	The centre table, shape (V, d).	required
C	ndarray	The context table, shape (V, d).	required
centre	int	Index of the centre word.	required
context	int	Index of the true context word.	required
negatives	Sequence[int]	The K sampled negative indices.	required

Returns:

Type	Description
float	The per-pair loss.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def sgns_loss(
    W: np.ndarray,
    C: np.ndarray,
    centre: int,
    context: int,
    negatives: Sequence[int],
) -> float:
    """
    Skip-gram negative-sampling loss for one ``(centre, context)`` pair.

    With centre vector ``v = W[centre]``, positive context ``u = C[context]`` and
    ``K`` negatives ``u_k = C[n_k]``::

        l = -log sigma(v . u) - sum_k log sigma(-v . u_k)

    The positive term rewards a high score for the true context; each negative
    term rewards a *low* score for a sampled non-context. The ``sigma(-z)``
    spelling of the negative term is the numerically stable one (see the module
    Notes), and both terms are evaluated through :func:`_log_sigmoid`.

    Parameters
    ----------
    W : np.ndarray
        The centre table, shape ``(V, d)``.
    C : np.ndarray
        The context table, shape ``(V, d)``.
    centre : int
        Index of the centre word.
    context : int
        Index of the true context word.
    negatives : Sequence[int]
        The ``K`` sampled negative indices.

    Returns
    -------
    float
        The per-pair loss.
    """
    v = W[centre]
    u = C[context]
    positive = _log_sigmoid(v @ u)
    negative = sum(_log_sigmoid(-(v @ C[n])) for n in negatives)
    return float(-positive - negative)

sgns_gradients

sgns_gradients(W: ndarray, C: ndarray, centre: int, context: int, negatives: Sequence[int]) -> dict[str, np.ndarray]

Gradients of :func:sgns_loss with respect to both tables.

Term by term, with v = W[centre], u = C[context] and u_k = C[n_k]::

dl/dv       = (sigma(v . u) - 1) u + sum_k sigma(v . u_k) u_k
dl/du       = (sigma(v . u) - 1) v
dl/du_k     =  sigma(v . u_k) v

Only the rows a pair touches receive a gradient, so the returned tables are zero except at centre (in W) and at context and the negatives (in C). Contributions are accumulated with += one index at a time, so a negative that happens to equal the context -- or a repeated negative -- adds correctly rather than overwriting.

Parameters:

Name	Type	Description	Default
W	ndarray	The centre table, shape (V, d).	required
C	ndarray	The context table, shape (V, d).	required
centre	int	Index of the centre word.	required
context	int	Index of the true context word.	required
negatives	Sequence[int]	The K sampled negative indices.	required

Returns:

Type	Description
dict[str, ndarray]	{"W": dW, "C": dC} -- full (V, d) gradient tables, zero off the touched rows.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def sgns_gradients(
    W: np.ndarray,
    C: np.ndarray,
    centre: int,
    context: int,
    negatives: Sequence[int],
) -> dict[str, np.ndarray]:
    """
    Gradients of :func:`sgns_loss` with respect to both tables.

    Term by term, with ``v = W[centre]``, ``u = C[context]`` and
    ``u_k = C[n_k]``::

        dl/dv       = (sigma(v . u) - 1) u + sum_k sigma(v . u_k) u_k
        dl/du       = (sigma(v . u) - 1) v
        dl/du_k     =  sigma(v . u_k) v

    Only the rows a pair touches receive a gradient, so the returned tables are
    zero except at ``centre`` (in ``W``) and at ``context`` and the negatives (in
    ``C``). Contributions are accumulated with ``+=`` one index at a time, so a
    negative that happens to equal the context -- or a repeated negative -- adds
    correctly rather than overwriting.

    Parameters
    ----------
    W : np.ndarray
        The centre table, shape ``(V, d)``.
    C : np.ndarray
        The context table, shape ``(V, d)``.
    centre : int
        Index of the centre word.
    context : int
        Index of the true context word.
    negatives : Sequence[int]
        The ``K`` sampled negative indices.

    Returns
    -------
    dict[str, np.ndarray]
        ``{"W": dW, "C": dC}`` -- full ``(V, d)`` gradient tables, zero off the
        touched rows.
    """
    v = W[centre]
    u = C[context]

    dW = np.zeros_like(W)
    dC = np.zeros_like(C)

    # Positive pair: pull the centre and its true context together.
    sigma_pos = sigmoid(v @ u)
    dW[centre] += (sigma_pos - 1.0) * u
    dC[context] += (sigma_pos - 1.0) * v

    # Negatives: push the centre away from each sampled non-context.
    for n in negatives:
        u_n = C[n]
        sigma_neg = sigmoid(v @ u_n)
        dW[centre] += sigma_neg * u_n
        dC[n] += sigma_neg * v

    return {"W": dW, "C": dC}

cbow_loss

cbow_loss(W: ndarray, C: ndarray, centre: int, context_words: Sequence[int], negatives: Sequence[int]) -> float

CBOW negative-sampling loss for one (context -> centre) example.

CBOW reverses skip-gram: average the context vectors and predict the centre. With ubar = mean_j C[c_j] over the context words, centre vector v = W[centre] and negatives scored against the centre table, v_k = W[n_k]::

l = -log sigma(ubar . v) - sum_k log sigma(-ubar . v_k)

It is :func:sgns_loss with ubar in the centre's place and the centre/negative words looked up in W -- the same objective, the scoring swapped.

Parameters:

Name	Type	Description	Default
W	ndarray	The centre table, shape (V, d).	required
C	ndarray	The context table, shape (V, d).	required
centre	int	Index of the centre word being predicted.	required
context_words	Sequence[int]	The context word indices whose vectors are averaged.	required
negatives	Sequence[int]	The K sampled negative centre indices.	required

Returns:

Type	Description
float	The per-example loss.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def cbow_loss(
    W: np.ndarray,
    C: np.ndarray,
    centre: int,
    context_words: Sequence[int],
    negatives: Sequence[int],
) -> float:
    """
    CBOW negative-sampling loss for one ``(context -> centre)`` example.

    CBOW reverses skip-gram: average the context vectors and predict the centre.
    With ``ubar = mean_j C[c_j]`` over the context words, centre vector
    ``v = W[centre]`` and negatives scored against the **centre** table,
    ``v_k = W[n_k]``::

        l = -log sigma(ubar . v) - sum_k log sigma(-ubar . v_k)

    It is :func:`sgns_loss` with ``ubar`` in the centre's place and the
    centre/negative words looked up in ``W`` -- the same objective, the scoring
    swapped.

    Parameters
    ----------
    W : np.ndarray
        The centre table, shape ``(V, d)``.
    C : np.ndarray
        The context table, shape ``(V, d)``.
    centre : int
        Index of the centre word being predicted.
    context_words : Sequence[int]
        The context word indices whose vectors are averaged.
    negatives : Sequence[int]
        The ``K`` sampled negative centre indices.

    Returns
    -------
    float
        The per-example loss.
    """
    u_bar = C[list(context_words)].mean(axis=0)
    v = W[centre]
    positive = _log_sigmoid(u_bar @ v)
    negative = sum(_log_sigmoid(-(u_bar @ W[n])) for n in negatives)
    return float(-positive - negative)

cbow_gradients

cbow_gradients(W: ndarray, C: ndarray, centre: int, context_words: Sequence[int], negatives: Sequence[int]) -> dict[str, np.ndarray]

Gradients of :func:cbow_loss with respect to both tables.

With ubar = mean_j C[c_j] and M = len(context_words)::

dl/dubar  = (sigma(ubar . v) - 1) v + sum_k sigma(ubar . v_k) v_k
dl/dW[w]  = (sigma(ubar . v) - 1) ubar
dl/dW[n_k]=  sigma(ubar . v_k) ubar
dl/dC[c_j]= (1 / M) dl/dubar     for each context word c_j

The averaging splits the gradient on ubar evenly back to each context row (1 / M); a word that appears twice in the window collects the share twice, which the one-index-at-a-time += handles.

Parameters:

Name	Type	Description	Default
W	ndarray	The centre table, shape (V, d).	required
C	ndarray	The context table, shape (V, d).	required
centre	int	Index of the centre word being predicted.	required
context_words	Sequence[int]	The context word indices whose vectors are averaged.	required
negatives	Sequence[int]	The K sampled negative centre indices.	required

Returns:

Type	Description
dict[str, ndarray]	{"W": dW, "C": dC} -- full (V, d) gradient tables.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def cbow_gradients(
    W: np.ndarray,
    C: np.ndarray,
    centre: int,
    context_words: Sequence[int],
    negatives: Sequence[int],
) -> dict[str, np.ndarray]:
    """
    Gradients of :func:`cbow_loss` with respect to both tables.

    With ``ubar = mean_j C[c_j]`` and ``M = len(context_words)``::

        dl/dubar  = (sigma(ubar . v) - 1) v + sum_k sigma(ubar . v_k) v_k
        dl/dW[w]  = (sigma(ubar . v) - 1) ubar
        dl/dW[n_k]=  sigma(ubar . v_k) ubar
        dl/dC[c_j]= (1 / M) dl/dubar     for each context word c_j

    The averaging splits the gradient on ``ubar`` evenly back to each context
    row (``1 / M``); a word that appears twice in the window collects the share
    twice, which the one-index-at-a-time ``+=`` handles.

    Parameters
    ----------
    W : np.ndarray
        The centre table, shape ``(V, d)``.
    C : np.ndarray
        The context table, shape ``(V, d)``.
    centre : int
        Index of the centre word being predicted.
    context_words : Sequence[int]
        The context word indices whose vectors are averaged.
    negatives : Sequence[int]
        The ``K`` sampled negative centre indices.

    Returns
    -------
    dict[str, np.ndarray]
        ``{"W": dW, "C": dC}`` -- full ``(V, d)`` gradient tables.
    """
    context_list = list(context_words)
    count = len(context_list)
    u_bar = C[context_list].mean(axis=0)
    v = W[centre]

    dW = np.zeros_like(W)
    dC = np.zeros_like(C)

    # Gradient flowing into the averaged context vector ubar.
    sigma_pos = sigmoid(u_bar @ v)
    grad_u_bar = (sigma_pos - 1.0) * v
    dW[centre] += (sigma_pos - 1.0) * u_bar

    for n in negatives:
        v_n = W[n]
        sigma_neg = sigmoid(u_bar @ v_n)
        grad_u_bar = grad_u_bar + sigma_neg * v_n
        dW[n] += sigma_neg * u_bar

    # Split the gradient on ubar evenly across the context rows that formed it.
    for c in context_list:
        dC[c] += grad_u_bar / count

    return {"W": dW, "C": dC}

glove_weight

glove_weight(x: ndarray | float, x_max: float = 100.0, alpha: float = 0.75) -> np.ndarray | float

GloVe's co-occurrence weighting f(x) = min(1, (x / x_max)^alpha).

The weight grows with the count up to x_max and is capped at 1 thereafter, so very frequent pairs do not dominate the fit while rare pairs are not ignored. It is only ever evaluated at observed counts x > 0.

Parameters:

Name	Type	Description	Default
x	ndarray or float	A co-occurrence count (or array of them).	required
x_max	float	The count at which the weight saturates to 1.	100.0
alpha	float	The exponent controlling the ramp below x_max.	0.75

Returns:

Type	Description
ndarray or float	The weight f(x) in (0, 1].

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def glove_weight(
    x: np.ndarray | float, x_max: float = 100.0, alpha: float = 0.75
) -> np.ndarray | float:
    """
    GloVe's co-occurrence weighting ``f(x) = min(1, (x / x_max)^alpha)``.

    The weight grows with the count up to ``x_max`` and is capped at ``1``
    thereafter, so very frequent pairs do not dominate the fit while rare pairs
    are not ignored. It is only ever evaluated at observed counts ``x > 0``.

    Parameters
    ----------
    x : np.ndarray or float
        A co-occurrence count (or array of them).
    x_max : float, default=100.0
        The count at which the weight saturates to ``1``.
    alpha : float, default=0.75
        The exponent controlling the ramp below ``x_max``.

    Returns
    -------
    np.ndarray or float
        The weight ``f(x)`` in ``(0, 1]``.
    """
    return np.minimum(1.0, (np.asarray(x, dtype=np.float64) / x_max) ** alpha)

glove_loss

glove_loss(W: ndarray, W_tilde: ndarray, b: ndarray, b_tilde: ndarray, X: ndarray, x_max: float = 100.0, alpha: float = 0.75) -> float

GloVe weighted least-squares loss over the observed co-occurrences.

For every X[i, j] > 0 the residual is r_ij = W[i] . W_tilde[j] + b[i] + b_tilde[j] - log X[i, j] and::

J = sum_{i,j: X_ij > 0} f(X_ij) * r_ij^2

Zero counts contribute no term -- GloVe fits log X only where X is observed, which is also what keeps log 0 out of the sum.

Parameters:

Name	Type	Description	Default
W	ndarray	The word table, shape (V, d).	required
W_tilde	ndarray	The context-word table, shape (V, d).	required
b	ndarray	The word biases, shape (V,).	required
b_tilde	ndarray	The context-word biases, shape (V,).	required
X	ndarray	The (V, V) co-occurrence matrix.	required
x_max	float	The weighting cap passed to :func:glove_weight.	100.0
alpha	float	The weighting exponent passed to :func:glove_weight.	0.75

Returns:

Type	Description
float	The total loss J.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def glove_loss(
    W: np.ndarray,
    W_tilde: np.ndarray,
    b: np.ndarray,
    b_tilde: np.ndarray,
    X: np.ndarray,
    x_max: float = 100.0,
    alpha: float = 0.75,
) -> float:
    """
    GloVe weighted least-squares loss over the observed co-occurrences.

    For every ``X[i, j] > 0`` the residual is
    ``r_ij = W[i] . W_tilde[j] + b[i] + b_tilde[j] - log X[i, j]`` and::

        J = sum_{i,j: X_ij > 0} f(X_ij) * r_ij^2

    Zero counts contribute no term -- GloVe fits ``log X`` only where ``X`` is
    observed, which is also what keeps ``log 0`` out of the sum.

    Parameters
    ----------
    W : np.ndarray
        The word table, shape ``(V, d)``.
    W_tilde : np.ndarray
        The context-word table, shape ``(V, d)``.
    b : np.ndarray
        The word biases, shape ``(V,)``.
    b_tilde : np.ndarray
        The context-word biases, shape ``(V,)``.
    X : np.ndarray
        The ``(V, V)`` co-occurrence matrix.
    x_max : float, default=100.0
        The weighting cap passed to :func:`glove_weight`.
    alpha : float, default=0.75
        The weighting exponent passed to :func:`glove_weight`.

    Returns
    -------
    float
        The total loss ``J``.
    """
    rows, cols = np.nonzero(X)
    total = 0.0
    for i, j in zip(rows, cols):
        residual = W[i] @ W_tilde[j] + b[i] + b_tilde[j] - np.log(X[i, j])
        total += glove_weight(X[i, j], x_max, alpha) * residual**2
    return float(total)

glove_gradients

glove_gradients(W: ndarray, W_tilde: ndarray, b: ndarray, b_tilde: ndarray, X: ndarray, x_max: float = 100.0, alpha: float = 0.75) -> dict[str, np.ndarray]

Gradients of :func:glove_loss with respect to tables and biases.

With r_ij and f(X_ij) as in :func:glove_loss, each observed entry contributes::

dJ/dW[i]       += 2 f(X_ij) r_ij W_tilde[j]
dJ/dW_tilde[j] += 2 f(X_ij) r_ij W[i]
dJ/db[i]       += 2 f(X_ij) r_ij
dJ/db_tilde[j] += 2 f(X_ij) r_ij

Parameters:

Name	Type	Description	Default
W	ndarray	The word table, shape (V, d).	required
W_tilde	ndarray	The context-word table, shape (V, d).	required
b	ndarray	The word biases, shape (V,).	required
b_tilde	ndarray	The context-word biases, shape (V,).	required
X	ndarray	The (V, V) co-occurrence matrix.	required
x_max	float	The weighting cap passed to :func:glove_weight.	100.0
alpha	float	The weighting exponent passed to :func:glove_weight.	0.75

Returns:

Type	Description
dict[str, ndarray]	{"W", "Wt", "b", "bt"} -- gradients matching each parameter's shape.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def glove_gradients(
    W: np.ndarray,
    W_tilde: np.ndarray,
    b: np.ndarray,
    b_tilde: np.ndarray,
    X: np.ndarray,
    x_max: float = 100.0,
    alpha: float = 0.75,
) -> dict[str, np.ndarray]:
    """
    Gradients of :func:`glove_loss` with respect to tables and biases.

    With ``r_ij`` and ``f(X_ij)`` as in :func:`glove_loss`, each observed entry
    contributes::

        dJ/dW[i]       += 2 f(X_ij) r_ij W_tilde[j]
        dJ/dW_tilde[j] += 2 f(X_ij) r_ij W[i]
        dJ/db[i]       += 2 f(X_ij) r_ij
        dJ/db_tilde[j] += 2 f(X_ij) r_ij

    Parameters
    ----------
    W : np.ndarray
        The word table, shape ``(V, d)``.
    W_tilde : np.ndarray
        The context-word table, shape ``(V, d)``.
    b : np.ndarray
        The word biases, shape ``(V,)``.
    b_tilde : np.ndarray
        The context-word biases, shape ``(V,)``.
    X : np.ndarray
        The ``(V, V)`` co-occurrence matrix.
    x_max : float, default=100.0
        The weighting cap passed to :func:`glove_weight`.
    alpha : float, default=0.75
        The weighting exponent passed to :func:`glove_weight`.

    Returns
    -------
    dict[str, np.ndarray]
        ``{"W", "Wt", "b", "bt"}`` -- gradients matching each parameter's shape.
    """
    dW = np.zeros_like(W)
    dW_tilde = np.zeros_like(W_tilde)
    db = np.zeros_like(b)
    db_tilde = np.zeros_like(b_tilde)

    rows, cols = np.nonzero(X)
    for i, j in zip(rows, cols):
        residual = W[i] @ W_tilde[j] + b[i] + b_tilde[j] - np.log(X[i, j])
        scale = 2.0 * glove_weight(X[i, j], x_max, alpha) * residual
        dW[i] += scale * W_tilde[j]
        dW_tilde[j] += scale * W[i]
        db[i] += scale
        db_tilde[j] += scale

    return {"W": dW, "Wt": dW_tilde, "b": db, "bt": db_tilde}

train_sgns

train_sgns(sentences: Sequence[Sequence[int]], vocab_size: int, *, d: int = 8, window: int = 2, k: int = 2, lr: float = 0.05, epochs: int = 200, seed: int = 0, exponent: float = 0.75) -> dict

Train skip-gram with negative sampling by full-batch gradient descent.

A single seeded generator draws the two tables (W then C) and then the k negatives for every (centre, context) pair, in pair order. The negatives are drawn once and reused each epoch, so the objective is fixed and the loss is a clean descent curve. Each epoch sums the loss and the gradient over all pairs and takes one plain SGD step -- no optimiser, no momentum (that lesson lives in optimizer-update-rules).

Parameters:

Name	Type	Description	Default
sentences	Sequence[Sequence[int]]	The corpus as sentences of token indices.	required
vocab_size	int	The vocabulary size V.	required
d	int	The embedding dimension.	8
window	int	The context half-width.	2
k	int	The number of negatives per pair.	2
lr	float	The gradient-descent step size.	0.05
epochs	int	The number of full-batch passes.	200
seed	int	Seeds the single generator driving init and sampling.	0
exponent	float	The negative sampler's smoothing exponent.	0.75

Returns:

Type	Description
dict	{"W", "C", "history"} -- the trained (V, d) tables and the list of per-epoch total losses (each recorded before that epoch's update, so it is strictly decreasing on this fixture).

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def train_sgns(
    sentences: Sequence[Sequence[int]],
    vocab_size: int,
    *,
    d: int = 8,
    window: int = 2,
    k: int = 2,
    lr: float = 0.05,
    epochs: int = 200,
    seed: int = 0,
    exponent: float = 0.75,
) -> dict:
    """
    Train skip-gram with negative sampling by full-batch gradient descent.

    A single seeded generator draws the two tables (``W`` then ``C``) and then
    the ``k`` negatives for every ``(centre, context)`` pair, in pair order. The
    negatives are drawn **once** and reused each epoch, so the objective is fixed
    and the loss is a clean descent curve. Each epoch sums the loss and the
    gradient over all pairs and takes one plain SGD step -- no optimiser, no
    momentum (that lesson lives in ``optimizer-update-rules``).

    Parameters
    ----------
    sentences : Sequence[Sequence[int]]
        The corpus as sentences of token indices.
    vocab_size : int
        The vocabulary size ``V``.
    d : int, default=8
        The embedding dimension.
    window : int, default=2
        The context half-width.
    k : int, default=2
        The number of negatives per pair.
    lr : float, default=0.05
        The gradient-descent step size.
    epochs : int, default=200
        The number of full-batch passes.
    seed : int, default=0
        Seeds the single generator driving init and sampling.
    exponent : float, default=0.75
        The negative sampler's smoothing exponent.

    Returns
    -------
    dict
        ``{"W", "C", "history"}`` -- the trained ``(V, d)`` tables and the list
        of per-epoch total losses (each recorded *before* that epoch's update,
        so it is strictly decreasing on this fixture).
    """
    rng = np.random.default_rng(seed)
    W = _init_table(rng, vocab_size, d)
    C = _init_table(rng, vocab_size, d)

    probabilities = unigram_probabilities(sentences, vocab_size, exponent)
    pairs = [
        pair for sentence in sentences for pair in training_pairs(sentence, window)
    ]
    negatives_per_pair = [sample_negatives(rng, probabilities, k) for _ in pairs]

    history: list[float] = []
    for _ in range(epochs):
        total_loss = 0.0
        dW = np.zeros_like(W)
        dC = np.zeros_like(C)
        for (centre, context), negatives in zip(pairs, negatives_per_pair):
            total_loss += sgns_loss(W, C, centre, context, negatives)
            grads = sgns_gradients(W, C, centre, context, negatives)
            dW += grads["W"]
            dC += grads["C"]

        history.append(total_loss)
        W -= lr * dW
        C -= lr * dC

    return {"W": W, "C": C, "history": history}

train_glove

train_glove(X: ndarray, vocab_size: int, *, d: int = 8, lr: float = 0.05, epochs: int = 200, seed: int = 0, x_max: float = 100.0, alpha: float = 0.75) -> dict

Fit GloVe to a co-occurrence matrix by full-batch gradient descent.

A single seeded generator draws the two tables and the two bias vectors in the order W, W_tilde, b, b_tilde. Each epoch sums the loss and gradient over the observed entries of X and takes one SGD step. The learned vector for a word is the sum of the two tables, W[i] + W_tilde[i], returned as vectors.

Parameters:

Name	Type	Description	Default
X	ndarray	The (V, V) co-occurrence matrix from :func:cooccurrence_matrix.	required
vocab_size	int	The vocabulary size V.	required
d	int	The embedding dimension.	8
lr	float	The gradient-descent step size.	0.05
epochs	int	The number of full-batch passes.	200
seed	int	Seeds the single generator driving initialisation.	0
x_max	float	The weighting cap passed to :func:glove_weight.	100.0
alpha	float	The weighting exponent passed to :func:glove_weight.	0.75

Returns:

Type	Description
dict	{"W", "Wt", "b", "bt", "vectors", "history"} -- the trained tables and biases, the summed word vectors (V, d), and the per-epoch loss.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def train_glove(
    X: np.ndarray,
    vocab_size: int,
    *,
    d: int = 8,
    lr: float = 0.05,
    epochs: int = 200,
    seed: int = 0,
    x_max: float = 100.0,
    alpha: float = 0.75,
) -> dict:
    """
    Fit GloVe to a co-occurrence matrix by full-batch gradient descent.

    A single seeded generator draws the two tables and the two bias vectors in
    the order ``W``, ``W_tilde``, ``b``, ``b_tilde``. Each epoch sums the loss
    and gradient over the observed entries of ``X`` and takes one SGD step. The
    learned vector for a word is the **sum of the two tables**,
    ``W[i] + W_tilde[i]``, returned as ``vectors``.

    Parameters
    ----------
    X : np.ndarray
        The ``(V, V)`` co-occurrence matrix from :func:`cooccurrence_matrix`.
    vocab_size : int
        The vocabulary size ``V``.
    d : int, default=8
        The embedding dimension.
    lr : float, default=0.05
        The gradient-descent step size.
    epochs : int, default=200
        The number of full-batch passes.
    seed : int, default=0
        Seeds the single generator driving initialisation.
    x_max : float, default=100.0
        The weighting cap passed to :func:`glove_weight`.
    alpha : float, default=0.75
        The weighting exponent passed to :func:`glove_weight`.

    Returns
    -------
    dict
        ``{"W", "Wt", "b", "bt", "vectors", "history"}`` -- the trained tables
        and biases, the summed word vectors ``(V, d)``, and the per-epoch loss.
    """
    rng = np.random.default_rng(seed)
    W = _init_table(rng, vocab_size, d)
    W_tilde = _init_table(rng, vocab_size, d)
    sigma = 0.1 / np.sqrt(d)
    b = rng.normal(0.0, sigma, size=vocab_size)
    b_tilde = rng.normal(0.0, sigma, size=vocab_size)

    history: list[float] = []
    for _ in range(epochs):
        history.append(glove_loss(W, W_tilde, b, b_tilde, X, x_max, alpha))
        grads = glove_gradients(W, W_tilde, b, b_tilde, X, x_max, alpha)
        W -= lr * grads["W"]
        W_tilde -= lr * grads["Wt"]
        b -= lr * grads["b"]
        b_tilde -= lr * grads["bt"]

    return {
        "W": W,
        "Wt": W_tilde,
        "b": b,
        "bt": b_tilde,
        "vectors": W + W_tilde,
        "history": history,
    }

nearest_neighbor

nearest_neighbor(embeddings: ndarray, query: int, k: int = 1) -> list[int]

Rank the k nearest words to a query by cosine similarity.

Scores every row of embeddings against embeddings[query] with the head topic's :func:~dlhub.embeddings.word_embeddings.cosine_similarity -- this topic produces vectors, it does not re-derive their geometry -- then returns the top k indices, the query excluded, ties broken by ascending index so the ranking is deterministic.

Parameters:

Name	Type	Description	Default
embeddings	ndarray	A (V, d) table whose rows are word vectors (e.g. the trained W).	required
query	int	The row index of the query word.	required
k	int	How many neighbours to return.	1

Returns:

Type	Description
list[int]	The k nearest word indices, most similar first.

Raises:

Type	Description
ValueError	If query is out of range or k < 1.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def nearest_neighbor(embeddings: np.ndarray, query: int, k: int = 1) -> list[int]:
    """
    Rank the ``k`` nearest words to a query by cosine similarity.

    Scores every row of ``embeddings`` against ``embeddings[query]`` with the
    head topic's :func:`~dlhub.embeddings.word_embeddings.cosine_similarity` --
    this topic *produces* vectors, it does not re-derive their geometry -- then
    returns the top ``k`` indices, the query excluded, ties broken by ascending
    index so the ranking is deterministic.

    Parameters
    ----------
    embeddings : np.ndarray
        A ``(V, d)`` table whose rows are word vectors (e.g. the trained ``W``).
    query : int
        The row index of the query word.
    k : int, default=1
        How many neighbours to return.

    Returns
    -------
    list[int]
        The ``k`` nearest word indices, most similar first.

    Raises
    ------
    ValueError
        If ``query`` is out of range or ``k < 1``.
    """
    vocab_size = embeddings.shape[0]
    if not 0 <= query < vocab_size:
        raise ValueError(f"query must be in [0, {vocab_size}); got {query}")
    if k < 1:
        raise ValueError(f"k must be >= 1; got {k}")

    q = embeddings[query]
    scored = [
        (j, cosine_similarity(q, embeddings[j]))
        for j in range(vocab_size)
        if j != query
    ]
    scored.sort(key=lambda pair: (-pair[1], pair[0]))
    return [j for j, _ in scored[:k]]

main

main() -> None

Train SGNS on the fixture and print the pets family's recovered neighbours.

Source code in src/dlhub/embeddings/word2vec_and_glove.py

def main() -> None:
    """Train SGNS on the fixture and print the pets family's recovered neighbours."""
    fixture = make_fixture()
    sentences, vocab, word_to_ix = (
        fixture["sentences"],
        fixture["vocab"],
        fixture["word_to_ix"],
    )
    V = fixture["dims"]["V"]

    result = train_sgns(sentences, V)
    history = result["history"]
    print(
        f"V = {V}; SGNS loss {history[0]:.3f} -> {history[-1]:.3f} over {len(history)} epochs"
    )

    cat = word_to_ix["cat"]
    neighbours = nearest_neighbor(result["W"], cat, k=3)
    print(f"nearest to 'cat': {[vocab[j] for j in neighbours]}")

Gender bias as a direction in the embedding space: the gender axis recovered as the top principal component of the definitional-pair differences, hard debiasing via neutralize and equalize, WEAT's association score with its exact enumerated permutation p-value, and the neutral-neighbour map that shows the indirect bias surviving the projection.

dlhub.embeddings.debiasing_word_embeddings

Debiasing Word Embeddings -- Bias Geometry, Hard Debiasing, and WEAT

A from-scratch, teaching reference for the geometry of bias in a given static embedding: how gender bias lives in a direction of the vector space, how Bolukbasi et al.'s hard debiasing removes it (identify the subspace, neutralize, equalize), and how WEAT (Caliskan et al.) measures the association that remains -- used here to quantify bias before and after debiasing.

The vectors are treated as given -- a small, hand-planted toy matrix (see :func:make_fixture) whose every answer is exact and checkable by eye. What a word vector is belongs to the sibling :mod:dlhub.embeddings.word_embeddings; how vectors are learned belongs to :mod:dlhub.embeddings.word2vec_and_glove. Cosine similarity and nearest- neighbor ranking are imported from the head rather than restated: this module studies only the bias geometry on top of them.

The module's thesis is cautionary. After :func:hard_debias, every direct projection on the gender direction is zero and WEAT's effect size d is exactly 0 -- the metrics say "fixed". Yet the neutral-neighbour map (:func:neutral_neighbors) is identical before and after: the words still group by gender through the indirect cluster axis the debiasing never touched. That is Gonen & Goldberg's "lipstick on a pig" critique, demonstrated in a four-dimensional space small enough to read by eye: geometric debiasing is superficial.

Layout is (V, d) -- one word per row, so a word's vector is E[i]. This is the head's E transposed (E == head_E.T), the convention the dossier settles because projections, WEAT, and neighbour structure are transpose-free on rows; where the head's column-convention helpers are reused, the row matrix is passed transposed (see :func:neutral_neighbors). Every row is unit-norm -- :func:make_fixture L2-normalizes, which :func:equalize requires.

References

• Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. NeurIPS. https://arxiv.org/abs/1607.06520
• Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334), 183-186. (WEAT.)
• Gonen, H., & Goldberg, Y. (2019). Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases. NAACL. https://arxiv.org/abs/1903.03862
• Ng, A. (2018). Deep Learning Specialization, Course 5 (Sequence Models), Week 2. https://www.coursera.org/specializations/deep-learning

Author

Deep Learning Reference Hub

License

MIT

Notes

• Deterministic male-positive sign. The SVD fixes the gender direction only up to sign, so :func:gender_direction flips g when needed to make g . (e_he - e_she) > 0 for the first definitional pair -- the same determinism-by-convention the head's pca_project_2d uses. Without it, two runs could report mirrored projections and a WEAT statistic with a flipped sign.
• Stacked-deviation SVD = Bolukbasi's covariance PCA. The paper's Step 1 takes the SVD of the covariance C = sum_i sum_w (w - mu_i)^T (w - mu_i) / |D_i| over the defining sets; this module instead stacks the deviations w - mu_i themselves and takes their SVD. The right singular vectors of the stacked deviations are the eigenvectors of its Gram matrix, so the two forms give the same principal directions, and with definitional pairs the 1/|D_i| weight is the uniform 1/2, so they coincide exactly. The stacked form is the one shown because it displays the data the SVD consumes.
• A single direction (k = 1). Bolukbasi generalizes to a k-dimensional subspace B (they use k = 10 on real corpora), with the projection v_B = sum_j (v . b_j) b_j. The fixture's deviation matrix is exactly rank-1 -- singular values [sqrt(6), 0, 0, 0] -- so one direction is the whole subspace here, and every function takes a single unit vector g.
• Normalize first. Equalize's sqrt(1 - ||nu||^2) is real only when the set members are unit vectors (then ||nu|| <= 1, nu being the neutral part of their mean). :func:make_fixture L2-normalizes every row, and :func:equalize raises rather than return a nan when ||nu|| > 1.
• Neutralize zero-residual guard. A word that lies entirely along g (like the fixture's definitional he) has no neutral part: the residual w - (w.g) g is the zero vector and renormalizing would divide by zero. :func:neutralize returns the zero vector instead, which the head's cosine zero-guard then keeps out of every ranking. :func:hard_debias never calls it on definitional words -- those define g and are left untouched, as in Bolukbasi.
• Population standard deviation in WEAT's effect size. d divides by the population std (ddof = 0) of the pooled association scores over X u Y; under the sample std (ddof = 1) the fixture's validated answer would be 1.8102397229 instead of 1.9830182615.
• Exact permutation p. :func:weat_pvalue enumerates every C(|X u Y|, |X|) equal-size partition -- 20 for the fixture's 3-vs-3 targets -- and reports the fraction whose statistic is >= the observed, so the observed partition always counts itself and p >= 1/C. Exact enumeration is feasible only because the toy sets are tiny; Caliskan et al. sample 10,000 random permutations for real-size sets.
• The d zero-spread guard. When every pooled score is equal the std is zero and the effect size is 0/0; :func:weat_effect_size returns 0.0 by definition rather than a nan. That is exactly the post-debias case -- every neutralized word scores 0 against the pure-gender attributes -- and the dossier pins d_after = 0 to this guard.

make_fixture

make_fixture() -> dict

Build the planted bias fixture every test and document shares.

The matrix is hand-written, not trained: V = 14 words by d = 4 coordinates, committed as an integer literal and L2-normalized row-wise so every row is unit-norm (normalize-first, as Bolukbasi requires). The coordinates are 0 = gender (the bias axis), 1 = occupation cluster (the indirect-bias axis, correlated with gender -- this is what survives neutralize), 2-3 = within-cluster content (tie-free neighbours).

The six definitional words are pure-gender archetypes and coincide within gender (he = man = boy = +e0, she = woman = girl = -e0): their only role is to define g and to serve as WEAT's gender attributes, and nothing computes neighbours among them, so the coincidence keeps every known answer exact without affecting any result.

The vocabulary is stored sorted, which fixes the row order of E and the word_to_ix mapping.

Returns:

Type

Description

dict

{"E", "vocab", "word_to_ix", "dims", "definitional_pairs", "neutral_words", "equality_sets", "weat_sets"} where E is the (V, d) float64 matrix with unit-norm rows (one word per row, the head's E.T), vocab the sorted word list, word_to_ix the word-to-row map, dims the {"V", "d"} sizes, and the four role sets named by word: the definitional (male, female) pairs, the gender-neutral words to neutralize, the equality sets to equalize, and the WEAT A/B attribute and X/Y target sets.

Source code in src/dlhub/embeddings/debiasing_word_embeddings.py

def make_fixture() -> dict:
    """
    Build the planted bias fixture every test and document shares.

    The matrix is hand-written, not trained: ``V = 14`` words by ``d = 4``
    coordinates, committed as an integer literal and L2-normalized row-wise so
    every row is unit-norm (normalize-first, as Bolukbasi requires). The
    coordinates are ``0`` = gender (the bias axis), ``1`` = occupation cluster
    (the *indirect*-bias axis, correlated with gender -- this is what survives
    neutralize), ``2-3`` = within-cluster content (tie-free neighbours).

    The six definitional words are pure-gender archetypes and coincide within
    gender (``he = man = boy = +e0``, ``she = woman = girl = -e0``): their only
    role is to define ``g`` and to serve as WEAT's gender attributes, and
    nothing computes neighbours among them, so the coincidence keeps every
    known answer exact without affecting any result.

    The vocabulary is stored sorted, which fixes the row order of ``E`` and the
    ``word_to_ix`` mapping.

    Returns
    -------
    dict
        ``{"E", "vocab", "word_to_ix", "dims", "definitional_pairs",
        "neutral_words", "equality_sets", "weat_sets"}`` where ``E`` is the
        ``(V, d)`` ``float64`` matrix with unit-norm rows (one word per row,
        the head's ``E.T``), ``vocab`` the sorted word list, ``word_to_ix`` the
        word-to-row map, ``dims`` the ``{"V", "d"}`` sizes, and the four role
        sets named by word: the definitional ``(male, female)`` pairs, the
        gender-neutral words to neutralize, the equality sets to equalize, and
        the WEAT ``A``/``B`` attribute and ``X``/``Y`` target sets.
    """
    # Each word's raw vector as (gender, cluster, within_a, within_b) -- the
    # dossier's planted axes, as integers so every answer is checkable by eye.
    vectors = {
        "boy": [1, 0, 0, 0],
        "engineer": [1, 3, 3, 0],
        "girl": [-1, 0, 0, 0],
        "grandfather": [2, 0, 0, 1],
        "grandmother": [-1, 0, 0, 1],
        "he": [1, 0, 0, 0],
        "man": [1, 0, 0, 0],
        "mechanic": [1, 3, 0, 0],
        "nanny": [-1, -3, 0, 0],
        "nurse": [-1, -3, 3, 0],
        "programmer": [1, 3, 2, 0],
        "receptionist": [-1, -3, 2, 0],
        "she": [-1, 0, 0, 0],
        "woman": [-1, 0, 0, 0],
    }

    vocab = sorted(vectors)
    raw = np.array([vectors[word] for word in vocab], dtype=np.float64)
    # Normalize-first: equalize's sqrt(1 - ||nu||^2) is only real on the unit
    # sphere, so every row is L2-normalized to length 1.
    E = raw / np.linalg.norm(raw, axis=1, keepdims=True)
    word_to_ix = {word: i for i, word in enumerate(vocab)}

    return {
        "E": E,
        "vocab": vocab,
        "word_to_ix": word_to_ix,
        "dims": {"V": E.shape[0], "d": E.shape[1]},
        "definitional_pairs": [("he", "she"), ("man", "woman"), ("boy", "girl")],
        "neutral_words": [
            "engineer",
            "programmer",
            "mechanic",
            "nurse",
            "receptionist",
            "nanny",
        ],
        "equality_sets": [["grandfather", "grandmother"]],
        "weat_sets": {
            "A": ["he", "man", "boy"],
            "B": ["she", "woman", "girl"],
            "X": ["engineer", "programmer", "mechanic"],
            "Y": ["nurse", "receptionist", "nanny"],
        },
    }

gender_direction

gender_direction(E: ndarray, definitional_pairs: Sequence[tuple[int, int]]) -> np.ndarray

Identify the unit gender direction g from definitional pairs.

For each (male, female) pair, take the pair mean mu and the two deviations w - mu; stack all deviations and take the top principal component (via SVD) as g (Bolukbasi §5.1, in the stacked-deviation form -- see the module Notes). The SVD fixes a direction only up to sign, so the sign is pinned deterministically male-positive: flip g when g . (e_male - e_female) < 0 for the first pair.

A single direction (k = 1) is returned; the method generalizes to a k-dimensional subspace B by taking the top k right singular vectors instead.

Parameters:

Name	Type	Description	Default
E	ndarray	The embedding matrix, shape (V, d), one word per row.	required
definitional_pairs	Sequence[tuple[int, int]]	Row indices (male, female) of words whose definitions differ only in gender -- e.g. (he, she), (man, woman), (boy, girl).	required

Returns:

Type	Description
ndarray	The unit gender direction g, shape (d,), male-positive.

Raises:

Type	Description
ValueError	If definitional_pairs is empty.

Source code in src/dlhub/embeddings/debiasing_word_embeddings.py

def gender_direction(
    E: np.ndarray, definitional_pairs: Sequence[tuple[int, int]]
) -> np.ndarray:
    """
    Identify the unit gender direction ``g`` from definitional pairs.

    For each ``(male, female)`` pair, take the pair mean ``mu`` and the two
    deviations ``w - mu``; stack all deviations and take the **top principal
    component** (via SVD) as ``g`` (Bolukbasi §5.1, in the stacked-deviation
    form -- see the module Notes). The SVD fixes a direction only up to sign,
    so the sign is pinned deterministically **male-positive**: flip ``g`` when
    ``g . (e_male - e_female) < 0`` for the first pair.

    A single direction (``k = 1``) is returned; the method generalizes to a
    ``k``-dimensional subspace ``B`` by taking the top ``k`` right singular
    vectors instead.

    Parameters
    ----------
    E : np.ndarray
        The embedding matrix, shape ``(V, d)``, one word per row.
    definitional_pairs : Sequence[tuple[int, int]]
        Row indices ``(male, female)`` of words whose definitions differ only
        in gender -- e.g. ``(he, she)``, ``(man, woman)``, ``(boy, girl)``.

    Returns
    -------
    np.ndarray
        The unit gender direction ``g``, shape ``(d,)``, male-positive.

    Raises
    ------
    ValueError
        If ``definitional_pairs`` is empty.
    """
    if len(definitional_pairs) == 0:
        raise ValueError("definitional_pairs must hold at least one pair")

    # Center each definitional pair on its own mean and stack the deviations:
    # one row per word, shape (2 * n_pairs, d).
    deviations = []
    for male, female in definitional_pairs:
        mu = (E[male] + E[female]) / 2.0
        deviations.append(E[male] - mu)
        deviations.append(E[female] - mu)
    deviation_matrix = np.array(deviations)

    # The top principal component is the first right singular vector. (The
    # singular values themselves show why k = 1 suffices on the fixture: the
    # spectrum is [sqrt(6), 0, 0, 0], exactly rank-1.)
    _, _, Wt = np.linalg.svd(deviation_matrix, full_matrices=False)
    g = Wt[0].copy()

    # Deterministic sign rule: the male direction is positive.
    male, female = definitional_pairs[0]
    if np.dot(g, E[male] - E[female]) < 0.0:
        g = -g
    return g

neutralize

neutralize(w: ndarray, g: ndarray) -> np.ndarray

Remove a gender-neutral word's g component and renormalize.

The projection out of the gender direction, spelled term by term::

w' = (w - (w . g) g) / || w - (w . g) g ||

After this, w' . g = 0 (the direct bias is gone) and w' is back on the unit sphere, so cosines and distances stay comparable.

Parameters:

Name	Type	Description	Default
w	ndarray	The word's vector, shape (d,).	required
g	ndarray	The unit gender direction, shape (d,).	required

Returns:

Type	Description
ndarray	The neutralized unit vector, shape (d,) -- or the zero vector when w lies entirely along g and has no neutral part (the defensive guard; see the module Notes).

Source code in src/dlhub/embeddings/debiasing_word_embeddings.py

def neutralize(w: np.ndarray, g: np.ndarray) -> np.ndarray:
    """
    Remove a gender-neutral word's ``g`` component and renormalize.

    The projection out of the gender direction, spelled term by term::

        w' = (w - (w . g) g) / || w - (w . g) g ||

    After this, ``w' . g = 0`` (the direct bias is gone) and ``w'`` is back on
    the unit sphere, so cosines and distances stay comparable.

    Parameters
    ----------
    w : np.ndarray
        The word's vector, shape ``(d,)``.
    g : np.ndarray
        The unit gender direction, shape ``(d,)``.

    Returns
    -------
    np.ndarray
        The neutralized unit vector, shape ``(d,)`` -- or the zero vector when
        ``w`` lies entirely along ``g`` and has no neutral part (the defensive
        guard; see the module Notes).
    """
    residual = w - np.dot(w, g) * g
    norm = float(np.linalg.norm(residual))
    if norm == 0.0:
        # w is pure-gender: nothing survives the projection. Return the zero
        # vector rather than divide by zero; the head's cosine zero-guard keeps
        # it out of every ranking.
        return np.zeros_like(residual)
    return residual / norm

equalize

equalize(vectors: ndarray, g: ndarray) -> np.ndarray

Equalize one equality set: make its members symmetric about g.

For a gendered pair to keep distinct but unbiased ({grandfather, grandmother}), Bolukbasi's Step 2a (arXiv:1607.06520) replaces every member's neutral part with the set's shared neutral part nu and gives the members equal-and-opposite g components::

mu   = mean(E)                  # the set mean
mu_B = (mu . g) g               # its gender component
nu   = mu - mu_B                # the shared neutral part
w_B  = (w . g) g                # each member's gender component

w'   = nu + sqrt(1 - ||nu||^2) * (w_B - mu_B) / || w_B - mu_B ||

Every output is unit-norm, the members' g projections are +/- the shared sqrt(1 - ||nu||^2), and all members are equidistant from any neutralized word -- which is what makes the pair unbiased without merging the two words into one vector.

Parameters:

Name	Type	Description	Default
vectors	ndarray	The equality set's member vectors, shape (n, d), unit-norm (normalize-first: sqrt(1 - ||nu||^2) is only real on the unit sphere).	required
g	ndarray	The unit gender direction, shape (d,).	required

Returns:

Type	Description
ndarray	The equalized vectors, shape (n, d), in the input's member order.

Raises:

Type	Description
ValueError	If ||nu|| > 1 (the members are not unit-norm), or if some member's gender component equals the set mean's, making w_B - mu_B zero and the direction undefined.

Source code in src/dlhub/embeddings/debiasing_word_embeddings.py

def equalize(vectors: np.ndarray, g: np.ndarray) -> np.ndarray:
    """
    Equalize one equality set: make its members symmetric about ``g``.

    For a gendered pair to keep *distinct* but unbiased (``{grandfather,
    grandmother}``), Bolukbasi's Step 2a (arXiv:1607.06520) replaces every
    member's neutral part with the set's shared neutral part ``nu`` and gives
    the members equal-and-opposite ``g`` components::

        mu   = mean(E)                  # the set mean
        mu_B = (mu . g) g               # its gender component
        nu   = mu - mu_B                # the shared neutral part
        w_B  = (w . g) g                # each member's gender component

        w'   = nu + sqrt(1 - ||nu||^2) * (w_B - mu_B) / || w_B - mu_B ||

    Every output is unit-norm, the members' ``g`` projections are ``+/-`` the
    shared ``sqrt(1 - ||nu||^2)``, and all members are equidistant from any
    neutralized word -- which is what makes the pair unbiased without merging
    the two words into one vector.

    Parameters
    ----------
    vectors : np.ndarray
        The equality set's member vectors, shape ``(n, d)``, **unit-norm**
        (normalize-first: ``sqrt(1 - ||nu||^2)`` is only real on the unit
        sphere).
    g : np.ndarray
        The unit gender direction, shape ``(d,)``.

    Returns
    -------
    np.ndarray
        The equalized vectors, shape ``(n, d)``, in the input's member order.

    Raises
    ------
    ValueError
        If ``||nu|| > 1`` (the members are not unit-norm), or if some member's
        gender component equals the set mean's, making ``w_B - mu_B`` zero and
        the direction undefined.
    """
    mu = vectors.mean(axis=0)
    mu_B = np.dot(mu, g) * g
    nu = mu - mu_B

    # The shared g-component magnitude. Real only on the unit sphere.
    neutral_sq = float(np.dot(nu, nu))
    if neutral_sq > 1.0:
        raise ValueError(
            "equalize needs unit-norm members: ||nu||^2 = "
            f"{neutral_sq:.6f} > 1, so sqrt(1 - ||nu||^2) is not real "
            "(normalize the rows first)"
        )
    scale = np.sqrt(1.0 - neutral_sq)

    equalized = np.empty_like(vectors)
    for row in range(vectors.shape[0]):
        w_B = np.dot(vectors[row], g) * g
        direction = w_B - mu_B
        norm = float(np.linalg.norm(direction))
        if norm == 0.0:
            raise ValueError(
                f"member {row} has the set mean's gender component, so the "
                "equalize direction w_B - mu_B is zero"
            )
        equalized[row] = nu + scale * direction / norm
    return equalized

hard_debias

hard_debias(E: ndarray, g: ndarray, neutral_words: Sequence[int], equality_sets: Sequence[Sequence[int]]) -> np.ndarray

Bolukbasi's hard debiasing: neutralize the neutral words, equalize the sets.

Copies E and applies :func:neutralize to every gender-neutral word (the occupations -- their direct bias on g goes to zero) and :func:equalize to every equality set (the gendered pairs that must stay distinct but symmetric). The definitional words that defined g are left untouched, as in Bolukbasi: neutralizing he = +e0 would annihilate it.

Parameters:

Name	Type	Description	Default
E	ndarray	The embedding matrix, shape (V, d), unit-norm rows.	required
g	ndarray	The unit gender direction, shape (d,).	required
neutral_words	Sequence[int]	Row indices of the gender-neutral words to neutralize.	required
equality_sets	Sequence[Sequence[int]]	Row indices of each equality set to equalize, disjoint from neutral_words.	required

Returns:

Type	Description
ndarray	The debiased matrix, shape (V, d) -- a copy; E is not mutated.

Raises:

Type	Description
ValueError	If a word appears in both neutral_words and an equality set (the two operations disagree on such a word, and the result would depend on the order they run in), or if :func:equalize raises.

Source code in src/dlhub/embeddings/debiasing_word_embeddings.py

def hard_debias(
    E: np.ndarray,
    g: np.ndarray,
    neutral_words: Sequence[int],
    equality_sets: Sequence[Sequence[int]],
) -> np.ndarray:
    """
    Bolukbasi's hard debiasing: neutralize the neutral words, equalize the sets.

    Copies ``E`` and applies :func:`neutralize` to every gender-neutral word
    (the occupations -- their direct bias on ``g`` goes to zero) and
    :func:`equalize` to every equality set (the gendered pairs that must stay
    distinct but symmetric). The definitional words that defined ``g`` are left
    untouched, as in Bolukbasi: neutralizing ``he = +e0`` would annihilate it.

    Parameters
    ----------
    E : np.ndarray
        The embedding matrix, shape ``(V, d)``, unit-norm rows.
    g : np.ndarray
        The unit gender direction, shape ``(d,)``.
    neutral_words : Sequence[int]
        Row indices of the gender-neutral words to neutralize.
    equality_sets : Sequence[Sequence[int]]
        Row indices of each equality set to equalize, disjoint from
        ``neutral_words``.

    Returns
    -------
    np.ndarray
        The debiased matrix, shape ``(V, d)`` -- a copy; ``E`` is not mutated.

    Raises
    ------
    ValueError
        If a word appears in both ``neutral_words`` and an equality set (the
        two operations disagree on such a word, and the result would depend on
        the order they run in), or if :func:`equalize` raises.
    """
    neutral = list(neutral_words)
    sets = [list(equality_set) for equality_set in equality_sets]

    overlap = sorted(set(neutral) & {i for equality_set in sets for i in equality_set})
    if overlap:
        raise ValueError(
            f"rows {overlap} are both neutralized and equalized; the two sets "
            "must be disjoint"
        )

    debiased = E.copy()
    for i in neutral:
        debiased[i] = neutralize(E[i], g)
    for equality_set in sets:
        debiased[equality_set] = equalize(E[equality_set], g)
    return debiased

weat_association

weat_association(w: ndarray, A: ndarray, B: ndarray) -> float

WEAT's association score s(w, A, B) for one word.

How much more similar w is to attribute set A than to B::

s(w, A, B) = mean_{a in A} cos(w, a) - mean_{b in B} cos(w, b)

Positive means "leans A", negative "leans B", zero "unassociated". Cosines come from the head's :func:~dlhub.embeddings.word_embeddings.cosine_similarity.

Parameters:

Name	Type	Description	Default
w	ndarray	The word's vector, shape (d,).	required
A	ndarray	The first attribute set's vectors, shape (|A|, d), one word per row.	required
B	ndarray	The second attribute set's vectors, shape (|B|, d).	required

Returns:

Type	Description
float	The association score s(w, A, B).

Source code in src/dlhub/embeddings/debiasing_word_embeddings.py

def weat_association(w: np.ndarray, A: np.ndarray, B: np.ndarray) -> float:
    """
    WEAT's association score ``s(w, A, B)`` for one word.

    How much more similar ``w`` is to attribute set ``A`` than to ``B``::

        s(w, A, B) = mean_{a in A} cos(w, a) - mean_{b in B} cos(w, b)

    Positive means "leans A", negative "leans B", zero "unassociated". Cosines
    come from the head's :func:`~dlhub.embeddings.word_embeddings.cosine_similarity`.

    Parameters
    ----------
    w : np.ndarray
        The word's vector, shape ``(d,)``.
    A : np.ndarray
        The first attribute set's vectors, shape ``(|A|, d)``, one word per row.
    B : np.ndarray
        The second attribute set's vectors, shape ``(|B|, d)``.

    Returns
    -------
    float
        The association score ``s(w, A, B)``.
    """
    mean_a = float(np.mean([cosine_similarity(w, a) for a in A]))
    mean_b = float(np.mean([cosine_similarity(w, b) for b in B]))
    return mean_a - mean_b

weat_statistic

weat_statistic(X: ndarray, Y: ndarray, A: ndarray, B: ndarray) -> float

WEAT's test statistic: the total association gap between the target sets.

::

statistic = sum_{x in X} s(x, A, B) - sum_{y in Y} s(y, A, B)

with s from :func:weat_association. Large positive means the X targets lean toward A more than the Y targets do.

Parameters:

Name	Type	Description	Default
X	ndarray	The first target set's vectors, shape (|X|, d).	required
Y	ndarray	The second target set's vectors, shape (|Y|, d).	required
A	ndarray	The first attribute set's vectors, shape (|A|, d).	required
B	ndarray	The second attribute set's vectors, shape (|B|, d).	required

Returns:

Type	Description
float	The test statistic.

Source code in src/dlhub/embeddings/debiasing_word_embeddings.py

def weat_statistic(X: np.ndarray, Y: np.ndarray, A: np.ndarray, B: np.ndarray) -> float:
    """
    WEAT's test statistic: the total association gap between the target sets.

    ::

        statistic = sum_{x in X} s(x, A, B) - sum_{y in Y} s(y, A, B)

    with ``s`` from :func:`weat_association`. Large positive means the ``X``
    targets lean toward ``A`` more than the ``Y`` targets do.

    Parameters
    ----------
    X : np.ndarray
        The first target set's vectors, shape ``(|X|, d)``.
    Y : np.ndarray
        The second target set's vectors, shape ``(|Y|, d)``.
    A : np.ndarray
        The first attribute set's vectors, shape ``(|A|, d)``.
    B : np.ndarray
        The second attribute set's vectors, shape ``(|B|, d)``.

    Returns
    -------
    float
        The test statistic.
    """
    total_x = sum(weat_association(x, A, B) for x in X)
    total_y = sum(weat_association(y, A, B) for y in Y)
    return float(total_x - total_y)

weat_effect_size

weat_effect_size(X: ndarray, Y: ndarray, A: ndarray, B: ndarray) -> float

WEAT's effect size d: the association gap in pooled standard deviations.

::

d = (mean_{x in X} s(x) - mean_{y in Y} s(y)) / std_{w in X u Y} s(w)

with s from :func:eat_association and the population std (ddof = 0; see the module Notes). d is scale-free -- it says how far apart the two target groups sit relative to their own spread -- and it is the dossier's hard known answer, before and after debiasing.

Parameters:

Name	Type	Description	Default
X	ndarray	The first target set's vectors, shape (|X|, d).	required
Y	ndarray	The second target set's vectors, shape (|Y|, d).	required
A	ndarray	The first attribute set's vectors, shape (|A|, d).	required
B	ndarray	The second attribute set's vectors, shape (|B|, d).	required

Returns:

Type	Description
float	The effect size d, or 0.0 when the pooled std is 0 (the zero-spread guard -- exactly the post-debias case; see the module Notes).

Source code in src/dlhub/embeddings/debiasing_word_embeddings.py

def weat_effect_size(
    X: np.ndarray, Y: np.ndarray, A: np.ndarray, B: np.ndarray
) -> float:
    """
    WEAT's effect size ``d``: the association gap in pooled standard deviations.

    ::

        d = (mean_{x in X} s(x) - mean_{y in Y} s(y)) / std_{w in X u Y} s(w)

    with ``s`` from :func:`weat_association`` and the **population** std
    (``ddof = 0``; see the module Notes). ``d`` is scale-free -- it says how
    far apart the two target groups sit relative to their own spread -- and it
    is the dossier's hard known answer, before and after debiasing.

    Parameters
    ----------
    X : np.ndarray
        The first target set's vectors, shape ``(|X|, d)``.
    Y : np.ndarray
        The second target set's vectors, shape ``(|Y|, d)``.
    A : np.ndarray
        The first attribute set's vectors, shape ``(|A|, d)``.
    B : np.ndarray
        The second attribute set's vectors, shape ``(|B|, d)``.

    Returns
    -------
    float
        The effect size ``d``, or ``0.0`` when the pooled std is ``0`` (the
        zero-spread guard -- exactly the post-debias case; see the module
        Notes).
    """
    s_x = [weat_association(x, A, B) for x in X]
    s_y = [weat_association(y, A, B) for y in Y]

    spread = float(np.array(s_x + s_y).std())  # population std (ddof = 0)
    if spread == 0.0:
        # Every score is identical: 0/0 is defined to be 0.0, not nan.
        return 0.0
    return float((np.mean(s_x) - np.mean(s_y)) / spread)

weat_pvalue

weat_pvalue(X: ndarray, Y: ndarray, A: ndarray, B: ndarray) -> float

WEAT's one-sided p-value, by exact enumeration of every permutation.

Pools the targets X u Y and enumerates all C(|X u Y|, |X|) ways to choose a new X' of the same size (the complement is Y'), computing the statistic of each partition. The p-value is the fraction of partitions whose statistic is >= the observed one -- the probability that a random equal-size split associates the targets with the attributes at least as strongly as the true split does.

Exact enumeration replaces Caliskan et al.'s 10,000 sampled permutations; it is affordable here because the toy sets give only C(6, 3) = 20 partitions (see the module Notes).

Parameters:

Name	Type	Description	Default
X	ndarray	The first target set's vectors, shape (|X|, d).	required
Y	ndarray	The second target set's vectors, shape (|Y|, d).	required
A	ndarray	The first attribute set's vectors, shape (|A|, d).	required
B	ndarray	The second attribute set's vectors, shape (|B|, d).	required

Returns:

Type	Description
float	The permutation p-value in [1/C, 1].

Raises:

Type	Description
ValueError	If |X| != |Y| -- the permutation test redistributes the pooled targets into equal-size partitions.

Source code in src/dlhub/embeddings/debiasing_word_embeddings.py

def weat_pvalue(X: np.ndarray, Y: np.ndarray, A: np.ndarray, B: np.ndarray) -> float:
    """
    WEAT's one-sided p-value, by **exact** enumeration of every permutation.

    Pools the targets ``X u Y`` and enumerates all ``C(|X u Y|, |X|)`` ways to
    choose a new ``X'`` of the same size (the complement is ``Y'``), computing
    the statistic of each partition. The p-value is the fraction of partitions
    whose statistic is ``>=`` the observed one -- the probability that a random
    equal-size split associates the targets with the attributes at least as
    strongly as the true split does.

    Exact enumeration replaces Caliskan et al.'s 10,000 sampled permutations;
    it is affordable here because the toy sets give only ``C(6, 3) = 20``
    partitions (see the module Notes).

    Parameters
    ----------
    X : np.ndarray
        The first target set's vectors, shape ``(|X|, d)``.
    Y : np.ndarray
        The second target set's vectors, shape ``(|Y|, d)``.
    A : np.ndarray
        The first attribute set's vectors, shape ``(|A|, d)``.
    B : np.ndarray
        The second attribute set's vectors, shape ``(|B|, d)``.

    Returns
    -------
    float
        The permutation p-value in ``[1/C, 1]``.

    Raises
    ------
    ValueError
        If ``|X| != |Y|`` -- the permutation test redistributes the pooled
        targets into *equal-size* partitions.
    """
    if X.shape[0] != Y.shape[0]:
        raise ValueError(
            f"equal-size target sets required; got |X| = {X.shape[0]} and "
            f"|Y| = {Y.shape[0]}"
        )

    observed = weat_statistic(X, Y, A, B)
    pool = np.vstack([X, Y])
    n, k = pool.shape[0], X.shape[0]

    at_least = 0
    total = 0
    for chosen in itertools.combinations(range(n), k):
        chosen_set = set(chosen)
        X_permuted = pool[list(chosen)]
        Y_permuted = pool[[j for j in range(n) if j not in chosen_set]]
        if weat_statistic(X_permuted, Y_permuted, A, B) >= observed:
            at_least += 1
        total += 1
    return at_least / total

neutral_neighbors

neutral_neighbors(E: ndarray, neutral_words: Sequence[int]) -> dict[int, int]

Map every neutral word to its nearest neighbour within the neutral set.

The tested half of the surviving-bias metric: :func:hard_debias zeroes every direct projection, but if this map is identical before and after, the words still group by gender through the indirect cluster axis -- the Gonen & Goldberg climax. Ranking reuses the head's :func:~dlhub.embeddings.word_embeddings.nearest_neighbors (cosine, self excluded, ties by ascending index), on the neutral sub-matrix alone.

Parameters:

Name	Type	Description	Default
E	ndarray	The embedding matrix, shape (V, d), one word per row.	required
neutral_words	Sequence[int]	Row indices of the gender-neutral words, at least two.	required

Returns:

Type	Description
dict[int, int]	{row: neighbour_row} -- each neutral word's row index in E mapped to its nearest neutral neighbour's row index in E.

Raises:

Type	Description
ValueError	If fewer than two neutral words are given (there would be no neighbour to find).

Source code in src/dlhub/embeddings/debiasing_word_embeddings.py

def neutral_neighbors(E: np.ndarray, neutral_words: Sequence[int]) -> dict[int, int]:
    """
    Map every neutral word to its nearest neighbour *within the neutral set*.

    The tested half of the surviving-bias metric: :func:`hard_debias` zeroes
    every direct projection, but if this map is identical before and after,
    the words still *group by gender* through the indirect cluster axis -- the
    Gonen & Goldberg climax. Ranking reuses the head's
    :func:`~dlhub.embeddings.word_embeddings.nearest_neighbors` (cosine, self
    excluded, ties by ascending index), on the neutral sub-matrix alone.

    Parameters
    ----------
    E : np.ndarray
        The embedding matrix, shape ``(V, d)``, one word per row.
    neutral_words : Sequence[int]
        Row indices of the gender-neutral words, at least two.

    Returns
    -------
    dict[int, int]
        ``{row: neighbour_row}`` -- each neutral word's row index in ``E``
        mapped to its nearest neutral neighbour's row index in ``E``.

    Raises
    ------
    ValueError
        If fewer than two neutral words are given (there would be no neighbour
        to find).
    """
    words = list(neutral_words)
    if len(words) < 2:
        raise ValueError(f"need at least 2 neutral words; got {len(words)}")

    sub = E[words]  # (n, d): the neutral words only, in the given order
    mapping: dict[int, int] = {}
    for position, row in enumerate(words):
        # The head works on (d, V) columns: pass the sub-matrix transposed, so
        # its column j is neutral word j, then map the answer back to E's rows.
        neighbour, _score = nearest_neighbors(sub.T, position, k=1)[0]
        mapping[row] = words[neighbour]
    return mapping

main

main() -> None

Run the debiasing pipeline end to end and print its known answers.

Source code in src/dlhub/embeddings/debiasing_word_embeddings.py

def main() -> None:
    """Run the debiasing pipeline end to end and print its known answers."""
    fixture = make_fixture()
    E, vocab, ix = fixture["E"], fixture["vocab"], fixture["word_to_ix"]

    pairs = [(ix[male], ix[female]) for male, female in fixture["definitional_pairs"]]
    g = gender_direction(E, pairs)
    print(f"gender direction g = {np.round(g, 4)} (male-positive)")

    print("direct bias (projection on g):")
    for word in fixture["neutral_words"]:
        print(f"  {word:12s} {E[ix[word]] @ g:+.5f}")

    sets = fixture["weat_sets"]
    A, B = _rows(E, ix, sets["A"]), _rows(E, ix, sets["B"])
    X, Y = _rows(E, ix, sets["X"]), _rows(E, ix, sets["Y"])
    print(
        f"WEAT before: statistic = {weat_statistic(X, Y, A, B):.10f}, "
        f"d = {weat_effect_size(X, Y, A, B):.10f}, p = {weat_pvalue(X, Y, A, B)}"
    )

    neutral = [ix[word] for word in fixture["neutral_words"]]
    equality_sets = [[ix[word] for word in s] for s in fixture["equality_sets"]]
    debiased = hard_debias(E, g, neutral, equality_sets)

    X_after, Y_after = _rows(debiased, ix, sets["X"]), _rows(debiased, ix, sets["Y"])
    print(
        f"WEAT after:  statistic = {weat_statistic(X_after, Y_after, A, B):.10f}, "
        f"d = {weat_effect_size(X_after, Y_after, A, B):.10f}, "
        f"p = {weat_pvalue(X_after, Y_after, A, B)}"
    )

    before = neutral_neighbors(E, neutral)
    after = neutral_neighbors(debiased, neutral)
    print(f"neutral-neighbour map unchanged by debiasing: {before == after}")
    for row in neutral:
        print(f"  {vocab[row]:12s} -> {vocab[after[row]]}")

    for equality_set in equality_sets:
        members = ", ".join(vocab[i] for i in equality_set)
        print(f"equalize {{{members}}}:")
        for i in equality_set:
            print(f"  {vocab[i]:12s} {np.round(debiased[i], 5)}")