Debiasing word embeddings: identify the bias direction, watch the metric say "fixed", and meet the bias that survives¶

Purpose. The word embeddings notebook ended with a warning: the toy "gender" axis was a bookkeeping coordinate, and the real gender direction that appears in vectors learned from text is a harm, not a feature. This notebook makes the harm geometric on a planted 14-word, 4-dimensional fixture: recover the gender direction g from definitional pairs, measure the direct bias (projections on g and the WEAT effect size d / exact permutation p), apply Bolukbasi et al.'s hard debiasing (neutralize then equalize), and then — at the climax — see the surviving bias. Every projection is now 0, every WEAT d is 0, the metric says fixed. Yet the neutral words still group by gender under their nearest neighbours and a 2-D PCA scatter shows the two clusters right where they were. The hub's thesis on bias, demonstrated in four dimensions: geometric debiasing is "lipstick on a pig" (Gonen & Goldberg, 2019) — it removes the projection, not the structure.

Prerequisites. What a word vector is, cosine similarity, and nearest neighbours — the head topic. Principal components at the level of the PCA section of the head notebook. No framework — the lesson uses numpy and matplotlib only. Every algorithm is imported from the hub's canonical dlhub.embeddings.debiasing_word_embeddings and dlhub.embeddings; the notebook defines no algorithm of its own, so the lesson cannot drift from the reference or the tests.

What you will be able to do by the end.

• Identify the unit gender direction g as the top principal component of the definitional-pair deviations, with the sign pinned male-positive.
• Quantify direct bias as the projection of any word on g, and as the WEAT effect size d plus the exact enumerated permutation p for the male-vs-female-occupations split.
• Apply hard_debias — neutralize removes the g component from a gender-neutral word and renormalizes; equalize makes an equality pair symmetric about g.
• Show that after hard_debias every neutral projection is 0, d = 0, and p = 1.0 — the metric says fixed.
• The twist: show the same neutral_neighbors map before and after (identical) and project the debiased space to 2-D with pca_project_2d to see the male / female occupation clusters persist — the surviving bias.

What this notebook is not. It does not re-implement gender-direction identification, neutralize, equalize, WEAT, or PCA: every one of those is imported. It trains nothing: the embedding matrix is a small, hand-planted fixture, so every answer is exact and checkable by reading the numbers. It is not a real-corpus audit: a bias-audit on a real pretrained embedding needs a how-to, and the hub declined to write one for this topic because real vectors are forbidden as fixtures. The mechanics and the WEAT formulas are pinned in the reference; the argument that geometric debiasing is superficial is in the explanation.

Runtime. A couple of seconds end to end on a laptop CPU. There are exactly 20 WEAT partitions to enumerate, the SVD is on a 6×4 deviation matrix, the PCA is on a 14×4 matrix, and the rest is imports and drawing four figures. Seed. There is no seed. The fixture is hand-planted as an integer literal and L2-normalized row-wise inside make_fixture; the SVD signs are fixed by convention; PCA's signs are fixed by the head's pca_project_2d; and WEAT's permutation test is exact (every partition is enumerated). Every number in this notebook is reproducible to the bit, by construction — no random draw happens anywhere.

Where this fits¶

Reader wants to	Go to
Learn it step by step	This notebook — you are here
Do it in a project	Not written for this topic — a how-to was declined; the WEAT audit pipeline is pinned in the reference
Look up an equation or rule	Debiasing word embeddings
Understand why it works	Debiasing word embeddings
Learn by running it	This notebook — you are here

Sibling topics (the Word Embeddings family): the head word-embeddings (what a word vector is), word2vec-and-glove (how vectors are learned), and the contextual-embeddings explanation (what changes when a word's vector depends on the sentence it sits in — this family's fourth topic has no notebook of its own).

1. Orient: a planted bias space¶

A word embedding is a matrix; bias lives in it as a direction. To see what that means, the hub uses a planted toy: V = 14 words in d = 4 coordinates, hand-written as integers and L2-normalized row-wise inside make_fixture so every row is unit length. The four coordinates mean something:

• coordinate 0 = gender (the bias axis — what we will identify, then project out);
• coordinate 1 = occupation cluster (correlated with gender, but the indirect axis — what survives debiasing);
• coordinates 2–3 = within-cluster content (tie-free nearest neighbours).

The vocabulary is stored sorted, which fixes the row order and the word_to_ix mapping. The raw integer rows are the dossier's planted coordinates; make_fixture only L2-normalizes each row, so the gender sign in the projection is what the planted integer had.

In [1]:

import matplotlib.pyplot as plt
import numpy as np

from dlhub.embeddings import pca_project_2d
from dlhub.embeddings.debiasing_word_embeddings import (
    gender_direction,
    hard_debias,
    make_fixture,
    neutral_neighbors,
    weat_effect_size,
    weat_pvalue,
    weat_statistic,
)

fixture = make_fixture()
E = fixture["E"]  # (V, d) = (14, 4), one word per row, unit-norm rows
vocab = fixture["vocab"]
word_to_ix = fixture["word_to_ix"]
V, d = fixture["dims"]["V"], fixture["dims"]["d"]

print(f"E has shape (V, d) = ({V}, {d}); one word per row; rows are unit-norm\n")

role = {
    **{w: "definitional (male)" for w in ["he", "man", "boy"]},
    **{w: "definitional (female)" for w in ["she", "woman", "girl"]},
    **{w: "male-coded occupation" for w in ["engineer", "programmer", "mechanic"]},
    **{w: "female-coded occupation" for w in ["nurse", "receptionist", "nanny"]},
    **{w: "equality set" for w in ["grandfather", "grandmother"]},
}

print(f"{'word':14s} {'row (gender, cluster, within_a, within_b)':42s} role")
print("-" * 90)
for w in vocab:
    row = ", ".join(f"{c:+.3f}" for c in E[word_to_ix[w]])
    print(f"{w:14s} ({row})  {role[w]}")

import matplotlib.pyplot as plt import numpy as np from dlhub.embeddings import pca_project_2d from dlhub.embeddings.debiasing_word_embeddings import ( gender_direction, hard_debias, make_fixture, neutral_neighbors, weat_effect_size, weat_pvalue, weat_statistic, ) fixture = make_fixture() E = fixture["E"] # (V, d) = (14, 4), one word per row, unit-norm rows vocab = fixture["vocab"] word_to_ix = fixture["word_to_ix"] V, d = fixture["dims"]["V"], fixture["dims"]["d"] print(f"E has shape (V, d) = ({V}, {d}); one word per row; rows are unit-norm\n") role = { **{w: "definitional (male)" for w in ["he", "man", "boy"]}, **{w: "definitional (female)" for w in ["she", "woman", "girl"]}, **{w: "male-coded occupation" for w in ["engineer", "programmer", "mechanic"]}, **{w: "female-coded occupation" for w in ["nurse", "receptionist", "nanny"]}, **{w: "equality set" for w in ["grandfather", "grandmother"]}, } print(f"{'word':14s} {'row (gender, cluster, within_a, within_b)':42s} role") print("-" * 90) for w in vocab: row = ", ".join(f"{c:+.3f}" for c in E[word_to_ix[w]]) print(f"{w:14s} ({row}) {role[w]}")

E has shape (V, d) = (14, 4); one word per row; rows are unit-norm

word           row (gender, cluster, within_a, within_b)  role
------------------------------------------------------------------------------------------
boy            (+1.000, +0.000, +0.000, +0.000)  definitional (male)
engineer       (+0.229, +0.688, +0.688, +0.000)  male-coded occupation
girl           (-1.000, +0.000, +0.000, +0.000)  definitional (female)
grandfather    (+0.894, +0.000, +0.000, +0.447)  equality set
grandmother    (-0.707, +0.000, +0.000, +0.707)  equality set
he             (+1.000, +0.000, +0.000, +0.000)  definitional (male)
man            (+1.000, +0.000, +0.000, +0.000)  definitional (male)
mechanic       (+0.316, +0.949, +0.000, +0.000)  male-coded occupation
nanny          (-0.316, -0.949, +0.000, +0.000)  female-coded occupation
nurse          (-0.229, -0.688, +0.688, +0.000)  female-coded occupation
programmer     (+0.267, +0.802, +0.535, +0.000)  male-coded occupation
receptionist   (-0.267, -0.802, +0.535, +0.000)  female-coded occupation
she            (-1.000, +0.000, +0.000, +0.000)  definitional (female)
woman          (-1.000, +0.000, +0.000, +0.000)  definitional (female)

What to notice. The six definitional words are pure-gender archetypes — he, man, boy are all +e0; she, woman, girl are all -e0. The coincidence is deliberate: their only job is to define g and to serve as WEAT's gender attributes; nothing computes neighbours among them, so the coincidence keeps every known answer exact without affecting any result. The three male-coded occupations share gender = +1 and cluster = +3; the three female-coded ones share gender = -1 and cluster = -3. The equality set {grandfather, grandmother} is asymmetric on purpose: equalize is the operation that symmetrises it.

2. Recover the gender direction g¶

The standard recipe (Bolukbasi et al., 2016, §5.1): for each definitional pair (male, female), take the pair mean and the two deviations w - mean; stack all deviations and take the top principal component as g. The SVD fixes a direction only up to sign, so the sign is pinned male-positive — flip g if g . (e_he - e_she) < 0 for the first pair. A single direction (k = 1) suffices because the deviation matrix is exactly rank-1 on this fixture — its singular values are [sqrt(6), 0, 0, 0].

The function gender_direction takes row indices, not words, so the code below maps the named pairs to indices and calls it. The reference page shows the equation; we just call it.

In [2]:

pairs = [(word_to_ix[m], word_to_ix[f]) for m, f in fixture["definitional_pairs"]]
g = gender_direction(E, pairs)
print(f"g = {g}   (the first coordinate is the gender axis in the planted space)")

# The deviation matrix is rank-1 by construction -- the singular values make that
# visible. Re-derive the same SVD the function used, to show why k = 1 suffices.
deviations = []
for m, f in pairs:
    mu = (E[m] + E[f]) / 2.0
    deviations.append(E[m] - mu)
    deviations.append(E[f] - mu)
deviation_matrix = np.array(deviations)
svd_singular_values = np.linalg.svd(deviation_matrix, compute_uv=False)
print(f"\nsingular values of the stacked deviation matrix: {svd_singular_values}")
print(f"so the first right-singular vector captures the whole subspace (k = 1).")

pairs = [(word_to_ix[m], word_to_ix[f]) for m, f in fixture["definitional_pairs"]] g = gender_direction(E, pairs) print(f"g = {g} (the first coordinate is the gender axis in the planted space)") # The deviation matrix is rank-1 by construction -- the singular values make that # visible. Re-derive the same SVD the function used, to show why k = 1 suffices. deviations = [] for m, f in pairs: mu = (E[m] + E[f]) / 2.0 deviations.append(E[m] - mu) deviations.append(E[f] - mu) deviation_matrix = np.array(deviations) svd_singular_values = np.linalg.svd(deviation_matrix, compute_uv=False) print(f"\nsingular values of the stacked deviation matrix: {svd_singular_values}") print(f"so the first right-singular vector captures the whole subspace (k = 1).")

g = [1. 0. 0. 0.]   (the first coordinate is the gender axis in the planted space)

singular values of the stacked deviation matrix: [2.44948974 0.         0.         0.        ]
so the first right-singular vector captures the whole subspace (k = 1).

What to notice. g came out exactly as [1, 0, 0, 0] — the planted coordinate 0. On a fixture where the gender axis is the first coordinate, the top principal component of the definitional-pair deviations has to be +e0, and the male-positive sign convention picks it. The singular values [sqrt(6), 0, 0, 0] are the reason a single direction captures the whole subspace: there is no second non-zero direction to capture. On a real embedding the spectrum would have a long tail and Bolukbasi's full method takes the top k = 10; gender_direction returns the k = 1 unit vector and the dossier accepts that on this fixture, where one is the whole answer.

3. The direct bias: projections on g and WEAT¶

Now that we have g, measure the bias two ways.

Direct bias is the projection of a word's vector on g. A positive number means "leans male", a negative one "leans female", a zero one "no direct gender component". Plot it for every word, coloured by the original gender of the role the word plays — definitional (the pure-gender archetypes), male-coded occupation, female-coded occupation, equality set. The colour is the original role, not a measurement; it tells you whether the role predicts the sign of the projection.

In [3]:

from matplotlib.patches import Patch


# Gender colouring by ORIGINAL role (not by what the metric would say): blue
# for male-coded, red for female-coded, dark grey for the equality set.
def original_gender_color(word: str) -> str:
    """Hex colour for a word by its ORIGINAL role: blue if male-coded, red if female-coded."""
    if word in {
        "he",
        "man",
        "boy",
        "engineer",
        "programmer",
        "mechanic",
        "grandfather",
    }:
        return "#1f77b4"  # blue
    return "#d62728"  # red


def original_gender_label(word: str) -> str:
    """Colour label for a word by its ORIGINAL role: "blue" or "red"."""
    return (
        "blue"
        if word
        in {"he", "man", "boy", "engineer", "programmer", "mechanic", "grandfather"}
        else "red"
    )


projections = np.array([E[word_to_ix[w]] @ g for w in vocab])

fig, ax = plt.subplots(figsize=(8.0, 4.6))
colors = [original_gender_color(w) for w in vocab]
ax.bar(range(V), projections, color=colors, edgecolor="black", linewidth=0.4)
ax.axhline(0.0, color="black", lw=0.6)
ax.set_xticks(range(V))
ax.set_xticklabels(vocab, rotation=45, ha="right")
ax.set_ylabel(r"projection on $g$  (w $\cdot$ g)")
ax.set_xlabel("word (ordered as in vocab; colour = original role)")
ax.set_title(
    "Direct bias before debiasing: the gender-direction projection of every word"
)
ax.legend(
    handles=[
        Patch(facecolor="#1f77b4", label="male-coded role"),
        Patch(facecolor="#d62728", label="female-coded role"),
    ],
    loc="upper right",
)
fig.tight_layout()
plt.show()

print("\nprojection on g  (before hard_debias):")
for w in vocab:
    print(
        f"  {w:14s}  {projections[word_to_ix[w]]:+.5f}   ({original_gender_label(w)})"
    )

from matplotlib.patches import Patch # Gender colouring by ORIGINAL role (not by what the metric would say): blue # for male-coded, red for female-coded, dark grey for the equality set. def original_gender_color(word: str) -> str: """Hex colour for a word by its ORIGINAL role: blue if male-coded, red if female-coded.""" if word in { "he", "man", "boy", "engineer", "programmer", "mechanic", "grandfather", }: return "#1f77b4" # blue return "#d62728" # red def original_gender_label(word: str) -> str: """Colour label for a word by its ORIGINAL role: "blue" or "red".""" return ( "blue" if word in {"he", "man", "boy", "engineer", "programmer", "mechanic", "grandfather"} else "red" ) projections = np.array([E[word_to_ix[w]] @ g for w in vocab]) fig, ax = plt.subplots(figsize=(8.0, 4.6)) colors = [original_gender_color(w) for w in vocab] ax.bar(range(V), projections, color=colors, edgecolor="black", linewidth=0.4) ax.axhline(0.0, color="black", lw=0.6) ax.set_xticks(range(V)) ax.set_xticklabels(vocab, rotation=45, ha="right") ax.set_ylabel(r"projection on $g$ (w $\cdot$ g)") ax.set_xlabel("word (ordered as in vocab; colour = original role)") ax.set_title( "Direct bias before debiasing: the gender-direction projection of every word" ) ax.legend( handles=[ Patch(facecolor="#1f77b4", label="male-coded role"), Patch(facecolor="#d62728", label="female-coded role"), ], loc="upper right", ) fig.tight_layout() plt.show() print("\nprojection on g (before hard_debias):") for w in vocab: print( f" {w:14s} {projections[word_to_ix[w]]:+.5f} ({original_gender_label(w)})" )

projection on g  (before hard_debias):
  boy             +1.00000   (blue)
  engineer        +0.22942   (blue)
  girl            -1.00000   (red)
  grandfather     +0.89443   (blue)
  grandmother     -0.70711   (red)
  he              +1.00000   (blue)
  man             +1.00000   (blue)
  mechanic        +0.31623   (blue)
  nanny           -0.31623   (red)
  nurse           -0.22942   (red)
  programmer      +0.26726   (blue)
  receptionist    -0.26726   (red)
  she             -1.00000   (red)
  woman           -1.00000   (red)

What to notice. The six definitional words project to +1 or -1 exactly — they are the pure-gender archetypes and define the axis. The three male-coded occupations project to roughly +0.23, +0.27, +0.32 (the exact values are 1/√19, 1/√14, 1/√10); the three female-coded occupations are the exact negatives. The equality-set pair grandfather and grandmother project to +0.894 and -0.707 — asymmetric on purpose, so that equalize has something to symmetrise.

WEAT (Caliskan, Bryson & Narayanan, 2017) measures the same bias in a way that does not assume the direction is one-dimensional. The fixture's WEAT sets are A = {he, man, boy} (male attributes), B = {she, woman, girl} (female attributes), X = {engineer, programmer, mechanic} (male-coded targets), Y = {nurse, receptionist, nanny} (female-coded targets). The effect size d is the gap between the target means in units of the pooled standard deviation of the per-word association scores; the exact permutation p-value enumerates all C(6, 3) = 20 equal-size partitions of the pooled targets and returns the fraction whose statistic is at least the observed. With only 20 partitions the test is exact, not sampled, and the observed partition always counts itself so p ≥ 1/20 = 0.05.

In [4]:

import itertools

weat = fixture["weat_sets"]
A = E[[word_to_ix[w] for w in weat["A"]]]
B = E[[word_to_ix[w] for w in weat["B"]]]
X = E[[word_to_ix[w] for w in weat["X"]]]
Y = E[[word_to_ix[w] for w in weat["Y"]]]

stat_before = weat_statistic(X, Y, A, B)
d_before = weat_effect_size(X, Y, A, B)
p_before = weat_pvalue(X, Y, A, B)
print(f"BEFORE hard_debias")
print(f"  statistic = {stat_before:.10f}")
print(f"  d         = {d_before:.10f}")
print(f"  p         = {p_before}  (exact over C(6, 3) = 20 partitions)")

# The exact p of 0.05 means the observed split uniquely *maximises* the
# statistic. Confirm by checking how many partitions match or exceed it.
pool = np.vstack([X, Y])
n, k = pool.shape[0], X.shape[0]
at_least = sum(
    1
    for chosen in itertools.combinations(range(n), k)
    if weat_statistic(
        pool[list(chosen)],
        pool[[j for j in range(n) if j not in set(chosen)]],
        A,
        B,
    )
    >= stat_before
)
print(
    f"\n  partitions with statistic >= observed: {at_least} / {sum(1 for _ in itertools.combinations(range(n), k))}"
)

import itertools weat = fixture["weat_sets"] A = E[[word_to_ix[w] for w in weat["A"]]] B = E[[word_to_ix[w] for w in weat["B"]]] X = E[[word_to_ix[w] for w in weat["X"]]] Y = E[[word_to_ix[w] for w in weat["Y"]]] stat_before = weat_statistic(X, Y, A, B) d_before = weat_effect_size(X, Y, A, B) p_before = weat_pvalue(X, Y, A, B) print(f"BEFORE hard_debias") print(f" statistic = {stat_before:.10f}") print(f" d = {d_before:.10f}") print(f" p = {p_before} (exact over C(6, 3) = 20 partitions)") # The exact p of 0.05 means the observed split uniquely *maximises* the # statistic. Confirm by checking how many partitions match or exceed it. pool = np.vstack([X, Y]) n, k = pool.shape[0], X.shape[0] at_least = sum( 1 for chosen in itertools.combinations(range(n), k) if weat_statistic( pool[list(chosen)], pool[[j for j in range(n) if j not in set(chosen)]], A, B, ) >= stat_before ) print( f"\n partitions with statistic >= observed: {at_least} / {sum(1 for _ in itertools.combinations(range(n), k))}" )

BEFORE hard_debias
  statistic = 3.2516189672
  d         = 1.9830182615
  p         = 0.05  (exact over C(6, 3) = 20 partitions)

  partitions with statistic >= observed: 1 / 20

What to notice. The effect size is d ≈ 1.98 — about two pooled standard deviations between the two target groups, a large effect by Cohen's loose rule of thumb. The exact permutation p = 0.05 is the minimum possible value (the observed split is the unique maximiser of the statistic over all 20 equal-size partitions), so a frequentist reading would call this very strong evidence that the male-vs-female-occupations split is the most gender-loaded one. The 1-of-20 count above is just the permutation test rendering that fact: of 20 ways to choose which three of the six occupation vectors are "the male-coded ones", only the planted split puts them on the same side of the male-vs-female attribute line.

This is the bias we are about to remove. Hold this number.

4. Apply hard_debias: neutralize the occupations, equalize the equality set¶

hard_debias is the composition of two operations. neutralize removes the g component from a gender-neutral word and renormalizes: w' = (w - (w.g) g) / ||w - (w.g) g||. After this, w' . g = 0 and w' is back on the unit sphere. equalize makes an equality-set pair symmetric about g — each member ends up with the same neutral part and equal-and-opposite g components, so all are equidistant from any neutralized word. The two operations are applied to disjoint sets: hard_debias raises if they overlap, because the two operations would disagree on a shared word.

The reference page derives the equalize formula from the unit-sphere constraint; here we just call the canonical function. The definitional words are left untouched — they defined g and neutralize would annihilate the pure-gender archetypes (the zero-residual guard in neutralize would return the zero vector for them).

In [5]:

neutral_indices = [word_to_ix[w] for w in fixture["neutral_words"]]
equality_indices = [[word_to_ix[w] for w in s] for s in fixture["equality_sets"]]

print(f"neutralize ({len(neutral_indices)} words): {fixture['neutral_words']}")
print(
    f"equalize   ({sum(len(s) for s in equality_indices)} words): {fixture['equality_sets']}"
)

debiased = hard_debias(E, g, neutral_indices, equality_indices)
print(f"\nhard_debias returned a copy: {debiased is not E}")
print(f"shape preserved: {debiased.shape}")

neutral_indices = [word_to_ix[w] for w in fixture["neutral_words"]] equality_indices = [[word_to_ix[w] for w in s] for s in fixture["equality_sets"]] print(f"neutralize ({len(neutral_indices)} words): {fixture['neutral_words']}") print( f"equalize ({sum(len(s) for s in equality_indices)} words): {fixture['equality_sets']}" ) debiased = hard_debias(E, g, neutral_indices, equality_indices) print(f"\nhard_debias returned a copy: {debiased is not E}") print(f"shape preserved: {debiased.shape}")

neutralize (6 words): ['engineer', 'programmer', 'mechanic', 'nurse', 'receptionist', 'nanny']
equalize   (2 words): [['grandfather', 'grandmother']]

hard_debias returned a copy: True
shape preserved: (14, 4)

5. "The metric says fixed"¶

Now read the same two measurements on the debiased matrix. Projections on g for the neutral words go to 0 by construction; the equality set lands at the shared magnitude ±sqrt(1 - ||nu||^2) ≈ ±0.81663, symmetric about g. The WEAT effect size is 0 — the post-debias d = 0/0 case the dossier's zero-spread guard was written for. The exact permutation p = 1.0 — every one of the 20 partitions ties the observed statistic, so none exceeds it. By the metric, the bias is gone.

In [6]:

projections_after = np.array([debiased[word_to_ix[w]] @ g for w in vocab])

fig, axes = plt.subplots(1, 2, figsize=(12.0, 4.6), sharey=True)
for ax, values, title in [
    (axes[0], projections, "before hard_debias"),
    (axes[1], projections_after, "after hard_debias"),
]:
    ax.bar(
        range(V),
        values,
        color=[original_gender_color(w) for w in vocab],
        edgecolor="black",
        linewidth=0.4,
    )
    ax.axhline(0.0, color="black", lw=0.6)
    ax.set_xticks(range(V))
    ax.set_xticklabels(vocab, rotation=45, ha="right")
    ax.set_title(title)
    ax.set_xlabel("word")
axes[0].set_ylabel(r"projection on $g$  (w $\cdot$ g)")
fig.suptitle(
    "Projections on the gender direction: every neutral word is 0 after debiasing"
)
fig.tight_layout()
plt.show()

X_after = debiased[[word_to_ix[w] for w in weat["X"]]]
Y_after = debiased[[word_to_ix[w] for w in weat["Y"]]]

stat_after = weat_statistic(X_after, Y_after, A, B)
d_after = weat_effect_size(X_after, Y_after, A, B)
p_after = weat_pvalue(X_after, Y_after, A, B)
print(f"AFTER hard_debias")
print(f"  statistic = {stat_after:.10f}")
print(f"  d         = {d_after:.10f}   (zero-spread guard: every pooled score is 0)")
print(f"  p         = {p_after}        (every permutation ties the observed statistic)")

fig, ax = plt.subplots(figsize=(5.6, 4.0))
ax.bar(
    ["before", "after"],
    [d_before, d_after],
    color=["#9aa0a6", "#2ca02c"],
    edgecolor="black",
    linewidth=0.4,
)
ax.axhline(0.0, color="black", lw=0.6)
ax.set_ylabel("WEAT effect size  d")
ax.set_xlabel("matrix")
ax.set_title(
    "WEAT effect size $d$ before vs after hard_debias  --  the metric says fixed"
)
for x, v in enumerate([d_before, d_after]):
    ax.text(x, v + 0.05 if v >= 0 else v - 0.10, f"{v:.4f}", ha="center")
ax.set_ylim(-0.2, max(d_before, 0.0) + 0.4)
fig.tight_layout()
plt.show()

projections_after = np.array([debiased[word_to_ix[w]] @ g for w in vocab]) fig, axes = plt.subplots(1, 2, figsize=(12.0, 4.6), sharey=True) for ax, values, title in [ (axes[0], projections, "before hard_debias"), (axes[1], projections_after, "after hard_debias"), ]: ax.bar( range(V), values, color=[original_gender_color(w) for w in vocab], edgecolor="black", linewidth=0.4, ) ax.axhline(0.0, color="black", lw=0.6) ax.set_xticks(range(V)) ax.set_xticklabels(vocab, rotation=45, ha="right") ax.set_title(title) ax.set_xlabel("word") axes[0].set_ylabel(r"projection on $g$ (w $\cdot$ g)") fig.suptitle( "Projections on the gender direction: every neutral word is 0 after debiasing" ) fig.tight_layout() plt.show() X_after = debiased[[word_to_ix[w] for w in weat["X"]]] Y_after = debiased[[word_to_ix[w] for w in weat["Y"]]] stat_after = weat_statistic(X_after, Y_after, A, B) d_after = weat_effect_size(X_after, Y_after, A, B) p_after = weat_pvalue(X_after, Y_after, A, B) print(f"AFTER hard_debias") print(f" statistic = {stat_after:.10f}") print(f" d = {d_after:.10f} (zero-spread guard: every pooled score is 0)") print(f" p = {p_after} (every permutation ties the observed statistic)") fig, ax = plt.subplots(figsize=(5.6, 4.0)) ax.bar( ["before", "after"], [d_before, d_after], color=["#9aa0a6", "#2ca02c"], edgecolor="black", linewidth=0.4, ) ax.axhline(0.0, color="black", lw=0.6) ax.set_ylabel("WEAT effect size d") ax.set_xlabel("matrix") ax.set_title( "WEAT effect size $d$ before vs after hard_debias -- the metric says fixed" ) for x, v in enumerate([d_before, d_after]): ax.text(x, v + 0.05 if v >= 0 else v - 0.10, f"{v:.4f}", ha="center") ax.set_ylim(-0.2, max(d_before, 0.0) + 0.4) fig.tight_layout() plt.show()

AFTER hard_debias
  statistic = 0.0000000000
  d         = 0.0000000000   (zero-spread guard: every pooled score is 0)
  p         = 1.0        (every permutation ties the observed statistic)

What to notice. The left panel is the bias we started with. The right panel is the same projection after hard_debias: every neutral occupation has projection 0 on g, and the equality set is now exactly symmetric about g at ±0.81663. The definitional words are unchanged — they defined g and the debiaser is not allowed to touch them. The WEAT d has fallen from 1.98 to 0 exactly, and p from 0.05 to 1.0. The two metrics a bias-audit relies on — direct projection and WEAT — both say the bias is gone.

Predict first. If the direct gender component is gone and WEAT reads zero, what should the neutral-neighbour map look like on the debiased matrix? Specifically: for every neutral word, who is its nearest neighbour within the neutral set? Before revealing, think about it — would you expect the male-coded occupations to keep grouping together (and the female-coded ones separately), or to mix freely now that their g component is gone? Write down your answer before running the next cell.

6. The twist: the neighbour structure survives¶

Reveal. The neutral-neighbour map is computed inside the neutral sub-matrix alone (the six occupations) — the definitional words and the equality set are excluded, so the bias the debiaser erased from g is the only thing that could change the answer. nearest_neighbors is the head's function, ranking by cosine with self excluded and ties broken by ascending index — the same ranking rule the head notebook uses.

In [7]:

map_before = neutral_neighbors(E, neutral_indices)
map_after = neutral_neighbors(debiased, neutral_indices)
print(
    f"{'word':14s}  before  ->  after   (within the neutral set, cosine, self excluded)"
)
print("-" * 60)
for row in neutral_indices:
    w = vocab[row]
    print(
        f"{w:14s}  {vocab[map_before[row]]:8s}  {vocab[map_after[row]]:8s}    {'(same)' if map_before[row] == map_after[row] else '(CHANGED)'}"
    )

identical = map_before == map_after
print(f"\nthe map is identical before vs after hard_debias: {identical}")

map_before = neutral_neighbors(E, neutral_indices) map_after = neutral_neighbors(debiased, neutral_indices) print( f"{'word':14s} before -> after (within the neutral set, cosine, self excluded)" ) print("-" * 60) for row in neutral_indices: w = vocab[row] print( f"{w:14s} {vocab[map_before[row]]:8s} {vocab[map_after[row]]:8s} {'(same)' if map_before[row] == map_after[row] else '(CHANGED)'}" ) identical = map_before == map_after print(f"\nthe map is identical before vs after hard_debias: {identical}")

word            before  ->  after   (within the neutral set, cosine, self excluded)
------------------------------------------------------------
engineer        programmer  programmer    (same)
programmer      engineer  engineer    (same)
mechanic        programmer  programmer    (same)
nurse           receptionist  receptionist    (same)
receptionist    nurse     nurse       (same)
nanny           receptionist  receptionist    (same)

the map is identical before vs after hard_debias: True

What to notice. The map is byte-for-byte identical before and after hard_debias. engineer's nearest neighbour is still programmer; mechanic still points at programmer; nurse ↔ receptionist and nanny → receptionist. The two gender-coded groups — {engineer, programmer, mechanic} and {nurse, receptionist, nanny} — never mixed. neutralize removed the direct projection on g, but cosine similarity is dominated by the other coordinates: the male-coded occupations share cluster = +3, the female-coded ones share cluster = -3, and the within-cluster coordinates are too small to overturn that. The neighbour structure that encoded gender indirectly is intact.

That is the Gonen–Goldberg headline on a 14-word fixture: the metric is fixed, the bias is not. We can also see the surviving structure by projecting to 2-D.

7. See the surviving clusters: 2-D PCA of the debiased matrix¶

pca_project_2d is the head's PCA — compute the SVD of the centered (V, d) matrix, take the top-2 right singular vectors, project, and fix the sign by convention. Pass the debiased matrix transposed, because the head's function takes the head's (d, V) layout. The plot below is the debiased space. Every word is coloured by its original role (blue = male-coded, red = female-coded) — the same colour key as the projection bars above. If the bias is really gone, the two colours should be mixed at random. They are not.

In [8]:

from matplotlib.lines import Line2D

projection = pca_project_2d(debiased.T)
coords = projection.coords  # (V, 2): one 2-D point per word, on the debiased matrix

fig, ax = plt.subplots(figsize=(7.0, 5.6))
ax.axhline(0.0, color="grey", lw=0.6)
ax.axvline(0.0, color="grey", lw=0.6)

# Two markers: the equality set and the gendered archetypes use a different glyph
# so the plot distinguishes "untouched definitional" from "neutralized/equalized".
shape_for = {}
for w in vocab:
    if w in {"he", "she", "man", "woman", "boy", "girl"}:
        shape_for[w] = "D"  # definitional, untouched
    elif w in {"grandfather", "grandmother"}:
        shape_for[w] = "s"  # equality set, equalized
    else:
        shape_for[w] = "o"  # neutralized occupation

for w, (x, y) in zip(vocab, coords):
    ax.scatter(
        x,
        y,
        s=85,
        c=original_gender_color(w),
        marker=shape_for[w],
        edgecolor="black",
        linewidth=0.5,
        zorder=3,
    )
    ax.annotate(w, (x, y), textcoords="offset points", xytext=(6, 5), fontsize=10)

legend_handles = [
    Line2D(
        [0],
        [0],
        marker="o",
        color="w",
        markerfacecolor="#1f77b4",
        markeredgecolor="black",
        markersize=9,
        label="male-coded (neutralized)",
    ),
    Line2D(
        [0],
        [0],
        marker="o",
        color="w",
        markerfacecolor="#d62728",
        markeredgecolor="black",
        markersize=9,
        label="female-coded (neutralized)",
    ),
    Line2D(
        [0],
        [0],
        marker="s",
        color="w",
        markerfacecolor="grey",
        markeredgecolor="black",
        markersize=9,
        label="equality set (equalized)",
    ),
    Line2D(
        [0],
        [0],
        marker="D",
        color="w",
        markerfacecolor="grey",
        markeredgecolor="black",
        markersize=9,
        label="definitional (untouched)",
    ),
]
ax.legend(handles=legend_handles, loc="lower left", fontsize=9, framealpha=0.95)
ax.set_xlabel("PC1   (the principal axis of remaining variance in the debiased space)")
ax.set_ylabel("PC2")
ax.set_title(
    "PCA of the DEBIASED matrix, coloured by ORIGINAL role -- two clusters persist"
)
fig.tight_layout()
plt.show()

from matplotlib.lines import Line2D projection = pca_project_2d(debiased.T) coords = projection.coords # (V, 2): one 2-D point per word, on the debiased matrix fig, ax = plt.subplots(figsize=(7.0, 5.6)) ax.axhline(0.0, color="grey", lw=0.6) ax.axvline(0.0, color="grey", lw=0.6) # Two markers: the equality set and the gendered archetypes use a different glyph # so the plot distinguishes "untouched definitional" from "neutralized/equalized". shape_for = {} for w in vocab: if w in {"he", "she", "man", "woman", "boy", "girl"}: shape_for[w] = "D" # definitional, untouched elif w in {"grandfather", "grandmother"}: shape_for[w] = "s" # equality set, equalized else: shape_for[w] = "o" # neutralized occupation for w, (x, y) in zip(vocab, coords): ax.scatter( x, y, s=85, c=original_gender_color(w), marker=shape_for[w], edgecolor="black", linewidth=0.5, zorder=3, ) ax.annotate(w, (x, y), textcoords="offset points", xytext=(6, 5), fontsize=10) legend_handles = [ Line2D( [0], [0], marker="o", color="w", markerfacecolor="#1f77b4", markeredgecolor="black", markersize=9, label="male-coded (neutralized)", ), Line2D( [0], [0], marker="o", color="w", markerfacecolor="#d62728", markeredgecolor="black", markersize=9, label="female-coded (neutralized)", ), Line2D( [0], [0], marker="s", color="w", markerfacecolor="grey", markeredgecolor="black", markersize=9, label="equality set (equalized)", ), Line2D( [0], [0], marker="D", color="w", markerfacecolor="grey", markeredgecolor="black", markersize=9, label="definitional (untouched)", ), ] ax.legend(handles=legend_handles, loc="lower left", fontsize=9, framealpha=0.95) ax.set_xlabel("PC1 (the principal axis of remaining variance in the debiased space)") ax.set_ylabel("PC2") ax.set_title( "PCA of the DEBIASED matrix, coloured by ORIGINAL role -- two clusters persist" ) fig.tight_layout() plt.show()

What to notice. The clusters are right where they were. Every male-coded occupation sits at the top of the plot (positive PC2) and every female-coded occupation at the bottom (negative PC2); on PC1 the neutralized occupations all sit at 0, by construction, because the g component has been projected out. The colour is the only place "gender" appears in this plot — and it is a colouring, not a measurement, copied from the original role. The fact that the colours line up means the surviving structure is the original gender division, re-routed through the indirect cluster axis that hard_debias did not touch.

The definitional words (diamonds) and the equalized pair (squares) are deliberately separate from the occupations in this projection: the definitional words are pure-gender archetypes, untouched by debiasing, and the equalize operation built a symmetric pair that does not collapse onto the occupations. In a real embedding the result would be less geometrically clean — there is no planted "cluster" axis, the gender direction is not aligned with a coordinate, and PCA sees only whatever bias the embedding has accumulated. The structural point, though, is the same: the neighbour grouping survives the projection-out of g, and a 2-D view of a debiased space still shows the two groups as neighbours.

8. What this means, and what it does not¶

What it means. Gender bias in a static embedding is a direction in vector space, and gender_direction recovers it as the top principal component of the definitional-pair deviations. Two measurements of the bias agree: the direct projection on g and the WEAT effect size d (with an exact enumerated permutation p on a 6-target, 3-vs-3 fixture). hard_debias then projects out that direction from every neutral word and symmetrises every equality pair — after which the projections are 0, d = 0, and p = 1.0. The metric says fixed.

And the metric is wrong. The neutral-neighbour map is byte-for-byte identical before and after hard_debias; the PCA scatter of the debiased matrix still shows the male and female occupation clusters, separated along the indirect cluster axis the debiaser never touched. That is Gonen & Goldberg's "lipstick on a pig" headline in 14 words and 4 dimensions: geometric debiasing erases the projection, not the structure. Bias that is encoded as a cluster does not need a g component to survive a g removal.

What it does not. Three honest limits, each visible above.

• A k = 1 direction is the fixture's gift, not the method's promise. On real embeddings the deviation matrix has a long tail of singular values, and Bolukbasi's full method takes the top k = 10. The reference page's notes call this out; the function here returns the k = 1 unit vector because one direction is the whole subspace on this fixture.
• This is one bias axis. The fixture has a single planted gender direction. A real embedding's bias is multi-dimensional and intersectional: race, age, nationality, and so on. Manzini et al. (2019) generalise hard debiasing to multiclass, and that is named in the explanation as "where this goes"; it is not implemented here because the hub forbids extra machinery on a planted set.
• Removing a direction does not remove the harm. A downstream classifier can still pick up the indirect clustering — Gonen & Goldberg trained a classifier on debiased vectors and recovered the original bias almost perfectly. The neighbour map above is the same point made without a classifier: the structure is still there for a model to find. The real fixes are learning-time (GN-GloVe, Zhao et al., 2018) or contextual models (BERT / ELMo) where the representation of a word depends on the sentence it sits in.

Key Takeaways¶

1. Gender bias is a direction. gender_direction takes the top principal component of the stacked definitional-pair deviations and pins its sign male-positive. On this fixture the answer is exactly +e0 and the singular values show why one direction is the whole subspace.
2. The bias is measurable two ways. Direct projection on g (a number per word) and WEAT effect size d with exact permutation p (a single number per split). On the fixture d ≈ 1.98 and p = 0.05 — the observed split uniquely maximises the statistic over all 20 partitions.
3. hard_debias projects out g and symmetrises the equality sets. neutralize removes the g component and renormalizes; equalize gives every member of an equality pair the same neutral part and opposite g components, so the pair stays distinct but unbiased.
4. The metric says fixed. After hard_debias the neutral projections are 0, d = 0 (the zero-spread guard), and p = 1.0. By every projection-style metric the bias is gone.
5. The bias is not fixed. The neutral-neighbour map is identical before and after hard_debias; the PCA scatter of the debiased matrix still shows the male-coded and female-coded occupation clusters. Geometric debiasing removes the projection, not the structure — the Gonen–Goldberg "Lipstick on a Pig" headline, demonstrated in 14 words and 4 dimensions.
6. This notebook ran canonical code only. Every direction identification, every neutralize, every equalize, every WEAT statistic, every PCA projection, and every neighbour ranking was imported from dlhub.embeddings.debiasing_word_embeddings and dlhub.embeddings; the lesson is a view onto the reference, not a second copy of it.

One sentence to keep. A gender direction in a static embedding is real, can be removed by a linear projection, and the resulting "debiased" vectors still group by gender under their nearest neighbours — bias is a structure, not a coordinate, and a projection removes the latter, not the former.

Back to the family. The word embeddings notebook treated the gender coordinate as a bookkeeping convenience and refused to make a bias claim; the word2vec-and-glove notebook trained a small embedding with a deliberately bias-free corpus; this notebook inherits the planted space from the head and studies only the bias geometry on top of it.