Word2Vec and GloVe: watch word vectors get learned from a corpus¶
Purpose. The word embeddings notebook took the vectors as given — a hand-planted matrix — and read meaning off their geometry. It left one honest gap: nothing in it put the structure there. This notebook closes that gap. You will start from nothing but a stream of words, build the co-occurrence counts a window sees, and then train two tables with skip-gram negative sampling until the planted family structure reappears in the learned geometry. Then you will do the same with GloVe and compare.
Prerequisites. What a word vector is and what you read off one — cosine similarity, nearest neighbours, the analogy rule — all from the head topic. Gradient descent on a scalar loss. No prior NLP beyond that. Every objective, gradient, and sampler this notebook uses is imported from the hub's canonical dlhub.embeddings.word2vec_and_glove; the notebook defines no algorithm of its own, so the lesson cannot drift from the code the reference documents and the tests check.
What you will be able to do by the end.
- Turn a list of sentences into the windowed co-occurrence matrix $X$ and see its block-diagonal, within-family structure.
- Read the (shifted) PMI matrix off $X$ and name what skip-gram is secretly factorising (Levy & Goldberg).
- Train skip-gram with negative sampling and confirm the loss descends monotonically and the run is reproducible to the bit.
- See the learned neighbours recover the planted families —
cat's nearest words are the other pets. - Predict and then measure how the
3/4sampler exponent reshapes the negative distribution. - Run GloVe on the same counts and see why "fits the counts" and "recovers the neighbourhoods" are not the same thing on a tiny corpus.
What this notebook is not. It does not reimplement Word2Vec or GloVe — train_sgns, train_glove, the sampler, and nearest_neighbor are all imported. It is also not a production trainer: the corpus is a 36-token hand-planted toy, chosen so every count is known and every claim is checkable. How to train on your corpus is the how-to; the equations are in the reference; the why is in the explanation.
Runtime. About 20 seconds end to end from a clean kernel on a laptop CPU — the training itself is only a few seconds (two short skip-gram runs and one GloVe run over 72 training pairs, 200 epochs each); the rest is imports and drawing the figures. Seed. Everything is seeded: train_sgns(seed=0) drives initialisation and negative sampling from one generator, so a run is byte-identical across calls, and np.random.seed(0) is set below for any stray draw.
Where this fits¶
| Reader wants to | Go to |
|---|---|
| Learn it step by step | This notebook — you are here |
| Do it in a project | Train word vectors on your own corpus |
| Look up an equation or rule | Word2Vec and GloVe (reference) |
| Understand why it works | Word2Vec and GloVe (explanation) |
| Learn by running it | This notebook — you are here |
1. Orient: a corpus, a window, and a block-diagonal count matrix¶
Nothing is trained yet. First meet the raw material. The fixture is a planted toy corpus — 12 three-word sentences (36 tokens) over a vocabulary of $V = 16$, arranged as four semantic families of four words each: pets, food, actions, places. Every sentence stays inside one family, so when a size-2 window slides over a sentence it never leaves the family. That single design choice is the whole plant: co-occurrence will be dense within a family and exactly zero across families, and that is the structure training has to recover.
The corpus is a literal committed inside make_fixture — never downloaded — so every frequency and count below is known by construction.
import time
import matplotlib.pyplot as plt
import numpy as np
from dlhub.embeddings import analogy, cosine_similarity
from dlhub.embeddings.word2vec_and_glove import (
cooccurrence_matrix,
make_fixture,
nearest_neighbor,
train_glove,
train_sgns,
unigram_probabilities,
)
np.random.seed(0) # training seeds its own generator; this pins any stray draw
fixture = make_fixture()
corpus = fixture["corpus"]
sentences = fixture["sentences"] # the same corpus, mapped to integer indices
vocab = fixture["vocab"] # the sorted vocabulary, so an index is fixed by the corpus
word_to_ix = fixture["word_to_ix"]
families = fixture["families"]
V = fixture["dims"]["V"]
print(f"{len(corpus)} sentences, {sum(len(s) for s in corpus)} tokens, V = {V} words\n")
for name, words in families.items():
print(f" {name:8s} {words}")
print("\nfirst three sentences:", corpus[:3])
12 sentences, 36 tokens, V = 16 words pets ['cat', 'dog', 'fish', 'bird'] food ['bread', 'meat', 'seed', 'corn'] actions ['chase', 'eat', 'feed', 'catch'] places ['home', 'yard', 'pond', 'field'] first three sentences: [['cat', 'dog', 'fish'], ['cat', 'dog', 'bird'], ['dog', 'fish', 'bird']]
# The window the objectives see is turned into counts by the canonical builder.
# X[i, j] = how many times word j falls inside word i's window, over the corpus.
X = cooccurrence_matrix(sentences, V) # (V, V) integer counts, window = 2
print(f"total (center, context) pairs: {int(X.sum())}\n")
pets = families["pets"]
print("co-occurrence block for the pets family (raw counts):")
print(" " + "".join(f"{w:>6}" for w in pets))
for r in pets:
i = word_to_ix[r]
print(f" {r:>5} " + "".join(f"{X[i, word_to_ix[c]]:6d}" for c in pets))
cross = X[word_to_ix["cat"], word_to_ix["bread"]]
print(f"\ncross-family count X[cat, bread] = {cross} (no sentence mixes families)")
total (center, context) pairs: 72
co-occurrence block for the pets family (raw counts):
cat dog fish bird
cat 0 2 1 1
dog 2 0 2 2
fish 1 2 0 1
bird 1 2 1 0
cross-family count X[cat, bread] = 0 (no sentence mixes families)
# Reorder rows/cols so the four families are contiguous, then the plant is visible.
family_order = [word_to_ix[w] for words in families.values() for w in words]
ordered_labels = [vocab[i] for i in family_order]
family_bounds = np.cumsum([len(w) for w in families.values()])[:-1] # 4, 8, 12
def draw_family_grid(ax):
"""Draw red separators and family-ordered word ticks on a (V, V) heatmap."""
for b in family_bounds:
ax.axhline(b - 0.5, color="crimson", lw=1.0)
ax.axvline(b - 0.5, color="crimson", lw=1.0)
ax.set_xticks(range(V))
ax.set_xticklabels(ordered_labels, rotation=90, fontsize=8)
ax.set_yticks(range(V))
ax.set_yticklabels(ordered_labels, fontsize=8)
X_ordered = X[np.ix_(family_order, family_order)]
fig, ax = plt.subplots(figsize=(6.6, 5.8))
im = ax.imshow(X_ordered, cmap="Blues")
draw_family_grid(ax)
ax.set_xlabel("context word")
ax.set_ylabel("center word")
ax.set_title("Co-occurrence counts $X$, words grouped by family")
fig.colorbar(im, ax=ax, label="count $X_{ij}$")
fig.tight_layout()
plt.show()
What to notice. The counts form four blocks down the diagonal and nothing anywhere else — the red lines mark the family boundaries. Within the pets block, the second word dog is the family hub: it co-occurs twice with each of cat, fish, bird, while cat co-occurs twice with dog but only once each with fish and bird. That asymmetry is why cat's single clearest neighbour is dog. Every off-block entry is zero because the window never crosses a sentence, and no sentence mixes families. This block-diagonal matrix is the only thing training gets to see.
2. The structure is already in the counts: the PMI matrix¶
Before training anything, it is worth asking what the best a method could do with these counts even looks like. Levy & Goldberg (2014) answer it exactly: skip-gram with negative sampling, at its optimum, drives each inner product $v_w \cdot u_c$ to the pair's pointwise mutual information shifted by $\log K$,
$$v_w \cdot u_c \;\to\; \mathrm{PMI}(w, c) - \log K, \qquad \mathrm{PMI}(w, c) = \log\frac{\#(w, c)\,|D|}{\#(w)\,\#(c)}.$$
So skip-gram is implicitly factorising a PMI matrix built from exactly the counts above — "predict, don't count" turns out to be counting after all. We are not asserting that identity here (the full derivation is in the explanation); we are just showing the target. Build PMI from $X$ and look at it. This is a demonstration quantity — the canonical module has no PMI function because training never forms this matrix.
# A short derived demonstration (not a module algorithm): PMI straight from X.
# #(w, c) = X[w, c]; #(w) = row sum; #(c) = column sum; |D| = total pairs.
D = X.sum()
hashtag_w = X.sum(axis=1, keepdims=True) # (V, 1) center totals
hashtag_c = X.sum(axis=0, keepdims=True) # (1, V) context totals
with np.errstate(divide="ignore"): # the cross-family zeros send PMI to -inf
pmi = np.log(X * D / (hashtag_w * hashtag_c))
print("PMI(center, context) for the pets family (nats):")
print(" " + "".join(f"{w:>7}" for w in pets))
for r in pets:
i = word_to_ix[r]
row = []
for c in pets:
value = pmi[i, word_to_ix[c]]
row.append(" -inf" if np.isneginf(value) else f"{value:7.2f}")
print(f" {r:>5} " + "".join(row))
print(
f"\ncross-family PMI, e.g. PMI(cat, bread) = {pmi[word_to_ix['cat'], word_to_ix['bread']]}"
)
PMI(center, context) for the pets family (nats):
cat dog fish bird
cat -inf 1.79 1.50 1.50
dog 1.79 -inf 1.79 1.79
fish 1.50 1.79 -inf 1.50
bird 1.50 1.79 1.50 -inf
cross-family PMI, e.g. PMI(cat, bread) = -inf
pmi_ordered = pmi[np.ix_(family_order, family_order)]
masked = np.ma.masked_invalid(pmi_ordered) # hide the -inf cells
cmap = plt.cm.viridis.copy()
cmap.set_bad("lightgrey")
fig, ax = plt.subplots(figsize=(6.6, 5.8))
im = ax.imshow(masked, cmap=cmap)
draw_family_grid(ax)
ax.set_xlabel("context word")
ax.set_ylabel("center word")
ax.set_title(r"Shifted-PMI target: finite within family, $-\infty$ (grey) across")
fig.colorbar(im, ax=ax, label="PMI (nats)")
fig.tight_layout()
plt.show()
What to notice. The PMI matrix has the same block-diagonal shape as the raw counts, but now it says something sharper: within a family the values are positive (the hub pairs reach $\log 6 \approx 1.79$, the rest $\log 4.5 \approx 1.50$), while every cross-family pair is $-\infty$ (grey) because those words never co-occur. This positive within-family block is precisely the target skip-gram will chase: push $v_w \cdot u_c$ up where PMI is positive, down where it is negative. If training works, the learned vectors should reproduce this grouping. Keep this picture in mind — we will compare the learned geometry against it.
3. Train skip-gram with negative sampling¶
Now the smallest meaningful run. train_sgns draws two $(V \times d)$ tables — a center table $W$ and a context table $C$ — from one seeded generator, then takes plain full-batch gradient-descent steps on the negative-sampling loss. For each of the 72 pairs it also draws $K = 2$ negatives from the smoothed unigram sampler once and reuses them every epoch, which fixes the objective and makes the loss a clean descent curve. The defaults are the fixture's settings — d=8, window=2, k=2, lr=0.05, epochs=200, seed=0 — the same ones the module's test pins.
start = time.perf_counter()
sgns = train_sgns(sentences, V) # d=8, window=2, k=2, lr=0.05, epochs=200, seed=0
elapsed = time.perf_counter() - start
W = sgns["W"] # the (V, d) center table -- these are the learned word vectors
history = sgns["history"]
print(f"trained SGNS over {int(X.sum())} pairs in {elapsed:.2f} s")
print(
f"loss: {history[0]:.1f} (epoch 0) -> {history[-1]:.1f} (epoch {len(history) - 1})"
)
monotone = all(history[i] > history[i + 1] for i in range(len(history) - 1))
print(f"strictly decreasing every epoch: {monotone}")
W_again = train_sgns(sentences, V)["W"]
print(f"a second seeded run is byte-identical: {np.array_equal(W, W_again)}")
trained SGNS over 72 pairs in 0.67 s loss: 149.7 (epoch 0) -> 39.8 (epoch 199) strictly decreasing every epoch: True
a second seeded run is byte-identical: True
fig, ax = plt.subplots(figsize=(6.4, 4.2))
ax.plot(range(len(history)), history, color="#1f77b4")
ax.set_xlabel("epoch")
ax.set_ylabel("total SGNS loss over all 72 pairs")
ax.set_title("Skip-gram negative-sampling loss (seed 0, 200 epochs)")
ax.grid(True, alpha=0.3)
fig.tight_layout()
plt.show()
What to notice. The loss falls monotonically — strictly down every epoch, not just on average — and flattens as the vectors settle. That smoothness is a direct consequence of fixing the negatives: because the same $K$ negatives are reused each epoch, the objective does not move under the model's feet, so full-batch descent on it cannot go uphill. (Classic Word2Vec resamples negatives every epoch and gets a noisier curve.) The byte-identical second run is the determinism the test depends on: one generator, one seed, a fixed draw order. A flat or rising curve would mean a broken gradient — but the module's finite-difference checks have already ruled that out, which is why a loss like this is trustworthy evidence that learning happened.
4. Inspect: the learned neighbours recover the families¶
A falling loss only says the model got good at its game. The real question is whether the vectors carry the planted structure. Read off the nearest neighbours of a few words — nearest_neighbor ranks every other word by cosine in the learned center table $W$, exactly the geometry the head topic defined, self excluded.
print("nearest neighbours by cosine in the learned center table W (self excluded):\n")
for query in ["cat", "home", "bread", "chase"]:
neighbours = nearest_neighbor(W, word_to_ix[query], k=3)
family_of = next(name for name, ws in families.items() if query in ws)
print(f" {query:>6} ({family_of:7s}) -> {[vocab[j] for j in neighbours]}")
nearest neighbours by cosine in the learned center table W (self excluded):
cat (pets ) -> ['dog', 'fish', 'bird']
home (places ) -> ['pond', 'yard', 'field']
bread (food ) -> ['meat', 'feed', 'seed']
chase (actions) -> ['feed', 'field', 'eat']
# Family-ordered cosine heatmap of the learned vectors -- compare to X and PMI.
cos_W = np.array([[cosine_similarity(W[i], W[j]) for j in range(V)] for i in range(V)])
cos_ordered = cos_W[np.ix_(family_order, family_order)]
fig, ax = plt.subplots(figsize=(6.6, 5.8))
im = ax.imshow(cos_ordered, cmap="RdBu_r", vmin=-1, vmax=1)
draw_family_grid(ax)
ax.set_xlabel("word")
ax.set_ylabel("word")
ax.set_title("Cosine between learned SGNS vectors (grouped by family)")
fig.colorbar(im, ax=ax, label="cosine similarity")
fig.tight_layout()
plt.show()
def avg_within_cross(table):
"""Average cosine for same-family pairs vs different-family pairs (self excluded)."""
family_of = {w: name for name, ws in families.items() for w in ws}
within, across = [], []
for i in range(V):
for j in range(V):
if i == j:
continue
bucket = within if family_of[vocab[i]] == family_of[vocab[j]] else across
bucket.append(cosine_similarity(table[i], table[j]))
return float(np.mean(within)), float(np.mean(across))
sgns_within, sgns_across = avg_within_cross(W)
print(
f"SGNS average cosine: within-family {sgns_within:.3f}, across-family {sgns_across:.3f}"
)
SGNS average cosine: within-family 0.605, across-family 0.115
What to notice. cat's nearest word is dog, and its top three are exactly the three other pets — the planted family, recovered from the pairs rather than put there by hand. home's neighbours are all places. The heatmap shows the same four warm blocks as the co-occurrence and PMI matrices: within-family cosine averages about $0.6$ while across-family sits near $0.1$, a clear separation. It is not flawless on 36 tokens — a hub word like dog, which co-occurs with everything in its family, occasionally pulls one stray neighbour from outside — which is why the module's test anchors on cat's family and asserts a rank, never a cosine value. Ranks are stable; absolute cosines on a toy corpus are not.
Does the parallelogram survive learning?¶
The head notebook solved man : woman :: king : ? exactly, because its vectors were planted to make the analogy a perfect parallelogram. Here the vectors are learned from 36 tokens, so the same arithmetic is fragile. The four families share an identical internal shape (a first word, a hub, two others), so a family-parallel analogy sometimes lands. Try two — analogy runs 3CosAdd on the head's $(d \times V)$ convention, which is this topic's $W$ transposed ($E = W^{\top}$).
E = W.T # the head's (d, V) matrix is the trained center table transposed
def solve(a, b, c):
"""Return the word 3CosAdd predicts for the analogy a : b :: c : ?."""
return vocab[analogy(E, word_to_ix[a], word_to_ix[b], word_to_ix[c])]
print("analogy a : b :: c : ? (3CosAdd on the learned vectors)")
print(
f" cat : dog :: home : {solve('cat', 'dog', 'home')!r:9} (want 'yard', the places hub)"
)
print(
f" cat : dog :: bread : {solve('cat', 'dog', 'bread')!r:9} (want 'meat', the food hub)"
)
analogy a : b :: c : ? (3CosAdd on the learned vectors) cat : dog :: home : 'yard' (want 'yard', the places hub) cat : dog :: bread : 'feed' (want 'meat', the food hub)
What to notice. One analogy lands on the right word and one misses. That is the honest result of asking a 36-token corpus to pin a cross-family parallelogram: the within-family grouping is solid (section 4), but the finer claim that the "first word $\to$ hub" offset is the same vector in every family is more than this little corpus can support. On real corpora trained on billions of tokens the analogy property re-emerges as a genuine empirical regularity — see the head topic's exact version. The lesson: neighbours (a coarse, rank-based read) survive small data; analogies (a fine, direction-based read) need more.
5. Vary exactly one thing: the sampler's 3/4 exponent¶
Everything above used the default sampler. Now change one knob and nothing else — the exponent that shapes where negatives come from — and watch its single effect on the negative distribution. Negatives are drawn from the smoothed unigram
$$P(w) = \frac{f(w)^{\beta}}{\sum_{w'} f(w')^{\beta}},$$
where $f(w)$ is the raw corpus frequency and $\beta$ is the exponent. In the fixture the four family hubs occur 3 times and the other twelve words 2 times, so the only thing $\beta$ controls is the ratio $P(\text{hub}) / P(\text{other})$ — here, $P(\text{dog}) / P(\text{cat})$.
Predict first. At $\beta = 1$ (draw in proportion to raw frequency) the ratio is $3/2 = 1.5$. At $\beta = 0$ (uniform) every word is equally likely, so the ratio is $1$. The Word2Vec default is $\beta = 3/4$, strictly between. Before running: will the $3/4$ ratio land closer to $1.5$ or to $1.0$ — and does that over- or under-sample the rare words relative to raw frequency?
exponents = [0.0, 0.5, 0.75, 1.0]
hub, other = word_to_ix["dog"], word_to_ix["cat"] # frequencies 3 and 2
ratios = []
for beta in exponents:
probabilities = unigram_probabilities(sentences, V, exponent=beta)
ratio = probabilities[hub] / probabilities[other]
ratios.append(ratio)
tag = " <- Word2Vec default" if beta == 0.75 else ""
print(f" beta = {beta:>4}: P(dog) / P(cat) = {ratio:.3f}{tag}")
beta = 0.0: P(dog) / P(cat) = 1.000 beta = 0.5: P(dog) / P(cat) = 1.225 beta = 0.75: P(dog) / P(cat) = 1.355 <- Word2Vec default beta = 1.0: P(dog) / P(cat) = 1.500
fig, ax = plt.subplots(figsize=(6.4, 4.2))
ax.plot(exponents, ratios, "o-", color="#d62728")
ax.axhline(1.0, color="grey", ls="--", lw=0.9, label="uniform (ratio 1.0)")
ax.axhline(1.5, color="grey", ls=":", lw=0.9, label="raw frequency (ratio 1.5)")
ax.annotate("3/4", (0.75, ratios[2]), textcoords="offset points", xytext=(6, -14))
ax.set_xlabel(r"sampler exponent $\beta$ (0 = uniform, 1 = raw frequency)")
ax.set_ylabel("negative-draw ratio P(dog) / P(cat)")
ax.set_title("The 3/4 power flattens the negative distribution toward uniform")
ax.legend(loc="center right")
fig.tight_layout()
plt.show()
Reveal. At $\beta = 3/4$ the ratio is about $1.36$ — pulled down from $1.5$ toward $1$, but not all the way. The sub-unit power compresses large frequencies more than small ones, so the distribution flattens: the frequent hub is still sampled as a negative more often than a rare word, just less ruinously so. That is the point of the exponent. Draw negatives at raw frequency ($\beta = 1$) and they are dominated by the most common words, which co-occur with everything and so teach little; draw them uniformly ($\beta = 0$) and you waste effort separating words that were never going to be confused. The value $3/4$ is empirical — Mikolov et al. found it worked best — and GloVe reaches for the very same power to cap frequent pairs in its weighting. We changed one variable; the response was one number moving along one curve.
6. A second route to the same target: GloVe¶
Section 2 said both methods aim at the same PMI target. GloVe takes the count-based route: instead of streaming pairs, it fits the log of the global co-occurrence counts directly by weighted least squares, over the same $X$ we already built. Its final vector for a word is the sum of its two tables, $W_i + \tilde{W}_i$. Run it on the identical counts and compare what comes out.
glove = train_glove(X, V) # same (V, V) counts, d=8, lr=0.05, epochs=200, seed=0
glove_vectors = glove["vectors"] # W + W_tilde, the (V, d) GloVe vectors
glove_history = glove["history"]
print(
f"GloVe loss: {glove_history[0]:.3f} -> {glove_history[-1]:.3f} (fits log-counts almost exactly)\n"
)
cat_neighbours = nearest_neighbor(glove_vectors, word_to_ix["cat"], k=3)
print(f"GloVe nearest neighbours of 'cat': {[vocab[j] for j in cat_neighbours]}")
glove_within, glove_across = avg_within_cross(glove_vectors)
print(
f"GloVe average cosine: within-family {glove_within:.3f}, across-family {glove_across:.3f}"
)
GloVe loss: 0.675 -> 0.008 (fits log-counts almost exactly) GloVe nearest neighbours of 'cat': ['bread', 'dog', 'fish'] GloVe average cosine: within-family 0.086, across-family -0.025
labels = ["within family", "across families"]
x = np.arange(len(labels))
width = 0.36
fig, ax = plt.subplots(figsize=(6.4, 4.2))
ax.bar(
x - width / 2,
[sgns_within, sgns_across],
width,
label="SGNS (skip-gram)",
color="#1f77b4",
)
ax.bar(
x + width / 2, [glove_within, glove_across], width, label="GloVe", color="#ff7f0e"
)
ax.axhline(0, color="black", lw=0.6)
ax.set_xticks(x)
ax.set_xticklabels(labels)
ax.set_ylabel("average cosine similarity")
ax.set_title("Family separation in the learned vectors: SGNS vs GloVe")
ax.legend()
fig.tight_layout()
plt.show()
What to notice. GloVe drives its loss almost to zero — it fits the log-counts nearly perfectly — yet its neighbourhoods do not cleanly recover the families: cat's nearest word here is not even a pet, and the within-vs-across cosine gap (orange) is a sliver next to skip-gram's (blue). This is not a bug in GloVe; it is what 36 tokens buy. GloVe's weighted least squares has only a handful of nonzero cells to fit, with tiny counts, so it nails the arithmetic without pinning the geometry. Skip-gram, pushing against sampled negatives pair by pair, extracts more usable structure from the same counts at this scale. Both target the same (shifted) PMI matrix in the limit — the measured gap between them is mostly data and hyperparameters, as Levy, Goldberg & Dagan (2015) showed — but "fits the counts" and "recovers the neighbourhoods" are plainly not the same claim on a small corpus. Treat the GloVe run as a demonstration of the mechanics; the clean family recovery is skip-gram's, on this corpus, at these settings.
7. What this means, and what it does not¶
What it means. You watched word vectors get learned. Starting from a block-diagonal count matrix — the only input — skip-gram with negative sampling optimised a seeded objective until the learned geometry reproduced the planted families: cat's nearest words are the other pets, within-family cosine sits far above across-family, and the loss descended monotonically and reproducibly to get there. The head topic's "given" geometry is this topic's output: the trained center table $W$ is the head's $E$ transposed. And the surprise under it all is that this predictive game is implicitly factorising the PMI matrix you built by hand in section 2.
What it does not mean. Three honest limits, each visible above.
- Small data recovers coarse structure, not fine. Rank-based neighbours survived 36 tokens; the direction-based analogy was fragile. The parallelogram is a large-corpus regularity, not a toy-corpus guarantee.
- "Fits the counts" is not "recovers the meaning." GloVe drove its loss to near zero and still missed
cat's family. The objective value is not the thing you care about; the geometry is. - This is a teaching reference, not a trainer. The corpus is planted so every number is checkable. Real vectors need a real corpus, a larger $d$, frequent-word subsampling, and often subwords for unseen words — the how-to names the knobs and where to go next.
Key Takeaways¶
- Co-occurrence is the raw material. A window over sentences becomes a block-diagonal count matrix $X$; because no sentence mixes families, co-occurrence is dense within a family and zero across — the structure training must recover.
- Skip-gram implicitly factorises PMI. The counts already define a (shifted) PMI target with a positive within-family block; the learned inner products chase it (Levy & Goldberg, 2014), which is why "predict, don't count" is secretly counting.
- Training works and is reproducible.
train_sgnsdescends monotonically because the negatives are drawn once and reused, and one seeded generator makes a run byte-identical across calls. - The vectors recover the families.
cat's nearest neighbours are the other pets, and within-family cosine ($\approx 0.6$) dwarfs across-family ($\approx 0.1$) — asserted as a rank, never a cosine value. - The
3/4exponent flattens the negative distribution. Varying only $\beta$ moved $P(\text{dog})/P(\text{cat})$ from $1.5$ (raw frequency) through $1.36$ ($3/4$) to $1.0$ (uniform), oversampling rare words as negatives without erasing frequency. - GloVe reaches the same target by a different route, and lands differently at this scale. It fits log-counts almost exactly yet does not cleanly recover the families on 36 tokens — same PMI target, different sample efficiency.
- This notebook ran canonical code only. Every objective, gradient, sampler, and trainer came from
dlhub.embeddings.word2vec_and_glove; the lesson is a view onto the reference, not a second copy of it.
One sentence to keep. Word vectors are a low-rank factorisation of a co-occurrence matrix's mutual information — and on this toy corpus you can watch skip-gram learn that factorisation from nothing but counts, recover the planted neighbourhoods, and still leave the finer analogy geometry to a bigger corpus.