Word embeddings: measure meaning as geometry, then watch averaging lose word order¶
Purpose. Turn "a word embedding is a dense vector whose geometry carries meaning" from a sentence you have read into numbers you have measured. You will select a word's vector two equivalent ways, read the similarity between words off a heatmap, solve king − man + woman → queen as plain vector arithmetic, see that analogy become a parallelogram in a 2-D projection, and then meet the one thing this representation throws away — word order — in the place it hurts most.
Prerequisites. Vectors, dot products, and cosine as the angle between two vectors. No prior NLP is needed. Every equation and rule this notebook uses is derived in the explanation and pinned in the reference; every operation is imported from the hub's canonical dlhub.embeddings module, so the lesson cannot drift from the code it teaches.
What you will be able to do by the end.
- Select a word's vector two equivalent ways — the one-hot matrix product $E\,o_w$ and a column gather — and see they agree exactly.
- Read the similarity between any two words as the cosine of the angle between their vectors, straight off a heatmap.
- Rank a word's nearest neighbours and solve an analogy as vector arithmetic.
- Project the space to 2-D with PCA and recognise the analogy as a parallelogram.
- Show that averaging word vectors is order-blind —
"not good"scores like"good"— and name precisely which models fix that.
What this notebook is not. It does not implement any of these operations. The embedding matrix, cosine similarity, nearest neighbours, the analogy solver, and the from-scratch PCA projection are all imported from dlhub.embeddings — the same code the reference documents and the tests check. It trains nothing: the vectors are a small hand-planted toy, so every answer is exact and checkable by eye. How vectors are learned (Word2Vec, GloVe) and the bias geometry of real embeddings are separate topics.
Runtime. Under a second on a laptop CPU — there is no training. Every result is a closed-form read-off of a $4 \times 6$ matrix. Seed. np.random.seed(0) is set once below, though nothing here is random: the fixture is hand-planted and the PCA sign is fixed by convention, so every number is reproducible by construction.
Where this fits¶
| Reader wants to | Go to |
|---|---|
| Learn it step by step | This notebook — you are here |
| Do it in a project | Not written for this topic — training vectors is the word2vec & GloVe topic |
| Look up an equation or rule | Word embeddings (reference) |
| Understand why it works | Word embeddings (explanation) |
| Learn by running it | This notebook — you are here |
1. Orient: a word is a column of a small planted matrix¶
An embedding matrix $E$ stores one vector per word. We use Andrew Ng's convention — shape $(d \times V)$, with $d$ features down the rows and $V$ words across the columns — so a word's vector is a column of $E$.
To keep every answer exact and checkable, we do not train anything. We use a planted toy matrix of $V = 6$ words in $d = 4$ dimensions, in which only two coordinates carry signal:
- coordinate 0 — royalty (higher = more royal),
- coordinate 1 — gender ($+1$ masculine, $-1$ feminine; a bookkeeping convention of this toy, nothing learned),
- coordinates 2–3 — zero.
Because only two coordinates move, the six words really live on a 2-D plane inside $\mathbb{R}^4$ — a fact we will cash in when we project with PCA. Build the fixture and read it.
import matplotlib.pyplot as plt
import numpy as np
from dlhub.embeddings import (
analogy,
cosine_similarity,
lookup,
make_fixture,
nearest_neighbors,
one_hot,
pca_project_2d,
)
np.random.seed(0) # nothing here is random; the determinism is built in (see above)
fixture = make_fixture()
E = fixture["E"]
vocab = fixture["vocab"]
word_to_ix = fixture["word_to_ix"]
d, V = fixture["dims"]["d"], fixture["dims"]["V"]
print(f"E has shape (d, V) = ({d}, {V}) -> one word per column\n")
header = "word".ljust(10) + "royalty gender c2 c3"
print(header)
print("-" * len(header))
for w in vocab:
royalty, gender, c2, c3 = E[:, word_to_ix[w]]
print(f"{w.ljust(10)}{royalty:6.0f} {gender:6.0f} {c2:4.0f} {c3:4.0f}")
E has shape (d, V) = (4, 6) -> one word per column word royalty gender c2 c3 ------------------------------------- king 2 1 0 0 man 0 1 0 0 prince 1 1 0 0 princess 1 -1 0 0 queen 2 -1 0 0 woman 0 -1 0 0
The lookup identity¶
How do you get a word's vector out of $E$? Multiply $E$ by the word's one-hot column $o_w$ — a vector that is $1$ at the word's index and $0$ everywhere else:
$$e_w \;=\; E\,o_w \;=\; E_{:,\,i} \;\in\; \mathbb{R}^{d}.$$
Because $o_w$ is zero everywhere but one entry, the product simply selects column $i$ and nothing else. That is the topic's spine: an embedding layer is a bias-free linear layer whose input is one-hot. No real implementation forms the product — a gather by index returns the identical column far more cheaply. The reference exposes both forms; confirm they agree.
i = word_to_ix["king"]
o = one_hot(i, V)
print(f'one-hot column for "king" (index {i}): {o}')
print(f"E @ one_hot -> {E @ o}")
print(f"lookup(E, {i}) -> {lookup(E, i)}")
identity_holds = all(np.array_equal(E @ one_hot(j, V), lookup(E, j)) for j in range(V))
print(f"\nE @ one_hot(j) == lookup(E, j) for every word j: {identity_holds}")
one-hot column for "king" (index 0): [1. 0. 0. 0. 0. 0.] E @ one_hot -> [2. 1. 0. 0.] lookup(E, 0) -> [2. 1. 0. 0.] E @ one_hot(j) == lookup(E, j) for every word j: True
2. Read similarity off the geometry: cosine¶
A dense vector is only useful if distance in the space means something. The standard meter is cosine similarity — the cosine of the angle between two vectors, $+1$ for the same direction, $0$ for orthogonal, $-1$ for opposite, and blind to length. Start with the two pairs you can check by hand.
king, queen = word_to_ix["king"], word_to_ix["queen"]
man, woman = word_to_ix["man"], word_to_ix["woman"]
print(f"cos(king, queen) = {cosine_similarity(E[:, king], E[:, queen]):.4f}")
print(f"cos(man, woman) = {cosine_similarity(E[:, man], E[:, woman]):.4f}")
cos(king, queen) = 0.6000 cos(man, woman) = -1.0000
king and queen share a large royalty coordinate and differ only in gender, so they point mostly the same way: $\cos = 0.6$. man and woman have no royalty and opposite gender, so they point in exactly opposite directions: $\cos = -1$. Now do all $6 \times 6$ pairs at once.
cos_matrix = np.array(
[[cosine_similarity(E[:, a], E[:, b]) for b in range(V)] for a in range(V)]
)
fig, ax = plt.subplots(figsize=(6.2, 5.2))
im = ax.imshow(cos_matrix, cmap="RdBu_r", vmin=-1, vmax=1)
ax.set_xticks(range(V))
ax.set_xticklabels(vocab, rotation=45, ha="right")
ax.set_yticks(range(V))
ax.set_yticklabels(vocab)
for a in range(V):
for b in range(V):
value = cos_matrix[a, b]
ax.text(
b,
a,
f"{value:.2f}",
ha="center",
va="center",
fontsize=9,
color="white" if abs(value) > 0.55 else "black",
)
fig.colorbar(im, ax=ax, label="cosine similarity")
ax.set_title("Pairwise cosine similarity of the six word vectors")
fig.tight_layout()
plt.show()
What to notice. The diagonal is $1$ (every word is identical to itself). The warm (red) block links the royal words king, prince, queen, princess; the cool (blue) corners mark opposite-gender pairs with no shared royalty, bottoming out at cos(man, woman) = -1. Similarity here is direction, not distance: king is closer to queen ($0.6$) than to man ($0.45$), because shared royalty outweighs shared gender.
3. Rank the nearest neighbours¶
"Which word is most like this one?" is just: score every other word by cosine, sort descending, and take the top. nearest_neighbors does exactly that, excluding the word itself and breaking ties by ascending index so the ranking is deterministic.
print("nearest neighbour of each word (top-1, the word itself excluded):\n")
for query, word in enumerate(vocab):
neighbour, score = nearest_neighbors(E, query, k=1)[0]
print(f" {word.ljust(9)} -> {vocab[neighbour].ljust(9)} (cos = {score:.3f})")
nearest neighbour of each word (top-1, the word itself excluded): king -> prince (cos = 0.949) man -> prince (cos = 0.707) prince -> king (cos = 0.949) princess -> queen (cos = 0.949) queen -> princess (cos = 0.949) woman -> princess (cos = 0.707)
What to notice. Royalty dominates the ranking. king's nearest word is prince, not man; queen's is princess, not woman. Even man's nearest word is prince — the royal word that still shares its gender — because the royalty coordinate is simply larger than the gender coordinate for most words. The neighbour relation is not symmetric: king → prince, but prince → king.
4. The analogy: predict, then reveal¶
The property that made word vectors famous: meaning-differences are directions, so an analogy becomes arithmetic. "man is to woman as king is to ?" is solved by 3CosAdd — build the target
$$t \;=\; e_{\text{woman}} - e_{\text{man}} + e_{\text{king}}$$
and return the word whose vector is most cosine-similar to $t$ (excluding the three input words).
Predict first. The displacement woman − man is a pure gender flip (royalty $0 \to 0$, gender $+1 \to -1$). Add it to king $= (\text{royalty } 2, \text{gender } +1)$. What royalty and gender should the answer have — and which of the six words is that? Decide before running the next cell.
def solve(a, b, c):
"""Solve ``a : b :: c : ?`` with the imported 3CosAdd analogy solver."""
return vocab[analogy(E, word_to_ix[a], word_to_ix[b], word_to_ix[c])]
print(f"man : woman :: king : {solve('man', 'woman', 'king')}")
print(f"prince : princess :: king : {solve('prince', 'princess', 'king')}")
target = lookup(E, woman) - lookup(E, man) + lookup(E, king)
print(f"\ntarget t = woman - man + king = {target}")
for word in ["queen", "princess"]:
score = cosine_similarity(target, E[:, word_to_ix[word]])
print(f" cos(t, {word.ljust(8)}) = {score:.4f}")
man : woman :: king : queen prince : princess :: king : queen target t = woman - man + king = [ 2. -1. 0. 0.] cos(t, queen ) = 1.0000 cos(t, princess) = 0.9487
What to notice. The target vector is $[2, -1, 0, 0]$ — royalty $2$, gender $-1$ — which is exactly queen, so $\cos(t, \text{queen}) = 1$. The runner-up princess is close ($\cos \approx 0.949$) but loses, because its royalty is only $1$. The vital detail that makes this work is excluding the input words: without it, so much of king remains in $t$ that king would be its own nearest point and the analogy would answer itself.
5. See the whole space: PCA to 2-D¶
Four dimensions are two too many to plot, so project onto the two directions of greatest variance with PCA (computed from scratch via the SVD). Because the data genuinely lives on a 2-D plane, this projection is loss-less — it recovers the plane exactly, it does not approximate it.
projection = pca_project_2d(E)
coords = projection.coords # (V, 2): one 2-D point per word
fig, ax = plt.subplots(figsize=(6.4, 5.4))
ax.axhline(0, color="grey", lw=0.6)
ax.axvline(0, color="grey", lw=0.6)
ax.scatter(coords[:, 0], coords[:, 1], s=70, color="#1f77b4", zorder=3)
for word, (x, y) in zip(vocab, coords):
ax.annotate(word, (x, y), textcoords="offset points", xytext=(7, 5), fontsize=11)
# Draw the analogy as a parallelogram: the gender displacement (woman - man)
# is the same arrow applied at man and at king.
for start, end in [("man", "woman"), ("king", "queen")]:
x0, y0 = coords[word_to_ix[start]]
x1, y1 = coords[word_to_ix[end]]
ax.annotate(
"",
xy=(x1, y1),
xytext=(x0, y0),
arrowprops=dict(arrowstyle="->", color="#d62728", lw=1.8),
)
ax.set_xlabel("PC1 (≈ gender: masculine → )")
ax.set_ylabel("PC2 (≈ royalty: ↑ )")
ax.set_title("Word vectors projected to 2-D by PCA (loss-less: the data is rank-2)")
fig.tight_layout()
plt.show()
reconstruction = coords @ projection.components + projection.mean
print(
f"max reconstruction error over all 24 numbers: {np.abs(reconstruction - E.T).max():.2e}"
)
print(
f"variance on each axis (singular values squared): {projection.singular_values[:2] ** 2}"
)
max reconstruction error over all 24 numbers: 0.00e+00 variance on each axis (singular values squared): [6. 4.]
What to notice. The horizontal axis (PC1) is the gender direction — masculine words king, prince, man on the right, feminine queen, princess, woman on the left — and it carries the most variance, so PCA chose it first. The vertical axis (PC2) is royalty: king/queen on top, man/woman at the bottom. The two red arrows are the same displacement woman − man drawn from man and from king; that they are parallel and equal is the analogy king − man + woman → queen seen as a parallelogram. The reconstruction error is zero to machine precision — exactly what "loss-less" means.
6. Vary exactly one thing: the word order¶
Everything so far has been about single words. To represent a whole phrase with one vector, the simplest recipe is a bag-of-embeddings: average the vectors of its words. It is a real baseline for tasks like sentiment classification — and it has one fatal property we can expose by changing exactly one thing and nothing else: the order of the words.
The planted royalty/gender space has no sentiment words, so for this one demonstration we plant a throwaway 2-coordinate lexicon: coordinate 0 is sentiment polarity ($+$ good, $-$ bad), and "not" is a near-neutral function word with little mass of its own.
Predict first. A bag-of-embeddings averages the word vectors. If you keep the words the same but reverse their order, what happens to the average — and so, what will a sentiment reader make of "not good"?
# A throwaway lexicon, planted by hand for THIS illustration only -- not the
# canonical fixture. Coordinate 0 = sentiment polarity; "not" is near-neutral.
lexicon = {
"good": np.array([1.0, 0.0]),
"bad": np.array([-1.0, 0.0]),
"not": np.array([0.1, 0.1]),
}
def phrase_vector(phrase):
"""Average a phrase's word vectors -- the order-blind bag-of-embeddings baseline.
This is the naive baseline on trial here, not a ``dlhub.embeddings``
algorithm; ``np.mean`` ignores word order, which is exactly the point.
"""
return np.mean([lexicon[w] for w in phrase.split()], axis=0)
# (a) Vary ONLY the word order, nothing else:
print(f'bag("not good") = {phrase_vector("not good")}')
print(f'bag("good not") = {phrase_vector("good not")}')
same = cosine_similarity(phrase_vector("not good"), phrase_vector("good not"))
print(f"cosine between the two orderings = {same:.4f}\n")
# (b) The consequence: does "not good" read as good, or as bad?
to_good = cosine_similarity(phrase_vector("not good"), lexicon["good"])
to_bad = cosine_similarity(phrase_vector("not good"), lexicon["bad"])
print(f'cos( "not good" , good ) = {to_good:.4f}')
print(f'cos( "not good" , bad ) = {to_bad:.4f}')
bag("not good") = [0.55 0.05]
bag("good not") = [0.55 0.05]
cosine between the two orderings = 1.0000
cos( "not good" , good ) = 0.9959
cos( "not good" , bad ) = -0.9959
What to notice. Reversing the order leaves the vector byte-for-byte identical (cosine $1.0000$), because the average of a set of vectors cannot depend on their order — addition is commutative. That is not a bug in this lexicon; it is true of every bag-of-embeddings. The cost lands in part (b): "not good" sits at cosine $\approx 0.996$ to good and $\approx -0.996$ to bad, so a cosine sentiment reader confidently calls "not good" positive. Averaging has no way to let "not" act on the word after it — the negation is simply averaged in as mild noise.
This is the precise limit that motivates the next family of models. A representation that does respect order has to read the words as a sequence, carrying state from one to the next — which is exactly what a recurrent network does. The order-aware story begins with the recurrent neural networks notebook.
7. What this means, and what it does not¶
What it means. A static word embedding turns meaning into geometry you can measure. Selecting a vector is a one-hot matrix product that reduces to a column gather; similarity is a cosine; "most similar" is a sort; an analogy is vector addition that lands, in this planted space, exactly on the right word; and a space that looks four-dimensional can be a 2-D plane that PCA recovers without loss. Every number above is reproducible by construction, because all of it is imported from dlhub.embeddings and the vectors are planted, not trained.
What it does not mean. Three honest limits, each visible above.
- Averaging discards order. A bag-of-embeddings reads
"not good"as"good", because the mean of a set cannot encode sequence or composition. Order-aware models — recurrent networks, and later attention — exist to fix exactly this. - These vectors are given, not learned. The toy matrix is hand-planted so the answers are clean. How real vectors are fitted to a corpus — skip-gram, negative sampling, GloVe — is the separate word2vec & GloVe topic, and it is where the analogy property stops being a plant and becomes an empirical surprise.
- The "gender" axis is a convenience, not a claim. Here it is a bookkeeping coordinate that makes one clean analogy. When a gender direction appears in vectors learned from real text it is a harm, not a feature — and removing it is harder than it looks. That is the debiasing topic, deliberately out of scope here.
One sentence to keep. A word embedding is a bias-free linear layer's worth of learned geometry: enough to measure similarity, rank neighbours, and do analogies by arithmetic — but, on its own, blind to the order of the words it represents.