Vectors of meaning
A language model stores every token as an embedding: a list of numbers positioned so that similar meanings sit close together. Distance is measured with cosine similarity: the cosine of the angle between two vectors, a·b / (|a||b|), which is 1.0 for parallel vectors, 0 for unrelated ones and negative for opposites. Below is a toy 4-dimensional space (real models use hundreds of dimensions). Run the cell: king and queen come out at 0.787, king and cabbage at a distant 0.097.
Because meanings are now geometry, you can do arithmetic on them. The famous test: take king, subtract man, add woman: the result should land near queen. Run it and check the ranking: queen wins at 0.903. Nobody programmed that; it falls out of where the vectors sit.
💡 Try it: add a sixth word, "prince": np.array([0.85, 0.55, 0.04, 0.08]), to E in the first cell and re-run both. Then predict what prince − man + woman should land on before testing it. What breaks if you set every dimension of a word to zero?
Bias in the space
Embeddings are learnt from human text, so they absorb human associations, including ones we would rather they didn't. A standard audit: build a gender axis as the unit vector man − woman, then project profession vectors onto it with a dot product. Positive projection = the space treats the word as male-leaning; negative = female-leaning. Run it: engineer lands at +0.499, nurse at −0.552, even though a profession has no gender.
Why does this matter downstream? Imagine a CV-screening tool that ranks applicants by embedding similarity to the job title. Run the cell: the space says a man "fits" engineer at +0.711 while a woman scores −0.454, a huge gap driven entirely by the gender dimension, not by any skill. A system built on these vectors inherits the prejudice invisibly, at scale, with a veneer of mathematical objectivity.
💡 Try it: debias engineer by subtracting its gender component (v - (v @ axis) * axis), then re-project it and re-run the cosine comparison. Does the man/woman gap close? Note what information you did not lose: the royalty, vegetable and skill dimensions are untouched. Real debiasing is harder, but this is the core idea.
A pixel grid meets a filter
To a computer, an image is a grid of numbers: here a 5×5 patch where 9 is bright and 0 is dark, with a vertical edge down the middle. A convolution slides a small 3×3 kernel across the grid; at each position it multiplies overlapping numbers and sums them. This kernel is a vertical-edge detector: it responds where left ≠ right. Run it: the feature map lights up with 27s exactly along the edge and stays 0 in the flat region. This one operation, stacked and repeated with learnt kernels, is the entire idea of a convolutional neural network (CNN).
Kernels are orientation-specific. Rotate the detector 90° and run it on the same image: every output is 0, because the image never changes from top to bottom. A trained CNN holds thousands of kernels: some find edges, later layers combine edges into textures, shapes, and eventually "cat". Training is just gradient descent choosing the kernel numbers (which is exactly where the next section picks up).
💡 Try it: redraw image so the bright region is the top two rows instead of the left three columns, predict both feature maps, then run the cells to check. Bonus: make an image containing a corner. Which positions do both detectors fire on?
Forward pass & loss
A neural network is a stack of matrix multiplications with a squashing function between them. Here is a complete 2-2-1 network: two inputs, two hidden neurons, one output. Each layer computes weights @ inputs + bias, then applies the sigmoid activation 1/(1+e^−z) to squash the result into (0, 1). The forward pass runs the data through; the loss then scores how wrong the prediction is: squared error (ŷ − y)² against the target y = 1. Run it: the untrained network predicts 0.530, giving a loss of 0.221.
How does training work? For every weight, ask: if I nudge this number, does the loss go up or down? Try it on w2[0]. Nudging it down to 0.1 raises the loss to 0.294; nudging it up to 1.1 drops it to 0.160. So "increase w2[0]" is the downhill direction. Gradient descent does exactly this, but computes the slope for all weights simultaneously with calculus (backpropagation), then takes a small step downhill. Repeat thousands of times and the loss walks to a minimum.
💡 Try it: nudge w2[1] instead. Which direction is downhill for it? Then change the target to y = 0.0 in the first cell and re-run both: every downhill direction should flip. Finally, shrink the nudge to ±0.01; the tiny loss differences you see are (approximately) the gradient itself.
Train a tiny classifier
Time to train for real. We generate two clouds of 2-D points (class 0 centred at (1, 1), class 1 at (3, 3)) with a seeded random generator so every run is identical, then split them: 30 training points the model may learn from, 10 test points it never sees. The split is the whole discipline of machine learning: performance on unseen data is the only honest score.
The model is logistic regression: a weighted sum squashed by sigmoid, trained by the real gradient-descent update (the calculus version of section 4's nudge test). Watch the log-loss fall from 0.693 (exactly the loss of pure 50/50 guessing) to 0.242. Then the honest scores on unseen data: accuracy 0.90, precision 0.80 (of the points it called class 1, 80% really were; one false positive), recall 1.00 (it missed no true class-1 point). Note train accuracy is 0.97: the small train→test drop is a mild sign of overfitting, the model fitting quirks of its 30 examples. More data, fewer parameters or earlier stopping all shrink that gap.
💡 Try it: move the clusters closer (centre red at (2.0, 2.0)) and re-run both cells. Which metric collapses first? Then train for 3 000 epochs instead of 300: does the test accuracy improve as much as the train loss suggests it should? That divergence is overfitting made visible.
Temperature sampling
At the end of the transformer pipeline sits a score (a logit) for every token in the vocabulary. Softmax turns scores into probabilities; temperature T divides the logits first, reshaping the distribution. Here are toy logits for "The cat sat on the ___". Run it: at T = 0.2 the distribution collapses onto mat (0.993); at T = 1.5 it flattens, giving roof and even moon a real chance. Low temperature sharpens, high temperature flattens; the logits never change.
Now actually sample 20 next tokens at each setting (seeded, so your output matches this text). At T = 0.2 you get mat nineteen times out of twenty: reliable, predictable, dull. At T = 1.5 the same model, same logits, produces a mix including moon and roof: more surprising, less dependable. That is the usefulness-versus-reliability trade-off in one knob: low T for facts and code, higher T for brainstorming and poetry.
💡 Try it: what happens as T → 0? Try T = 0.01 (this is effectively greedy decoding: always the top token). Then try T = 5 and check the five probabilities: they approach 0.2 each, i.e. uniform randomness. Where would you set T for a medical-advice chatbot, and why?
RAG in 20 lines
A language model answers from its training data, which may be stale, or silent about your documents. Retrieval-augmented generation (RAG) fixes this: embed the documents as vectors, embed the question, retrieve the closest documents by cosine similarity, and paste them into the prompt so the model answers from evidence. Here is the retrieval half, using simple word-count vectors as a stand-in for learnt embeddings. Run it: the query about meeting times pulls D1 to the top at 0.667, and D4, which shares no useful words, scores 0.000.
Now the generation half: build a grounded prompt that hands the model only the retrieved sources and demands citations. Compare the two behaviours below. Ungrounded, a model must produce something: the next-token objective rewards fluency, not truth, so it may confidently invent "Fridays at 4 pm". Grounded, the correct answer is sitting in the context window with an ID to cite, and a wrong answer is now checkable against [D1]. RAG doesn't make a model honest; it makes the evidence auditable.
💡 Try it: ask "How much does membership cost?". Does the right document win? Then ask "What must I wear in the lab?" and note the failure: wear matches nothing, even though D4 is the answer. Word-count vectors miss synonyms; learnt embeddings (section 1) place wear near shoes, which is precisely why real RAG uses them. What retrieval metric would you report for this system, and what counts as a "hit"?
Audit & model card
A single overall accuracy can hide a lot. The audit that finds it: compute the metrics per group. Below, a screening model's predictions for ten applicants from group A and ten from group B; each group has six genuine positives. Recall asks: of the people who truly deserved a yes, how many did the model catch? A false negative here is a person wrongly passed over. Run it: group A gets recall 0.83, group B only 0.50: a gap of 0.33. Same model, same threshold, very different error rates.
Finding the gap is half the job; documenting it is the other half. A model card is the standard artefact of AI governance: what the model does, what it was trained on, its measured metrics (including the uncomfortable ones), its known limitations, and how it may be used. Transparency turns a hidden failure into an accountable design decision that a regulator, a school, or the affected group can challenge.
💡 Try it: compute precision per group as well. Does it tell the same story? Then decide: would you ship this model with a warning, hold it until the gap closes, or redesign the training data? Defend your call in two sentences. And a closing thought on scale: this audit cost you one loop; the systems it stands in for are trained on billions of examples, at real energy, water and labour cost, and their errors land on real people. Every build in this notebook (embeddings, gradients, sampling, retrieval) is the same machinery, only larger. Knowing how it works is what makes you qualified to question it.