Tokens & numbers
A language model never sees words; it sees numbers. First the text is chopped into tokens, then each token is swapped for an ID from a vocabulary. Run the cell to tokenise a sentence yourself. Notice that the appears twice but gets one ID: same token, same number, every time.
Now the famous gotcha. Ask a chatbot how many r’s are in strawberry and it may say two. Why? A real tokeniser often stores a rare word as a few chunks, so the model receives about three chunk-IDs; it never sees the ten individual letters at all. Python, counting letters directly, gets it right: 3.
💡 Try it: change the sentence to "the cat and the dog and the bird" and predict the ID list before you run. Then count the s’s in "mississippi" with .count("s").
Pictures as numbers
Images get the same treatment as text: they become numbers. Here a 5×5 picture is just a list of rows, where 1 means a dark pixel and 0 means a light one. The print loop redraws the numbers as squares. Run it and a diamond appears, made of 13 dark pixels.
Because the picture is numbers, editing the picture means doing maths. One line of Python computes 1 - p for every pixel, and the whole image inverts, like a photo negative. Every filter on your phone is number-crunching like this, just on millions of pixels with more shades than 0 and 1.
💡 Try it: edit the 0s and 1s to draw your own initial, then re-run both cells. (The second cell reuses image from the first; that’s the shared notebook memory at work.)
Embeddings lite
Once words are IDs, models go one better: each word gets a position in space, called an embedding. Words with similar meanings sit close together. Real models use hundreds of dimensions; ours uses two, furriness and love of water, so you can picture it. The distance formula is just Pythagoras: cat to dog comes out at 1.41, while cat to shark is a distant 9.22.
This is exactly how AI answers “which word is most similar?”: it finds the nearest neighbour in the space. Run it and check the pairs make sense. Then look at towel: its nearest neighbour is dog! With only two dimensions there is nowhere sensible for a towel to sit, one reason real embeddings need so many dimensions.
💡 Try it: add "otter": (7, 8) to the dictionary and re-run both cells. Who is otter’s nearest neighbour, and whose nearest neighbour does otter steal?
A single neuron
Every neural network, including the giant ones behind chatbots, is stacked from one tiny building block. A neuron multiplies each input by a weight, adds a bias, then an activation decides whether to fire. Ours uses the simplest activation there is: a step. Total ≥ 0 → output 1; otherwise → 0. With weights 1, 1 and bias -0.5, this neuron computes OR: it fires if either input is 1.
💡 Try it: without touching the function, change only the numbers on the w1, w2, bias = … line so the neuron computes AND, firing only for 1 1. (Hint: keep the weights, make the bias more negative, e.g. -1.5.) When a network learns, this is all that changes: the weights and biases get nudged until the outputs come out right.
Learn from data
Time to build a real classifier. This is supervised learning: every training example comes with a label. Each animal is a point, (weight, ear length), labelled rabbit or capybara. Our learner is 1-nearest-neighbour: to classify a new animal, find the closest training point and copy its label. Note the golden rule: the test animals are kept separate, so we can check the model on data it has never seen.
Now evaluate: run every test animal through the model and tally the results. Accuracy comes out at 3 / 4 = 0.75, and the confusion counts show exactly where it went wrong: one real rabbit was called a capybara. Look at the culprit: (6, 4) is an unusually big, small-eared rabbit sitting in capybara territory. Real models fail on unusual cases in exactly this way.
💡 Try it: add a fifth training rabbit at (6, 5) and re-run both cells. Does the accuracy reach 1.0? More (good) data is often worth more than a cleverer algorithm.
Be the LLM
A language model is, at heart, a next-word predictor. Training means reading text and remembering what tends to follow what. This cell builds a bigram model from a four-sentence story: for each word, it collects every word that ever came next. Repeats matter: cat appears twice in the followers of the, so it is twice as likely to be picked.
Generating is just predicting on repeat: pick a next word, then a next word after that. The first loop picks at random from the follower list: common followers win more often, but anything can happen. That randomness is the intuition behind an LLM’s temperature setting. The second loop is temperature-zero: always take the most common follower, which here gives the same sentence every time: the cat ate a bone .
💡 Try it: change the seed and re-run a few times: the sampled sentence changes, the greedy one never does. Then add a sentence of your own to the story (keep the . spaced out) and watch the model pick up your words. Notice the model has no idea what a cat is; it only knows what follows what.
Attention, tiny
Our bigram model only ever looked at one previous word. Modern transformers use attention: for each new token they look back at all the earlier words and blend their meanings, giving more weight to the relevant ones. Take “the fish saw the net and it hid”: what does it mean? Below, each context word has a tiny meaning-vector, and the attention weights (which always sum to 1) say how much of each to mix in. With 0.7 on fish, the blend lands near fish’s vector.
Now shift the attention to net and the very same sentence “means” something different: the blend [3.5, 7.0] sits right next to net’s own vector [3, 8]. Which word the model attends to changes what it thinks it refers to, and therefore which token it predicts next. (Real attention computes these weights from the vectors themselves; here we set them by hand to see the effect.)
💡 Try it: set the weights to a perfectly even 1/3 each. Where does the blend land? Then invent weights that make it mean saw. Remember the rule: the weights must sum to 1.
A tiny agent
Finally, put it together. A plain chatbot just predicts text, and Section 6 showed that predicting-what-sounds-right is a risky way to do arithmetic. An agent runs a loop: think about the goal, act by calling a tool, observe the result, then answer. Here the tool is a percentage calculator, so “17% of 240” comes back as exactly 40.8: computed, not guessed.
A better agent also knows its limits. This version checks whether the question fits its one tool: if yes, it parses out the numbers, calls the tool, and, crucially, verifies the observation with a sanity check before answering. If not, it refuses rather than bluffing. That final habit is the whole of Pillar 5 in one line: a confident-sounding answer is not evidence: check before you trust, whether the answerer is an AI or you.
💡 Try it: ask agent2("What is 35% of 80?"). Check the answer by hand (it should be 28.0). Then find a question shaped like "What is X% of Y?" that fools the crude parser, and suggest one line that would fix it. Congratulations, you have now built every layer of a modern AI system in miniature: representation, learning, generation, attention and agency.