Tokens and Predictions: Why AI Writes Word by Word

Your phone sends a short text. Somewhere in a data center, an AI spins up and, in a blink, makes not one choice but billions of tiny ones—just to decide its next word. Yet it never “sees” the whole sentence. How does something so blind to the big picture sound so fluent?

You’re watching the reply appear on your screen, one fragment at a time, but under the hood something much smaller than “a word” is doing the real work. Modern models don’t actually think in words at all—they think in tokens. A token might be a whole word, half a word, punctuation, or even a single character, depending on how often it shows up in real text. Common pieces like “ing”, “pre”, or “tion” get their own entries, while rare names might be split into several chunks. Each of these chunks is turned into numbers, shuffled through layers of computation, and turned back into text. This matters because it sets a hard budget: every question and answer must fit inside a fixed token window. Go past that, and earlier parts of the conversation start to fall off the edge, changing what the model can “remember” and how it responds.

Behind the scenes, each token lives as a long list of numbers called a vector—a kind of GPS coordinate in a huge abstract space. Nearby coordinates tend to mean related ideas: “doctor” might sit closer to “nurse” than to “banana.” During training, the model repeatedly adjusts these coordinates so that contexts predicting similar continuations pull tokens into meaningful clusters. This is where “understanding” emerges: not from rules, but from patterns of proximity. When you ask a question, your prompt reshapes this space on the fly, nudging which regions become more likely for the next step.