What are tokens? How AI reads and writes text

In Part 2, GPT-2 small practiced guessing what came after “The cat sat on the ___”. It was guessing chunks of text called tokens. Think of Lego bricks: a small set of reusable pieces can be combined into many different things. A token might be a word, part of a word, or a punctuation mark.

GPT-2 splits “The cat sat on the mat.” into seven tokens: “The”, “ cat”, “ sat”, “ on”, “ the”, “ mat”, and the period. The leading spaces belong to their tokens. The same model splits “Understanding AI is easier than it sounds.” into eight tokens: “Understanding”, “ AI”, “ is”, “ easier”, “ than”, “ it”, “ sounds”, and the period. Less common words, however, can break into smaller parts. “unbelievably” becomes “un”, “bel”, “iev”, and “ably”. One token is therefore not necessarily one word.

Like a cook chopping ingredients before using them, a tool called a tokenizer splits text into tokens and assigns each an identification number. It works from a fixed list, or vocabulary, of available pieces. Think of a phrasebook whose entries can be combined into longer expressions. GPT-2's vocabulary has 50,257 tokens.

One way to build such a list is byte-pair encoding, or BPE. It repeatedly joins common neighboring pieces into larger ones, much as “t” and “h” could be joined. Frequent sequences can become whole-word tokens, while rarer words are assembled from smaller pieces.

The numbers from Part 1 appear here too. The seven tokens in “The cat sat on the mat.” have IDs 464, 3797, 3332, 319, 262, 2603, and 13. An ID is like a library catalog number: it identifies an entry, but says little by itself. Inside GPT-2 small, each token is represented by an embedding, a richer list of 768 numbers, like an index card with more information about the entry.

The model can consider only so many tokens at once. Its context window is that maximum amount, rather like the portion of a long conversation a person can keep in mind at one time. GPT-2's window holds 1,024 tokens.

Within that limit, it predicts a likely next token, adds it to the text, and predicts again, much like repeatedly accepting phone autocomplete. This is next-token prediction; the model does not plan the entire answer in advance.

Token counts affect what fits in a window and how some services price usage, a bit like paying by the brick. OpenAI estimates that an English token averages about four characters, but actual splits vary. They also help explain why a model can stumble when asked to count letters inside a word it handled as a few chunks. Different models may split the same text differently; all the splits here belong to GPT-2. Across the series, the picture is now complete: a model is arranged numbers, training sets them, and tokens are the pieces of text it reads and writes.