How is AI trained? Guess, score, nudge, repeat

Imagine buying a new radio and finding every knob spun to a random position. Switch it on and all that comes out is static. The 124 million dials inside GPT-2 small, from Part 1, start out the same way. A brand-new model begins at random settings, so a request for a sentence produces nonsense. Training is the work of turning those random dials into useful settings by showing the model example after example and nudging the dials a little each time.

Those examples work like a giant practice workbook filled with real writing from the internet, and they are the model's training data: the text it learns from. For GPT-2 that meant about 8 million web pages, roughly 40 GB of text. OpenAI built the set, called WebText, from Reddit links that other users had upvoted at least a few times.

With a page in front of it, the model plays a fill-in-the-blank game where the blank is always at the end. Given "The cat sat on the ___", it hides what comes next and guesses. Strictly speaking, it guesses chunks of text called tokens, which Part 3 explains.

Every guess then gets a score, and lower is better, as on a golf card. If the hidden word was "mat" and the guess was far off, the score is high. A close guess scores low. That score is the loss: a measure of how wrong the guess was. Training tries to make the loss smaller.

To shrink it, the model uses gradient descent. Picture walking downhill in thick fog toward a valley floor. The bottom is hidden, yet the slope underfoot shows which way is down, so each step is a small move downhill, then another. Gradient descent lowers the loss by repeatedly nudging parameters a little in that downhill direction. The steps stay tiny on purpose, because big jumps could overshoot.

GPT-2 small has about 124 million dials, and each one needs its own nudge. Backpropagation works out the direction and size of those nudges by tracing the mistake backward through every layer, the way a cook whose soup came out too salty goes back through the recipe to see which ingredient to cut.

Put together, the training loop is guess, score the error, work out the nudges, and apply them, then repeat across millions of pages. Any one nudge barely changes anything. The model improves because the loop runs an enormous number of times, tuning dials for better guessing in general rather than memorizing every page.

That first huge round is called pretraining: predicting text across a large general dataset. For GPT-2, pretraining was the whole story. Today's chat assistants add later stages, a bit like coaching on the job after finishing school. Fine-tuning is extra training on a smaller, targeted set, such as helpful answers to questions, to specialize a pretrained model. Reinforcement learning from human feedback, or RLHF, has testers rank several answers so the model can be steered toward the ones people preferred. Those steps shape behavior, though they cannot make a model always truthful.

Training, then, is a huge loop of guess, measure the error, and nudge every dial slightly to make the error smaller. What those chunks of text look like is the subject of Part 3, "What are tokens?"