How a Model Produces Text: Strengths and Limits

You do not need mathematics to understand how a language model works, and a simple mental picture will save you from many disappointments. In this lesson you will see how text is cut into pieces, how the model chooses each next piece, how it learned to do that, and why this mechanism explains both its remarkable strengths and its surprising mistakes.

In the previous lesson you learned that a chat assistant is a language model wrapped in a conversation interface. Now let us open the hood, without any equations.

Text, cut into "tokens"

A language model does not read words the way you do. Before it processes anything, your text is split into small pieces called tokens. A token can be a short word ("cat"), part of a longer word ("un" + "believ" + "ably"), a number, or a punctuation mark. Common words are often a single token, while rare or long words are broken into several.

Why does this matter to you?

• Length limits are counted in tokens, not in words or pages. The amount of text a model can handle at once is called its context window. We cover it in detail in chapter 2.
• Some tasks are awkward for models. Counting the letters in a word or reversing a string of characters is hard when the model does not "see" individual letters but chunks of them.
• Languages are not equal. A text in some languages is cut into more tokens than the same text in another, which can affect cost and limits.

Predicting what comes next, again and again

The core principle is surprisingly simple. Given all the tokens already present (your prompt, then whatever the model has already written), the model computes a probability for each possible next token. It picks one, adds it to the text, and starts again with the longer text. It repeats this until the answer is complete.

Text so far:  "The sky is"
Likely next:  " blue" (very probable), " clear" (probable),
              " falling" (unlikely), " a teapot" (very unlikely)

Think of the autocomplete on your phone, but trained on a vast amount of text and capable of taking a very long context into account. The model does not plan the whole answer in advance and then write it down: it builds it piece by piece, each piece depending on all the previous ones.

Why the same question can give different answers

When the model has computed its probabilities, it does not always choose the single most probable token. A controlled amount of randomness is introduced. Many tools expose this as a setting often called temperature:

• Low temperature: the model sticks to the most probable choices. Answers are more predictable and consistent. Useful for extraction, classification or factual rewriting.
• High temperature: the model more often picks less probable tokens. Answers are more varied and surprising. Useful for brainstorming or creative writing, but with a higher risk of drifting.

Most chat interfaces set this for you and do not show it. What matters is the consequence: asking the same question twice can give two different answers. That is normal, not a malfunction. It also means that if you get a result you like, you should save it rather than count on getting it again.

Training: learning patterns from text

How does the model know which continuations are likely? Through training, which has two main stages.

1. Pre-training. The model is shown gigantic amounts of text. For each snippet, it tries to predict the next token, compares its guess with the real one, and slightly adjusts its internal parameters to do better next time. Repeated an astronomical number of times, this teaches it grammar, styles, common facts, ways of structuring arguments, even how to write code.
2. Tuning for conversation. A second phase makes the model more helpful and safer in dialogue: it learns to follow instructions, to answer in a useful format, to be polite and to decline some requests. Human feedback often plays a role here.

The key point: the model does not look up a source document when it answers. There is no little library inside it. What it holds is a huge set of learned statistical regularities. When you ask a question, it reconstructs a plausible answer from those regularities. Some products add a search or document-reading step on top (we will meet this idea again in chapter 5), but that is an extra layer, not the base mechanism.

Knowledge has a date

Training stops at some point. The model's built-in knowledge therefore reflects the world as it was in its training data, and it may not know about recent events, new laws, new products or updated prices. Some tools can browse the web or read files you provide, which compensates for this. When recency matters, check whether your tool is using up-to-date sources, and verify the facts.

Strengths

This prediction mechanism, applied at huge scale, is surprisingly good at:

• Writing and rewriting: drafting, rephrasing, shortening, changing the tone.
• Summarizing and extracting from a text you provide.
• Translating and adapting between registers (formal, friendly, technical).
• Exploring ideas: lists of options, outlines, angles, analogies.
• Explaining a concept at different levels, with examples.
• Helping with code, formulas and spreadsheets.

Limits and hallucinations

Because the model looks for the plausible continuation rather than the true one, it can state false information in a very confident tone. This is called a hallucination: an invented reference, a quotation that nobody said, a wrong figure, a function that does not exist in a programming library.

Hallucinations are more likely when:

• you ask about very specific, rare or recent facts;
• you ask for precise references, numbers, quotes or links;
• the question contains a false assumption that the model goes along with;
• the prompt is vague and the model fills the gaps.

Other typical limits: sensitivity to wording (a slightly different question can yield a different answer), difficulty with long or exact calculations, inconsistent behavior over long conversations, and no real experience of the physical world.

A useful analogy

Picture a very well-read colleague who has read an immense amount, who answers instantly, never says "I'm tired," and loves to be helpful. But this colleague has no way of looking things up in the middle of the conversation, rarely says "I don't know," and sometimes fills a gap with something that sounds right. You would happily ask for a draft, a list of ideas or an explanation. You would not sign a legal document on their word alone.

Your turn: test the model on what you know

Exercise 1. Ask an assistant a question whose answer you know very well (your hometown, your profession, a hobby, a book you love). Mark each statement in its answer as correct, approximate or invented.

Exercise 2. Ask the same question twice in two separate conversations. Compare the answers: what is identical, what changes?

Project. In your prompt notebook, add a section "What I learned about reliability" and write two sentences describing what you observed. You will use this as a reminder when deciding what to double-check.

See a solution

Typical observations: the general outline of the answer is usually right, but details (dates, names, exact numbers, titles) are sometimes approximate or invented. Across the two conversations, the main ideas often match while the wording, order and examples differ, which illustrates the probabilistic nature of generation.

A good note for your notebook: "The AI is reliable for the general idea and style, less for precise facts. I always check dates, figures, names and quotes, and I save the answers I like instead of expecting to get them again."