BM
Bashar ModallalSOFTWARE · AI · SYSTEMS
← All field notes

AI / الذكاء الاصطناعي

Tokens and Attention: How LLMs Actually Generate Text

A clear guide to subword tokens, causal self-attention, next-token probabilities and sampling—and why fluent prediction is not factual verification.

Abstract token tiles flow through a lime attention lattice into a generated sentence on a dark forest background.

Arabic edition: اقرأ بالعربية.

A language model does not usually read a sentence as a row of whole words, then retrieve a ready-made reply. It converts text into token IDs, builds contextual representations with a neural network, and repeatedly scores what token could come next. That mechanism is useful, but it is not the same thing as a human understanding a fact.

Text becomes tokens

A tokenizer maps text to discrete units from a model's vocabulary. A unit may be a whole common word, a word fragment, punctuation, whitespace plus letters, or bytes represented through a subword scheme. The split is model-specific: “attention” might be one token in one vocabulary and several in another. A token count is therefore not a word count, and changing one character can change the segmentation around it.

GPT-2 described byte-level byte-pair encoding, which starts from bytes and iteratively combines frequent sequences. This lets a vocabulary represent many strings without needing a separate entry for every word or script. In deployed APIs, message structure and non-text input can also affect the total token count; use the target model's current tokenizer or counting method for capacity planning rather than a generic character estimate. See the GPT-2 language-model paper and the OpenAI token-counting guide.

Attention builds context

In a Transformer, each position is projected into query, key, and value vectors. Attention compares a position's query with keys for available positions, scales the scores, applies a softmax, and uses the resulting weights to combine value vectors. Multiple attention heads can learn different relationships. Position information helps the model distinguish order.

For a causal language model, a mask prevents a position from attending to future tokens. When the model predicts the next token, it can use the prompt and previously generated tokens, but not tokens it has yet to produce. The original Transformer paper introduced scaled dot-product attention and the multi-head architecture; GPT-2 applied a Transformer language model to left-to-right text generation.

One token at a time

The model produces scores, often called logits, over its vocabulary. Softmax turns them into a conditional probability distribution for the next token. A decoding rule then chooses or samples a candidate. The selected token is appended to the context, and the process repeats until a stopping condition is reached.

Consider a deliberately tiny, illustrative vocabulary after the prompt “The greenhouse stayed warm because the”:

Candidate Illustrative probability
glass 0.52
vents 0.21
door 0.16
quickly 0.07
bicycle 0.04

These numbers are invented to show the mechanics, not measured model output. A greedy rule selects “glass”; a sampling rule may select another candidate. The next distribution is then computed from the longer context “The greenhouse stayed warm because the glass”. A generated word can take several token steps, and the token boundary need not line up with a word boundary.

Temperature changes the sharpness of the distribution before sampling. Top-k keeps a fixed number of candidates; nucleus (top-p) sampling keeps a variable-sized set whose probabilities reach a chosen cumulative mass. These controls alter selection behavior, not the information the model has. Research on neural text degeneration shows why decoding choices matter: maximizing likelihood can become repetitive, while sampling the entire low-probability tail can become incoherent.

Prediction is not verification

The model is trained to assign probabilities to token sequences. A likely continuation can be fluent and still be false, outdated, or unsupported. Attention describes information flow through the network; it does not establish that a cited source is true or that a statement was checked against the world.

For engineering use, treat generated text as a proposal. Retrieve authoritative documentation or data for claims that matter, preserve links to that evidence, and validate outputs with tests or domain checks. In short: tokenization defines the units, attention mixes context, and decoding chooses a path through the model's predictions. None of these steps alone turns prediction into factual verification.

References

#LLM

YOUR TAKE

What did you think?

Choose one reaction for this article.

Was this useful?

A cookie helps limit repeat responses. Your feedback is private.

Share this noteLinkedIn ↗X ↗Email ↗

Keep exploring.