Tokens and Attention: How LLMs Actually Generate Text
A clear guide to subword tokens, causal self-attention, next-token probabilities and sampling—and why fluent prediction is not factual verification.
AI / الذكاء الاصطناعي
A practical guide to structure-aware RAG chunking, overlap, context and evaluation, with methods for preserving meaning and traceable source locations.

Arabic edition: اقرأ بالعربية.
A chunk is the unit a retrieval system searches and sends onward to a model. If a splitter cuts at an arbitrary character count, it may separate a heading from its explanation, split a table row, or leave a pronoun without its referent. If it keeps entire documents intact, a broad document can dilute a focused passage. Chunking is the design choice between those failure modes.
Chunking is not a goal by itself. It is useful when a source exceeds a model's input limit or when one large representation mixes several subjects. The original RAG paper describes a retriever selecting passages for an input and a generator conditioning on those passages; the quality of the units available to the retriever therefore shapes what can reach generation (Lewis et al.).
First define what a good retrieved result must contain. For a policy question, it may be a complete rule plus its scope and exception. For a troubleshooting guide, it might be one diagnostic step with its precondition and expected result. Keep the smallest passage that answers a realistic query without forcing the model to reconstruct missing context.
A practical splitter can follow a hierarchy:
This hybrid approach avoids treating a heading and its body as unrelated text while preventing very long sections from exceeding model limits. Azure AI Search documents fixed-size, structure-aware and semantic strategies and notes that overlap and size should vary with content and query shape; its published defaults are starting points, not universal settings (chunking guidance). Its document-layout guidance also describes preserving heading fields alongside chunk text for section-aware indexing (Document Layout).
A chunk copied from the middle of a report may contain “this limit” or “the second option” without naming the subject. Preserve a heading path and source metadata, or prepend a short context line such as “Deployment guide → rollback limits.” Keep this generated or inherited context separate from the verbatim source text so citations can still point to the original passage.
Anthropic's contextual retrieval work describes adding a concise chunk-specific explanation using the whole document before building both embedding and BM25 indexes. It also reports that retrieval outcomes depend on chunk boundaries, overlap, embedding choice and reranking, so contextual text complements a sound split rather than replacing it (Contextual Retrieval).
A useful chunk record might look like this:
{
"documentId": "guide-42",
"documentVersion": "2026-11",
"sectionPath": ["Deployments", "Rollback"],
"pageStart": 8,
"pageEnd": 9,
"sourceUri": "https://docs.example/guide-42",
"text": "The rollback window is measured from deployment completion..."
}
The values are illustrative. In production, source IDs, versions, page locations and access controls must come from the ingestion system, not be invented by a language model.
A table should usually stay with its column headings. If it is too large, split by rows and repeat the header in each chunk. Keep code with its language, filename or section heading; split at functions or logical blocks rather than in the middle of a statement. For prose, overlap can preserve a transition, but every duplicated token also creates repeated index content and can make the same evidence appear multiple times in retrieval results.
There is no universally correct size or overlap percentage. Compare a few candidate strategies using representative questions with known supporting passages. Measure whether retrieval includes the complete answer-bearing evidence, whether neighboring chunks create duplicates, and whether citation locations remain traceable. Tune the strategy by document type, not only by a global token target.
Build a small reviewed query set with expected source passages. For each query, inspect retrieved chunk IDs and source locations before evaluating the generated answer. Separate failures into extraction, boundary choice, metadata, retrieval ranking and answer synthesis. A better chunk may help retrieval, but it cannot repair text lost during OCR or guarantee a correct answer.
A reliable chunk retains enough local meaning to be found, enough provenance to be checked, and a stable link to its parent document. Start from structure, cap length with model-aware limits, preserve context explicitly, and validate against real queries.
YOUR TAKE
Choose one reaction for this article.
A cookie helps limit repeat responses. Your feedback is private.