BM
Bashar ModallalSOFTWARE · AI · SYSTEMS
← All field notes

AI / الذكاء الاصطناعي

Chunking Documents Without Losing Meaning

A practical guide to structure-aware RAG chunking, overlap, context and evaluation, with methods for preserving meaning and traceable source locations.

A long document divided into connected, meaning-preserving sections on a dark green background.

Arabic edition: اقرأ بالعربية.

A chunk is the unit a retrieval system searches and sends onward to a model. If a splitter cuts at an arbitrary character count, it may separate a heading from its explanation, split a table row, or leave a pronoun without its referent. If it keeps entire documents intact, a broad document can dilute a focused passage. Chunking is the design choice between those failure modes.

Start with the retrieval job

Chunking is not a goal by itself. It is useful when a source exceeds a model's input limit or when one large representation mixes several subjects. The original RAG paper describes a retriever selecting passages for an input and a generator conditioning on those passages; the quality of the units available to the retriever therefore shapes what can reach generation (Lewis et al.).

First define what a good retrieved result must contain. For a policy question, it may be a complete rule plus its scope and exception. For a troubleshooting guide, it might be one diagnostic step with its precondition and expected result. Keep the smallest passage that answers a realistic query without forcing the model to reconstruct missing context.

Prefer document structure, then enforce a size limit

A practical splitter can follow a hierarchy:

  1. Preserve the source structure: document, heading path, paragraph, list, table, code block, and page location.
  2. Treat a section or coherent group of paragraphs as the initial chunk.
  3. Split an oversized section at paragraph or sentence boundaries.
  4. Use a token limit as a final bound, not as the only boundary rule.
  5. Add modest overlap only where a boundary can separate a definition from its use.

This hybrid approach avoids treating a heading and its body as unrelated text while preventing very long sections from exceeding model limits. Azure AI Search documents fixed-size, structure-aware and semantic strategies and notes that overlap and size should vary with content and query shape; its published defaults are starting points, not universal settings (chunking guidance). Its document-layout guidance also describes preserving heading fields alongside chunk text for section-aware indexing (Document Layout).

Keep the missing context attached

A chunk copied from the middle of a report may contain “this limit” or “the second option” without naming the subject. Preserve a heading path and source metadata, or prepend a short context line such as “Deployment guide → rollback limits.” Keep this generated or inherited context separate from the verbatim source text so citations can still point to the original passage.

Anthropic's contextual retrieval work describes adding a concise chunk-specific explanation using the whole document before building both embedding and BM25 indexes. It also reports that retrieval outcomes depend on chunk boundaries, overlap, embedding choice and reranking, so contextual text complements a sound split rather than replacing it (Contextual Retrieval).

A useful chunk record might look like this:

{
  "documentId": "guide-42",
  "documentVersion": "2026-11",
  "sectionPath": ["Deployments", "Rollback"],
  "pageStart": 8,
  "pageEnd": 9,
  "sourceUri": "https://docs.example/guide-42",
  "text": "The rollback window is measured from deployment completion..."
}

The values are illustrative. In production, source IDs, versions, page locations and access controls must come from the ingestion system, not be invented by a language model.

Handle tables, code and overlap deliberately

A table should usually stay with its column headings. If it is too large, split by rows and repeat the header in each chunk. Keep code with its language, filename or section heading; split at functions or logical blocks rather than in the middle of a statement. For prose, overlap can preserve a transition, but every duplicated token also creates repeated index content and can make the same evidence appear multiple times in retrieval results.

There is no universally correct size or overlap percentage. Compare a few candidate strategies using representative questions with known supporting passages. Measure whether retrieval includes the complete answer-bearing evidence, whether neighboring chunks create duplicates, and whether citation locations remain traceable. Tune the strategy by document type, not only by a global token target.

Evaluate the whole path

Build a small reviewed query set with expected source passages. For each query, inspect retrieved chunk IDs and source locations before evaluating the generated answer. Separate failures into extraction, boundary choice, metadata, retrieval ranking and answer synthesis. A better chunk may help retrieval, but it cannot repair text lost during OCR or guarantee a correct answer.

A reliable chunk retains enough local meaning to be found, enough provenance to be checked, and a stable link to its parent document. Start from structure, cap length with model-aware limits, preserve context explicitly, and validate against real queries.

References

#RAG

YOUR TAKE

What did you think?

Choose one reaction for this article.

Was this useful?

A cookie helps limit repeat responses. Your feedback is private.

Share this noteLinkedIn ↗X ↗Email ↗

Keep exploring.