AI / الذكاء الاصطناعي / SELECTED NOTE
RAG and LLMs: Building Answers Grounded in Your Data
A practical introduction to RAG and large language models: retrieval, source grounding, permissions, evaluation, and Arabic–English search.

تفضّل العربية؟ اقرأ النسخة العربية.
An AI assistant can explain a software concept clearly and still give the wrong answer about your product. Your latest release notes, internal APIs, and operating procedures may never have been part of its training.
For a useful assistant, I want to know which evidence supports the answer and whether the person asking is allowed to see it. Retrieval-augmented generation, or RAG, gives us an architecture for bringing that evidence into the conversation.
What an LLM brings to the system
A large language model (LLM) generates text using patterns learned during training and the context supplied with a request. It can explain, summarize, translate, and connect ideas. A fluent answer, however, does not establish that its claims match your current documentation. Google's introduction to LLMs explains the underlying language-modeling concepts.
An LLM does not automatically have access to your private knowledge base or a live view of your application. Your system must provide that access deliberately.
What RAG adds
RAG retrieves relevant information and supplies it to the model as context for generating an answer. The original RAG paper combined a language model with an external retrieval component for knowledge-intensive tasks.
In a typical application, this means searching approved documents, selecting useful passages, and asking the LLM to answer from that material. Updating the searchable documents can update the available evidence without retraining the model, provided your ingestion and indexing pipeline processes the changes.
Retrieval can use keyword search, vector similarity, a database lookup, or a combination. A vector database is one implementation choice, rather than a requirement for every RAG system.
A concrete example
Imagine a fictional support assistant for a reporting platform. Someone asks:
How long are exported reports available to download?
The current documentation says seven days; an older page says thirty. A useful answer needs the applicable version and a traceable source. Finding a related paragraph without checking its status could produce a confident, outdated response.
That example shapes the architecture I would start with:
PREPARATION
Approved documents → parse → meaningful chunks + metadata → index
ANSWERING
Question + authenticated identity
→ retrieve within allowed scope
→ optionally rerank and select evidence
→ LLM generates an answer
→ check source support and return citations
I would preserve a document ID, section title, source URL, version, update time, and access-control information alongside each passage. These fields help the application decide which evidence belongs in an answer.
Three decisions that shape retrieval
Preserve the context of each passage
A sentence such as “files expire after seven days” loses meaning when separated from its product, feature, or version. Start with section-aware chunks and keep useful headings. Evaluate chunk length and overlap against your documents rather than choosing one universal number.
Anthropic's Contextual Retrieval explores adding explanatory context to chunks before indexing. It is a useful technique to test when isolated passages lose important meaning.
Test keyword and semantic search together
Semantic search can find related ideas across different wording. Keyword search can be valuable for exact identifiers such as ERR_EXPORT_410, product names, and version numbers. Hybrid retrieval combines the two; Microsoft's hybrid search overview describes one implementation using rank fusion.
I would compare keyword, vector, and hybrid baselines on representative questions before adding a reranker. Keep the option that improves the results enough to justify its latency and cost.
Keep freshness and permissions in the retrieval path
Expired documents, deleted pages, and revoked permissions need to propagate through the index and caches. Resolve the user's access scope on the server and enforce it before any passage reaches the model. Microsoft's security-filter pattern illustrates document filtering; the application still has to authenticate the caller and apply the correct scope consistently.
An application flow
This TypeScript-style sketch shows where those responsibilities sit. The functions represent application services:
const scope = await permissions.forUser(authenticatedUser.id);
const candidates = await retrieve(question, {
scope,
activeDocumentsOnly: true,
});
const evidence = await selectEvidence(candidates, {
tokenBudget: config.contextBudget,
});
if (!hasEnoughEvidence(question, evidence)) {
return { status: "insufficient_evidence" };
}
return generateAnswer({
question,
evidence,
includeSourceReferences: true,
});
The evidence gate needs criteria tested on real questions. A similarity score alone cannot establish that a passage answers the question. I would also define how to handle conflicting versions and when to ask a clarifying question.
RAG and fine-tuning
RAG changes the evidence available during a request. Fine-tuning trains model parameters using additional examples, as described in Google's fine-tuning guide.
For an assistant answering from frequently updated documents, retrieval is a sensible starting point. If the remaining problem concerns a consistent output format or specialized task behavior, fine-tuning may be worth evaluating. The two approaches can work together.
Retrieved content needs a trust boundary
A retrieved document may contain malicious instructions. Treat its contents as data, separate them from application instructions, and enforce tool permissions in code. The OWASP prompt-injection guidance explains why RAG does not eliminate this risk.
Ask for citations, then check that they exist and support the associated claims. A citation marker and confident wording are insufficient evidence of correctness. When the available sources cannot support an answer, the product should communicate that limitation clearly.
Evaluate retrieval and answers separately
I would build a small evaluation set before changing models or adding more infrastructure:
| Layer | Questions to test |
|---|---|
| Retrieval | Did we find the relevant passage and the applicable document version? |
| Answer | Are the claims supported, correct, and responsive to the question? |
| Citations | Can each important claim be traced to evidence that supports it? |
| Access | Do restricted passages stay out of unauthorized requests? |
| Missing evidence | Does the assistant abstain or clarify appropriately? |
| Operations | What are end-to-end latency and cost per useful answer? |
The Ragas paper provides a research starting point for evaluating retrieval and generation along different dimensions. Automated scores can help identify regressions; human review remains useful for ambiguous or consequential answers.
Arabic and English need their own test cases
For bilingual systems, include Arabic questions, English questions, and mixed queries containing identifiers. Test diacritics, character variants, and multilingual retrieval on the actual corpus. Microsoft's multilingual indexing guidance describes language-aware analysis and multilingual vectors. I would preserve original text for display and avoid normalization rules that damage names or IDs.
Where I would begin
Start with a narrow document collection and a clear set of questions. Make the sources inspectable, keep permissions explicit, and record where failures occur. Then improve the weakest part of the pipeline.
The useful outcome is an assistant whose answers people can check against evidence, and whose limits are clear when that evidence is missing.