BM
Bashar ModallalSOFTWARE · AI · SYSTEMS
← All field notes

AI / الذكاء الاصطناعي

Embeddings and Semantic Search: What Similarity Can and Cannot Tell You

Embeddings help semantic search find related passages, but a nearest neighbor is not necessarily relevant. Learn how to interpret similarity, add exact filters, and evaluate retrieval.

A glowing query point in a vector space connects to a nearby teal neighbor while an amber point sits farther away.

Embeddings and Semantic Search: What Similarity Can and Cannot Tell You

An embedding represents an item of text as a vector: a list of numbers produced by a model. A semantic search system embeds both stored passages and a query, then retrieves passages whose vectors are near the query vector. This can find related wording even when the two texts share few exact terms. It does not prove that a passage answers the question.

From text to neighbors

Sentence-BERT showed how sentence embeddings can be compared efficiently with cosine similarity. Dense Passage Retrieval used separate query and passage encoders to retrieve candidate passages for open-domain question answering. These designs make retrieval practical at scale, but their vectors are useful only within the representation space they were trained to create. Vectors from unrelated models, versions, or preprocessing pipelines should not be compared as if they shared coordinates.

For normalized vectors, cosine similarity is the dot product; cosine distance is often expressed as one minus that similarity. Consider a tiny illustrative example: a query vector q = [1, 0], a passage vector a = [0.9, 0.1], and b = [0, 1]. Cosine similarity ranks a closer to q than b. The numbers only demonstrate the geometry. They are not model output, a relevance score, or evidence that a is useful.

A ranking answers “which candidates are nearest under this representation and metric?” It does not answer “is this result correct, current, authorized, or sufficient?” A nearest neighbor can share a broad subject while missing an exact product code, version, date, negation, or exception.

Similarity is not confidence

A similarity score is not automatically a probability of relevance. Scores depend on the model, corpus, metric, normalization, and search implementation. Some services transform the raw metric before returning a rank score. A universal cutoff such as “accept anything above 0.8” has no reliable meaning without task-specific evaluation and calibration.

Keep metadata filters for hard requirements: tenant, permissions, language, document status, version, and date. Use lexical retrieval for exact names, identifiers, error codes, and quoted wording. A hybrid system can combine lexical matches and vector neighbors, then rerank the merged candidates. Microsoft’s vector-search documentation describes both vector and hybrid queries and the behavior of its returned scores; check the selected service’s own definition instead of assuming every score is raw cosine.

Evaluate the retrieval job

BEIR was introduced to test information-retrieval models across heterogeneous datasets and zero-shot settings. Its premise is useful operationally: performance on one familiar collection does not establish generalization to another domain. Build a reviewed query set from the material your application actually searches. For each query, record the relevant passages and assess metrics such as Recall@k, mean reciprocal rank, or nDCG, along with whether retrieved text contains the full evidence needed.

Inspect failures, not only aggregate scores. Look for exact-match misses, stale copies, false friends, language mismatches, and chunks that omit a qualifying sentence. Compare an embedding model and metric on the same test set. Re-run evaluation when the model, chunking, filters, corpus, or query path changes.

A robust semantic search system treats embedding similarity as candidate generation. It preserves source IDs and versions, applies access controls independently, and exposes evidence for downstream review. The similarity value helps order possibilities; relevance still has to be measured against the retrieval task.

Arabic edition: https://basharmodallal.com/blog/embeddings-semantic-search-limits-ar

References

#RAG

YOUR TAKE

What did you think?

Choose one reaction for this article.

Was this useful?

A cookie helps limit repeat responses. Your feedback is private.

Share this noteLinkedIn ↗X ↗Email ↗

Keep exploring.