Embeddings, Vector Databases and RAG

Retrieval-augmented generation answers questions from your documents rather than from the model's memory. A search step finds the most relevant passages, and the model writes its answer from them, citing what it used. This article walks through each stage, from embeddings to evaluation, and the places where real systems usually go wrong.

Why retrieval?

An LLM's knowledge is frozen when training ends and spread across billions of parameters. It cannot see your internal wiki, last week's price change or a customer's contract, and when asked about them it may produce a confident guess: a hallucination. Retrieval-augmented generation (RAG) fixes this by fetching relevant text at query time and placing it in the prompt. The model's job changes from "remember" to "read and summarise", which it does far more reliably.

Retrieval brings four practical benefits:

• Fresh knowledge. Update a document and the next answer reflects it, with no retraining.
• Private data. Documents stay in your own store and can be filtered by each user's permissions.
• Citations. The answer is built from known passages, so users can check the sources.
• Cost. Teaching a model facts through fine-tuning is expensive, slow to update and unreliable at recall. Fine-tuning suits style and format; for facts, retrieval is cheaper and easier to audit.

A RAG system has two halves that share one index: indexing runs offline whenever documents change, and the query pipeline runs on every request.

Embeddings and similarity

An embedding is a fixed-length vector, typically a few hundred to a few thousand numbers, that represents the meaning of a piece of text. An embedding model is trained so that texts with similar meaning produce vectors pointing in similar directions. "How many days of holiday do I get?" and "annual leave entitlement" share almost no words, yet their embeddings lie close together.

Closeness is usually measured with cosine similarity: the cosine of the angle between two vectors, from −1 (opposite) to 1 (same direction). If you normalise vectors to unit length when you store them, cosine similarity becomes a plain dot product, which is what vector indexes compute fastest. Some models are trained for dot product or Euclidean distance instead, so use the metric the model's documentation recommends.

Embedding models are not the chat model

Retrieval embeddings come from a separate model, much smaller than the LLM that writes the answer. Most are bi-encoders in the style of Sentence-BERT: a transformer encoder reads the text and pools its token vectors into a single vector. They are cheap enough to embed millions of chunks and fast enough to embed every query. (The token embeddings inside an LLM, described in What Is a Large Language Model?, are a different thing.)

Three rules follow. Embed queries and documents with the same model, because vectors from different models live in unrelated spaces. Expect a model change to mean re-embedding the whole corpus. And check whether the model expects different prefixes for queries and documents; skipping them lowers quality without any visible error. Open-weight embedding models run well on modest hardware, which helps when documents cannot leave your network (see running models locally).

Chunking and metadata

Documents are too long to embed whole. One vector cannot represent fifty pages, and the prompt could not hold them anyway. Chunking splits each document into passages that are embedded and retrieved on their own. The chunk is the unit the model will read, so its boundaries decide what context survives.

• Fixed-size windows of a few hundred tokens, with some overlap so that a sentence cut at a boundary appears in both neighbours. Simple and predictable, but blind to structure.
• Structure-aware splitting on headings, paragraphs, list items, table rows or code functions. Usually better for manuals, policies and source code.
• Semantic splitting, which starts a new chunk where the meaning of consecutive sentences shifts sharply. Useful for long unstructured prose, at extra cost.

Small chunks match queries precisely but lose their surroundings; large chunks keep context but dilute the embedding. A common compromise is to retrieve small chunks and expand each hit to its parent section before it goes into the prompt. Another is to prepend the document title and section heading to every chunk, so that "the limit is 30 days" carries the fact that it is about refunds.

Store metadata with every chunk: source, URL, section, date, language and access permissions. Metadata lets you filter during search ("only the current policy"), render citations, and trace a bad answer back to the passage that caused it.

Vector databases and approximate search

Finding the nearest vectors exactly means comparing the query with every stored vector: fine for tens of thousands of chunks, too slow for tens of millions. A vector database uses an approximate nearest-neighbour (ANN) index that gives up a little recall in exchange for far less work per query.

The most widely used index is HNSW (Hierarchical Navigable Small World). It builds a layered graph in which each vector links to a handful of close neighbours, and each higher layer holds fewer nodes with longer links. A search enters at the sparse top layer, greedily walks towards the query, then drops down a layer and refines, much like zooming in on a map. Two settings matter: how many links each node keeps (memory and build time) and how many candidates a search explores (recall versus latency). Alternatives include IVF, which searches only the nearest clusters, and product quantization, which compresses vectors.

As of 2026, the options fall into three broad groups:

For many teams pgvector is a sensible default. Chunks, metadata and permissions live next to the rest of the application data, and filters are ordinary WHERE clauses. Move to a dedicated engine when scale, latency or operational needs justify running a second system.

The query pipeline

Basic RAG embeds the question, takes the top k chunks and pastes them into a prompt. Production systems add a few stages, each fixing a specific weakness.

Query rewriting

User questions are often short, ambiguous or dependent on the conversation: "and what about contractors?" means nothing on its own. A cheap LLM call can rewrite the question into a self-contained query, expand acronyms, or split a compound question into several searches. A related technique, HyDE, asks the model to draft a hypothetical answer and searches with its embedding, because answers tend to resemble documents more than questions do.

Hybrid search

Embeddings capture meaning but are weak on exact tokens: error codes, part numbers, names and rare jargon. Classic keyword search with BM25 is the opposite, precise on exact terms and blind to paraphrase. Hybrid search runs both and merges the two ranked lists, often with reciprocal rank fusion (RRF), which scores each document by summing 1/(k + rank) across the lists. Because RRF uses only rank positions, the two systems' scores never need to be calibrated against each other.

Reranking

First-stage search is built for speed and recall, not precision. A reranker, usually a cross-encoder, reads the query and one candidate passage together and outputs a relevance score. Comparing the texts token by token makes it far more accurate than comparing two precomputed vectors, and far too slow for the whole corpus, so it re-orders a few dozen candidates and keeps the best handful.

Prompt assembly and citations

The final prompt holds instructions, the retrieved chunks with identifiers and source labels, and the question. Tell the model to answer only from the sources, to cite them by identifier, and to say plainly when the sources do not contain the answer. The core retrieval step needs nothing more than a matrix and a dot product:

import numpy as np

def embed(texts: list[str]) -> np.ndarray:
    """Placeholder: call your embedding model or API here.
    Must return one vector per text, shape (len(texts), dim)."""
    raise NotImplementedError

def normalise(v: np.ndarray) -> np.ndarray:
    return v / np.linalg.norm(v, axis=-1, keepdims=True)

# Indexing (offline): embed every chunk once and keep unit-length vectors.
chunks = [
    {"id": "hr-12#3", "source": "Leave policy", "text": "Employees accrue 25 days of annual leave..."},
    {"id": "it-04#1", "source": "IT standard",  "text": "Laptops must use full-disk encryption..."},
    # ... thousands more
]
index = normalise(embed([c["text"] for c in chunks]))   # shape (n_chunks, dim)

def retrieve(query: str, k: int = 5) -> list[dict]:
    q = normalise(embed([query]))[0]                     # same model as the index
    scores = index @ q                                   # cosine similarity for unit vectors
    top = np.argsort(-scores)[:k]
    return [{**chunks[i], "score": float(scores[i])} for i in top]

def build_prompt(question: str, hits: list[dict]) -> str:
    sources = "\n\n".join(
        f"[{n}] ({h['source']}) {h['text']}" for n, h in enumerate(hits, start=1)
    )
    return (
        "Answer using only the numbered sources below and cite them like [1]. "
        "If they do not contain the answer, say that you do not know.\n\n"
        f"{sources}\n\nQuestion: {question}"
    )

Once the corpus grows, swap the exhaustive dot product for a vector-database query; the rest of the pipeline stays the same. Because every chunk keeps its identifier, the application can check that each [n] in the answer points to a passage that was actually retrieved, and render it as a link.

Evaluating a RAG system

RAG has two parts that fail independently, so measure them separately. When an answer is wrong, you need to know whether the right passage never arrived or arrived and was ignored.

• Retrieval quality. Recall@k asks, for each test question, whether at least one relevant chunk appears in the top k. Rank-aware metrics such as MRR or nDCG also reward putting it near the top. These need labelled relevant chunks but no LLM.
• Generation quality. Faithfulness asks whether every claim in the answer is supported by the retrieved passages; correctness asks whether the answer matches a reference. Both are usually scored by an LLM judge and spot-checked by people.

Start with a small golden set: a few dozen to a few hundred real questions, each with the passages that answer it and a reference answer. Take questions from search logs or support tickets rather than inventing them, and include some the corpus cannot answer. Run the set after every change to chunking, embedding model, k or prompt. These evals turn "it feels better" into numbers. Tools such as Ragas package common metrics, but a script and a spreadsheet are enough to begin.

Long context, agents and RAG

Do long context windows make RAG obsolete?

A context window of hundreds of thousands of tokens makes it tempting to skip retrieval and paste everything in. For one long document or a small, stable corpus that can work well, and it removes retrieval errors entirely. But every request pays for every token, latency grows with input length, and most knowledge bases are far larger than any window. Quality suffers too: Liu et al. showed in "Lost in the Middle" that models use information at the start and end of a long context more reliably than information buried in the middle.

In practice the two combine: retrieve generously, rerank, and use the larger window for whole sections instead of fragments, with the strongest evidence first.

Agentic RAG

The pipelines above retrieve once, before the model starts writing. In agentic RAG, search becomes a tool that the model invokes through tool calling. It decides what to look for, reads the results, and searches again with a refined query or turns to another source, such as a SQL database. This handles multi-hop questions ("which suppliers delivered late last quarter, and what do their contracts say about penalties?") that no single search answers. The price is more LLM calls, higher latency and less predictable behaviour, so reserve it for questions where single-shot retrieval measurably fails. The loop itself is covered in AI agents, tool calling and MCP.

Common failure modes

Most RAG problems are retrieval or data problems that surface as bad answers. Start debugging from the symptom: