Embeddings and Vector Search

An embedding is a dense numeric representation of text. Similar meaning maps to nearby points in vector space, so a query can retrieve related passages without exact keyword overlap. Vector search finds the closest stored vectors to a query vector, usually as the first stage in retrieval-augmented generation.

Embeddings power semantic recall. They do not replace permissions, citations, reranking, or generation quality on their own.

How the pieces fit together

From documents to vector retrieval

flowchart LR
Docs["Source documents"] --> Chunk["Chunk text"]
Chunk --> EmbedDoc["Embedding model"]
EmbedDoc --> Index["Vector index"]
Question["User question"] --> EmbedQuery["Same embedding model"]
EmbedQuery --> Search["Nearest-neighbor search"]
Index --> Search
Search --> Candidates["Top-k chunks"]
Candidates --> Rerank["Optional rerank"]
Rerank --> RAG["RAG generation"]

The query and the indexed chunks must use the same embedding model and preprocessing rules. Changing either side without reindexing breaks retrieval quality.

Embeddings

An embedding model converts text into a fixed-length vector, often a few hundred to a few thousand dimensions. Vectors that point in similar directions represent related concepts, paraphrases, or nearby topics.

Practical implications:

Same model at ingest and query time. Mixing models compares incompatible spaces.
Language and domain matter. A general model may miss product-specific jargon unless fine-tuned or paired with lexical search.
Model upgrades are migrations. Swapping embedding models usually requires re-embedding the corpus and rerunning evaluation.
Cost and latency. Embedding every chunk at ingest is cheap relative to generation, but large corpora still need batching and monitoring.

Chunking before embedding

Indexes store chunks, not whole libraries. Chunk design changes retrieval more than index choice.

Smaller chunks improve precision for pinpoint facts but may lose surrounding context.
Larger chunks preserve narrative but can bury the answer in noise.
Overlap between adjacent chunks can improve recall when answers span boundaries.
Metadata such as title, product, version, tenant, and permissions should travel with each chunk.

Chunking is a product decision. Legal clauses, API docs, and support tickets often need different sizes and overlap rules.

Vector indexes and similarity

A vector index stores embeddings and supports fast approximate nearest-neighbor search at query time. Common options include dedicated vector databases, search engines with vector fields, and pgvector-style extensions.

Similarity is usually measured with:

Cosine similarity, angle between vectors, common for normalized embeddings.
Dot product, related to cosine when vectors are normalized; some systems optimize for it.
Euclidean distance, less common for text embeddings but available in some stores.

Indexes trade recall, latency, memory, and operational complexity. For many RAG products, a managed index plus good chunking beats self-hosting exotic ANN tuning early on.

Vector search alone is not enough

Pure vector search is strong at semantic match and weak at exact identifiers.

Use hybrid retrieval when queries include:

Error codes, UUIDs, SKUs, and version numbers.
Person names, company names, and product names that embeddings may blur.
Rare terms that appear rarely in the corpus.

Hybrid search combines vector results with lexical search such as BM25 or inverted-index keyword matching, then merges or reranks the candidate set. This pattern appears throughout RAG because real user questions mix concepts with exact strings.

Minimal retrieval shape

async function retrieveChunks(question, { limit = 8, filters = {} }) {
  const queryVector = await embedText(question);

  const vectorHits = await vectorIndex.search(queryVector, {
    limit: limit * 2,
    filters,
  });

  const lexicalHits = await keywordIndex.search(question, {
    limit: limit * 2,
    filters,
  });

  return mergeAndDedupe(vectorHits, lexicalHits).slice(0, limit);
}

type SearchFilters = Record<string, string | number | boolean>;

async function retrieveChunks(
  question: string,
  { limit = 8, filters = {} }: { limit?: number; filters?: SearchFilters } = {},
) {
  const queryVector = await embedText(question);

  const vectorHits = await vectorIndex.search(queryVector, {
    limit: limit * 2,
    filters,
  });

  const lexicalHits = await keywordIndex.search(question, {
    limit: limit * 2,
    filters,
  });

  return mergeAndDedupe(vectorHits, lexicalHits).slice(0, limit);
}

Filters enforce tenant boundaries and permissions before generation. Retrieval guardrails belong here, not only in the final prompt.

Metadata, freshness, and security

Every stored chunk should carry the metadata needed to filter safely:

Tenant, workspace, or user visibility.
Product, environment, language, and document version.
Source URL or document ID for citations.
Ingest timestamp or content hash for freshness checks.

Stale indexes produce confident wrong answers. Treat reindexing as part of the release process when source documents change often.

Retrieved text is untrusted input. Chunks can carry prompt injection payloads even when the user question looks benign.

When to use vector search

Vector search helps when:

Users ask in natural language and exact keywords vary.
The corpus is large enough that prompt stuffing is impossible.
Answers depend on semantic similarity across documents.

Skip or narrow it when:

The task is exact lookup by ID or key.
The dataset is tiny and structured search or SQL is simpler.
Wrong retrieval is costlier than no retrieval.

Trade-offs

Embeddings add ingest pipelines, storage, and another moving part to monitor.
Approximate indexes can miss the best chunk, which is why reranking and hybrid search matter.
Embedding quality drifts silently when models, chunking, or source documents change.
Hybrid retrieval costs more engineering than vector-only demos, but it matches production query behavior.

Embeddings are the retrieval primitive under many RAG systems. Get chunking, permissions, hybrid search, and evaluation right, and vector search becomes a dependable source of evidence instead of a demo trick.