Keyword search matches strings: a search for “car” misses a document that only says “automobile.” Vector search matches meaning. By converting content into numerical vectors whose geometric closeness reflects semantic similarity, you can ask “find things that mean the same as this” — and that is the capability behind modern semantic search, recommendation, and the retrieval step that grounds large language models. A vector database is the system that stores those vectors and answers “what is nearest to this?” at scale.
In the interview: the trigger phrases are “semantic search,” “similar items / recommendations,” “search by meaning,” “find duplicates,” and anything involving an LLM that needs to look things up (RAG). When you hear them, reach for embeddings + a vector database with an ANN index, and be ready to discuss the one real trade-off: approximate search buys speed at the cost of a little recall.
Embeddings: meaning as geometry
An embedding model (a neural network) maps an input — a sentence, a product, an image — to a fixed-length vector of floating-point numbers, typically a few hundred to a few thousand dimensions. The model is trained so that inputs with similar meaning map to vectors that are close together, and dissimilar inputs map far apart. “car” and “automobile” land next to each other; “car” and “banana” land far away.
Closeness is measured by a distance function — usually cosine similarity (the angle between vectors) or dot product. So the fundamental query becomes purely geometric: given the query’s vector, which stored vectors have the smallest distance to it? That is a k-nearest-neighbour (kNN) search.
Why you need an approximate index
The obvious approach — compare the query vector against every stored vector and keep the closest k — is exact but O(n) per query. With a few thousand vectors that is fine; with a hundred million 768-dimensional vectors it is hopeless, and the curse of dimensionality means the clever low-dimensional tricks (k-d trees) stop helping.
The answer is Approximate Nearest Neighbour (ANN) search: accept a result that is almost always the true nearest neighbours in exchange for an enormous speed-up. Two families dominate:
- HNSW (Hierarchical Navigable Small World). Builds a layered graph where each vector links to its near neighbours, with sparse “express lane” layers on top. A query greedily walks the graph toward the query vector, descending layers — reaching the neighbourhood in roughly logarithmic hops. Excellent recall and latency; higher memory and build cost. It is the default in most vector databases.
- IVF (Inverted File / clustering). Clusters the vectors (e.g. with k-means) into cells; a query only searches the few nearest cells instead of the whole set. Cheaper memory, tunable via how many cells you probe. Often combined with product quantization (PQ) to compress vectors and shrink the index.
The knob both expose is recall vs latency/memory: search more of the graph or more cells and you find the true neighbours more often but slower. Stating that trade-off explicitly — “I’ll tune HNSW’s search breadth for ~95% recall at single-digit-millisecond latency” — is exactly the level of answer interviewers want.
The architecture
A vector search system has two paths that must use the same embedding model:
- Indexing (offline / write path). For each document, run the embedding model to get its vector, and store the vector plus its metadata (id, source, timestamps, tags, and often the original text) in the vector database, which builds the ANN index. Re-embedding is needed whenever you change the model.
- Querying (online / read path). Embed the query with the same model, ask the ANN index for the top-k nearest vectors, then:
- Filter by metadata — restrict to a tenant, a language, a date range, or an access-control scope (pre- or post-filtering has real performance implications worth mentioning).
- Rerank — optionally pass the top candidates through a heavier, more accurate model (a cross-encoder) to reorder the final few.
# Indexing
vec = embed(document.text) # same model both sides
store(id, vec, metadata={...})
# Querying
qvec = embed(query)
hits = index.search(qvec, k=20, filter={"tenant": t})
results = rerank(query, hits)[:5]Operationally these systems must handle scale by sharding vectors across nodes and replicating for availability; handle updates (ANN indexes are happiest with batch inserts — frequent single updates can degrade an HNSW graph, so many systems rebuild segments in the background, much like an LSM compaction); and decide between a dedicated store and a bolt-on.
The ecosystem and where it fits
| Option | Shape | Reach for it when |
|---|---|---|
| pgvector | Postgres extension | You already run Postgres and want vectors alongside relational data |
| FAISS | A library, not a server | You need an in-process, high-performance ANN index to embed in an app |
| Pinecone | Managed service | You want vector search without operating it |
| Milvus / Weaviate / Qdrant | Dedicated vector databases | Large scale, metadata filtering, hybrid (keyword + vector) search |
The dominant use cases:
- Semantic search — retrieve by meaning, often blended with keyword (BM25) search in a hybrid ranker.
- Recommendations and “similar items” — nearest neighbours in an item-embedding space.
- De-duplication and clustering — near-duplicate detection by vector proximity.
- Retrieval-Augmented Generation (RAG) — the big one: to answer a question with an LLM, embed the question, retrieve the most relevant document chunks from the vector store, and stuff them into the prompt so the model answers from real, current data instead of hallucinating. Any “design a system that answers questions over our docs with an LLM” question has a vector database at its heart.
Key Takeaways
-
Embeddings turn meaning into geometry. An embedding model maps content to a vector where closeness equals semantic similarity, measured by cosine or dot-product distance.
-
The core operation is nearest-neighbour search. Exact kNN is O(n) and too slow at scale, so vector databases use approximate (ANN) indexes.
-
HNSW and IVF are the two ANN families. HNSW is a navigable graph (great recall/latency, more memory); IVF clusters and probes a few cells (cheaper, tunable), often with product quantization.
-
The real trade-off is recall vs latency/memory. Search more of the index for higher recall at higher cost — tune it to your needs and say so.
-
Embed documents and queries with the SAME model, then filter and rerank. Index offline; at query time embed, retrieve top-k, filter by metadata, optionally rerank with a heavier model.
-
It powers semantic search, recommendations, and RAG. When an LLM needs to look up grounded, current information, a vector database is the retrieval layer.