Vector Databases in 2023: Choosing the Right Store for RAG

The vector database market spent the last three years convincing everyone they needed a specialised database. In 2023, the more useful question is the opposite: do you? For a large share of the RAG systems we build, the honest answer is that Postgres with the pgvector extension is enough, and reaching for a dedicated vector database adds an operational dependency the workload never justified. For another share, a purpose-built store is the difference between a system that works at ten thousand documents and one that works at a hundred million. Knowing which situation you are in is the whole skill.

This is a practitioner's guide to choosing, drawn from having shipped both — and having migrated more than one client off the wrong choice.

The choice is really about four things

Underneath the marketing, a vector store decision comes down to four concrete axes. Get clear on where you sit on each before you read a single benchmark:

• Scale — thousands, millions, or hundreds of millions of vectors? This alone eliminates most options.
• Filtering — do you need to combine similarity search with metadata filters (tenant, date, permission)? This is where many systems quietly break.
• Freshness — is your corpus static, or are you inserting and deleting continuously?
• Operational surface — how much infrastructure can your team actually run and maintain?

Nobody was ever fired for having one fewer database to operate. The best vector store is often the one your team already knows how to back up, monitor and restore at 3 a.m.

Option 1: pgvector, and why it's the default we start from

If you already run Postgres — and most teams do — pgvector lets you store embeddings next to your relational data and query them with SQL. The advantages compound: transactional consistency between your documents and their vectors, metadata filtering using the full power of SQL and existing indexes, one system to operate, and joins to the rest of your data model for free.

With HNSW indexing, pgvector comfortably serves millions of vectors with good latency. Its real strength is filtered search: because filters are just SQL predicates on indexed columns, "find similar chunks belonging to this tenant, created after this date, that this user may see" is a natural query rather than a feature you have to hope your vector database implemented well.

-- HNSW index for fast approximate search
CREATE INDEX ON chunks
  USING hnsw (embedding vector_cosine_ops);

-- Similarity search WITH metadata filtering
SELECT id, content
FROM chunks
WHERE tenant_id = $1
  AND created_at >= $2
ORDER BY embedding <=> $3   -- cosine distance to query vector
LIMIT 8;

Where pgvector strains is at very high scale (hundreds of millions of vectors), extreme write throughput, or when you need advanced retrieval features — sparse-dense hybrid search, learned rerankers — as first-class primitives. That is the signal to look further.

Option 2: dedicated vector databases

Purpose-built stores — the Qdrant, Weaviate, Milvus family — are engineered from the ground up for approximate nearest-neighbour search at scale. They shine when vector search is genuinely the core of your workload rather than a feature bolted onto an application. You get horizontal scaling to billions of vectors, fine-grained control over the index (quantisation, ef/M tuning), and native hybrid search that fuses keyword and semantic signals.

The cost is a new system to run, keep consistent with your source of truth, secure and back up. The critical question to ask a dedicated store is not "how fast is unfiltered search?" but "how does it perform with selective metadata filters?" Many implementations degrade sharply when a filter excludes most of the corpus, because the graph index and the filter fight each other. Benchmark that path with your real filter distribution, not the vendor's.

Option 3: managed search platforms

Elasticsearch/OpenSearch and the managed cloud search services now offer strong vector support alongside mature keyword search, security and operations. If you already run one of these, or you need robust hybrid retrieval with battle-tested filtering and access control, they are a pragmatic choice. You trade some raw vector performance and index tunability for maturity, ecosystem and one less bespoke system. For enterprises with existing search infrastructure and strict permission models, this is frequently the path of least resistance.

The mistake almost everyone makes: ignoring filtered search

Benchmarks obsess over unfiltered recall and queries-per-second. Real RAG systems almost never do unfiltered search. Every production query we run carries filters — tenant isolation in multi-tenant products, document-level permissions, recency windows, source restrictions. And filtered vector search is a fundamentally harder problem: the approximate index is built for the whole space, but you only want results from a subset.

There are two strategies, and both have failure modes:

1. Pre-filtering — restrict the candidate set first, then search within it. Correct, but can be slow if the filter is expensive or the subset is large.
2. Post-filtering — search the whole index, then drop results that fail the filter. Fast, but if your filter is selective you may retrieve the top-K globally and find that zero of them match, silently returning too few results.

The right approach depends on filter selectivity, and a store that handles this well for your access pattern matters far more than a marginally higher benchmark score. Test with your actual filters, at your actual selectivity.

Embeddings and index tuning outlast the database choice

Two decisions have more impact on retrieval quality than which vector store you pick, and both travel with you across any database:

• The embedding model defines the ceiling of what similarity search can find. A better model lifts every downstream metric; a mismatched one caps you no matter how you index. And changing it means re-embedding your entire corpus — plan for that migration before you commit.
• Index parameters govern the recall-versus-speed trade-off. For HNSW, higher ef_search and M raise recall at the cost of latency and memory. Quantisation shrinks memory and cost but can erode recall. These knobs deserve deliberate tuning against a real evaluation set, not defaults copied from a tutorial.

Your vector database sets the operational envelope; your embedding model and index tuning set the quality. Teams over-invest in the first decision and under-invest in the second two.

How we actually decide

Our default: start with pgvector if the client runs Postgres and the corpus is in the millions or fewer. It removes a moving part and its filtered search is excellent. We escalate to a dedicated vector database when scale, write throughput, or advanced retrieval features clearly exceed what Postgres serves comfortably — and we prove that need with numbers, not vibes. We choose a managed search platform when strong hybrid retrieval, mature permissions and existing infrastructure point that way.

The uncomfortable meta-point is that the vector database is rarely the bottleneck people fear. Retrieval quality is dominated by chunking, embedding choice, and reranking; retrieval reliability is dominated by filtering correctness and operational maturity. Pick the store that lets your team ship and sleep, tune the parts that actually move the metrics, and treat any vendor claim about raw QPS as the least interesting number in the decision.