Multi-tenant RAG: the isolation bugs that don't show up in the demo

Single-tenant RAG demos great. Ship it to five customers sharing one vector index and you’ve built a system where a well-placed query from Customer A can return Customer B’s contract terms as “relevant context.” Nobody wired that on purpose — it’s what happens when tenant isolation is implicit instead of enforced.

The uncomfortable fact: no major vector database isolates tenants by default. Pinecone, Weaviate, Milvus, and pgvector all support namespaces, partitions, or metadata filters — but every one of them requires you to apply the filter correctly on every single query. Omit it once, in one code path, and similarity search will happily return the nearest neighbors regardless of who they belong to. Research on ungated retrieval pipelines has found cross-tenant leakage in 98–100% of probes when the filter is missing. That’s not a tail risk, it’s the default outcome of forgetting one WHERE clause.

Where the leak actually happens

Isolation fails at boundaries, not in the obvious places:

• Ingestion: a batch job re-indexes documents and drops the tenant_id metadata because the pipeline was written for a single-tenant POC and never updated.
• Retrieval: the query embeds correctly, but the similarity search call doesn’t pass a namespace or filter — often because a caching layer or retry path bypassed the code path that sets it.
• Reranking: you re-score the top 50 chunks with a cross-encoder for quality, and the reranker call doesn’t know about tenant boundaries at all — it just sees text.
• The KV cache: if you’re using prompt caching or shared inference infrastructure, prefix-sharing across requests is a real side channel — identical system prompts across tenants can end up sharing cached state in ways that weren’t designed with tenant boundaries in mind.
• The LLM’s own context window: even if retrieval is clean, a badly scoped tool call (e.g., “search all documents”) executed by an agent can pull cross-tenant data directly, no vector search involved.

The fix is architectural, not promptable

Telling the model “only use documents from this tenant” in the system prompt is not access control. Models don’t reliably refuse to use context that’s already sitting in front of them, and you shouldn’t be relying on instruction-following for a security boundary anyway.

tenant_id must be enforced at the data layer, not the prompt layer:

  request → auth (JWT, scoped claim: tenant_id)
          → retrieval (query filter: tenant_id = claims.tenant_id)
          → rerank (input already pre-filtered, never re-widened)
          → context assembly (assert every chunk.tenant_id == claims.tenant_id)
          → LLM call

That last assertion step is the one teams skip. Add a hard check right before you build the prompt: every chunk you’re about to hand the model gets its tenant_id compared against the request’s claim, and anything that doesn’t match gets dropped and logged as an incident, not silently filtered. This catches the ingestion bug, the retry-path bug, and the reranker bug in one place, regardless of which upstream step introduced them.

Picking a partition strategy

• Separate indexes per tenant — strongest isolation, worst blast radius from a shared bug (you’d need one), but real operational cost at hundreds of tenants: index sprawl, uneven utilization, slower cold starts for small tenants.
• Namespaces (Pinecone-style) — one call scopes one tenant; cheap, but only as strong as the code path that sets the namespace. Good default for most SaaS scale.
• Row-level security (pgvector) — one table, a tenant_id column, and a Postgres RLS policy that filters at the database layer regardless of what the application code does. This is the one option where a forgotten WHERE clause still doesn’t leak, because the database enforces it below your query. If you’re already on Postgres, this is worth the migration.

What to test, not just build

Isolation bugs don’t show up in functional tests because functional tests use one tenant’s data. Write tests that seed two tenants with near-duplicate content — same structure, different values — and assert that Tenant A’s queries never return Tenant B’s chunks, at the retrieval layer and after reranking. Run it in CI on every change to the retrieval or ingestion code, not just once at launch. The bug that gets through is never the one you tested for; it’s the one where someone touched the ingestion job eight months later and didn’t know isolation was a requirement.

If you’re bringing on your third enterprise customer and still filtering tenants at the application layer with no database-level backstop, that’s the week to fix it — before it’s the week a customer finds it for you.