Retrieval that cannot cross a tenant boundary

A similarity search has no concept of a customer. It takes a vector, finds the nearest ones, and returns them. Whether those neighbours belong to the person who asked is not something the index knows or can be persuaded to care about.

Where you put the filter is therefore the entire question, and the two ways of getting it wrong produce symptoms that look identical to a ranking problem.

The failure

The natural way to write it is the wrong way:

1. Embed the question.
2. Retrieve the twenty nearest chunks.
3. Drop the ones that do not belong to this workspace.
4. Send what is left to the model.

Step three arrives too late. The twenty nearest were drawn from every customer's documents, so what survives is whatever fraction happened to be yours. For a large tenant that might be fifteen chunks and nobody notices. For a small tenant it is two, or none.

The support ticket says the assistant could not find anything about our refund policy, even though the policy is indexed. The team tunes the prompt, raises k, adjusts chunk size. None of it helps consistently, because the problem is not relevance — it is that the tenant is competing with everyone else for the same twenty slots.

Now the same bug with step three omitted, or applied to the wrong identifier. Retrieval returns another company's onboarding document, the model summarises it confidently, and the answer appears in a customer-facing chat. It reads as a good answer. It is a disclosure, and nothing in the logs looks like an error.

Same root cause. One version files as a bug. The other is reportable.

Putting the boundary where it cannot be forgotten

Filter inside the query. The tenant predicate belongs in the same statement as the distance ordering, so the k nearest are k nearest within the tenant. This is the minimum, and it is correct — but be aware that a highly selective filter interacts badly with approximate indexes. The index is built over everything; asking it for neighbours that also satisfy a narrow predicate can mean scanning far more candidates than expected, or quietly returning fewer results than requested. Correct and slow is a much better failure than fast and wrong, but you should know which one you have before it is under load.

Partition the index. Give each tenant its own namespace, partition or partial index. Isolation stops depending on a predicate being present in every query and starts being structural — there is no shared space to leak across. The cost is operational: thousands of tenants means thousands of partitions, and small tenants each carry index overhead.

Separate the storage entirely. A database per tenant makes the question disappear at the cost of everything else, which is why it is usually reserved for the few customers who contractually require it.

The rule of thumb: if isolation depends on a WHERE clause that a developer must remember to write, it will eventually be forgotten in one code path. Structure beats discipline.

Three leaks that survive correct retrieval

Fixing retrieval is necessary and not sufficient.

The synthesis step. Retrieval scoped correctly, then a batch job assembles several tenants' context into one prompt for efficiency. The isolation was real right up to the last call.

Caches. A response cache keyed on the question text alone will serve one tenant's answer to another asking the same thing. What is our refund window is not a distinctive question.

Derived artefacts. Summaries, evaluation sets, prompt logs and traces all contain retrieved content. If the primary store is partitioned and the observability pipeline is not, the boundary holds in the place people audit and fails in the place they do not.

The test that actually catches it

Assertions about response shape will not find this. The test has to be adversarial about content.

Seed two tenants with near-identical documents differing in exactly one distinctive token — a made-up product name is ideal. Ask tenant A a question that both documents answer. Assert that the retrieved chunk identifiers all belong to A, and that B's token appears nowhere in the response.

Then repeat it with tenant A holding one document and tenant B holding several hundred. That second case is where post-filtering shows itself, because A's single relevant chunk now has to beat several hundred of B's for a place in the top twenty. A suite that only tests balanced tenants passes while the bug is live.

Both cases belong in the suite permanently, not in a one-off review. The filter is one line, and one line is exactly what gets lost in a refactor.

Why this is worth being careful about

Most retrieval bugs are quality problems that annoy people. This one changes category: it turns a support feature into a disclosure of one customer's internal documents to another, discovered by the recipient rather than by monitoring.

It is also cheap to prevent and expensive to remediate, because remediation means reconstructing which answers were assembled from which chunks, for how long — which is only possible if the retrieval trace was recorded, and it usually was not.