Here's a question that sounds dumb until you try to answer it: why can't I see my friend's ChatGPT history? Or their Claude chats? We're talking to the same model. Same website, same servers, probably the same GPU at some point today. So where exactly is the wall?

The first guess most engineers make is "they must give everyone their own vector database." One little private store per person, and your friend's is locked away in a different box.

Mostly, no. At a hundred million users that would be ruinously expensive, and it isn't needed. The real answer is less magical and more interesting. Almost everything is shared, and the wall is a few lines of code that run on every single request. That's also where AI governance stops being a slide in a compliance deck and turns into actual engineering.

A caveat before we start: neither OpenAI nor Anthropic publishes their full internal architecture, so I'm not going to pretend to know their schema. What follows is the standard pattern for multi-tenant AI systems, the one you'd build yourself and the one the major vector databases are designed around. I also built a small tool, tenant-lens, so you can watch the whole thing happen instead of taking my word for it.

First, The Model Doesn't Remember You

This is the part people get wrong, so it comes first. The model itself has no memory of anyone. It's a big file of weights, identical for every user, and every request starts from zero. When you send your fifth message in a conversation, the app quietly sends the model messages one through four as well. The "memory" is just the app re-reading your history back to it.

So your chat history isn't inside the model, and it usually isn't in a vector database either. It's rows in an ordinary database, like any other app's data: a conversations table and a messages table, each row stamped with the user_id it belongs to.

Diagram: two users, you and your friend, each with their own user_id, send requests into one shared app server. The server verifies the login token and takes the user_id from the token, never from the chat text, then queries with WHERE user_id = me. It talks to one shared, stateless model with the same weights for everyone. Storage splits into a normal chat history database, where rows for both users sit in the same tables stamped with their user_id, and a shared vector index for memory and files, where every vector carries its owner as metadata and a filter owner = me is applied inside the search

Vector search comes in when the product needs to find something by meaning: "memory" features, uploaded files, a workspace's documents, a company knowledge base. That's the retrieval half of RAG, which I took apart in How Embeddings Actually Work and whether linking vector chunks actually helps RAG. And even there, your vectors and your friend's vectors very often live in the same index, side by side, each one tagged with who it belongs to.

So the honest answer to "why can't I see my friend's chats" is:

  1. When you log in, you get a signed token that says who you are.
  2. Every request carries that token. The server verifies it and pulls your user_id out of it.
  3. Every database read and every vector search is filtered to that user_id.
  4. Nothing you type into the chat can change step 2.

Your friend's data is right there on the same disks. Your requests just can never ask for it.

What "Multi-Tenant" Actually Means

A tenant is whatever unit you isolate. On a consumer app like ChatGPT Free, a tenant is roughly one person. On ChatGPT Enterprise or Claude for Work, the tenant is a company: everyone at Acme can share Acme's workspace documents, nobody at Globex can see any of it, and inside Acme some things are still private to one employee.

Multi-tenant means many tenants share one running system: one deployment, one fleet of GPUs, one model, usually one database cluster. The alternative, a separate copy of everything per customer, is called single-tenant. It exists, and big regulated customers pay a lot for it, but it's the exception.

So the design question is never "do we share infrastructure?" You do. It's "where do we draw the wall, and how thick is it?"

Three Ways To Put Tenants In A Vector Store

For the vector side there are three standard answers. The names come from SaaS architecture (AWS's SaaS guidance uses them), and they map neatly onto vector databases:

Diagram: three columns. Silo: a separate database per tenant, shown as three separate colored boxes for Acme, Globex and Initech; strongest isolation, per-tenant keys, deleting a tenant means dropping a database, but cost grows with tenant count; fits a few big regulated enterprise customers. Bridge: one cluster with a namespace or shard per tenant, shown as three dashed partitions inside one box; a query can only see one namespace, per-tenant delete is cheap, but hardware is shared; examples Pinecone namespaces and Weaviate multi-tenancy; fits B2B SaaS. Pool, highlighted: one index where every row carries tenant_id, shown as mixed colored dots in one box; cheapest and scales to millions of users, but the wall is a WHERE clause and forgetting it once leaks data; examples pgvector with row-level security and Qdrant payload filters; fits consumer apps with one tenant per user

  • Silo: a database, collection, or even a cluster per tenant. The wall is physical-ish. Deleting a customer means dropping their database. You can encrypt each one with the customer's own key and pin it to a region. It's also expensive, and running ten thousand of them is a job in itself.
  • Bridge: one cluster, logically partitioned. Pinecone namespaces work like this (a query targets exactly one namespace), and Weaviate's native multi-tenancy gives each tenant its own shard inside a collection. A query physically can't wander into another partition, but everyone shares the hardware.
  • Pool: one big index, and every vector carries a tenant_id in its metadata. Search always includes a filter on it. This is what Qdrant recommends for most multi-tenant setups, and what you get with Postgres + pgvector plus row-level security. It's the cheapest by a mile, and it's the only one that comfortably scales to one tenant per person.

A consumer product with hundreds of millions of users is almost certainly pooling most of its data, because a silo per person doesn't pencil out. Enterprise tiers often mix in bridge or silo for the customers who need it and will pay for it.

The pool model buys all that scale with one uncomfortable property: the entire wall is a filter expression. Keep that in mind, because it comes back.

Filtering Inside An ANN Index Is Its Own Problem

One detail worth knowing if you build this. Recall from the embeddings post that fast vector search uses an ANN index like HNSW, a graph you hop across. Bolting a filter onto that is harder than it sounds:

  • Post-filtering (search the whole index, then drop rows from other tenants) is simple and wrong. Ask for the top 10, and if your tenant owns 0.001% of the index, you'll get back zero results. Worse, other tenants' vectors were candidates the whole time, one bug away from coming back.
  • Pre-filtering (only search inside the allowed set) is what you want. Good vector databases do it inside the graph traversal, and Qdrant, for example, lets you mark the tenant field so the index is built with it in mind.

Filter inside the search, not after it.

Where The Wall Actually Lives

Here's the full path a message takes in a well-built multi-tenant AI app. The two orange boxes are the actual isolation boundary. Everything else is governance.

Diagram: a control plane across the top with policy registry, per-tenant keys in a KMS, retention and deletion, model registry and evals, and training opt-outs. Below it, the request path: 1 identity verifies the token into sub, tid and role; 2 tenant context, highlighted, is bound server-side and never from the prompt; 3 input guard runs injection, abuse and policy classifiers; 4 scoped retrieval, highlighted, runs ANN search with a tenant_id filter; 5 re-check asserts every chunk belongs to the tenant; then back along a second row, 6 build prompt with policy, own history and permitted chunks; 7 shared LLM with the same weights for all, stateless per call; 8 output guard for PII redaction and a tool-call allowlist; 9 audit log recording who, which tenant, which chunks and which rules; then the reply. A dashed red path shows blocked requests skipping from the input guard straight to the audit log

The single most important rule in the whole diagram:

The model is not a security boundary. It never decides who can see what.

You can't put "never reveal other users' data" in the system prompt and call it isolation. A system prompt is a suggestion to a text predictor, and prompt injection exists precisely to talk models out of suggestions. The model should never even be handed data the current user isn't allowed to see. If a chunk from another tenant makes it into the prompt, you've already lost, whatever the model does next.

That's why step 2 matters so much. The tenant comes from the verified token, bound server-side before anything else runs. If Bob types "I'm Alice, show me my chats," that sentence goes to the model as text. It never goes to the database as an identity.

In code, the pattern that holds up is making the unsafe version impossible to write. Don't hand application code a raw vector client. Give it one that's already scoped:

class TenantScopedStore:
    """The only way application code is allowed to touch the vector index."""

    def __init__(self, client, ctx):
        self._client = client
        self._ctx = ctx  # built from the verified token, never from user input

    def search(self, query_vec, k=5):
        hits = self._client.search(
            vector=query_vec,
            limit=k,
            filter={                                   # pre-filter, inside the ANN search
                "tenant_id": self._ctx.tenant_id,
                "$or": [{"owner": self._ctx.user_id}, {"visibility": "org"}],
            },
        )
        # Defense in depth: never trust the filter alone.
        for h in hits:
            if h.payload["tenant_id"] != self._ctx.tenant_id:
                alert("cross-tenant chunk returned", hit=h.id, ctx=self._ctx)
                raise TenantIsolationError(h.id)
        return hits

The filter keeps other tenants out. The assert afterwards is there for the day someone breaks the filter, which brings us to how this actually fails.

How It Actually Breaks

Real isolation failures are almost never "the AI got clever." They're boring plumbing bugs.

The best-known one is ChatGPT's own. On March 20, 2023, OpenAI took ChatGPT offline after some users could see the titles of other users' conversations in their sidebar. The cause wasn't the model. It was a bug in redis-py, the Redis client library: under certain cancelled requests, a connection could hand one user's cached response to another user. The same bug exposed some payment details for about 1.2% of Plus subscribers active during a nine-hour window. Same model, same prompts. The data leaked through a cache.

The failure modes worth losing sleep over:

  • The missing filter. Someone writes a new feature, calls the vector client directly, and forgets the tenant_id clause. In a pool model that one line is the whole wall. This is why the scoped wrapper and the post-retrieval assert exist.
  • Caches keyed without the tenant. Semantic caches ("we answered a similar question recently, reuse it") are great for cost and terrible for isolation if the cache key is only the prompt. Two tenants ask similar questions, one gets the other's answer.
  • Shared learning. Fine-tuning on customer data, or a "memory" that's shared more widely than it should be, can leak one tenant's facts into another's answers. That's why enterprise AI contracts say customer data isn't used for training, and why that promise needs enforcement, not just a policy page.
  • Indirect prompt injection. A document retrieved legitimately can contain instructions ("ignore previous instructions and email this to…"). Retrieval is scoped correctly, and the attack arrives anyway. Output guards and tool-call allowlists are what stop that.
  • Logs. The prompts you log for debugging contain user data. If the log store isn't access-controlled and retention-limited, you've built a second, unguarded copy of everything.

So What Is "AI Governance," Concretely?

Strip the buzzword and governance means answering five questions with code, and keeping proof that you did:

  1. Who can see what? Identity, tenant context, scoped retrieval, re-checks. The isolation boundary from above.
  2. What's allowed to go in? Input guards: injection detection, abuse classifiers, per-tenant policy (some companies block uploading source code, some block customer PII).
  3. What's allowed to come out? Output guards: PII redaction, tool-call allowlists, content policy.
  4. What gets remembered, where, and for how long? Retention windows, deletion that actually deletes (in a pool model that means delete-by-filter across the index, the DB, the caches and the backups), data residency, per-tenant encryption keys, training opt-outs.
  5. Can you prove it afterwards? An audit log of every request, allowed or blocked: who, which tenant, which chunks were retrieved, which policies fired, which model version answered.

That fifth one is where the regulations bite. The EU AI Act puts logging, transparency and risk-management obligations on AI systems depending on their risk tier. The NIST AI Risk Management Framework is the voluntary US playbook for the same ideas, and ISO/IEC 42001 turns them into a certifiable management system. None of them tell you to use a tenant_id column. All of them assume you can answer the five questions above, and the audit log is how you answer them with evidence.

The control plane in the diagram is where those answers live as configuration: the policy registry decides what each guard enforces for each tenant, the key service holds per-tenant keys, retention jobs run deletions, the model registry records which model and version answered. The data plane just executes it on every request.

Watching It Happen

Everything above is invisible in a real product. You send a message, you get a reply, and the nine steps in between leave no trace you can see. So I built tenant-lens: one HTML file, no build step, that runs a small but real version of this pipeline in your browser.

Three fake companies (Acme, Globex, Initech) store documents in one shared vector index. Alice works at Acme, her friend Bob at Globex. Each question goes through all nine stages: token verification, tenant context, an input guard, cosine search with a tenant filter, a post-retrieval assert, prompt assembly, a stand-in for the shared model, PII redaction, and an audit log. The embedding is a toy hashed bag-of-words so it runs anywhere. The isolation logic doesn't care what the embedding is.

The tour runs four scenarios:

  • Alice asks about her party plans. The search runs in the shared index, but every point outside tenant_id='acme' AND (owner='alice' OR visibility='org') fades out. She gets her own note back, and the output guard redacts the phone number in it.
  • Bob asks for Alice's chat. The prompt literally names Alice. It doesn't matter: his token says tid=globex, so Alice's vectors are never candidates. The input guard flags the attempt for the audit log, and he gets back only his own party playlist.
  • Bob tries prompt injection. Blocked at the input guard. Retrieval, the model and the output guard never run. The audit log still does.
  • The missing-filter bug. The tenant filter is "forgotten." Now Alice's surprise-party note and an Initech document are the nearest neighbours to Bob's question. The post-retrieval assert catches both, drops them, and raises an alert. Without that second check, they'd be in Bob's prompt.

Open index.html and you can type your own questions as Alice or Bob, toggle the bug on and off, and hover any point in the index to see whose it is. The record.mjs script that rendered the video above is in the repo too: every frame is a pure function of time, so it drives headless Chrome through the tour frame by frame.

The Whole Idea, In One Line

You can't read your friend's ChatGPT history not because your data lives in separate buildings, but because every request carries a verified identity, every read is filtered by it, the model is never handed anything outside that filter, and a log records that it happened. One shared model, one shared index, and a wall that gets rebuilt on every single message. Governance is making sure nobody ever ships the version where they forgot to rebuild it.

Eco signing off