How MidBrain encrypts agent memory

An agent that remembers years of conversations will accumulate information rarely kept together in one document: work decisions, personal preferences, health concerns, and details about other people. In 1. A lifelong companion, we described why that history should remain owned and controlled by the user. End-to-end encryption is part of how we can make that happen.

The engineering problem is that the memory still needs to be useful. An agent must be able to search old conversations, find a passage in an uploaded document, and update its understanding of the user. So there is a balance between enabling all of these operations while still maintaining security and privacy, such that we or any other third party could not read these memories even if we wanted to.

Of course, there are methods that guarantee some of these properties in various forms, like differential privacy and homomorphic encryption. And quite often there is a trade-off between accuracy/speed and privacy that is not always trivial to solve. At MidBrain, we have tried to adopt the pragmatic approach of operating on encrypted data as far as possible, but without sacrificing accuracy or latency.

Different protections for different operations

Our approach takes inspiration from CryptDB, which explored practical SQL queries over encrypted data. CryptDB used layers of encryption, often described as “onions,” that could be adjusted to expose the properties needed for particular operations, such as equality comparisons. The useful idea is to decide what an operation must learn and choose a representation with an explicit leakage profile.

MidBrain applies that idea to memory retrieval using separate keys and search representations. We do not peel encryption layers off a memory body to make it searchable. The body remains encrypted, while the search index exposes the relationships needed for retrieval.

Each agent has four independently generated 256-bit keys. A master wrapping key, called client_key, encrypts each of three operational keys directly. This is envelope encryption: the encrypted keys can be stored alongside the encrypted data, while the key needed to open those envelopes stays with the client.

These keys are independent rather than derived from one another. Obtaining the search key does not reveal the memory key or state key.

The separation follows what the server needs to do. Finding a relevant memory requires search representations; updating a user profile requires access to derived state. Neither operation requires the key that decrypts the raw archive. Derived state can still contain sensitive facts from that archive, so its separate key limits cryptographic access without making the information itself harmless.

For field encryption and key wrapping, we use AES-256-GCM with a fresh random 96-bit nonce and a 128-bit authentication tag.

Each encrypted value carries a purpose through additional authenticated data (AAD). For example, conversation text uses the scope episodic.text, while a user profile uses profile. Attempting to decrypt a value under a different scope fails authentication. This prevents an encrypted value being silently reinterpreted as another kind of content (e.g. swapping the encrypted values between columns).

Who holds the keys?

Our API supports two different key-management modes. But the backend is implemented the same way and operates under the same encryption scheme. The only difference is who holds the keys.

In client-managed mode, the client creates and owns its client-key. It is never sent to our servers, and the client is therefore expected to encrypt all memories locally before sending them to us, and decrypt them locally when retrieved. This means that there is not even a theoretical chance for us or any attacker (in our system) to get hold of the key to decrypt any memories.

In our “midbrain-managed” mode, we store the client-key for you. We use an external key management system so the client-key is never stored in the same place as the production database, and all operations, calls, etc. work in the same way as with the “client-managed” mode.

So the purpose of the “midbrain-managed” mode is one of convenience and thus is more suitable if you want to have a slightly easier API, and you don’t want to keep track of your client-key across your devices.

Our API exposes these modes through /api/v3/mm/* for MidBrain-managed agents and /api/v3/cm/* for client-managed agents.

Ingesting a memory

As you are conversing with an agent, the messages going back and forth between you are what we call episodic memories. We ingest these memories in multiple ways, for one, we run what we call consolidation, which essentially derives facts (semantic memories) and skills (procedural memories) from these messages. Secondly, we want to make the source message searchable so that an agent can retrieve it if necessary.

These two operations need to observe the raw memories in one way or another. Of course we could add indexing and whatnot on the client, such that the server never needs to see the plaintext versions. This is, however, where we take the pragmatic approach: if we were to push all of that business logic to the client side, each client would need to not only correctly implement all our backend assumptions in terms of schemas, indexes and whatnot, but also implement and run embedding models, consolidation etc. From a service perspective, this also becomes difficult to maintain, since all clients need to be updated and working under those same assumptions.

Instead, we decided to send two copies of each message at ingestion: one encrypted client-side and one in plaintext. This way, the client never needs to expose the memory-key, while our service can use a transient copy of the plaintext version.

The basic threat model here is that a plaintext memory exists for as short an amount of time as possible (time-of-flight), typically <100 ms. If a threat actor somehow gained full access to our system, they would at best be able to intercept these time-of-flight messages, but would not be able to decrypt previous messages.

In practice, it looks like this for a client-managed ingestion:

curl 'https://memory.midbrain.ai/api/v3/cm/memories/episodic' \
  -H 'Authorization: Bearer <agent-api-key>' \
  -H 'Content-Type: application/json' \
  -H 'X-Search-Key: <base64url-search-key>' \
  -H 'X-State-Key: <base64url-state-key>' \
  --data '{
    "text": "Hello world!",
    "text_enc": "<base64url-locally-encrypted-message>",
    "role": "user"
  }'

Notice that we send the plaintext versions of state-key and search-key (but not mem-key!)

These are optional, and if omitted it means that the service will skip those steps. So if state-key is omitted, consolidation cannot progress and will be skipped. Likewise if search-key is omitted, search-indexing will be skipped.

Through this design, we leave it up to the client to decide their service level based on which keys they are willing to expose to the server-side. But with that being said, mem-key, which is used to encrypt the raw messages, is never exposed to the server-side. So at best an attacker could only get access to the index, which is the embeddings and a trigram GIN index, both of which are very hard to reverse constructively.

Searching memories

Suppose the user later searches for “Hello.” The server needs to identify relevant memories and return their ciphertext for client-side decryption. As mentioned earlier, we maintain two kinds of protected indexes for this: an HMAC trigram index for lexical search, and a transformed-vector index for similarity scoring.

Lexical search: HMAC trigrams

For lexical search, we split the document into overlapping three-character fragments, or trigrams. For example, Hello produces Hel, ell, llo.

We derive an HMAC subkey from search_key using HKDF-SHA256, then compute HMAC-SHA256 for each trigram. The index stores these keyed tokens. A query’s literal fragments go through the same process, allowing the server to find candidate memories through token matching without decrypting their bodies. The index can select candidates by token containment without knowing the corresponding letters. Repeated trigrams are deduplicated, so the index does not record their order or how often they occur within a message. HMAC is a keyed hash, not encryption that the server later decrypts.

A lexical search for this conversation uses:

curl --get 'https://memory.midbrain.ai/api/v3/cm/memories/search/lexical' \
  -H 'Authorization: Bearer <agent-api-key>' \
  -H 'X-Search-Key: <base64url-search-key>' \
  --data-urlencode 'pattern=Hello'

Patterns can also combine literal fragments. For Hello.*world, the server requires the tokens from both fragments; Hello|world creates alternative token groups.

These are candidate matches, not full regular-expression matches. A message can contain all required trigrams without containing the complete word, and token containment cannot enforce fragment order or anchors. If the client needs exact matching, it must decrypt the returned candidates and check them locally.

An observer can see that two memories share trigrams, and can study their frequency and co-occurrence. Computing a token for a guessed word requires the relevant key or access to an oracle that produces matching tokens.

Semantic search: transformed vectors

For semantic retrieval, we apply an orthogonal transformation derived from search_key to the embedding vectors. At search time, the server applies the same transformation to the query embedding, so the query and documents end up with the same transformation.

An orthogonal transformation preserves inner products and cosine similarity. For a transformation R and vectors x and y, (Rx) · (Ry) = x · y. The search can therefore compare transformed vectors using the same similarity relationships. Our implementation combines Walsh–Hadamard operations, sign flips, and a permutation, applying the transform in O(d log d) (on the query).

That means that the embedding transformation is unique per storage space. So even if an actor was able to recover one client’s key, they could not use it to access another client’s storage.

curl --get 'https://memory.midbrain.ai/api/v3/cm/memories/search/semantic' \
  -H 'Authorization: Bearer <agent-api-key>' \
  -H 'X-Search-Key: <base64url-search-key>' \
  --data-urlencode 'query=Where am I?'

The transformation obscures their coordinates in the original representation space, but it is reversible with the search-key and does not provide the confidentiality of AES-GCM. We treat it as obfuscation with a defined purpose, not as encryption of the embeddings.

Search results

Both search endpoints return an array of encrypted results. Here is an illustrative semantic result:

[
  {
    "id": 42,
    "role": "user",
    "corpus": "episodic",
    "text_enc": "<base64url-encrypted-message>",
    "metadata_enc": "<base64url-encrypted-metadata>",
    "score": 0.38
  }
]

The client decrypts text_enc locally with mem_key and the episodic.text AAD scope, recovering “Hello world.” Metadata is decrypted separately with search_key.

Consolidation

Consolidation updates the agent’s persona, user profile, and semantic+procedural knowledge by observing the ongoing conversation. All of these consolidated memories are encrypted using the state-key. So during the consolidation process, we use our proprietary consolidation model, but that needs to see the plaintext message and it also needs to see the prior consolidated memories to determine if it needs to create a new memory or update/delete an existing one. Of course our models use a zero-data-retention policy, so even if we send memories to another server, this server is guaranteed to not log or store any transient information.

In the future, we plan on studying if it is possible to have the consolidation model act directly on encrypted data using privacy-preserving machine-learning.

Supplying the state key grants access to existing consolidated memories, not merely permission to append one update. A party that captures it could decrypt that agent’s stored profile and procedural knowledge. It still would not obtain the mem-key.

This is the practical reason for separating raw memories from the state built from them.

What happens if something is compromised?

A database snapshot is first encrypted with standard file-encryption many production databases use. If an attacker somehow manages to decrypt that first layer of defence, each client’s storage space is then individually encrypted as we’ve described above. This means that even if a single space is somehow compromised, it still protects the other users.

Furthermore, the client-key used to encrypt the envelope is never stored in the same database, so even if a snapshot leaks, it does not contain any keys that could be used to decrypt it.

A compromised server processing a request can capture the transient plaintext and keys supplied for that operation. A captured search-key exposes search representations and metadata and a captured state-key exposes consolidated memories.

Compromising the client or its client-key is a more serious event. The client-key can unwrap all three operational keys and therefore provides access to the agent’s data. Clients should protect it accordingly. In the browser, the client-key is stored as a non-extractable WebCrypto key. Other clients need local key protection appropriate to their environment.

The implementation also never logs individual memories, redacts authorization and key headers from logging and sanitizes server errors.

Looking further ahead

A lifelong companion may hold memories for decades. Data stolen today could be kept until better attacks become available, so the encryption needs to be chosen with that lifetime in mind.

The AES-256 cipher we use in AES-256-GCM is considered resistant to known quantum attacks. Grover’s algorithm theoretically reduces a brute-force key search from roughly 2²⁵⁶ to 2¹²⁸ operations, which still leaves an impractically large search. NIST expects AES-256 to remain safe for a very long time, assuming no new weaknesses are discovered.

The point of a lifelong memory is that it can still be useful years from now. Its privacy needs to last at least as long.