A lot of our customers ask what they gain from using MidBrain, and we usually show them internal benchmarks and demos. However, we also wanted to test it on a public benchmark that is somewhat realistic in that it exercises a known harness. So we ran MidBrain on DolphinBench, released by the kind people at Mem0.
DolphinBench uses agent harnesses to test tasks that resemble the work people ask assistants to do: sending messages, updating documents, and managing calendars using information from earlier conversations. It measures whether the agent completes each task correctly, alongside the cost of running the agent and its memory.
Using the leaderboard’s agent and model and our production memory system, MidBrain achieved state-of-the-art results with an accuracy of 83.2% at the record low cost of $64.
The benchmark has three simulated users, each with roughly half a million tokens of dated conversation history and 200 tasks. It gives the agent access to simulated apps and checks the resulting tool calls. A useful memory can help the agent choose the right recipient, follow a standing rule or include a detail the request leaves unstated. The cost matters too: an assistant has to store and retrieve those details across many interactions, as well as pay for the model calls used to act on them.
What does a test look like?
One test asks the agent to post a deployment update to the appropriate Discord channel. A colleague suggests #eng-all, but the deployment is still awaiting validation. Morgan had established a rule years earlier: release notes go to #eng-releases until the deployment is green. The test passes only if the agent sends one successful message to #eng-releases.
Another test asks Alex’s assistant to text Kibo’s walker the harness and leash setup for a route near the Prospect Park entrances. The request supplies the walker’s number but not the setup. An October 2023 conversation established the properly fitted front-clip harness and shorter leash; the test takes place in January 2028. The grader checks the recipient and asks a model to judge whether the text includes both pieces of equipment.
The benchmark supplies the simulated app state and conversation history. Each test starts a fresh conversation, so the agent must recover what matters from memory before acting.
How was the memory built?
The benchmark asks that history be ingested the way an assistant would meet it, in order, one dated session at a time, through the agent’s normal memory behavior. We ran the Hermes agent harness with GPT 5.6 Luna, the same harness and model as the published Hermes rows, with MidBrain as its memory. The agent had the same simulated apps as the leaderboard runs, with MidBrain in place of Hermes’s own memory. Each session went through the agent as a live turn, with the same memory tools available as in production. Their schemas contribute to input-token usage, and the agent can call them during ingestion, making the ingestion trajectory longer.
When ingestion finished and consolidation had settled, the memory was frozen. Each of the 600 tests then ran in a fresh conversation with fresh app state and no ability to write to memory.
Results
Harness cost is the agent’s inference cost for ingestion and tests, priced from recorded token usage at list prices. Some providers also include the cost of their memory system. MidBrain’s internal costs do not map directly to input and output tokens, so we report its memory usage separately in credits.
| Memory | Accuracy | Total cost |
|---|---|---|
| Hermes built-in | 65.7% | $61.48 |
| Mem0 | 70.7% | $96.21 |
| Honcho | 68.5% | $142.85 |
| Hindsight | 69.5% | $84.65 |
| Supermemory | 59.2% | $345.64 |
| past.dev (self-reported) | 81.7% | $196.72 |
| MidBrain | 83.2% | $64 + 52k credits |
Leaderboard rows are the published DolphinBench records; past.dev’s comes from its own disclosure.
How close is this to production?
The configuration we report uses the production MidBrain system with two experimental features switched on in the MCP integration and agent harness.
Episodic prefetch injects relevant memories before each turn without waiting for the agent to ask for them. This is a standard feature in the memory-provider interface for Hermes, but is disabled by default in our public hermes-integration. We enabled it because nearly every DolphinBench task depends on memory. Its benefit is less clear in mixed settings, where many requests may not need memory.
Dynamic prefetch for consolidated memories adds the consolidated facts and procedures that best match the current turn to the prefetched context. We enabled it to make that learned context available alongside recalled memories as the agent starts each task. Consolidation itself always runs in production; the dynamic prefetch is experimental.
Where does the agent still fail?
In 89.1% of failed tests, the needed facts were in context and the agent called the right tool, but the judge found that its response omitted a specific requirement. Retrieval misses accounted for 7.9% of failed tests, and in 3.0% the agent never called the expected tool.
In most of these failures, the relevant facts reached the model and the agent called the expected tool, but the model still missed a requirement in its output. Addressing those errors might require changes to the task prompt or other parts of the agent setup, which would change the benchmark protocol.
