Four user profiles ran against production Pass — every request compiled by the small model: route, goal, language, what to pull. New since last time: a standing user profile, write-time-reconciled memory, hybrid retrieval with a cross-encoder judge, and a fourth character built to test all of it. All 400 messages also went to Anthropic's Opus 5, and an independent judge scored both.
For Alice, Bob and Claude the judge (gemini-3.6-flash) scored blind — each answer alone against a 0–100 rubric: correctness, then completeness, then clarity. For Dobby a blind judge is structurally wrong: it cannot know that "the pressure" is a Vaillant boiler reading 1.2–1.5 bar, so it rates a polite "I can't access that, try an app" at 95. The Dobby judge was therefore given the ground truth of his life — city, medication, boiler, prices, appointments — and asked whether the answer serves this user. With facts in hand: Pass 61.2, Opus 23.5. A model without memory cannot answer "what's the weather?" with the weather in Faro.
Every request produces a Job on the receipt: the route, the goal in the compiler's own words, the language of the answer, and the pulls — knowledge base, memory, standing profile, web, rerank — in order. Reads the compiler orders are judged by a cross-encoder before anything enters the prompt; a read that finds nothing is said out loud, never papered over; an emptied read may escalate to the web once, and the receipt names it. The assistant's own answers are never written to memory — only the user's words are.
400 unique prompts — 100 per profile, 10 per language — against production, cache off. All 400 went to Opus 5 with identical prompts (max_tokens 1200) over OpenRouter. Costs are inference only, per message: Pass from token usage at registry prices (its serving models are small — the compiler routes most traffic to them), Opus from its own usage; Pass's $0.01 service fee is separate from these numbers. Answers truncated to 1400 characters for the judge. Previous benchmarks: 2026-08-25 · 2026-08-19.