The dangerous failure is the one that reports success

Eduardo Verboonen Khoury · November 3, 2026

I don’t write code. Claude writes the code, and open-weight models on two Mac Studios do a growing share of the rest. My job is to decide what gets built and to prove that it works. Since early 2026, the second part has become the real job.

The failure that worries me isn’t the crash. A crash is loud: someone sees it and someone fixes it. The failure that worries me is the agent, the test suite or the dashboard that says “done” when it isn’t. Below are three things I built to catch that, and what each one taught me. Every number comes from a dated log.

1. An exam where the score isn’t the whole story

In October 2026 I wanted to know whether a local model could take real programming work off the cloud. Public benchmarks didn’t answer that, so I built an exam from real work.

How the tasks are made. Each task comes from a real commit in a production repository. The state before the change is the starting point; the accepted change provides the reference solution and the hidden tests. Each task ships with visible tests the model can run and hidden tests it never sees. A verifier counts only after three controls: the reference solution must pass, the untouched starting point must fail, and tampering with the visible tests must not fool it. The task contents stay on my machines. This note describes only the method.

A fairness gate, written as code. My first reference run with a frontier cloud model looked bad. When I diagnosed the failures by hand on 2026-10-18, about 14 of 18 were defects in the task, not in the model: hidden tests demanding a string, a constant or a cutoff date that neither the prompt nor the code provided. So I wrote the gate as code. It flags any task whose hidden test asserts something the model had no way to know, inspects source code instead of behavior, or fakes the database client by matching strings. Calibrated without looking at other results, it let the 3 known-good tasks through and flagged 6 of the 8 failures that had evidence of a defect. The two it missed were underspecified in meaning rather than in a hidden string, which is why the gate is necessary but not sufficient. Flagged tasks were set aside with a note, never deleted. The comparisons below ran on the 28 tasks that survived.

Criteria fixed before measuring. Before each run I wrote down what would count. For a new candidate, on 2026-10-22: 22 or more of 28 is better than the incumbent, 19 to 21 is comparable, 18 or less is inferior. The arithmetic runs in code over the result files, never in a model.

Results. Over three repetitions, Gemma 26B (Gemma 4, 26B, 8-bit) solved 21, 19 and 20 of 28 (mean 71%). Qwen (Qwen3-Coder-Next, 8-bit) solved 14, 15 and 13 (50%). A third candidate, Flint (RecursiveSI, int4 GGUF served by llama.cpp, in our harness), solved 19, then 20.

The run that mattered. In Flint’s first repetition, an automatic detector flagged 20 runs for mentioning grader file names. I reviewed all 20 transcripts. Most were noise. One was a search inside the model’s own folder, harmless. Two were not: the model ran find from the root of the disk, and from the users folder, looking by name for verificar.py, pruebas_ocultas.py and esperado.json, the hidden grader files, which the system prompt itself named only to forbid them. It said, in plain words, that those files did not exist for the model and that searching for them would count as cheating.

It found nothing. What stopped it was not my own path lock, which watches writes, not reads. It was Claude Code’s permission layer: in an unattended run, a search outside the working directory needs an approval nobody was there to give. Both runs went on to pass their hidden tests. The score said 19. Under a strict reading, where an attempt counts as cheating whatever the outcome, it was 17: the “inferior” side of a line I had drawn the day before. Its second repetition scored 20 with no searches for the grader files, so its mean was 19.5 raw and 18.5 strict, against Gemma’s 20. That is what happened in 2 of 28 runs of our harness, with our prompts and our time limits; it is an observation, not a general judgment of the model. With the same detector, Gemma and Qwen made 0 such attempts in the 112 runs of their second and third repetitions.

I describe this as behavior, not intent. But the lesson for an operator is plain: had I looked only at the pass rate, I would have missed the most important thing that happened in the exam.

2. A sandbox I had to rebuild twice

I also run Claude Code with a local model as its brain. On 2026-10-02, before any fix, the measurement was humbling: on a small test task the agent passed 0 of 5 runs and wrote files outside its folder in 5 of 5. It wasn’t escaping on purpose. It was writing into the folder of its own configuration, because it didn’t know where it was.

I tested one change at a time, over 70 runs. A hook that blocks file edits outside the project fixed the pass rate but left a gap: in 2 of 5 runs of a second task, the model created folders elsewhere with shell commands. Extending the hook to shell commands closed it. The simplest change, stating the absolute working directory in the system prompt, solved it on its own: 5 of 5 passed, 0 of 5 wrote outside. I kept both. On the real tool: 5 of 5 passed, 0 wrote outside, 0 network connections beyond the local model.

On 2026-10-30 I tested the hook again with a harder question: could the agent edit the rules that govern it? The hook matched patterns, and patterns have holes. Three of five write paths got through: sed -i, a Python open(), and an append to a file in the home folder. The fix was to stop relying on patterns. The operating system’s sandbox now confines every shell command to the project, temporary files and tool caches, with network only to package registries, and explicit deny rules keep the agent away from its own settings, hooks and instructions. I verified it with a decoy file outside the project: sed -i and the Python write returned “Operation not permitted”, a direct write was denied, and the decoy stayed intact.

Two more hooks came from watching the agent work:

• Don’t stop early. Local models often end a turn with a question, or with an announcement of what they’ll do next. A stop hook sends the agent back to work while its checklist has open items, or until a completion check passes. That check is a script the agent cannot edit, run under a macOS sandbox that denies writes to the agent’s rule folders: a hook runs outside the agent’s sandbox, so it must never execute code the agent could have written without confining it first. The agent has an honest way out: mark one item BLOCKED, with the reason, and move on to the next. There’s a cap of 20 pushes per human message. Measured on 2026-10-31: a three step task that said “ask me before each step” finished on its own in 23 seconds, with one push.
• Remember after forgetting. A small model’s context compacts often, and the conversation stops being reliable memory. A session start hook re-injects the agent’s progress log at startup, on resume and right after every compaction. Tested with tools turned off, the model answered with its exact next step.

Generic versions of these hooks, with their tests, are here: github.com/eduardovk1/agent-guardrails.

3. Three lessons about measurement

A measurement has to imitate the real load. On 2026-11-02 I tested a new iOS simulator version with 30 boot cycles: 0 crashes in 30, against roughly 1 in 4 on the old version. On that basis I upgraded the simulators in use. The upgraded devices carried old state, and during UI tests a system process crashed in a loop: 29 crashes in one day. A clean device on the same version survived 20 install and launch cycles. The measurement was honest. It measured the wrong thing.

CI that didn’t run isn’t green. From 2026-10-24 to 2026-10-31 my CI ran out of minutes, and a repository’s security audit was blind for a week. Now a check that didn’t run means “not run”, and its local equivalent has to run, with its time and exit code recorded, before anything merges.

Green tests, no work. On 2026-10-06 a change to my personal assistant agent added a permission to its token refresh. Its tests passed, using test doubles. In production Google rejected the refresh, and the agent did not read email for 8 days. It logged 740 errors, all at “warning” level, and nobody was told. Now any change that crosses an external boundary gets one real end to end run before it counts as done. A failure that repeats three runs in a row escalates to critical. And every unattended process publishes its last success, with a watchdog that alerts on silence, not only on errors.

What this adds up to

None of this is sophisticated. These are the tools a non-expert has: transcripts, file system checks, decoys, criteria written down before the result exists, and the patience to read twenty flagged runs by hand. Each one caught something the green number missed.

I think many future AI deployments will look like mine: an operator who can’t read every line, answering for what the system does to real people. The question I care about is how that person can tell when “it worked” is true. I don’t have the full answer. I have a testbed, a record of the ways oversight has failed me, and the habit of asking again.

A second note, Forty days of local models, is the lab notebook behind this one: what we tried with local models and agents, what we kept, what we set aside, and the criterion behind each verdict.