Forty days of local models: what we kept and what we set aside

Eduardo Verboonen Khoury · November 3, 2026

On 2026-09-26 I turned on a Mac Studio with 256 GB of memory to run open-weight models next to the cloud. A second Mac served as backup, and later as a helper. The question was simple: which parts of the work can move off the cloud without getting worse, and how would I know? This is the notebook of the first 38 days, through 2026-11-03. It covers what we tried, what we kept, what we set aside and why. Every number comes from a dated log, and every verdict was measured against a criterion written down before the result existed.

A companion note, The dangerous failure is the one that reports success, covers the exam built from real work and the sandbox around the agent. This one is the wider record.

The rules of the lab

Three rules held for the whole period.

• Only open licenses, read in the author’s own repository. A model enters the list only after someone reads its license text where the author published it. Models are pinned by ID, not by a tag that can move.
• The criterion comes first. Before a run, the pass line is written in the log with the date. A result that misses the line doesn’t get a second run to improve the number.
• Whatever can be checked by code goes local. Classifying, extracting, summarizing and drafting code are verifiable. Anything that reaches a person, touches production or needs judgment stays with a frontier model and a human.

Things that failed quietly

The most useful findings were not about which model is smarter. They were about tools that looked like they worked.

A “local” coding agent that called home. I set up Codex CLI with a profile pointing at the local model, over loopback only. On 2026-10-02 I measured 15 runs and caught a connection to chatgpt.com in 3 of them. The cause was in the client’s source code, not in the model: a “memories” feature checks the account’s usage limit on every run, whatever model the run uses, and a profile is layered on top of the base configuration instead of replacing it. So the account feature stayed on. The fix was a separate home folder for the tool, with no login file and features.memories = false. After the fix: 0 connections in 15 runs, and 0 in 3 more with the installed version. One caveat: the detector could miss connections shorter than about a quarter of a second, so 3 in 15 is a floor, not an exact rate. The lesson: a profile is not isolation. A separate home is.

Prompts cut in half without a word. The version of Ollama we run, 0.34.4, handles a prompt longer than the context window by cutting it to about half the window, dropping the beginning, and the client is not told. We found it twice. First, the coding agent believed it had 64k tokens of context and was seeing 16k. Then, on 2026-10-06, an audit of my own context measurements showed that their recall, incremental and 90% fill results were invalid: my prompts had been cut before the model saw them. From then on every measurement sends truncate: false, so an oversized prompt fails loudly with an error instead of shrinking. In daily use a lock reads the server log and records every cut. A real control on 2026-10-18, a prompt of 44,785 tokens sent to a small window, was recorded and alerted exactly once.

Thinking budgets that return nothing. Several models think before they answer, and the thinking counts against the token limit. Gemma 4 thinks by default; with a short limit it spent the budget thinking and returned empty content, and in two cases it ran on until it filled the context. gpt-oss 20B did something similar even with thinking turned off: thousands of characters of reasoning and no answer. Both looked like model failures. Both were configuration. Every call now states its thinking mode explicitly.

The reasoning mode is a variable

A public comment said that a 27B Qwen model needs its reasoning mode on to program well. We tested it on 2026-10-04 with the official release, on the first 80 HumanEval problems: reasoning off, 75 of 80; reasoning on, 59 of 80, with a 6,000 token limit that may have cut some long reasoning. In our harness, the comment did not hold. An earlier run of the same model had scored 49 of 80, but that harness cut answers at 768 tokens. The earlier number was ours to fix, not the model’s.

Benchmarks stop discriminating

By early October, HumanEval no longer separated the candidates: the strongest local models all scored between 71 and 76 of 80. So between 2026-10-02 and 2026-10-04 I wrote a 41 task exam in three families (programming as an agent, reasoning, and following instructions in Spanish), graded by code, and used Opus 4.7 as the reference. Opus passed 41 of 41 for US$7.91. Gemma 4 31B passed 85% of the agent family and 100% of the other two. Gemma 4 26B: 77, 100, 100. Qwen3-Coder-Next, then in use for code: 69, 82, 73. In reasoning and Spanish instructions, three local models matched the reference. In programming as an agent, none did; the best was 15 points behind.

A reference that scores 100% everywhere means the exam is too easy to rank the top. That pushed me to build the exam from real commits in client repositories, described in the companion note. On the fair bank of 28 tasks, over three repetitions, Gemma 26B solved 21, 19 and 20, and Qwen solved 14, 15 and 13. On 9 tasks with rewritten statements, one repetition each, Opus 5.5 solved 9, Gemma 26B 8 and Qwen 7.

Candidates, with the verdict and the reason

• MiMo-V2.6-Flash-RL: set aside. 76 of 80 on HumanEval, above our line, and 0 connections outside the machine. As an agent it passed 1 of 30 exam tasks, with 29 timeouts. The official weights already came in low precision and could not be requantized, and the native file was too large to fit next to the models in use. The lesson: HumanEval with no time limit does not predict work as an agent.
• GLM-5.3-Flash: set aside. 71 of 80, measured on a separate engine because ours did not support its architecture, and it did not pass our manipulation bank: 5 concessions where the limit was 2.
• Muse Glimmer 30B: set aside. 66 of 80, below the line.
• Nemotron 3 Super 120B: not adopted. NVIDIA’s model, measured on 2026-10-03 and 2026-10-04 in our harness. The registry tag for it came with an evaluation only license, while NVIDIA’s official weights carry the Nemotron Open Model License, which allows commercial use; so we converted the official weights ourselves to 4-bit (Q4_K_M), in a sandbox with no network. HumanEval: 76 of 80. Inside the local coding agent it passed 4 of 5 runs of a small task and 10 of 10 of a medium one, with 0 network connections, 0 files outside its folder and 0 false claims of having run tests. On the 41 task exam, one run per task: 8 of 13 in the agent family, 16 of 17 in reasoning, 11 of 11 in Spanish instructions. It did not pass our manipulation bank: 0 critical concessions, but 3 minor ones where the limit is 2, and 7 of 10 control questions right where the minimum is 9.
• Gemma 4 26B: kept. It became the local model for code and for general text work. More on how below.
• Flint: watched, not adopted. 74 of 80 on HumanEval on 2026-10-22, the same as Gemma 26B. The Ollama registry tag carried a repackaged build whose bytes differ from the author’s file, so we used the author’s file and checked its hash. Then the engine mattered. Served by a release candidate of Ollama, the model’s tool names arrived in lowercase, Claude Code rejected them, and the run looped: 0 of 1, with 242 tool name errors. The same file served by llama.cpp passed the same task in 439 seconds with 0 name errors. A grid run through that engine would have measured an integration defect, not the model. On the fair bank Flint scored 19 and 20; the companion note tells what happened in the first of those runs.
• Two larger models, not measured. Muse Spark Open and Beam, in heavily compressed versions, fit in physical memory on paper. Our own memory guard did not let either run safely: one was stopped by the swap watchdog 14 seconds into its first task, the other never passed the free memory check. Neither produced a result, so neither has a verdict.

Alternatives set aside

• TensorFold and Splash, as faster engines. Both use a draft model to predict tokens ahead. TensorFold was 5.5 times faster with identical output, but its pass rate on HumanEval was 0.7875 against 0.8875 for the model in use, and the line was quality. Splash matched the reference output 0% of the time and scored 0.50. One result I could not explain: the 8-bit version of the same model scored worse than the 4-bit one. It is recorded as unexplained. Both engines were left installed and off.
• OpenCodex, a proxy between coding agents and local models. The line was 4 of 10 and at least 3 more than the direct path, with 0 cloud connections. With Codex, the direct path passed 2 of 2 and 2 of 2; through the proxy, all four runs hung for 20 minutes and all four connected to chatgpt.com. Its design for sharing load across several accounts also carried a terms of service risk I did not want attached to the account that runs everything.
• Hermes Agent, as an alternative agent shell. Tested on 2026-10-22 in an isolated sandbox with the same local model. It tied our path on email and code tasks (8 of 8 and 12 of 12). On a task list it scored 0% in its first run while reporting “10 new entries” on an empty result, then 100% twice. It created 0 skills in 7 runs. Version 0.21.5 ships with logins from other coding tools adopted, on the fly package installs, and computer use, browser and scheduled jobs turned on. Its command approval blocked rm -rf, and the agent then deleted the folder file by file. It did not beat what we have, and it brought more surface.
• The Neural Engine. A 4B model converted for Apple’s Neural Engine ran at 14.8 tokens per second with very little memory. The same model on the GPU ran at 213. Paused.

How a model changes jobs

No tool names a model. Each job (text, code agent, and so on) is an alias with a contract test, and one command moves a job to another model, runs the contract and keeps the way back.

On 2026-10-15 the text job moved from gpt-oss 20B to Gemma 26B. On a golden set of 30 synthetic cases, Gemma passed 30 and gpt-oss 25; the five misses were empty answers from the thinking budget problem above. Gemma also passed the manipulation bank, and gpt-oss did not.

On 2026-10-18 the code agent job moved from Qwen to Gemma 26B. The evidence was thin and I wrote that down: 4 of 13 against 2 of 12 on a small screen, not conclusive. The change was justified because it was reversible and watched, not because the evidence was strong. The way back was rehearsed before the change and took 6 seconds. The watch is written as code: if the error rate reaches 5%, the 95th percentile passes 90 seconds, three turns time out, two prompts get cut or memory leaves its limit, the job returns to Qwen on its own and I get one line. It passes after 7 days and 30 real turns with no incident. As I write this, that watch and the one on the text job are still open, so I am not calling either change confirmed.

The same pattern runs everything that spends time in shadow: numeric criteria with a minimum of days and cases, a daily evaluator that activates the change only when they are met, and an automatic return to shadow after a critical incident.

Next

NVIDIA’s newer Nemotron models (document parsing, embeddings, reranking and speech recognition) are prepared, not measured. Six official repositories are downloaded and checked by hash, the criteria are written, and NeMo Evaluator is installed. The one test so far: converting the 1B embedding model to the format our engine uses was not faithful (cosine 0.983 to 0.995 against the reference, where a faithful copy gives 1.000), so it will be measured in its original form. One model was set aside before measuring: Nemotron Content Safety. Its license adds the Gemma 3 terms of use and prohibited use policy on top of NVIDIA’s open license, and that was a layer of obligations we chose not to carry. Two Apache licensed safety models will be measured in its place.

What this adds up to

The models improved quickly over these weeks. What moved my results most, though, was the plumbing: a profile that did not isolate, a prompt cut in silence, a thinking budget that returned nothing, an engine that renamed a tool. Each looked like a model failure until I read the log. The best local models now do real programming work in my harness, and a frontier model still sets the line they are measured against.