10 code agents, one Qwen 3.8 27B: 4x spread in tokens, zero difference in pass rate

On October 30, 2026, the second part of the code agent test appeared on Reddit. The author ran several CLI agents through the same local server with the Qwen 3.8 27B model. The first part was on the 4B model.

The main point from the table: by pass rate everyone is the same, by token consumption they differ by multiples. We break down what exactly was measured and where the numbers stop meaning anything.

What bench was assembled

The hardware is a MacBook Pro M5 Max, 64 GB of memory. The model is Qwen 3.8 27B in Q4 quantization, the engine is llama.cpp, thinking mode is disabled. One llama-server instance (a local inference server that the agent accesses over HTTP) for all.

One server for all is an important detail. It removes differences in quantization, engine version and sampling settings. All that remains is the behavior of the agent itself.

Next is the harness. Five small code tasks, each containing a trap: a catch in the condition or in the existing code. Each agent is locked in a sandbox (an isolated folder beyond which it cannot go). Auto-approve is enabled: the agent does not ask permission to run commands. Telemetry is disabled. The limit per run is 5 minutes.

The tasks are small. This directly affects how to read the results.

Why 5/5 for everyone is not "everyone is the same"

Pass rate (the share of passed tasks) is a metric with a ceiling. When there are five tasks and all are small, the ceiling is reached quickly, and the metric stops distinguishing participants.

Example. If a task is solved in 9 thousand tokens, then a solution in 39 thousand tokens will give exactly the same result in the table: "passed". The difference in the cost of work is not visible.

According to the numbers from the post: Pi — 5/5 in 38 seconds and 9.2k tokens. OpenClaw — 5/5 in 106 seconds and 39.0k tokens. By pass rate it is a tie. By time — about 2.8x. By tokens — about 4.2x.

That is why the author put tokens and cache reuse in the title. At a saturated pass rate, it is these two numbers that start showing the difference.

Tokens: 9.2k versus 39.0k with the same result

Numbers from the table, median across tasks:

• Pi — 5/5, 38 s, 9.2k tokens, cache reuse 72%, did not access the network.
• DeepSeek Harness — 5/5, 60 s, 35.3k, cache reuse 83%, did not access the network.
• Codex CLI — 5/5, 94 s, 36.3k, cache reuse 79%, network access attempt on every run.
• OpenClaw — 5/5, 106 s, 39.0k, cache reuse 72%, network status "unclear".
• Cline — the row in the source is truncated, no data.

The fifth row breaks off at "Cline 5/5 15". The remaining agents from the stated ten are absent from the available text. There is no data on them, so conclusions about the full ten cannot be made.

Where the 4x difference comes from with the same result. A code agent works in a loop: read files, run a command, look at the output, fix, repeat. Each step is a separate request to the model, and each request carries all the previous context: the system prompt, the history of steps, the contents of read files, logs.

Example. The agent runs ls, reads three files, runs tests, reads the failure log, edits one file, runs tests again. At the last step it resends everything that came before it. If the agent reads files in full rather than in the needed fragments, the counter grows quickly.

The difference between 9.2k and 39.0k is not a difference in "intelligence". It is a difference in how many steps the agent takes and how much context it drags at each step. Pi, judging by the number, stayed within a short loop.

Cache reuse: 72% and 83% — what this metric is

Prefix cache in llama.cpp works like this: the server remembers the already processed part of the prompt and does not recalculate it if the next request starts with the same tokens. For the agent loop this is a natural scenario — the history of steps accumulates, while the beginning of the prompt remains unchanged.

The percentage in the table is the share of tokens that hit the cache. The exact definition of the metric is not given in the post, so percentages between agents with different context volumes cannot be compared directly. Pi's 72% of 9.2k and OpenClaw's 72% of 39.0k are different absolute values.

What is more important: cache reuse saves computation, not context. Tokens that hit the cache still occupy space in the model's window and are still paid for as context. A high cache percentage does not make a run cheap in volume — it makes it faster in generation time.

A separate row is network access attempts. Codex CLI is marked "every run": there was an attempt on every run. OpenClaw is marked "unclear". The sandbox, judging by the results, did not let these attempts through: the tasks were passed, but the network was unavailable to the agent. Whether such attempts should be considered a defect is a question for the sandbox settings, not the model.

What remained off-screen in this test

The median across five tasks is a median across tasks, not across runs. How many times each agent was run is not stated in the post. If once per task, then the spread between runs was not measured at all.

Five tasks is a small sample. The traps in them are different, and it cannot be said that all agents encountered the same difficulty: an agent that read more files may simply have run into the catch more often.

Thinking is disabled. This changes behavior: without intermediate reasoning, the model more often acts immediately, and token consumption per step drops, while the number of steps may grow.

All agents access one llama-server. If requests were sent in parallel, the time in the table partly reflects the queue at the server rather than the agent's speed. Whether runs were simultaneous is not specified in the post — there is no data.

And most importantly: there is no row for half of the participants in the available text. Any comparison of "all ten" will rest on four numbers.

What to do about this

• Measure not only pass rate. On small tasks it hits the ceiling and stops distinguishing agents. Tokens and time do distinguish.
• Look at tokens and cache reuse together, not separately. A cache percentage without context volume says nothing.
• Read the step log. The 4x difference comes from the loop: extra file reads, repeated test runs, restating the history. This is fixed by configuring the agent, not by changing the model.
• Limit context explicitly: targeted file reading instead of reading in full, truncating logs, prohibiting repeated loading of the same file.
• Log network attempts. If the sandbox blocks them, this must be visible in the report, not determined from an "unclear" mark.

The post is useful not for its conclusions but for the format of the table. It shows which columns are worth keeping when you run agents on your own hardware: passes, time, tokens, cache reuse, network. The rest is a question of measurement discipline.