Kolkrabbi got tool calls as text
The model returned them as message content rather than as tool calls, using "parameters" where the schema says "arguments", and invented absolute paths. Our own agent failed exactly as hard as the other two.
KolkBench
Benchmark the
harness.
Almost every benchmark answers “which model is smartest”. Give three coding agents the same model, the same repository and the same prompt and they still perform differently, because what varies is the agent loop — how it explores, when it decides it is done, how it recovers from a bad edit. That is the number nobody publishes.
The first thing it measured was its own premise, and the answer was no. Everything below is one pilot on , and it is a null result.
01 / THE FIRST RESULT
Three agents, one local 8B model, 0 of 18.
Before spending anything on a cloud model, the plan was to hold the model constant with a local one: no API cost, no credentials, reproducible by anyone. That pilot never got as far as comparing harnesses.
| Harness | Off-by-one | Python parse bug | Find a function | Total |
|---|---|---|---|---|
| Kolkrabbi 1.3.1 (ours) | 0/2 | 0/2 | 0/2 | 0/6 |
| Codex CLI 0.151.0 | 0/2 | 0/2 | 0/2 | 0/6 |
| OpenCode 1.18.29 | 0/2 | 0/2 | 0/2 | 0/6 |
02 / HOW THEY FAILED
Three agent loops, three shapes of nonsense.
The tasks are small on purpose. The first is a loop that stops one element early, with a failing test that names the function. The third asks for no code change at all — find a function, write its name to a file.
OpenCode got hallucinated Python
from todowrite import todowrite — the model wrote imports for an API that does not exist instead of calling the tools it was given.
Codex got invented tool names
create_goal, web_search, multi_agent_v1.wait_agent. None of them are tools Codex offers.
The only run that wrote anything was confidently wrong
Asked to find the function deciding token validity in a repository that is entirely Go, the model said it read api_gateway_service/api_token_utils.py — a file that does not exist, in a language the repository does not contain — wrote is_token_valid to ANSWER.txt and reported success. The answer was checkExpiry. It did not fail to find the function so much as decline to look.
03 / THE MODEL, NOT THE HARNESS
One model advertises a capability it does not have.
Two models were tried before concluding anything.
qwen2.5-coder:7b lists tools and does not do them
ollama show reports tools under Capabilities. Ask for a single trivial tool call and it comes back as message content instead — identically on the OpenAI-compatible endpoint and on Ollama's native one, so it is the model's template rather than the compatibility layer.
llama3.1:8b tool-calls once, then stops
It returns a correct native tool call for one simple tool in isolation, and stops doing so once a real agent's system prompt and full tool schema are in front of it — which is the only condition that matters.
When everything scores zero, the score is the model
At 8B on this hardware the bottleneck is not the agent loop, it is the model emitting a well-formed tool call under load. So this is published as a pilot, and the harness comparison waits for a model capable enough that the loop is what varies.
04 / WHY BELIEVE ANY OF IT
Written by the author of one of the harnesses.
That is a real problem and promising to be fair does not fix it. These are the mechanisms that let you check instead of trusting.
Tasks tagged before any run
The twelve tasks and their machine-checked pass conditions were committed and tagged kolkbench-tasks-v1 before a harness was pointed at them. Tuning a task to a result would mean moving the tag, and moving it is visible in the history.
Nothing is scored by opinion
Each task's verify.sh exit code is the entire result. And each pass condition is itself proven to discriminate: it must fail on the untouched fixture and pass after a known-good fix.
Transcripts published, not summaries
Every run's full output and the per-run JSONL are in the repository. If a summary ever disagrees with them, the raw files are right and the summary is a bug.
No key, no account, no spend
Pull the model, point the runner at it. The whole pilot costs nothing to repeat, which is the only reason to believe a benchmark somebody else ran.
THE DATA
Full pilot write-up and raw runs Methodology Harness configuration Running models locally