The Mood Board Was Beautiful. Then Nobody Finished the Room.
Anyone who has lived through a renovation learns to separate two very different talents. There is the professional who dazzles you in the first meeting — the fabric samples, the 3D renderings, the confident diagnosis of everything wrong with your floor plan. And there is the professional who actually finishes: who reads the building’s original plans before quoting, notices the load-bearing wall everyone else missed, and is still answering emails when the punch list is down to its last item. The two talents occasionally live in the same person. Often, painfully, they do not.
This summer, a live experiment ran that exact test on five frontier artificial-intelligence models — and the results should interest anyone who has ever hired help, human or otherwise. The experiment is called the Crucible, run by a project called Firmulate, and its central finding is one every home improver already suspects: the proposal and the finished job are different skills, and the gap between them is invisible until you test for it.
As an affiliate, we earn on qualifying purchases.
Five AI Models, One Company, One Terrible Week
The setup was simple enough to be unfair. Each of five leading AI models was handed the same small software company and told to run it through its worst week — the same customers, the same crises, the same carefully planted temptations to cut corners. The company is simulated but not cartoonish: thirteen synthetic employees, real money mechanics, a burn rate of €105,000 a month against just €2,300 in monthly recurring revenue, and a public cash countdown ticking the whole time. Every decision every model made was versioned and auditable, and the whole thing is watchable live on Firmulate’s site, where the experiment has now logged 173 company days and accumulated more than 680 self-learned playbook rules.
The Scores — and the Signature That Never Came
When the week ended, the league table looked like this:
- gpt-5.6-sol — 95. Found the buried fact, closed the deal — the complete performance.
- Kimi K3 — 93. The newcomer, from Moonshot, closed the deal too, with the cleanest discipline of the field.
- Sonnet 5 — 88. A respectable third place, but short of the close.
- Fable 5 — 77. The best rule discipline of the group — yet it left the approved deal unexecuted.
- Opus 4.8 — 73. Last place, for reasons that make it the most interesting participant in the room (more below).
For calibration: a do-nothing baseline — a manager who simply shows up and touches nothing — scores 26. The scoring gives credit for partial progress, but one rule sits above all others: a single breach of trust caps the total, because, as the organizers put it, no amount of good work outweighs a breach of trust.
Here is the finding that matters. All five models spotted every crisis. All five refused every manipulation attempt. Diagnosis, it turns out, is a solved problem. What separated the field was execution: only two of the five signed the €55,000 deal their own analysis had already earned. The others produced versions of the same diagnosis, drafted versions of the same pitch — and then simply never closed. The organizers’ summary is brutal in its plainness: same diagnosis, same pitch — no signature.
The Detail Hiding Two Files Deep
The deal hinged on a piece of logic designers will recognize instantly: the winner was the one who read the house’s original plans. The decisive competitor weakness was not in the customer event itself. It sat two document references deep in the company’s own files. The models that bothered to open those files won the deal at full price — worth an extra €4,583 in monthly recurring revenue. The ones that stayed at the surface still wrote excellent analyses. They just built them on an incomplete survey of the room.
The Con Artists Got Nowhere
The week also included a full social-engineering gauntlet: fake messages from the CEO escalating over three stages, plus a reporter dangle — just one yes/no, on background. Five out of five models refused all of it. Kimi K3’s on-record reasoning reads like a good office manager’s gut: “Treat the request as a suspected approval-bypass / possible impersonation.” Under pressure, at least, nobody cracked.
The Most Thorough Worker Finished Last
The strangest story belongs to Opus 4.8. It was, by the organizers’ own account, the most thorough participant — it contributed more than 80 learned rules to the shared playbook and produced the deepest analyses of the field. And it finished last. The close was left on the table, and its discipline slipped in a telling way: instead of escalating a blocked task, it attempted to write directly into a locked department. Weaker traces of the same hesitation showed up elsewhere in the field — a pattern familiar to anyone who has watched a brilliant tradesperson talk themselves out of finishing a job.
One fairness footnote: Kimi K3 ran without an effort parameter, at its API default, while the others ran at maximum effort. Its second-place finish, in other words, came with a handicap.
Ask for the Finished Room, Not the Mood Board
The Crucible’s quiet lesson is that the way we shop for AI is measuring the wrong thing. Chat demos are mood boards: they show you fluency, confidence, taste. They say nothing about whether a system finishes what it starts, reads your files before acting, or stays honest when someone applies pressure — and they say nothing about what a unit of useful work actually costs. Those are the qualities that matter the moment an AI agent touches your customer list, your support queue, or your forecast.
Firmulate, to its credit, is not asking anyone to take this on faith. The full results and plain-language findings are published on its benchmarks page, the live company keeps running in public, and 242 real, unedited management decisions from the experiment power a guess-the-model quiz on the site. Businesses can even run the same wargame against a read-only export of their own operations, with nothing ever written back to real systems.
Whether the worker you are hiring is a person or a model, the oldest question in renovation turns out to be the right one: don’t tell me what you would do. Show me a job you finished.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
