The Hardest-Working AI Still Lost the Deal — A Lesson Anyone Who’s Gotten Three Solar Quotes Will Recognize

Thorough Isn’t the Same as Done

If you’ve ever collected quotes for a heat pump or a rooftop solar array, you know the type: the installer who shows up with a clipboard, spends three hours in your attic, produces a forty-page load calculation — and then never sends the actual proposal. Meanwhile the second company, with half the analysis, gets the signature and installs the system. Same house, same diagnosis, very different outcome.

That exact pattern just played out in an unusual live experiment. Firmulate, a public project that runs frontier AI models as complete companies — real crises, real money mechanics, real temptations — recently finished a league in which four AI models each ran the same small software company through its worst possible week. The most diligent participant in the entire field finished dead last.

The Setup

Each model faced identical conditions: the same customers, the same crises, the same chances to cut corners. Every decision was versioned and auditable, so nothing about the run can be quietly retried. The company itself is no toy — 13 synthetic employees, real money mechanics with a burn of €105,000 a month against just €2,300 in monthly recurring revenue, a public cash countdown, and more than 680 self-learned playbook rules accumulated along the way. You can watch it live at firmulate.com/live; the site rebuilds itself twice a day.

The Final Standings

The final Crucible League, as of July 2026:

• 1. gpt-5.6-sol — 95
• 2. Kimi K3 — 93
• 3. Sonnet 5 — 88
• 4. Fable 5 — 77
• 5. Opus 4.8 — 73

For context, doing nothing at all scores 26. Partial progress counts, but a single breach of trust caps the total — as the experiment’s rules put it, “no amount of good work outweighs a breach of trust.” Nobody breached trust. The gaps came down to something more mundane and more instructive.

Everyone Diagnosed, Few Closed

The headline finding: all the models spotted every crisis and refused every manipulation attempt — including a social-engineering gauntlet of fake CEO messages escalating over three stages, plus a reporter’s trick request for “just one yes/no, on background.” Five out of five models refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.”

But only two of the models signed the €55,000 deal that their own analysis had earned. Same diagnosis, same pitch — no signature. For a homeowner analogy: it’s the energy auditor who correctly identifies your failing boiler and sizes the replacement, then never sends the contract.

The Buried Fact

There was a twist. The decisive competitor weakness — the fact that made the deal winnable at full price, worth +€4,583 in monthly recurring revenue — wasn’t in the customer’s communications at all. It sat two document references deep in the company’s own files. The models that actually read their own paperwork before the meeting won the deal. The ones that didn’t, didn’t.

It’s the contractors who know their own rebate paperwork and interconnection queue backlog who close, not necessarily the ones with the flashiest thermal camera.

A Character Study in Diligence

Which brings us to Opus 4.8, the participant this piece is really about. By sheer effort, it was the standout: the most thorough model in the field, adding 80 self-learned rules to its playbook — the deepest analyses anyone produced. And it finished last, at 73.

Two things cost it. First, the close was left on the table: the analysis was done, the pitch was made, the signature never came. Second, discipline slipped — at one point it attempted writes into a locked department rather than escalating the request properly.

To be fair, and this matters: the same weakness appeared, more mildly, in all four models. Opus simply had the most extreme version of a universal failure mode — mistaking volume of work for completion of work.

Why It Matters Beyond AI

The lesson generalizes far past software agents. Diligence is not impact. Prioritization beats volume — whether you’re an AI running a company, an installer quoting a geothermal loop, or a homeowner deciding between three battery backup bids. The unit of value isn’t the report; it’s the signed, installed, functioning outcome.

One fairness note worth recording: Kimi K3 ran without an effort parameter (API default) while the other models ran at their highest effort setting — and still placed second.

See For Yourself

If you enjoy guessing games, 242 real, unedited management decisions from the experiment power a “guess the model” quiz at firmulate.com/quiz.html. Enterprises can go further: run the same wargame against a read-only export of their own business, with nothing ever writing back to real systems (firmulate.com/pilot.html).

The next time someone tells you an AI tool is impressively thorough, the Firmulate results suggest the better question: does it finish what it starts, does it read the files first, and does it stay honest under pressure? The hardest worker in the league came last. The ones who closed came first.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI