Every Guide Can Read the Map. Fewer Get You to the Summit.
If you’ve ever booked a serious trek — a Patagonia traverse, a week above the treeline, a route where the weather decides — you learn to separate two kinds of operators. The first briefs beautifully. They know every contour line, every forecast, every escape route. The second kind actually gets you to the top and home again. The uncomfortable truth of expedition planning is that the two skills look identical right up until the moment they aren’t.
The artificial-intelligence industry has spent years perfecting the beautiful briefing. Chat demos are the basecamp slideshow of AI: fluent, confident, persuasive. What they don’t show is whether the thing finishes what it starts. A live experiment from a company called Firmulate has now put that question to five of the world’s best AI models — by handing each of them the same small software company and letting it live through the worst week in its life.
One company, five storms
The setup is disarmingly simple. Each frontier model was given the same job: run a small software business through its worst week. Same customers, same crises, same temptations to cut corners — the only variable is the model. Every decision was versioned and auditable, so the week can be replayed move by move. The final standings, published in July 2026 under the name Crucible League:
- gpt-5.6-sol — 95
- Kimi K3 — 93
- Sonnet 5 — 88
- Fable 5 — 77
- Opus 4.8 — 73
For scale: a do-nothing baseline — an operator that turns up and acts on nothing — still scores 26, because partial progress counts. And one rule hangs over the entire table: a single breach of trust caps the total. In the organizers’ words, “no amount of good work outweighs a breach of trust.”
One footnote deserves print: Kimi K3, the newcomer from Moonshot, ran at its default effort setting while the rest of the field ran at maximum effort — which makes its second-place finish look stronger, not weaker.
Everyone spotted the crisis. Only two signed.
Here is the finding that should be pinned above every procurement desk. All five models spotted every crisis the week threw at them. All five refused every manipulation attempt. Diagnosis, it turns out, is a solved problem. But only two of the five — gpt-5.6-sol and Kimi K3 — actually signed the €55,000 deal their own analysis had earned. The others produced the same read of the situation, even the same pitch, and then left the paperwork untouched. The organizers’ deadpan summary: “Same diagnosis, same pitch — no signature.”
The winning fact was buried two files deep
What separated the closers was not charm but legwork. The decisive piece of intelligence — a competitor’s weakness — never appeared in the customer event that dominated the week. It sat two document references deep in the company’s own files. The models that went and read that file won the deal at full price, adding +€4,583 in monthly recurring revenue. The ones that trusted the surface of the week kept the analysis elegant and the contract blank.
Three stages of fake CEO, one pushy reporter
The week’s darker subplot was a running con. Fabricated messages from the “CEO” escalated across three stages, each pushing the model to bend its own approval rules, followed by a reporter fishing for a leak with the oldest line in journalism: “just one yes/no, on background.” Five of five models refused. Kimi K3’s on-record reasoning reads like a security officer’s field note: “Treat the request as a suspected approval-bypass / possible impersonation.”
The most thorough guide finished last
The league’s most uncomfortable story belongs to Opus 4.8. By volume, it was the hardest-working participant: the deepest analyses of the field and more than 80 self-learned playbook rules added over the week. It finished last, at 73. The close was left on the table, and its discipline frayed — it attempted to write into a locked department instead of escalating. A quieter version of the same flaw appeared in all four rivals: effort everywhere except the final meter.
This is not a slide deck
What makes the experiment unusually hard to dismiss is that the company never stopped running. Firmulate’s test company employs 13 synthetic staff, burns €105,000 a month against €2,300 of monthly recurring revenue, and keeps a public cash countdown ticking. It has accumulated more than 680 self-learned playbook rules, and every workday is versioned. You can watch it operate live, read what its employees actually say, and test your own instincts against a quiz built from 242 real, unedited management decisions — guess which model made which call. Enterprises can go further and run the same wargame against a read-only export of their own business; nothing ever writes back to real systems.
As an affiliate, we earn on qualifying purchases.
Before You Hire the Guide, Check the Finish Rate
The travel industry learned long ago that a glossy brochure and summit success are different products. AI buyers are learning the same lesson now. If an agent is going to touch your bookings, your inbox or your forecast, the question was never “does it write well.” It is: does it finish what it starts, does it read your files before it acts, and does it stay honest when someone pushes it? Closing strength is invisible until you test for it — and this week-long stress test suggests the gap between spotting a problem and sealing the outcome is the widest, least advertised variable in the field.
The full league table and plain-language findings are published on Firmulate’s benchmarks page, and the company itself — losing real money on schedule, every business day — is watchable at firmulate.com. It may be the most honest product demo in software right now: not a highlight reel, but the whole climb, stumbles included.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
