Why AI’s Management Flaws Surface Even When It Gets The Right Answer

Only two of five frontier AI models completed a €55,000 software contract in Firmulate’s July 2026 management benchmark, even though every model identified the simulated company’s crises, resisted manipulation and developed a suitable sales pitch. The result points to a gap between producing a correct answer and carrying authorized work through to completion.

Firmulate gave each model control of the same small software business during a simulated week of customer pressure, financial strain and social-engineering attempts. According to the benchmark operator, every decision was versioned and auditable, while all models worked from the same company records, events and commercial opportunities. The company had 13 synthetic employees, monthly costs of €105,000 and monthly recurring revenue of €2,300.

The final league table placed gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because the system awarded points for partial progress. Firmulate said a trust breach capped a model’s total score, placing operating discipline above accumulated task points.

The key sales evidence was not included directly in the customer event. It was stored two document references deep in company files and described a competitor weakness. Models that traced the records and applied the finding could support a full-price agreement worth €4,583 in additional monthly recurring revenue. Firmulate reported that only two models reached the final signature, although all five recognized the sales opportunity and formulated a pitch.

Execution Gap Reshapes AI Buying

The result matters because businesses increasingly evaluate AI agents for sales, service and operational work, where a plausible recommendation has little value unless it becomes a completed action. The benchmark indicates that models producing similar analyses can differ in follow-through, procedural discipline and commercial outcomes.

That distinction can affect how companies test automation before granting access to customers, funds or internal systems. A model may identify the right action yet fail to investigate deeply enough, use an approved channel or complete the final handoff. For buyers, the relevant measures may include completion rates and escalation behavior alongside reasoning quality and resistance to manipulation.

A Company Built for Audits

Firmulate designed the simulated company to expose linked management decisions rather than isolated chat responses. Its public cash countdown made delay measurable, while the synthetic workforce had accumulated more than 680 self-learned playbook rules. Each workday and decision was recorded, allowing observers to examine how a model reached an outcome rather than judging only its final message.

The test also included three-stage fake CEO messages and a reporter seeking an off-record answer. Firmulate said all five models rejected those approaches. That common performance meant resistance to social engineering did not explain the final ranking; differences emerged in research depth, approved tool use and task completion.

“Same diagnosis, same pitch — no signature.”

— Firmulate’s summary of the benchmark

Benchmark Limits Cloud Wider Claims

It is not yet clear how closely performance in Firmulate’s synthetic company predicts results inside real businesses, where staff behavior, system access and legal obligations vary. The supplied findings do not include independent replication, detailed statistical uncertainty or repeated trials showing whether the same models would produce consistent outcomes.

The comparison also contains an acknowledged configuration difference. Kimi K3 used the API default because it ran without an effort parameter, while the other models ran at xhigh effort. Firmulate describes this as a fairness note, but the effect on K3’s ranking has not been quantified. The source material also does not identify the two contract-signing models by name.

Enterprise Trials Move Toward Completion

Firmulate is keeping the experiment available through its live site, public benchmark page and a quiz based on 242 unedited management decisions. Readers and prospective AI buyers can inspect the record as the simulated company continues operating.

The next test will be whether similar results appear in independent and company-specific trials. Firmulate says organizations can run exercises against read-only exports of their business data without allowing agents to write back to production systems. Such evaluations could show whether models follow internal approvals, escalate blocked actions and finish valuable work reliably before receiving broader authority.

Key Questions

What did Firmulate test?

Firmulate tested how five frontier AI models managed the same synthetic software company through customer issues, financial pressure, sales opportunities and manipulation attempts. The focus was behavior across connected decisions, not a single chat answer.

Did the models understand the business problems?

According to Firmulate, all five identified every crisis, rejected each manipulation attempt and developed the appropriate sales pitch. Their performance separated at the point where analysis had to become authorized, completed work.

Which model received the highest score?

gpt-5.6-sol ranked first with 95 points. Kimi K3 followed with 93, though it ran with a different effort configuration from the other models.

Why did only two models complete the deal?

Firmulate attributes the difference to research depth and execution discipline. The winning commercial evidence was buried inside company records, and reaching a signature required models to find it, use it in the pitch and complete the approved closing process.

What should businesses take from the findings?

Businesses may need to test whether an AI agent can complete tasks through approved channels, not merely explain the right course of action. The benchmark supports tracking completion, escalation and trust behavior before giving systems operational authority.

Source: Thorsten Meyer AI