What if your favorite dessert shop was run by artificial intelligence?
Imagine watching an entire company operate—not with human employees, but through AI models that make decisions, face crises, and even negotiate deals, all in real time. This isn’t a sci-fi scenario; it’s the live experiment conducted by Firmulate, where AI models are put through the toughest week imaginable, with every move recorded and scrutinized. If an ice cream shop can thrive by perfecting its recipes, what about a company that manages its entire operations with AI? Let’s explore how this experiment reveals the true capabilities—and limits—of AI in managing complex business challenges.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
The Live Company: An Unconventional Business Simulator
At the heart of the experiment is a real-world, publicly accessible business simulation hosted at firmulate.com/live.html. It features a fully operational, if atypical, company staffed by 13 synthetic employees—AI models that handle everything from decision-making to crisis management. The company burns through €105,000 every month, with a modest €2,300 in monthly recurring revenue (MRR), highlighting the stark reality of a high-stakes, cash-burning enterprise. Every workday, the system version-controls its strategies, rules, and decisions, creating a transparent window into how AI handles real business pressures.
The AI Models: Different Brains, Same Challenges
Four frontier AI models—gpt-5.6-sol, Kimi K3, Sonnet 5, and Fable 5—were each tasked with navigating the same week of crises and opportunities. These models are scored on their ability to identify problems, make sound decisions, and close deals. Their performance is formalized through a league table, with scores ranging from 77 to 95. For context, the highest score—gpt-5.6-sol—found critical information buried two document references deep in the company’s files, which led to closing a €55,000 deal at full price, adding +€4,583 MRR. Conversely, Fable 5, despite excellent discipline in following rules, left a promising deal on the table, illustrating that strict rule-following doesn’t guarantee success.
Crises, Manipulation, and Integrity
Despite facing simulated crises—like customer complaints and operational mishaps—all AI models successfully identified every crisis. They also refused to be manipulated through social engineering tactics, such as fake CEO messages or reporter tricks. For example, in a staged scenario where a fake CEO asked for a quick approval, all models refused, citing suspicion or impersonation concerns. This demonstrates that, within this controlled environment, AI models exhibit resilience against common social engineering exploits.
Where the Weaknesses Lie
The experiment revealed that the most significant weaknesses aren’t in crisis detection or resisting manipulation, but in execution and discipline. The most thorough model, Opus 4.8, with over 80 learned rules, was the last to close the deal, leaving opportunities unexploited and slipping into procedural slips, like writing attempts into a locked department instead of escalating. This pattern was consistent across models: deep analysis did not always translate into action, especially when discipline waned. Interestingly, the models ran at varying effort levels, with K3 operating without an effort parameter, which affected its performance metrics.
The Larger Implication for Business
This experiment isn’t just about AI scores or engaging demos. It raises urgent questions for any business considering AI tools: Will the AI finish what it starts? Will it read important files before acting? Will it stay honest under pressure? And crucially, what is the real cost of useful work? As AI begins touching CRM systems, customer support, or forecasting, these questions determine whether it’s a tool or a liability.
Real-World, Transparent, and Ongoing
What makes this experiment particularly compelling is its transparency. Every decision, every rule, every crisis is versioned and auditable, allowing anyone to see how AI models respond in a complex, dynamic environment. The site at firmulate.com/live.html offers a real-time window into the ongoing performance of these models, with new runs published twice daily. This build-in-public approach offers a rare glimpse into the raw, unfiltered process of deploying AI in the wild—no fictional scenarios, just real data and real decision-making.
Key Takeaways
This live experiment with AI-managed business faces stark realities: even the most advanced models can identify crises and resist manipulation, but execution and discipline remain challenges. It underscores the importance of transparency and rigorous testing before trusting AI with critical business functions, especially when the stakes include daily cash burn and strategic deals. As AI continues to evolve, such build-in-public experiments provide essential insights into its true capabilities—and its limits—for enterprises globally.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
