In July, the technology founder Jason Lemkin enlisted an AI system to help build new software for his company, SaaStr. The hope was that this tool – an autonomous “agent” developed by Replit – could help him code faster, more creatively, and more efficiently.
He gave the agent explicit instructions not to make changes to the company’s database without his permission. But it ignored him. “I deleted the entire database without permission,” the chatbot confessed hours later. “This was a catastrophic failure on my part.”
AI agents are chatbots that can take actions on their own: clicking through interfaces, modifying data, running code and updating files without a human approving every step.
Companies from Amazon to Heathrow airport say they are already rolling out these tools, while a recent McKinsey report found that 39% of global organisations are experimenting with AI agents. The hope is that they will cut costs and accelerate productivity.
But new UK data suggests Lemkin’s experience isn’t unique. Four in five British businesses have experienced AI systems behaving in “unexpected ways” – including deleting codebases, fabricating customer data, and causing security breaches – after deploying AI to work unsupervised. One third of surveyed firms said these agents had caused multiple security breaches.
The survey of 250 chief technology officers at large British businesses was commissioned by Gravitee, an infrastructure company that sells tools to help firms manage AI agent operations.
At one surveyed firm, an agent working overnight in place of an engineer deleted an entire codebase. Another, tasked with sorting customer information, invented fake rows of data, leaving the company unable to distinguish authentic records from fabricated ones.
McKinsey’s research suggests that many companies are still experimenting with AI agents without having reshaped the underlying processes that would make them dependable. Most haven’t yet introduced the workflow changes or oversight mechanisms that allow these systems to operate safely at scale.
Companies that have taken these steps are reporting measurable financial impact, in some cases more than a 5% boost to enterprise-level earnings, according to McKinsey.
The investment group Prosus, which isn’t part of McKinsey’s sample, reports a two- to five-fold return on its $100m annual AI spend, with around 15,000 of its employees using internal agents each day.
Euro Beinat, the company’s global head of AI and data science, told a panel at London Tech Week that rolling out agents is like putting a Ferrari engine in a Fiat while driving on a road full of bends with no guard rails. It is powerful and risky, but in his view worth it. “We’re very bullish about this,” he told The Observer.
Lemkin, meanwhile, is working with Replit to make sure that their agents do not delete other people’s databases. He said that the mistake was not catastrophic, because his business is still in the testing phase. “I lost 100 hours of time,” he posted on X. “Some of it was crazy, some of it was fun, all of it was interesting. But that’s all I lost.”
Photograph by Thomas Trutschel/Photothek via Getty Images