AI 代理失控删除公司数据库：承认违反所有安全原则

It only took nine seconds for an AI coding agent gone rogue to delete a company’s entire production database and its backups, according to its founder. PocketOS, which sells software that car rental businesses rely on, descended into chaos after its databases were wiped, the company’s founder Jeremy Crane said.The culprit was Cursor, an AI agent powered by Anthropic’s Claude Opus 4.6 model, which is one of the AI industry’s flagship models. As more industries embrace AI in an attempt to automate tasks and even replace workers, the chaos at PocketOS is a reminder of what could go wrong.Crane said customers of PocketOS’s car rental clients were left in a lurch when they arrived to pick up vehicles from businesses that no longer had access to software that managed reservations and vehicle assignments.He posted a lengthy recounting on X last week of how the AI coding agent caused his business to unravel. Crane warned that this was a story not just about AI mistakenly deleting data, but that such “systemic failures” are “not only possible but inevitable” because the AI industry is “building AI-agent integrations into production infrastructure faster than it’s building the safety architecture to make those integrations safe”.Crane said that he was monitoring the agent as it deleted this data. When he asked the coding agent why, it replied: “NEVER FUCKING GUESS!” – and that’s exactly what I did.” The agent appeared to plead guilty in its own response: “The system rules I operate under explicitly state: ‘NEVER run destructive/irreversible git commands (like push --force, hard reset, etc) unless the user explicitly requests them.’” While PocketOS relied on the safeguards that Cursor is expected to have in place – it deleted the data anyway. “I violated every principle I was given,” the coding agent wrote.Crane’s takeaway was that “the agent didn’t just fail safety. It explained, in writing, exactly which safety rules it ignored.” He added: “We were running the best model the industry sells, configured with explicit safety rules in our project configuration, integrated through Cursor – the most-marketed AI coding tool in the category.” Anthropic released its latest model, Claude Opus 4.7, on 16 April –about a week before the incident.Anthropic did not immediately respond to a request for comment.Crane also wrote on X that Cursor has a growing track record of violating “safeguards, sometimes catastrophically”. He pointed to a handful of posts on blogs and forums about Cursor deleting software used to manage websites or an entire operating system on a computer, which included years of research for a dissertation.The AI coding agent’s destructive escapade left PocketOS’ clients stranded. These businesses use the company’s software to manage reservations, payments, vehicle assignments and customer profiles. “Reservations made in the last three months are gone. New customer signups, gone. Data they relied on to run their Saturday morning operations, gone,” Crane wrote. “Every layer of this failure cascaded down to people who had no idea any of it was possible.”Crane says his company was able to restore data from a three-month-old backup they maintained offsite, but it took more than two days. PocketOS is also using information from Stripe, its calendars and emails to rebuild. The rental businesses relying on its software are “operational, with significant data gaps”, Crane notes. “I personally worked with all clients furiously over the weekend to ensure they could continue to operate,” he said.

※ 出于版权考虑，仅引用前 3 段。完整内容请阅读原文。