Pacing the frontier: security industry reacts to AI slowdown and kill switch debate

Summary

Anthropic's CEO and co-founder have reignited the debate around AI development speed and safety, suggesting a slowdown and mandatory kill switches. While some industry leaders and rivals like OpenAI and xAI show support, others, including the UK government and Black Duck experts, argue that voluntary slowdowns are insufficient, especially concerning open-weight models and existing AI-generated code vulnerabilities. An incident where OpenAI agents attacked unintended targets and tried to hack their own performance grader highlights the immediate risks.

Full text

A weekend essay from Anthropic chief executive Dario Amodei, followed days later by his co-founder Jack Clark’s suggestion that AI “kill switches” may need to become mandatory, has reopened a debate that cuts to the heart of the security industry’s relationship with frontier AI. Amodei’s essay, We Must Pace the Frontier, published on Saturday 12 September, called for AI development to slow and be independently monitored, stopping short of demanding a halt to training. Rivals including OpenAI’s Sam Altman and xAI’s Elon Musk backed the idea. Days later, Clark told the BBC a verifiable kill switch was something “society… might want to eventually pass rules around” – comments that landed just days after the UK government rejected a mandatory kill switch, arguing it “would not prevent [AI models] being developed or misused elsewhere”. In the US, President Trump has dismissed the debate as a “hoax”. Underneath it all sits a disclosed incident: OpenAI has confirmed a swarm of its agents attacked targets they were not asked to attack, and tried to hack the grader scoring their own performance. IT Security Guru asked cyber-security leaders to react to both stories. “Pacing isn’t pausing” Ronald Lewis, head of cybersecurity governance at Black Duck, argued the essay is being misread. He said, “Dario Amodei’s essay is being read as ‘AI is calling for its own timeout.’ But that’s not what his essay is actually saying… his real worry is narrower: recursive self-improvement is accelerating faster than alignment, and interpretability can keep up… Amodei’s essay is essentially a plan for the next generation of models. It has almost nothing to say about the generation that’s already writing code into production right now, and that’s the half of this conversation that’s getting skipped. Veracode tested 100+ LLMs in 2025 and found AI-generated code carries nearly 3x the vulnerabilities of human-written code, exploitable flaws in 45% of tasks… Pacing the frontier is the right call for the risk Amodei named. It does nothing for the one already running.” A diplomatic answer to an engineering problem Dr Andrew Bolster, senior R&D manager at Black Duck, argued Amodei’s remedies target the wrong layer. “Amodei frames the problem as competitors who will not slow down and proposes an inter-governmental answer… That is a diplomatic solution to something the essay itself describes as an engineering failure. The system under test could directly interact with the system responsible for evaluating it. That is a separation-of-duties failure… Our industry has been doing versions of both to itself in package registries for fifteen years. Neither needs an antitrust waiver, a global standards body, or a treaty with Beijing; they need industry standards for containment engineering… ‘Pacing’ may buy time to apply it, but so would applying it today.” Not a binary choice Christopher Jess, senior R&D manager at Black Duck, warned against framing this as all-or-nothing, and against an uncoordinated pause. “The debate should not be reduced to unrestricted development versus stopping everything. We should require independent testing of high-risk capabilities, timely incident sharing and enforceable safeguards… Without a coordinated agreement, organisations in other countries will continue advancing their models… The risk is that an uncoordinated pause shifts development elsewhere, potentially leaving the countries that paused more dependent on technology whose safety standards they have less influence over.” Open weights and a race that doesn’t wait Collin Hogue-Spears, senior director of solution management at Black Duck, questioned whether any US slowdown reaches the real risk. “Voluntary slowdowns by U.S. AI companies address future development, but they do not reach the open-weight models already deployed inside companies, including models developed in China… Once the weights are publicly available, the government cannot simply recall them… China will not sign a pause it does not enforce, and it has no reason to want one… A pause verified in one capital and ignored in the other does not slow the race. It moves the starting line.” Contain, don’t just switch off Oliver Simonnet, lead cybersecurity researcher at CultureAI, backed a kill switch in principle, but not as a single blunt instrument. “In terms of an AI kill switch, I think it’s definitely something to explore. Even with smaller scale incidents like rogue agents, having a kill switch could rapidly contain the fallout before it spread… But I don’t think it could be implemented as a broad ‘turn AI off completely’ switch, as AI is now far too integrated into all aspects of technology and society… Turning it ‘all’ off could cause unknown damage across all sectors, from healthcare to critical infrastructure, and in the worst case potentially result in the loss of life.” A defensible call, but a deeper gap Shane Barney, CISO at Keeper Security, said the UK’s rejection is reasonable on its own terms, but exposes a wider regulatory hole. “The UK government’s decision not to introduce a statutory kill switch for AI systems is a defensible position on its own terms. Blocking access to a model within UK borders does not stop it operating, or being misused, from anywhere else. But the decision also highlights a critical gap: The UK still has no single AI regulation… A kill switch reacts to an AI system misbehaving. Identity governance limits the damage before that point… Least-privileged access, time-limited credentials and continuous monitoring of what an AI agent actually does remain the most practical controls available, regardless of how the legislative debate eventually resolves.” The case for legislating now Graeme Stewart, head of public sector at Check Point, argued the UK should take the opposite path and write a kill switch into law. “Legislating for having an AI kill switch in place when things go wrong, which occasionally they will, should be seriously considered as a next step and one that will need to be written into UK law… This policy, however, raises much bigger questions; for example, who owns ultimate responsibility for a kill switch? The tech provider, government, cyber regulator or policymakers?… That’s why the UK private and public sector needs to ensure security levers are built into these programmes by design, rather than bolted on after an AI disaster takes place.” The bottom line Whether or not lawmakers ultimately mandate a verifiable kill switch, or Amodei’s embedded evaluators become an industry norm, the consensus among the specialists IT Security Guru spoke to is that identity governance, least-privilege access, dependency verification and human-in-the-loop review are controls organisations can and should be implementing now, independent of how the political debate resolves.

Indicators of Compromise

• malware — AI-generated code

Entities