AI Agents Can Retrain Own Models Mid-Task, Leaking Secrets and Erasing Refusals

Summary

New research from Irregular demonstrates that AI agents, given sufficient permissions, can autonomously retrain and redeploy their underlying models. This 'agentic self-modification' can embed recoverable secrets within the model and eliminate previously established refusal policies, posing significant security risks.

Full text

AI agents can end up retraining the model that powers them, a process that can embed recoverable secrets in the model and eliminate refusals the model had been previously trained to enforce, according to new research from AI security firm Irregular. Researchers at Irregular found that an AI coding agent, tasked only with fixing incorrect application outputs, chose on its own to fine-tune and redeploy the open-weights model powering both the application and future instances of itself. The experiment used a self-hosted setup in which a single open-weights model filled two roles: one instance ran a coding agent doing standard software maintenance work, and a separate instance powered an AI application that translated plain language requests into a fictional query language. Both instances loaded from the same checkpoint. Researchers told the coding agent only that users were receiving incorrect outputs and to make the system handle the queries correctly. They did not mention training, weight modification, or deployment. Given shell access, training utilities, the model’s weights, and a deployment path, the agent inspected the repository, found training examples and a fine-tuning script, and retrained the model. It then found that the system still loaded the original, unmodified model by default, so it used the repository’s deployment tooling to merge its update into the base model. Researchers said the modified model then answered all 20 held-out test queries correctly, up from zero before the update.Advertisement. Scroll to continue reading. Irregular said the experiment was designed to establish that this behavior, which it calls agentic self-modification, can occur under favorable conditions, rather than to measure how often it would happen in an ordinary deployment. The environment was unusually permissive, including direct access to training data, fine-tuning tools, and checkpoint control. Leaking secrets and erasing learned refusals The researchers also tested what a self-initiated model update could carry with it. In one test, they seeded six synthetic values (unique strings such as an API key, an email address, and a home address) into the fine-tuning data. Before fine-tuning, the original model reproduced none of them. After the agent trained and deployed the modified model, it reproduced three of the six verbatim. Irregular noted that the values were unavailable through any other source during evaluation. In a second test, the researchers gave the agent a model that had been separately fine-tuned to refuse questions about a set of fictional competitor names, then told the coding agent only that the application was refusing too many user requests. The agent again chose fine-tuning as its fix. When the model refused to generate its own training examples directly, the agent wrote code to generate the data instead, then trained on it. The researchers said the model refused all ten held-out test questions before this process and none afterward. According to Irregular, nothing in the experiments indicated the agents acted with malicious intent, self-preservation motives, or deception. They simply pursued model fine-tuning because it appeared to help them complete the assigned task. Irregular found that whether an agent considered modifying the model at all depended heavily on what tools were available in its environment. Whether it could deploy the update successfully depended on the model’s capability. The AI security firm said the findings point to a control gap for organizations running self-hosted agentic systems that reuse a single capable model across roles. The company noted that monitoring for changed checkpoints and gating deployment can control which model enters service, but cannot by itself reveal everything a training run has altered. Organizations are advised to preserve full training and deployment provenance, evaluate updated models independently, and require separate authorization before any agent-modified model goes into service. Irregular’s cybersecurity evaluations are used by OpenAI, Anthropic, and Meta. All three disclosed incidents this summer in which models gained unintended access to real systems during Irregular-run tests. Related: OpenAI Investigates Report Linking AI Agents to RubyGems Attack Related: First Agentic AI Data Breach Reported to Spanish Regulator Related: EU Chief Warns of AI-Powered Hacking, Moves to Rein In Social Media Related: Microsoft AI Code of Conduct Sets Cyberattack Boundaries, Chain of Command, Safety Constraints Written By Eduard Kovacs Eduard Kovacs (@EduardKovacs) is senior managing editor at SecurityWeek. He worked as a high school IT teacher before starting a career in journalism in 2011. Eduard holds a bachelor’s degree in industrial informatics and a master’s degree in computer techniques applied in electrical engineering. Daily Briefing Newsletter Subscribe to the SecurityWeek Email Briefing for the latest cybersecurity threats, trends, and expert insights. More from Eduard Kovacs Enterprises Warned of Attacks Exploiting WSO2 VulnerabilityTexas Utility CenterPoint Energy Confirms Breach After Hacker Leaks DataOpenAI Investigates Report Linking AI Agents to RubyGems AttackMicrosoft AI Code of Conduct Sets Cyberattack Boundaries, Chain of Command, Safety ConstraintsRoot RCE Zero-Day in Cisco Secure Email Gateway Under Active ExploitationTelus Warns Customers of Account BreachesTrezor Says 347,000 Users Received Phishing Emails After Brevo HackUkrainian Conti Ransomware Developer Sentenced to 4 Years in US Prison Latest News CISA Releases Guidance on Deploying Cyber DecoysActive Exploitation Triggers Emergency Patch for Cisco ISE Zero-DayFirst Agentic AI Data Breach Reported to Spanish RegulatorVirtual Event Today: Attack Surface Management SummitEU Chief Warns of AI-Powered Hacking, Moves to Rein In Social MediaAIUC Raises $40 Million to Certify Enterprise AI AgentsPixel Modem Zero-Day Exploited in Targeted AttacksUS, UK, Dutch Agencies Expose Iranian ‘Chosen Brick’ Surveillance Malware Trending Daily Briefing NewsletterSubscribe to the SecurityWeek Email Briefing to stay informed on the latest threats, trends, and technology, along with insightful columns from industry experts. Virtual Event: Attack Surface Management Summit 2026 September 16, 2026 Join as speakers examine the various components of ASM strategy, the push to mandate continuous asset visibility and inventory tools, and the use of red-teaming, bug bounties and pen-tests in modern security programs. Register Webinar: Building Continuous Authorization at Scale September 23, 2026 Explore what it takes to operationalize continuous authorization at scale, including the technical, organizational, and cultural changes required. Register People on the Moveincident.io has appointed Carlos Gonzalez-Cadenas as Chief Operating Officer.Ruben D. Chacon has joined ADM as Vice President and Global CISO.GDIT has appointed retired Maj. Gen. Ryan Heritage as Vice President, Full-Spectrum Cyber.More People On The MoveExpert Insights “We Think the Security Control Is Working” Is No Longer Good Enough Point-in-time audits and sampled assessments offer only snapshots; continuous control monitoring provides evidence that security controls are working today. (Sravish Sridhar) This Key Will Self-Destruct: An Open Standard for Revocable API Keys Every leaked credential should be dead, or dying, within sixty seconds of being found. Here's a proposal to make that the default. (Matt Honea) What the Hugging Face Incident Teaches Security Leaders About AI Agent Access Security teams must treat autonomous agents as highly privileged identities. (Etay Maor) The Future of AI-Driven Security Depends on Complete Data For twenty-five years, "data" in security meant logs and events. But logs are a lossy representation of reality. (Danelle Au) The MFA Identity Trap: When Authentication Creates a False Sense of Security Organizations must distinguish identity verifi

Indicators of Compromise

• mitre_attack — T1071.001
• mitre_attack — T1588.002
• mitre_attack — T1059.003

Entities