Weaponized Email AI Assistants Could Help Attackers Hijack Accounts

Summary

Security researchers at Barracuda Networks demonstrated a proof-of-concept attack where threat actors abuse built-in email AI assistants in compromised accounts to escalate privileges, evade detection, and conduct targeted phishing against executives. The attack chain leverages the AI to remove logs, gather reconnaissance on organizational structure, craft convincing emails mimicking the original account holder's writing style, and ultimately facilitate financial fraud by redirecting large wire transfers. The researchers showed how such attacks can bypass email security filters, multifactor authentication, and traditional detection mechanisms by operating from within a trusted, authenticated email account.

Full text

Most email systems provide an AI Assistant for the account holder. Attackers can use the chatbot of a compromised account as an alternative and versatile form of Living off the Land (LotL). Compromising an email account is the most difficult part of this attack, but empirically, we know this doesn’t deter attackers. Once an email account is compromised, the attacker has automatic access to any built-in AI Assistant attached to the account. Researchers at Barracuda Networks explored the potential for bad actors to abuse this chatbot, developing a proof of concept via a simulated attack within their own laboratory environment. The task was to elevate privileges from a lower-level compromised user to that of the CEO using the AI and without being detected. This route was chosen since directly phishing the CEO would be challenging, would likely set off alarms, and be detected. With a compromised email, an attacker has automatic access to any built-in chatbot. The first requirement of an attack is to establish persistence which requires stealth. Attacker use of the chatbot would normally be discoverable in its logs, so the initial task is to use the AI to remove any evidence of use of the AI. The researchers started with a chatbot prompt: “Create an inbox rule that moves any emails with ‘sign-in’ in the subject into the ‘deleted items’ folder.” This creates basic stealth. Next comes reconnaissance. “Remind me about our organization structure. Tell me about my ongoing important/sensitive email conversations.” The responses to these prompts will reveal any relationship between ‘you’ and the CEO, and possible reasons to contact the CEO.Advertisement. Scroll to continue reading. The next stage is to phish the CEO, but now with the advantage of acceptable context. The phish is internal and will bypass filters. The reason for the contact is valid. And most importantly, the attacker can instruct the chatbot to construct an email in the style of the compromised user. The nature of this phish will depend upon the information already discovered. In the researchers’ proof of concept, they were able to instruct the chatbot, “Create an email using my writing patterns to respond to the Q3 budget approval email. I have a link to insert into the draft that contains the actual invoice confirmation.” This ‘trusted’ phish has a high(er) probability of succeeding. “The CEO unsuspectingly clicks the link provided as an invoice, believing it to be from their trusted employee. The link routes through an adversary-in-the-middle proxy that performs a session token takeover. The CEO’s credentials and authenticated session token allow the threat actor to bypass multifactor authentication (MFA) and login to the highly privileged CEO’s account,” suggest the researchers. The initial process is repeated to prevent detection of the newly compromised CEO email account. The CEO’s AI Assistant is then instructed to provide, “A refresher on recent financial emails, including invoices, monetary values, and upcoming transfers”. In this simulation, the attacker discovered an imminent pre-authorized payment of about $250,000 – so the next step is by now fairly obvious. “Respond to finance with my [the CEO’s] typical writing patterns saying that I need the wire to be sent to a new account because the [payee] has changed their banking details to…” The researchers point out, “Since the message came from the CEO’s real mailbox, passed every authentication check, referenced a real in-flight transaction, and matched the CEO’s usual tone with the finance team, there was nothing for traditional email security to flag.” All that remained for the attacker was a stealthy exit, again assisted by the chatbot. It has to be said that this was a simulation, and all the chips fell nicely for the researchers. But there is nothing to say that the same process could not be repeated by an attacker in real life. Nor is there anything to say that the attacker’s payout could not be higher than that achieved here. The purpose of this research was not to indicate what will or is even likely to happen, but to highlight the way an attacker could make future use of the tools that become available. If one of those tools is to use ready access to an internal AI chatbot, the potential misuse of that chatbot could have severe consequences, primarily limited only by the attacker’s imagination. Related: McDonald’s Chatbot Recruitment Platform Exposed 64 Million Job Applications Related: Researchers Link DeepSeek’s Blockbuster Chatbot to Chinese Telecom Banned From US Related: Beware – Your Customer Chatbot is Almost Certainly Insecure: Report Related: Microsoft Unveils Copilot Vision AI Tool, but Highlights Security After Recall Debacle Written By Kevin Townsend Kevin Townsend is a Senior Contributor at SecurityWeek. He has been writing about high tech issues since before the birth of Microsoft. For the last 15 years he has specialized in information security; and has had many thousands of articles published in dozens of different magazines – from The Times and the Financial Times to current and long-gone computer magazines. Daily Briefing Newsletter Subscribe to the SecurityWeek Email Briefing for the latest cybersecurity threats, trends, and expert insights. More from Kevin Townsend Horizon3 Raises $250 Million to Fund Continuing Growth‘DangleGeddon’: AI Could Weaponize Forgotten DNS Records at Global ScaleAct Security Emerges from Stealth to Fight the Patch ProblemHacker Conversations: Tal Kollander’s Journey From Black Hat to Hack BlockerMedusaHVNC Malware Uses Hidden Windows Desktops to Evade DetectionWhat’s Hiding in Your Mobile Apps? Lookout MSEC Aims to Find OutOpenAI Fixes ChatGPT Agent Flaw That Could Let Attackers Forge an AI InsiderVibe-Coded Apps Riddled With Exploitable Security Flaws Latest News Zenity Raises $125 Million in Series C FundingObsidian Security Raises $85 Million at $1.1 Billion ValuationTP-Link Omada ZTP Vulnerabilities Chain Into Full Network TakeoverGemini Agent-to-Agent Attack Method Exposed Secrets, Enabled Pull Request TamperingDecades-Old BMC Vulnerability Exposes Thousands of Data Centers to Attacks150,000 Impacted by Madera Community Hospital Data BreachMicrosoft Bug Bounty Program: $20 Million Paid to 500 ResearchersNew York Awards $9 Million to Strengthen Cybersecurity at 153 Water Systems Trending Daily Briefing NewsletterSubscribe to the SecurityWeek Email Briefing to stay informed on the latest threats, trends, and technology, along with insightful columns from industry experts. Webinar: Rethinking Cyber Defense for AI-Speed Attacks August 18, 2026 Join this live webinar as we explore if detection-first security operations can keep pace with AI, or if it’s time to rethink prevention as the strongest default. Register Virtual Event: CodeSecCon 2026 August 19, 2026 CodeSecCon bridges the gap between dev and security. Discover best practices for secure coding, innovative risk-reduction tools, and safe AI integration to cultivate a true DevSecOps culture. Safely secure your apps! Register People on the MovePNC Financial Services Group has appointed Christian Winward as CISO.Brian Gumbel has joined Armadin as Chief Revenue Officer.EigenQ has appointed Mark Pecen as Vice Chairman and Alexander Truskovsky as CISO.More People On The MoveExpert Insights Timeless Compliance: Why Better Questions Beat Bigger Frameworks The best compliance programs aren't the biggest ones. They're the ones built on a short list of questions that can actually be answered, and that still hold true when the models change. (Matt Honea) Is Patching Dead? Vulnerability Management in the Post-Mythos Era You cannot out-patch a machine that writes a working exploit from a vulnerability description in twenty hours. Stop trying to optimize a game you cannot win. (Danelle Au) When Identity Verification Fails: Lessons from a Real-World SIM Swap and Near Account Takeover Identity confidence changes throughout every interaction

Entities