Attackers Manipulate AI Chatbots in Mass Disinformation, Phishing Campaign

Threat actors are exploiting a fundamental trust gap in AI chatbot systems by poisoning publicly indexed web content with malicious links and fabricated data, causing models like ChatGPT, Gemini, and Google AI Overview to surface and amplify harmful material as if it were legitimate. This form of indirect prompt injection and content poisoning undermines the perceived authority of AI-generated responses, making phishing lures significantly more convincing to end users who trust AI outputs. The root problem lies in insufficient validation of training and retrieval data sources, combined with a lack of end-user awareness that AI outputs can be manipulated. As AI assistants become primary information interfaces for millions of users, the blast radius of a single poisoned content campaign scales dramatically. Organizations that fail to educate employees about AI output risks and configure AI tool usage policies are particularly exposed.