AI and Cybersecurity – Everything You Wanted to Know, But Were Afraid to Ask

Summary

This article explores the multifaceted impact of AI on cybersecurity, drawing insights from numerous experts. It delves into how AI is used for defense, its potential for misuse by insiders, and how adversaries are exploiting it. Key AI technologies discussed include Generative AI (gen-AI), Agentic AI, Shadow AI, Machine Learning (ML), and Artificial General Intelligence (AGI), with a focus on the capabilities and risks associated with gen-AI, LLMs, and deepfake generation.

Full text

To better understand the current state of artificial intelligence (AI) in cybersecurity, SecurityWeek spoke with dozens of security practitioners, researchers, vendors, analysts, and AI experts. The result is a comprehensive snapshot of how AI is being used across the security landscape today. Organized into five key topic areas, this report examines the role of AI through multiple lenses: whether it can be trusted, how organizations are using it, how it can be misused by legitimate insiders, how it is being exploited by cyber adversaries, and where the technology is likely headed next. The five topics are: Generative AI (gen-AI) Agentic AI Shadow AI Machine learning (ML) Artificial general intelligence (AGI) Taken together, these perspectives provide a practical assessment of AI’s opportunities, risks, and likely evolution in cybersecurity. Generative AI Generative AI (gen-AI) is the bedrock of contemporary AI, although it is technically and potentially born out of earlier machine learning (ML, see below). It does what it says: it generates new content (most commonly text) from an AI model (most usually a large language model or LLM). Chatbots are the users’ interface to the LLM, enabling questions (known as prompts) to be applied and responses received in natural language, and answers to be received in natural language. Chatbots are the interface, and LLMs are the reasoning engine. For most people in most direct use the two seem inseparable – just one big gen-AI application.Advertisement. Scroll to continue reading. “Gen-AI trains on massive data sets, learns statistical and relationship patterns, and then uses those patterns to synthesize original output from a prompt,” explains Ahmad Shadid, co-founder and CEO at ORGN.com. This is important. It does not create factually correct answers to prompts; it predicts probable answers based on the relationship patterns it has learned – but it does create linguistically correct and compelling responses. Four deep learning architectures power the training for modern gen-AI variants. Transformer architecture (the ‘T’ in GPT and BERT) is used for the LLMs such as ChatGPT, BERT and Claude. Diffusion training generates the variants that focus on creating high quality images and also audio and video. Fundamentally, this process starts with random noise. Mathematically (guided by the user’s prompt) it reduces and reshapes the noise into the required clear result. Diffusion reverses the process of destruction. The generated result is again based on probability – in this case, the probably correct distribution of pixels. Classic diffusion is evolving into diffusion transformer technology (Sora) and ‘flow matching’ (DALL-E 3 and Midjourney) which can be described as next-gen diffusion. Generative adversarial networks (GANs) are trained via two adversarial networks locked in a feedback loop. One creates fake data, while the other learns to detect flaws by repeatedly suggesting flaws and feeding them back to the creation. Both improve until the detector can find no more flaws in the creation. This approach is good at creating images, video and audio, but has largely been superseded by diffusion technology for business use. However, criminals still use GAN-based simple, fast, real‑time face‑swap and voice‑clone models to create deepfakes. The fourth architecture, variational autoencoders (VAEs), use an encoder-decoder architecture for synthetic data generation, data compression, and anomaly detection. “Their main applications are in medical imaging and molecular generation for drug discovery,” comments Shadid. Trust in gen-AI “Gen-AI is a prediction engine. It generates what’s statistically plausible based on patterns it has seen before,” explains Emanuel Salmona, CEO and co-founder at Nagomi Security. “This makes it good at exploration: generating exploit hypotheses, trying different inputs, and connecting a strange behavior to known vulnerability patterns,” expands Albert Ziegler, head of AI at XBOW. “It’s a tool companies can use to automate creative labor,” adds David Karandish, CEO and founder at Capacity. And because of this, “It is becoming closely embedded into security teams’ workflows, from summarizing incident reports to helping draft response plans,” continues Devvret Rishi, general manager of AI at Rubrik. The Premier Conference on Securing AI in the Enterprise Galina Kho, chief strategy officer at Cyberbay, describes the advent of gen-AI as an efficiency revolution. “It’s not that entirely new capabilities have emerged; it’s that existing ones have become dramatically easier to execute at scale.” The biggest question in the use of AI is whether you can trust an output that is based on probability rather than grounded in known truth. The answer here is 56 shades of ‘No’. “It can be considered both trustworthy and not trustworthy, depending on the intent, the models used and the overall data flow involved,” comments Melissa Ruzzi, senior director of AI at AppOmni. “Gen-AI is not inherently trustworthy,” says Yichuan Zhang, CEO and co-founder of Boltzbit. “It is prone to hallucinations (confident but false statements) and data leakage (reproducing the training content or the context content exactly).” Trever Falconi, director of security and IT operations at HOPPR, explains, “Deploying a gen-AI model is not like installing software. A model trained at one institution will behave differently at another because it learned from a specific set of data and workflows. Move it somewhere new and you’ve introduced a distribution shift: the real-world data it now encounters no longer matches what it was built on, and performance quietly degrades.” Trustworthiness is a complicated question, suggests Aaron Sant-Miller, VP of AI at Booz Allen. “The model is making its best guess at the right response, but it’s not perfect.” Since gen-AI is the bedrock of all AI, there is a trickle-down effect of its strengths and weaknesses into both agentic AI and shadow AI discussed later. Cyber defenders should always be aware that gen-AI can produce errors; but that should not prevent its use. However, as Ruzzi stresses in quoting from Henri Thiel’s 1971 book (Principles of Econometrics), “Models are to be used, not believed. AI should assist analysts, not replace judgment.” The danger is that human nature drives people to believe anything that is said with confidence, and gen-AI can outright lie with confidence. Randell McNair, an adjunct professor at Florida Polytechnic university, explains on LinkedIn, “[Gen-AI] is for all practical purposes a ‘smart’ kid that has been told its whole life it is ‘brilliant’ when in fact, it is just a nearly-8 year old that has never experienced (felt the pain of) a single tangible consequence for being wrong, and has no memory of having ever truly failed someone and had to genuinely regret the shame and embarrassment that should be part of the ‘learning from failure’ process.” Gen-AI use Zhang suggests three areas where gen-AI use offers benefit: SOC productivity (summarizing complex incident logs and writing initial draft reports); secure coding (assisting developers with boilerplate code that adheres to security standards); and vibe coding (assisting non-developers with coding software applications from scratch). “Many enterprises use these models to generate documents, write articles, generate software, or replicate the messages a human would send when orchestrating a larger workflow,” says Sant-Miller. “It helps draft emails, summarize information and reduce manual effort,” adds Travis Springer, president at Sagiss. “Medical imaging teams are piloting vision-language models to surface findings from imaging studies,” says Falconi, “and researchers use synthetic data generation to fill gaps where real patient data is scarce or sensitive to use at scale.” New uses for gen-AI are continually being developed, but within cybersecurity, the most effective use comes from agentic-AI (see below) which

Entities