What attackers do. This is a technique from MITRE ATLAS, the companion knowledge base for attacks on AI systems. Attackers create adversarial data, input subtly modified so a machine learning model misclassifies it or behaves in a way the attacker chooses, while it still looks normal to people. It can be used to evade detection models or manipulate decisions.
How to defend. Test models against adversarial examples during development, use adversarial training and input preprocessing where suitable, and avoid relying on a single model for security-critical decisions. Limit the detail returned by model APIs, such as confidence scores, and rate-limit queries.
Signs to look for. Look for high volumes of slightly varied queries, inputs near decision boundaries and a drop in model accuracy on real-world data. A common gap is that model inputs and outputs are not logged, so attacks cannot be reconstructed.