AI red-teaming
Adversarial testing of an AI system to surface failure modes a benign eval would not catch — bias, misuse, prompt injection, safety breaks.
AI red-teaming applies the security-industry red-team concept to AI systems: a dedicated group attempts to make the model behave badly through crafted inputs, roleplay, tool-chaining, obfuscation and context manipulation. For LLMs the output is a log of (prompt, response, severity, category) tuples used as evidence of safety testing. Red-teaming is required (by name or by implication) under ISO 42001 Annex A, the EU AI Act for high-risk systems, and the 2027 SOC 2 AI points of focus on prompt manipulation. Enterprise buyers increasingly ask for a red-team summary before signing.
Examples
- →A customer-support chatbot is red-teamed before launch: 200 adversarial prompts across prompt injection, jailbreak, PII disclosure, policy bypass — findings logged and triaged.
- →A code-generation agent is red-teamed quarterly, with the test suite versioned in git and the results summary attached to the ISO 42001 evaluation report.
Common mistakes
- ✗One-and-done. Red-teaming is a cadence, not a launch gate — models, prompts and tools change.
- ✗Only testing refusal. The harder tests are correctness under adversarial framing, not whether the model refuses outright.
Authoritative sources
Related terms
- Prompt injection — An attack that smuggles instructions into an LLM via untrusted input (a web page, document, user message) to override the system prompt.
- ISO 42001 Annex A — The 38 AI-specific controls that make ISO 42001 auditable — the AI analogue of ISO 27001 Annex A.
- AI Impact Assessment (AIIA) — A structured review of an AI system's potential harms, benefits and affected parties — required by ISO 42001 Annex A.