Defense and governance
AI red teaming
AI red teaming is the practice of deliberately attacking an AI system, with prompt injections, jailbreaks, and adversarial inputs, to find weaknesses before real attackers do.
Definition
What is AI red teaming?
AI red teaming stress-tests a model or agent by simulating an adversary. Testers run a corpus of prompt injections, jailbreaks, exfiltration attempts, and edge cases against the system and measure what gets through. For agents, the target is the action layer: can an attacker make the agent call a dangerous tool or leak data?
Done continuously rather than once, red teaming becomes a measurement of defense over time. It quantifies detection and false-positive rates, catches regressions when a model or policy changes, and produces the evidence that a system was tested, which is increasingly expected by security reviews and AI governance regimes.
FAQ
Common questions.
How is AI red teaming different from a one-time penetration test?
A pen test is a point-in-time assessment. AI red teaming is most valuable run continuously, because models and policies change; ongoing testing catches regressions and tracks detection and false-positive rates over time.
Related terms
Keep reading.
Govern the actions, not just the vocabulary
AxioRank scores every tool call your agents make for leaked secrets, PII, destructive operations, and prompt injection, checks it against your policy, and proves it in a tamper-evident audit log. Start free, no card.