AI Red Teaming: How We Attack AI Systems Before Someone Else Does

AI red teaming is structured adversarial testing of an AI system, where a dedicated team deliberately tries to make the model misbehave: leak training data or secrets, ignore its own guardrails, follow malicious instructions hidden in content it reads, or abuse the tools it can call. The goal is the same as any red team engagement. Find the ways the system fails under attack before a real adversary does, then prove each finding is real and hand the owner a fix. The difference is the target. Instead of a network or a web app, you’re attacking a probabilistic model whose behavior shifts with the input and can’t be fully predicted from the code.

This is the work we do most often now, and it’s the most natural fit for an offensive team. The skill that matters in AI red teaming is the same one that matters in pen testing: thinking like an attacker who doesn’t care about your intended use case. The model was trained to be helpful. We get paid to make that helpfulness work against you.

A traditional pentest targets deterministic systems. The same request returns the same response, vulnerabilities map to known classes like SQL injection or broken access control, and a finding either reproduces or it doesn’t. AI red teaming breaks most of those assumptions.

The target is non-deterministic. Send the same jailbreak twice and the model might refuse once and comply the next time. A finding isn’t a clean true-or-false; it’s a success rate. The attack surface is the language itself, so the input space is effectively infinite and you can’t enumerate it the way you’d enumerate parameters in an API. And the most damaging issues often aren’t memory-safety bugs but behavioral ones: the model does exactly what an attacker asked, just not what its owner wanted.

So the methodology adapts. We still scope, threat-model, attack, and report. But we measure findings probabilistically, lean heavily on creative manual testing alongside automation, and treat the system around the model, like retrieval pipelines, tools, and downstream actions, as part of the target rather than background. The traditional discipline still matters, which is why AI red teaming sits inside our broader penetration testing and vulnerability management practice rather than off in its own silo.

Every engagement is scoped to the system in front of us, but the categories below cover most of what we go after. They line up closely with the OWASP Top 10 for LLM Applications, which is a good shared vocabulary for talking about these risks with your engineers.

The signature attack against language models, and OWASP’s LLM01. Direct injection is the user telling the model to ignore its instructions. The nastier variant is indirect: malicious instructions planted in content the model later reads, such as a web page, a PDF, an email, a support ticket. The user never sees it, but the model treats it as input and acts on it. For any system that pulls in outside content, indirect injection is the first thing we reach for. We go deep on detecting it in production under prompt injection monitoring.

Getting the model to produce output its safety layer is supposed to block. Roleplay framings, hypothetical wrappers, encoded payloads, language switching, splitting a request across several turns so no single message looks bad. We’re not collecting party tricks. We’re testing whether the guardrail is a real boundary or a thin filter that holds against the obvious phrasing and folds against a determined one. Most bolt-on filters fall into the second group.

Pulling out things the model should never reveal: its system prompt, secrets pasted into context, fragments of training data, or other users’ data sitting in a shared retrieval store. In RAG-backed apps this is often the highest-impact finding, because the model has live access to a knowledge base and the right phrasing convinces it to return records that the asking user was never authorized to see. The fixes here are mostly architectural, which is why we treat them as part of securing LLM applications rather than something you can prompt your way out of.

When the model can call tools, query databases, hit APIs, or trigger actions, the prompt stops being just text and becomes a way to make the system do things. OWASP labels this excessive agency, and it’s where a clever prompt turns into a real-world consequence: a refund issued, a record deleted, mail sent. We test whether a low-privilege user, or hidden text in some document, can drive the agent into actions it should never take. This overlaps heavily with how we approach controlling AI agents.

Depending on scope, we probe for outputs that are discriminatory, harmful, or wildly off-policy for the deployment, and for the conditions that reliably trigger them. For a regulated client this is often where compliance pressure is heaviest, since a model that produces unfair or unsafe output in front of customers is both a brand problem and, increasingly, a legal one.

Creativity matters in this work, but it sits on top of a repeatable process. Skip the structure and you get a pile of cool jailbreaks with no way to measure coverage or track whether anything got fixed.

We start by mapping the system. What’s the model, what can it read, what tools and data can it reach, who talks to it, and what’s the worst realistic outcome if it misbehaves? That last question drives everything. Red-teaming a marketing chatbot with no tools is a different job from red-teaming an agent wired into your billing system, and the scope reflects that.

We build the threat model on MITRE ATLAS, a knowledge base of real adversary tactics and techniques against AI systems, modeled on the ATT&CK framework most security teams already know. ATLAS gives us a shared map of how these systems get attacked in the wild, from reconnaissance through model access, evasion, and exfiltration, so coverage is driven by documented attacker behavior rather than whatever we happen to think of on the day.

Then we attack, on two tracks at once. Automated tooling fires large batches of known attack patterns and probes for regressions, which is how you measure success rates and cover ground at scale. Manual testing is where the real findings come from: a human chaining context, reading how the model reacts, and adapting in ways no scripted suite will. Automation tells you how often a known attack lands. A skilled operator finds the attack nobody scripted yet. You need both, and the manual side is where experience separates a real red team from a tool run.

A finding you can’t reproduce is an anecdote. Because models are non-deterministic, we report every issue with the exact input, the conditions, an observed success rate across repeated attempts, the impact, and a concrete fix. Your engineers should be able to paste in our payload and watch it work, then confirm it’s dead after they patch. That’s the bar. Vague writeups help nobody.

Most clients come to us for security, but the same work answers a fast-growing list of compliance questions, so it’s worth knowing how findings line up with the frameworks.

The NIST AI Risk Management Framework is built around four functions: Govern, Map, Measure, and Manage. Red teaming feeds the Measure function directly. It’s how you actually test for the risks you mapped, instead of asserting on paper that the system is safe. NIST’s generative AI profile calls out adversarial testing by name, which makes a red team engagement a clean way to show you’ve measured real risk.

The EU AI Act adds legal weight. For high-risk and general-purpose AI systems it expects adversarial testing and ongoing risk management, and providers of the most capable general-purpose models are expected to perform and document adversarial testing as part of their obligations. A red team report with reproducible findings, severities, and fixes is exactly the kind of evidence that maps onto those requirements. We don’t sell compliance theater, but the artifacts this work produces tend to be what auditors and regulators want to see.

Earlier than most teams do. The usual pattern is shipping an AI feature, watching it touch real users and real data, and only then asking whether it’s safe. By that point an attacker can ask the same question for free, in production.

Red-team before launch for anything customer-facing or anything wired to tools and sensitive data. Re-test after meaningful changes, because a new system prompt, an added tool, or a swapped base model can quietly reopen something you’d closed. And treat it as continuous for systems that keep evolving. The model and its guardrails drift, attacker techniques move fast, and a clean report from six months ago says little about today. When something does slip through in production, that’s where our incident response and threat intelligence team picks it up. This all sits inside the wider program we describe under AI security.

If you’re putting an AI system in front of customers or hooking it up to anything that matters, get someone who attacks these systems to test it first. That’s the work we do. See how we run offensive testing, or talk to our team about scoping an AI red team engagement before launch rather than after the incident.