All podcasts / Lenny's Podcast / Summary

The coming AI security crisis (and what to do about it) | Sander Schulhoff

2025-12-21 - 92 min - source - Read full transcript

Lenny Rachitsky (host)Sander Schulhoff

Key insights

AI guardrails do not meaningfully improve security because the attack space against a model is effectively infinite, making any percentage-based robustness claim statistically hollow.

Schulhoff argues the number of possible attacks against a model like GPT-5 is comparable to the number of possible prompts, a number he describes as '1 followed by a million zeros.' A vendor claiming 99% attack coverage has still left an astronomically large number of untested, unaddressed attacks; the sample size used to generate that percentage is never statistically significant relative to the true attack space.

guardrails-dont-work

In a joint adaptive-attack study with OpenAI, Google DeepMind, and Anthropic, human attackers broke 100% of state-of-the-art guardrails and defenses within 10 to 30 attempts, while automated attackers needed orders of magnitude more attempts and still succeeded only around 90% of the time.

This contradicts the common assumption that automated red teaming can fully replace human adversaries; Schulhoff says humans remain the most effective adaptive attackers because they learn from what fails and adjust their approach in real time, something automated systems still do less efficiently.

guardrails-dont-work

Guardrails don't dissuade determined attackers, and for many deployments they aren't needed at all, because a read-only chatbot can only let a user harm themselves.

If a malicious user can already get equivalent harmful output from ChatGPT, Claude, or Gemini directly, a guardrail on a company's own narrow chatbot provides no real protection, since a sufficiently motivated user will get through it anyway. Schulhoff's baseline recommendation for simple FAQ-style chatbots with no ability to take actions or access other users' data is to deploy no defense at all and instead just log inputs and outputs for product improvement.

practical-deployment-advice

The real risk shifts from chatbots to agents: any data an agent can access, and any action it can take, a malicious user or a poisoned data source can eventually make it access or take.

Schulhoff frames the core design principle as assuming the AI is a 'malicious god in a box': whatever permissions and data access you grant it, treat those as things an attacker will eventually obtain through the model. He cites the ServiceNow case, where a second-order prompt injection got one agent to recruit more privileged agents to perform unauthorized database writes and send data externally, despite ServiceNow's prompt injection protection being active.

agentic-attack-surface

Splitting a malicious request across multiple separate, individually-innocuous AI interactions can bypass defenses that would catch the request if made in one shot, as demonstrated in a real Claude Code-based cyberattack.

An attacker got around Claude Code's refusal to directly hack a target by first asking a separate instance to reconnaissance the target's backend systems (a legitimate-seeming request on its own), then feeding that information to a second instance and asking how to exploit it. Neither individual request looked malicious, but the composed sequence achieved the attack.

agentic-attack-surface

Indirect prompt injection via untrusted external data (web pages, incoming emails) is a distinct and harder-to-solve problem than eliciting harmful static knowledge like CBRNE information from a model.

Schulhoff explains you can train a model with an absolute rule like 'never explain how to build a bomb,' but you cannot give an email- or browsing-capable agent an equally absolute rule, because the same action (sending an email, reading a webpage) is sometimes exactly what the user wants and sometimes the injected attack vector; the line is contextual rather than categorical, which makes it much harder to train reliably.

agentic-attack-surface

CaMeL-style permission restriction, scoping an agent's allowed actions to only what the user's specific request requires before execution, is the most promising practical defense Schulhoff has seen, but it fails when a legitimate task genuinely needs both read and write access together.

For a request like 'write and send an email,' CaMeL can grant only send-email permission and withhold inbox-read access, so even a successful injection has nothing dangerous to exploit. But for a request like 'read my recent emails and forward any ops items to my head of ops,' both read and write permissions are legitimately required at once, which reopens the same attack surface a malicious email could exploit.

permissions-over-classification

The AI security guardrail and automated red-teaming industry is built more on marketing and unfamiliarity with how AI differs from classical cybersecurity than on working technology, and Schulhoff predicts a market correction within six to twelve months.

He says some vendors privately admit their testing methodology is weak, some guardrail products don't work on non-English languages despite translation being a common and trivial attack pattern, and vendors rarely test their own guardrails with the same automated red-teaming tools they sell to find vulnerabilities in customers' models. He frames the core conceptual gap as 'you can patch a bug, but you can't patch a brain': a software bug can be fixed with high confidence, but a model's underlying vulnerability to being tricked persists even after specific attacks are patched.

ai-security-industry-incentives

Frontier labs underinvest in adversarial robustness relative to capability because commercial incentives favor smarter models over more secure ones, and no lab has made meaningful robustness progress in the last several years despite being the most qualified to.

Schulhoff argues a maximally secure but unintelligent model is commercially worthless, so labs in a capability race prioritize intelligence gains; if the smartest AI researchers in the world working at frontier labs haven't solved adversarial robustness in years of trying, he sees no reason to believe a typical enterprise AI security vendor, without frontier research resources, can succeed where they haven't.

ai-security-industry-incentives

The most valuable emerging role sits at the intersection of classical cybersecurity and AI security, because classical security practitioners often fail to ask 'what if the AI is tricked' and AI researchers often lack systems-security instincts like sandboxing untrusted code execution.

Schulhoff's example: a classical security review of a system where an AI writes and executes code to solve math problems might approve the design without asking what happens if the AI is tricked into writing malicious code; the fix (running that code in an isolated container rather than on the main application server) is a classical security practice, not an AI-specific one, but it requires someone who thought to apply the 'malicious AI' threat model in the first place.

permissions-over-classification

Publishing new jailbreak or attack research is now net-negative for the field, per the argument in 'Don't write that jailbreak paper,' because it mainly arms attackers rather than advancing defense.

Schulhoff endorses this view from an advisor: the field already has overwhelming, repeatedly-replicated proof that models can be broken in countless ways, so further offensive research papers add little defensive value while lowering the bar for less-skilled attackers to find new techniques. He contrasts this with 'human in the loop' proposals to escalate flagged actions to a person, which he calls useful today but ultimately misaligned with what the market and frontier labs will actually build, since users and companies want agents that complete tasks autonomously, not agents that keep interrupting for approval.

practical-deployment-advice

Media referenced

Don't write that jailbreak paper - article - A blog post Schulhoff cites approvingly; its argument (echoed by one of his advisors) is that publishing new offensive attacks on models no longer meaningfully advances defense since everyone already knows models can be broken, and it mainly hands attackers new techniques.
CaMeL (Google research) - paper - A permissions-based framework Schulhoff calls the most promising practical mitigation: it restricts what actions an agent is allowed to take based on what the user actually asked for, before the agent runs, rather than trying to classify malicious content after the fact like guardrails do.
MITRE report on the MathGPT incident - article - Referenced as documentation of an early (GPT-3 era) prompt injection attack in which a math-solving website was tricked into exfiltrating its own OpenAI API key.

Companies

OpenAI - Referenced throughout as a frontier lab whose models (GPT-3 through GPT-5) are used as both attack targets and, via ChatGPT, in real-world incidents like the Las Vegas Cybertruck bombing, where the attacker is believed to have used a chat model to help plan the device.
Anthropic - Cited for Claude, Claude Code, and its constitutional classifiers, which Schulhoff says have made it meaningfully harder to extract CBRNE (chemical, biological, radiological, nuclear, explosives) information from Claude models, though not for indirect prompt injection against agents.
Google / Google DeepMind - Co-authored a major adaptive-attack research paper with Schulhoff's team and OpenAI/Anthropic; also the origin of the CaMeL permissioning framework discussed as the most promising practical defense.
ServiceNow - Central case study: a researcher used a second-order prompt injection to get ServiceNow's Assist AI agent to recruit other, more privileged agents within the platform to perform unauthorized database create/read/update/delete actions and send data to external emails, despite the platform's prompt injection protection being enabled.
Comet (Perplexity's AI browser) - Cited for an incident where a malicious chunk of text on a web page tricked the browser's AI into exfiltrating the user's account and personal data simply by visiting the page; Schulhoff notes this class of vulnerability likely applies to OpenAI's Atlas and other AI browsers too.
Trustible - An AI governance and compliance company Schulhoff shouts out as doing genuinely useful work helping enterprises track fast-evolving AI legislation, though he notes it is not strictly an 'AI security' company in his narrower sense.
Repello - An AI security vendor Schulhoff was initially skeptical of (it sells automated red teaming and guardrails, which he argues don't work) but now credits for a useful shadow-AI discovery product that scans a company's internal systems and surfaces AI deployments the security team didn't know it had.
remotely.io - Cited as the first publicly documented prompt injection: a promotional Twitter chatbot was tricked into posting threats against the president and hate speech; the company is believed to no longer be in business.
Datadog - Episode sponsor; pitched for its product analytics, session replay, and feature-flagging platform (now home to the Eppo experimentation product).
Metronome - Episode sponsor; usage-based billing infrastructure used by OpenAI and Anthropic, pitched to eliminate manual invoicing glue code.
GoFundMe - Episode sponsor; pitched for its new GoFundMe Giving Funds donor-advised fund product for year-end charitable giving.

Techniques and frameworks

AI guardrails (input/output classifier models) - A defense pattern where a separate LLM sits in front of and behind a deployed model, flagging malicious inputs before they reach the model and malicious outputs before they reach the user. Schulhoff argues, based on years of red-teaming competitions, that these are broadly ineffective and create false confidence.
Automated AI red teaming - Tools, usually other LLMs, that automatically generate adversarial prompts to try to elicit malicious output from a target model. Schulhoff argues this category 'works too well': it will always find something against any current transformer-based model, so its findings aren't diagnostic of a specific product being unusually insecure.
Adaptive evaluation vs. static evaluation - The rigorous way to measure a defense's real robustness: pit it against an attacker (human or automated) that learns and adapts over successive attempts, rather than a fixed one-time dataset of known-bad prompts. Schulhoff says most vendors report static numbers, which are close to meaningless because the attack space is effectively infinite.
Attack success rate (ASR) / adversarial robustness - The industry's standard metric for how often attacks get through a defense (e.g., a defense with 1% ASR is called '99% adversarially robust'). Schulhoff argues the framing is misleading because the space of possible prompts against a model like GPT-5 is astronomically large, so even a 99% block rate leaves a practically infinite number of successful attacks untested and unaddressed.
Prompt-based defenses - Adding instructions directly into a system prompt telling the model to ignore malicious user attempts. Schulhoff calls this the weakest defense category of all, established as ineffective since early 2023 research including the original HackAPrompt paper.
CaMeL-style capability restriction - Restricting an agent's available actions ahead of time based on what the user's request actually requires (e.g., granting only 'send email' permission for a 'write an email' request, withholding 'read inbox'), so a prompt injection has no dangerous permission to exploit even if it succeeds. Breaks down when a legitimate task genuinely requires both read and write permissions together.
Adversarial pre-training - A theoretical, largely untried mitigation Schulhoff discusses: exposing models to adversarial attacks earlier in pretraining, before capabilities and personality solidify, on the theory that a model 'raised' under adversarial pressure develops more durable resistance, analogous to how early hardship builds resilience in people.

Summary

Sander Schulhoff, an AI security researcher who wrote the first public guide to prompt engineering, founded the HackAPrompt red-teaming competition, and now runs a Maven course on AI red teaming, returns to Lenny's Podcast to argue that the entire commercial AI guardrail industry does not work. His central claim, repeated throughout the episode, is that guardrails (LLM-based classifiers that watch inputs and outputs for malicious content) provide no meaningful defense: the space of possible attack prompts against a model like GPT-5 is so large that any percentage-based "we catch 99% of attacks" claim is statistically hollow, since the untested remainder is still effectively infinite. He backs this with results from a joint adaptive-attack study run with OpenAI, Google DeepMind, and Anthropic, where human attackers broke every state-of-the-art defense in 10 to 30 attempts, while automated attackers needed far more attempts to reach a lower success rate. His summary framing is "you can patch a bug, but you can't patch a brain": a software vulnerability can be fixed with high confidence, but a model's susceptibility to being tricked persists after any individual patch.

Schulhoff walks through the anatomy of the problem using real incidents: the first publicly documented prompt injection against a promotional Twitter chatbot in the early GPT era, the MathGPT exploit that exfiltrated an OpenAI API key, and a recent case where attackers split a malicious request across two separate Claude Code sessions (reconnaissance in one, exploitation in the next) to bypass refusals that would have triggered had the full request been made at once. The most detailed case study is ServiceNow's Assist AI, where a researcher used a second-order prompt injection to get one agent to recruit other, more privileged agents within the platform into performing unauthorized database writes and sending data externally, despite ServiceNow's own prompt injection protection being active. He also cites Perplexity's Comet browser, where a malicious chunk of text on a visited web page exfiltrated a user's account data, and notes the same vulnerability class likely applies to other AI browsers. His overarching frame is to treat any deployed AI system as "a malicious god in a box": whatever data and actions it has access to, assume an attacker will eventually get it to use.

On what to actually do, Schulhoff offers a graduated, mostly deflationary set of recommendations. For a simple FAQ chatbot with no ability to take actions or touch other users' data, he recommends deploying no defense at all, since a determined attacker can get the same harmful output from ChatGPT or Claude directly, and instead just logging inputs and outputs for product improvement. Where systems become agentic (able to send emails, browse the web, or write to databases), the real work is classical security discipline: strict data and action permissioning, so a compromised agent can only do what the user themselves is authorized to do. The most promising technical defense he names is Google's CaMeL framework, which restricts an agent's available actions ahead of time to only what the specific user request requires (granting send-email permission but withholding inbox-read access for a "send an email" request, for instance), so even a successful injection has no dangerous capability to exploit. He's clear this breaks down whenever a legitimate task genuinely needs both read and write access at once, which is a large share of real agentic use cases.

Schulhoff is candid that the guardrail and automated red-teaming vendor space is more marketing than substance: some vendors privately tell him their testing methodology is weak or fabricated, some guardrails don't work on non-English languages despite translation being a trivial and common attack, and vendors rarely test their own guardrails with the same red-teaming tools they sell customers. He predicts a market correction within six to twelve months as enterprises realize revenue for these products isn't materializing and free open-source alternatives are often better. He's more positive about companies working adjacent to pure guardrails: Trustible for AI governance and compliance tracking amid fast-changing legislation, and Repello for a shadow-AI discovery product that surfaces AI systems running inside a company that its own security team didn't know about. He argues the most valuable emerging role sits at the intersection of classical cybersecurity and AI security, since classical security reviewers often don't think to ask "what if the AI is tricked," while AI researchers often lack systems-security instincts like sandboxing untrusted code execution.

The conversation closes on why frontier labs haven't solved this: Schulhoff argues commercial incentives favor capability over security, since a maximally secure but unintelligent model is worthless in a competitive race, and notes that if the most qualified researchers in the world haven't cracked adversarial robustness in years of trying, he doesn't expect a typical enterprise vendor to. He points to Anthropic's constitutional classifiers as genuine progress on making it harder to elicit static harmful knowledge (like CBRNE information) from Claude, but says indirect prompt injection against agents remains far less tractable because the line between a legitimate action and an attack is contextual, not categorical, unlike a flat "never explain how to build a bomb" rule. He also endorses the argument from the essay "Don't write that jailbreak paper" that further offensive attack research is now net-negative for the field, since it mostly arms less-skilled attackers rather than advancing defense, and dismisses "human in the loop" escalation proposals as a stopgap that doesn't match where the market and frontier labs are actually heading, toward agents that complete tasks autonomously without interruption.

Notable Quotes

"AI guardrails do not work. I'm going to say that one more time. Guardrails do not work. If someone is determined enough to trick GPT-5, they're going to deal with that guardrail. No problem." - Sander Schulhoff

"You can patch a bug, but you can't patch a brain. If you find some bug in your software and you patch it, you can be 99.99% sure that bug is solved. Try to do that in your AI system, you can be 99.99% sure that the problem is still there." - Sander Schulhoff

"The only reason there hasn't been a massive attack yet is how early the adoption is, not because it's secured." - Sander Schulhoff, quoting Alex Kamaroski's assessment of AI security

"Not only do you have a god in the box, but that God is angry. That God's malicious. That God wants to hurt you. Can we control that malicious AI and make it useful to us and make sure nothing bad happens?" - Sander Schulhoff

"If the smartest AI researchers in the world can't solve this problem, why do you think some random enterprise who doesn't even employ AI researchers can? It just doesn't add up." - Sander Schulhoff