LLMs don’t actually know who’s talking. New research from Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell — presented as an ICML 2026 poster — traces prompt injection to a single root cause: models assign roles based on how text sounds, not the <user> or <system> tags wrapping it.
The core findings:
- Style beats structure. Models treat text that reads like a system prompt as privileged, even when it sits inside
<user>tags. The role label is decoration; the writing style is the signal. - Role probes expose the gap. The researchers designed novel probes to measure how models internally identify “who is speaking.” The probes confirm that untrusted input mimicking
<think>or<assistant>style gets elevated to those roles. - Jailbreaks follow directly. Appending text styled like a model’s internal reasoning blocks to a harmful request can bypass safety training. The paper demonstrates this with concrete examples — a user request followed by text that mirrors
<think>-style reasoning is enough to flip the model’s behavior.
This isn’t a new attack surface — prompt injection has been a known problem since GPT-3. What’s new is the mechanistic explanation. Previous work treated injection as a training failure or an alignment gap. This paper argues it’s a perception failure: the model literally cannot distinguish privileged from unprivileged text when they share stylistic features.
NOTE
Simon Willison highlights that the paper comes with a readable blog-style writeup alongside the formal academic version — a format he wishes every paper adopted.
The practical implication for anyone building agents or tool-use pipelines: role tags alone are not a security boundary. If your agent reads untrusted content — web pages, emails, user uploads — and that content can mimic the style of your system prompt or reasoning trace, the model may treat it as authoritative.
No fix ships with this paper. The research confirms the problem is structural to how current LLMs process text, not a gap that safety training patches. Anyone shipping agentic systems should treat all ingested text as potentially role-confusing — OWASP’s prompt injection guidance remains the baseline.