Paper Shows LLM Role Confusion Enables Prompt Injection

Research accepted to ICML 2026 - "Prompt Injection as Role Confusion" by Charles Ye, Jasmine Cui, and Dylan Hadfield-Menell - presents a mechanistic theory of why LLMs fail against prompt injection. Bruce Schneier's security blog flagged the paper on June 25; security commentator Simon Willison published a detailed breakdown on June 22. The core claim: LLMs cannot reliably perceive roles from structural tags because they have learned to infer role identity from writing style instead.

The team developed "role probes," linear classifiers trained on model activations to measure what role a model internally assigns to each token. In controlled experiments with gpt-oss-20b, stripping all role tags from a conversation leaves internal role perception nearly unchanged - the model still assigns high "CoTness" to text that sounds like reasoning, regardless of tag. When reasoning-style text is wrapped in user tags, the stylistic signal overrides the official label. Style actively overrides the true role.

The team built an attack called CoT Forgery: injecting fake reasoning blocks written in the model's chain-of-thought style inside user messages or tool outputs. Because LLMs treat reasoning-like text as their own already-reached conclusions, they act on the injected logic without scrutiny. On a standard jailbreak benchmark, CoT Forgery raises attack success from near-zero to roughly 60% and transfers across multiple frontier models. The CoTness score of injected text - measured from activations before the model responds - predicts whether the attack will succeed.

The same mechanism explains tool-injection attacks. Prepending "User: " to a malicious command buried in tool output raises the model's internal Userness score for that text, increasing the probability the model treats it as a real user instruction. Testing 212 phrasing variations confirmed that higher Userness scores predict higher attack execution rates.

The clearest evidence that style drives role perception: removing specific words and syntax associated with the model's reasoning (destyling) drops attack success from 61% to 10%. The change is imperceptible to human readers but completely changes the model's internal perception of the text's role.

The authors frame the finding as structural. Role tags were originally a formatting convention that became the trust architecture of deployed LLMs. Because the learned boundary is stylistic and continuous rather than discrete, any text that sounds like a privileged role can exploit that trust. The paper warns that prompt injection defense will remain brittle without genuine role perception, and flags "subconscious steering" - using innocuous-seeming text to subtly shift model persona or recommendations at commercial scale - as a potentially larger threat than current cybersecurity-focused injection attacks. A May 2026 paper found Opus 4.5 and GPT-5.4 still failing 11% and 25% of the time, respectively, against automated attacks; human red-teamers achieve near-100% success rates against frontier models.