Study Finds Transformer Attention Fails Stroop Test

Per the paper published in PNAS Nexus by Suketu Patel and colleagues, researchers administered the classic Stroop task to several leading transformer-based language models, including GPT-4o, GPT-5, Claude 3.5 Sonnet, Claude Opus 4.1, and Gemini 2.5 (PNAS Nexus). The experiment used lists of varying lengths (reported sizes include 5, 10, 20, and 40 items) across congruent, incongruent, and mixed conditions, the paper reports (TechXplore; NeuroscienceNews). The authors report that models achieved high accuracy on short incongruent lists but suffered a length-dependent collapse: for example, GPT-4o fell from 91% accuracy at five words to 57% at ten words and 15% at 40 words, while mixed lists produced near-0% accuracy on the mismatched items, according to the PNAS Nexus paper and corroborating press coverage (PNAS Nexus; TechXplore; NeuroscienceNews).

Per the published study, the Stroop stimuli required models to report ink color while ignoring the lexical color term, creating a conflict between an automatic lexical response and the instructed color-naming response (PNAS Nexus). The paper presents quantitative failure thresholds by model and condition; several outlets reproduced the numeric trajectories and example failure modes (TechXplore; NeuroscienceNews; Heise). The experiments included control trials with neutral tokens (for example, strings of "X") to separate lexical recognition from color-mapping performance, the authors report (PNAS Nexus).

Transformer architectures use attention to weight token relationships, but attention weights do not implement human-like executive control mechanisms for inhibiting automatic responses. Industry reporting frames the observed breakdown as a length-dependent inability of current transformer attention patterns to maintain task-focused inhibition under scaling interference (TechXplore; NeuroscienceNews). For practitioners, this suggests that tasks requiring sustained suppression of a dominant signal across long contexts can expose different failure modes than those visible in short-context benchmarks.

The study amplifies concerns about how transformer-based models handle sustained interference in long-context workflows, a setting increasingly common in retrieval-augmented generation, multi-step agents, and document-level reasoning. Reporting across outlets highlights that the failure is not limited to a single vendor or an older model generation; multiple frontier models and releases were tested and exhibited similar patterns (ScienceDaily; TechXplore; NeuroscienceNews). For applied teams, the result reframes certain robustness questions: high short-context performance does not guarantee stable behavior as input length and pattern complexity increase.

Observers should look for replication on newer model checkpoints and across modalities, follow-up work proposing attention or objective modifications, and benchmarking that includes interference-style tasks at scale. Industry groups and model authors may publish mitigation experiments such as training objectives that explicitly reward sustained inhibition, architectural attention variants, or evaluation suites incorporating classic cognitive paradigms. These developments will indicate whether the failure mode is addressable by training data and objectives or requires structural architectural changes.