Transformer Architectures Converge Across Eight Years

Jun Yu Tan published a dataset-driven analysis titled "The Crystallization of Transformer Architectures (2017-2025)" on his personal blog, covering architectural developments between 2017 and 2025. The article documents that the original transformer established the fundamental structure of alternating multi-head self-attention and position-wise feed-forward layers, a pattern that persists across many subsequent models.

Across published model designs and replication implementations, industry patterns show repeated reuse of a small set of architectural primitives: attention mechanisms, residual connections, normalization layers, and feed-forward blocks. Observable variations since 2017 have concentrated on attention sparsity, efficient attention approximations, positional encoding choices, and normalization placement. These trends reflect tradeoffs between computational cost, memory footprint, and ease of scaling rather than purely novel block-level inventions.

For practitioners, the practical effect of this crystallization is twofold. First, convergence on common primitives simplifies transfer of engineering best practices and toolchain support across models. Second, incremental variants-efficient attention kernels, fused operators, and memory reductions-become the primary avenues for performance or cost improvements rather than wholesale architectural rework. Observers comparing models should therefore separate core-block equivalence from peripheral optimizations when attributing gains.

Monitor benchmark suites and reproducibility studies that disaggregate wins from optimizer, data, and architecture; releases of efficient attention kernels in major libraries; and dataset-level analyses that quantify how small design choices affect scaling curves. Jun Yu Tan's article supplies an empirically oriented snapshot of this consolidation, useful as a baseline for such follow-ups.