Attention Is All You Need

Abstract summary

Introduces the Transformer architecture: a sequence-to-sequence model based entirely on attention mechanisms, dispensing with recurrence and convolutions. Demonstrates state-of-the-art results on English-to-German and English-to-French translation benchmarks with significantly less training time than the prior recurrent encoder-decoder models. The architecture's self-attention mechanism allows parallel processing of sequence elements and scales effectively with model size and data.

Our summary in our own words — see the canonical source links below for the original abstract.

Why we cite this paper

The Transformer is the architectural foundation of every modern frontier LLM — GPT, Claude, Gemini, Mistral, Llama, Qwen, DeepSeek. Our engineering-acceleration tracking of AI infrastructure and agentic AI categories operates on a substrate that did not exist before this paper. We cite it as the foundational reference for the AI-native engineering surface our /signal corpus covers.

Key findings

• 1Attention-only architectures match or exceed recurrent models on sequence-to-sequence tasks while training significantly faster.
• 2Self-attention scales effectively with model size, enabling the parameter regimes (1B–1T+) that define modern LLMs.
• 3Position encoding via learned or sinusoidal embeddings allows attention models to handle sequence order without recurrence.
• 4Multi-head attention captures different relationship types in parallel — a design choice that proved central to LLM expressiveness.

Canonical sources

Related glossary terms

Frequently Asked Questions

Other research papers