Language Models are Few-Shot Learners

Abstract summary

Introduces GPT-3, a 175B-parameter autoregressive language model, and demonstrates that scaling up a Transformer LM produces emergent few-shot in-context learning capability. Shows that a single model can perform many NLP tasks competitively without fine-tuning, simply by being shown a few examples in the prompt. Documents capability and scaling behaviors that defined the LLM era.

Our summary in our own words — see the canonical source links below for the original abstract.

Why we cite this paper

GPT-3 is the catalyst paper for the modern LLM era. The few-shot in-context learning paradigm became the dominant interaction pattern for AI products, and the 175B-parameter scale established the design space that frontier labs (Anthropic, OpenAI, Mistral, Cohere, Hugging Face) operate within. The applied-AI category we track at /signal/[anthropic/openai/etc.] would not exist without the GPT-3 demonstration.

Key findings

• 1Few-shot learning emerges as a capability of sufficiently-scaled language models, without explicit task-specific fine-tuning.
• 2Performance scales smoothly with model size, training data, and compute — establishing the scaling-law trend later formalized by Kaplan et al.
• 3A single LLM can perform translation, question-answering, summarization, arithmetic, and code generation competitively given few-shot prompting.
• 4The 175B-parameter scale established a new design space that frontier labs compete within.

Canonical sources

Related glossary terms

Frequently Asked Questions

Other research papers