The history of large language models How prediction became a general interface.

Shannon modeled a source as a process that selects messages with known probabilities. Entropy measured average uncertainty, while conditional probabilities captured how the statistical structure of a sequence reduced that uncertainty.

Cross-entropy, bits per character, and next-token loss descend from this framework. It measures predictive uncertainty without claiming that prediction alone captures meaning.

Shannon asked human participants to guess successive letters from an English passage while revealing more context after errors. Their guesses supplied empirical bounds on the entropy and redundancy of printed English.

The paper tied a language statistic to next-symbol prediction with context. That is recognizably close to the training task of an autoregressive language model, but the predictor here was a person.

Francis and Kučera assembled 500 samples of edited American English first printed in 1961, totaling about one million words. Its documented categories and machine-readable form let researchers ask repeatable questions of the same body of text.

Language modeling depends on a corpus before it depends on an architecture. Brown showed the value of explicit sampling, shared data, and corpus documentation, concerns that remain central at web scale.

Bahl, Jelinek, and Mercer framed continuous speech recognition as choosing the word sequence with the highest probability given an acoustic signal. Their system separated acoustic evidence from a probabilistic language generator and addressed sparse-data estimation.

The paper represents the statistical turn that made learned language probabilities central to deployed speech systems. N-gram models would remain industrial workhorses long after neural alternatives appeared.

Katz discounted observed n-gram counts, reserved probability mass for unseen events, and recursively backed off to shorter contexts. Good-Turing estimates determined how much probability to redistribute.

A finite corpus contains only a fraction of possible phrases. Katz backoff made sparse count models usable by assigning nonzero probabilities beyond the exact sequences in the data.

Kneser and Ney changed the lower-order distribution used during backoff. Instead of rewarding only frequent words, it rewarded words that appeared after many different histories, a better clue that they could continue an unseen phrase.

The method corrected a subtle failure in ordinary n-gram smoothing and became the core of a leading statistical language-model recipe. It is a reminder that data sparsity was a modeling problem, not merely a storage problem.

Hochreiter and Schmidhuber introduced gated memory cells with a protected error path. The design addressed the difficulty of learning dependencies across long intervals with ordinary recurrent backpropagation.

LSTMs made recurrent networks far more useful for language, speech, and translation. They became the main neural sequence architecture of the 2010s before attention-based models displaced them.

Bengio and colleagues trained a feed-forward network to learn a vector for each word and a probability distribution for the next word at the same time. Similar words could share statistical strength even when an exact sequence had never appeared in training.

The paper joined embeddings, neural probability estimation, and next-word prediction in one scalable recipe. That combination is a direct ancestor of today's pretrained language models.

Mikolov and colleagues applied a simple recurrent network to word prediction and speech recognition. Unlike a fixed-order n-gram, its recurrent state could in principle use context of arbitrary length.

The reported perplexity and word-error improvements helped move recurrent language models from an appealing idea toward a practical competitor to mature n-gram systems.

The word2vec paper introduced efficient continuous-bag-of-words and skip-gram architectures for learning word vectors from very large corpora. The learned coordinates exposed regularities in syntax and semantics without a labeled dictionary.

Embeddings became standard reusable artifacts and made distributional structure visible to practitioners. Later language models would replace one fixed vector per word with a different representation for each context.

Bahdanau, Cho, and Bengio let a translation decoder assign a soft weight to each source position for every output word. The model learned an alignment while it learned to translate, rather than compressing the whole source into one fixed vector.

Attention provided a content-dependent route through a sequence and a visible alignment map. The idea later became the central computation of the Transformer.

Sutskever, Vinyals, and Le used one multilayer LSTM to encode a source sentence into a fixed-length vector and another to decode the translation. Reversing the source word order made optimization easier by shortening relevant dependency paths.

The same trainable interface could map variable-length inputs to variable-length outputs. It removed much task-specific translation machinery, while its fixed-vector bottleneck exposed the problem that attention would address.

Sennrich, Haddow, and Birch represented rare words as learned sequences of frequent character units. Their byte-pair encoding adaptation let a neural translator generate unseen names and compounds without assigning every word its own vocabulary entry.

Subword tokenization became part of the basic LLM stack. It balances sequence length against vocabulary size, though its splits can still be awkward across languages, code, and numbers.

The Transformer replaced recurrent sequence processing with stacked self-attention and feed-forward blocks. Every token could directly combine information from every other position, while training could process positions in parallel.

The architecture shortened the path between distant tokens and scaled efficiently on modern accelerators. Decoder-only Transformers later became the main foundation for autoregressive large language models.

ULMFiT started with a general-domain language model, adapted it to the target text, then fine-tuned a classifier with schedules designed to preserve useful pretrained knowledge. The same procedure worked across several text-classification datasets.

The paper made the order of adaptation explicit and treated fine-tuning as an engineering discipline. Its gradual unfreezing and learning-rate schedules anticipated the post-training care required by larger models.

ELMo formed a word representation from the internal states of a bidirectional language model. The same written word therefore received different vectors in different sentences, and downstream models learned how to mix the pretrained layers.

Contextual representations displaced the idea that one word should always have one vector. ELMo also showed that a general language-model objective could transfer across several labeled NLP tasks.

OpenAI pretrained a decoder-style Transformer by predicting the next token in a large unlabeled corpus, then fine-tuned it on labeled tasks. Task inputs were converted into token sequences that the same model could process with few architectural changes.

The work joined autoregressive Transformer pretraining with supervised adaptation. It established the GPT line's core premise: next-token prediction could produce a reusable base model.

BERT trained a Transformer encoder to reconstruct masked tokens using context on both sides, alongside a next-sentence objective. Fine-tuning the same pretrained model with a small task head produced strong results across question answering and language understanding benchmarks.

BERT made pretrain-then-fine-tune the standard NLP workflow and triggered a broad ecosystem of encoder models. Its masked objective favored representation and understanding tasks rather than free-form left-to-right generation.

GPT-2 scaled a decoder-only Transformer to 1.5 billion parameters and trained it on WebText. Without gradient updates, the model could attempt translation, summarization, question answering, and reading comprehension when those tasks were expressed as text continuations.

The work shifted attention from a separately fine-tuned model for each task toward behavior elicited through context. It also made staged model release and misuse analysis part of the public history of language models.

T5 represented classification, translation, summarization, and question answering with the same input-text to output-text interface. The study systematically compared objectives, architectures, data, transfer methods, and scale using the cleaned C4 web corpus.

A common text interface simplified multitask training and made task prefixes an ordinary control mechanism. The paper also documented how corpus construction, training choices, and model size affected transfer.

Across a wide range of model sizes, dataset sizes, and training compute, the authors found approximate power-law relationships with cross-entropy loss. The fitted curves suggested how to allocate a fixed compute budget among those quantities.

Scaling moved from an intuition toward a measurable planning tool. The laws did not guarantee useful behavior, but they made the pretraining loss of a larger run less of a blind guess.

RAG combined a pretrained retriever over Wikipedia with a pretrained sequence-to-sequence generator. Retrieved passages acted as a non-parametric memory, and the system could be trained end to end for knowledge-intensive tasks.

Retrieval gave a language model evidence beyond what its weights stored and created a path to updating knowledge without retraining the entire generator. It also introduced new failure points in search, ranking, and evidence use.

GPT-3 scaled an autoregressive Transformer to 175 billion parameters and evaluated it in zero-shot, one-shot, and few-shot settings. Examples in the context changed its behavior without changing its weights.

In-context learning made the prompt a temporary programming surface. The model's breadth was striking, but its failures in consistency, factuality, bias, and sample efficiency were also plainly documented.

Annotators compared pairs of model-written summaries, a reward model learned those preferences, and reinforcement learning optimized the summarizer against that learned reward. Human evaluators preferred the resulting summaries to those from much larger supervised models.

The experiment showed that human comparisons could shape generation where a simple reference-text loss was inadequate. The same broad pipeline later became central to instruction-following chat models.

A learned router sent each token to one feed-forward expert instead of running every expert. This sparse activation let the model hold far more parameters while keeping the computation per token closer to a dense baseline.

Mixture-of-experts models separated total capacity from active compute. The design later became one route to scaling large language models, although routing balance and serving efficiency remained hard engineering problems.

LoRA froze a pretrained model and learned small low-rank matrices that modified selected weight layers. Each adapted task needed only a fraction of the trainable parameters and storage required by full fine-tuning.

The method made customized model variants cheaper to train, store, and exchange. It also separated the base weights from an adaptation, a pattern widely used for domain, style, and instruction tuning.

OpenAI fine-tuned GPT models on publicly available code and evaluated whether generated Python programs passed unit tests. The accompanying HumanEval benchmark measured functional correctness rather than surface similarity to a reference answer.

Codex showed that domain-specific continued training could turn a general language model into a useful code model. Executable tests also exposed a firmer evaluation signal than prose similarity, while pass rates still depended strongly on sampling many candidates.

FLAN fine-tuned a pretrained language model on natural-language instructions drawn from dozens of task datasets. It then evaluated the model on task clusters withheld from the instruction-tuning mixture.

Instruction tuning converted a collection of labeled tasks into practice at reading task descriptions. It became a core post-training step for making pretrained models easier to direct through ordinary language.

Few-shot prompts paired answers with written intermediate steps. On arithmetic, symbolic, and commonsense tasks, sufficiently large language models often produced their own intermediate text and reached more correct final answers than with answer-only examples.

The prompt could influence the answer and the amount of computation expressed before it. This helped establish reasoning traces and test-time prompting as major areas of LLM research.

The InstructGPT pipeline used human-written demonstrations for supervised fine-tuning, human rankings to train a reward model, and reinforcement learning to optimize the policy. Evaluators preferred outputs from the 1.3-billion-parameter aligned model to outputs from the 175-billion-parameter GPT-3 base model on the tested prompt distribution.

Post-training became as important to the user experience as pretraining. The work also documented alignment taxes, reward-model limitations, and behavioral failures rather than treating preference scores as a complete measure of safety.

Using hundreds of training runs, the authors estimated that compute-optimal model size and token count should grow together. Their 70-billion-parameter Chinchilla model used four times as much data as the larger Gopher model and outperformed it on many reported tasks at the same training compute.

The result redirected scaling toward more training tokens rather than parameter count alone. It also changed the economics of deployment because a smaller, better-trained model could be cheaper to serve.

HELM evaluated language models through a matrix of scenarios and metrics instead of one aggregate score. Its taxonomy included accuracy alongside calibration, robustness, fairness, bias, toxicity, and efficiency, with prompts and raw results recorded for inspection.

As general models crossed task boundaries, evaluation had to expose tradeoffs and missing coverage. HELM made reproducibility, prompt sensitivity, and transparent reporting part of the infrastructure rather than an appendix to a launch claim.

ChatGPT adapted a model from the InstructGPT family for multi-turn dialogue and released it as a free research preview. The chat interface carried prior turns forward, while users supplied a large stream of real conversations and feedback.

The release made instruction-following language models legible to a mass audience and established conversation as their default public interface. It also exposed hallucination, prompt sensitivity, and confident error at unprecedented scale.

A language model first critiqued and revised its own responses using a written list of principles. A second phase trained a preference model from AI-generated comparisons and used reinforcement learning from AI feedback.

The approach made part of the behavior specification explicit and reduced dependence on a human comparison for every training example. It also moved the hard question toward who writes the principles and how their effects are audited.

Meta trained models from 7 billion to 65 billion parameters on publicly available datasets and emphasized token volume over raw parameter count. The 13-billion-parameter model exceeded GPT-3 on most reported benchmarks, while the 65-billion model was competitive with several larger systems.

LLaMA helped make capable base-model weights a practical research substrate outside a few API providers. Its descendants accelerated local inference, fine-tuning, quantization, and independent study of model behavior.

GPT-4 accepted interleaved text and image inputs and generated text outputs. OpenAI reported broad exam and benchmark gains, a predictable-scaling effort, and six months of safety work before release, but withheld model size, architecture, compute, and dataset details.

The release made multimodal prompting part of a prominent general-purpose model and treated deployment evaluation as a first-class technical artifact. Its limited disclosure also marked a widening gap between public capability and reproducible research detail.

QLoRA backpropagated through a frozen four-bit quantized base model into LoRA adapters. Its NormalFloat data type, double quantization, and memory management let the authors fine-tune models up to 65 billion parameters on a single 48 GB GPU.

Fine-tuning frontier-scale open weights no longer required a full cluster or a full-precision copy of every parameter. The method widened access to post-training experiments and domain-specific adapters.

Direct preference optimization rewrote a common KL-constrained preference-learning objective as a classification loss on chosen and rejected responses. Training could update the policy directly, without fitting an explicit reward model and then running reinforcement learning against it.

DPO made one route to preference tuning simpler and more stable to implement. It quickly became a baseline for post-training, while preserving the deeper dependence on how preference pairs were collected and whose judgments they represented.

Gemini 1.5 combined a sparse mixture-of-experts Transformer with native text, image, audio, and video input. The initial Pro release offered a 128,000-token standard context and a one-million-token context to selected users, while the report studied retrieval across even longer sequences.

Long context let one prompt contain books, repositories, hours of media, or large document collections. It did not remove the need to measure retrieval, attention, latency, cost, and whether the model used distant evidence correctly.

GPT-4o was trained end to end across text, images, and audio, with one neural network handling the inputs and outputs. This replaced an earlier voice pipeline in which separate models transcribed audio, generated text, and synthesized speech, losing tone and timing between stages.

Native multimodality made voice, vision, and text part of one learned interaction rather than loosely connected services. It shifted attention from adding an image encoder to preserving real-time cues across modalities.

OpenAI described o1 as a language model trained with large-scale reinforcement learning to produce an internal chain of thought before its answer. Reported performance improved with more reinforcement learning during training and more reasoning compute during inference.

Scaling gained a second axis beyond pretraining: the system could spend more computation on a difficult request at answer time. That changed latency, cost, evaluation, and the value of tasks with checkable outcomes.

DeepSeek-R1-Zero applied large-scale reinforcement learning to a base model without supervised fine-tuning first, producing strong but sometimes unreadable reasoning behavior. DeepSeek-R1 then added cold-start data and a multi-stage training pipeline, and the team released model weights plus smaller distilled variants.

The release gave researchers inspectable reasoning weights and unusually detailed post-training documentation. It showed both what outcome-based reinforcement learning could elicit and why supervised data still mattered for readable, usable answers.

OpenAI defined GPT-5 as a unified system containing a fast model, a deeper reasoning model, and a real-time router. The router selected a path using conversation type, complexity, explicit intent, and tool needs; separate mini models handled some usage limits.

A flagship model name no longer identified one checkpoint or one inference budget. Evaluation and historical comparison now had to ask which component, router policy, reasoning setting, and surrounding safeguards produced an answer.

By 16 July 2026, leading public systems combined several controls once discussed separately: native multimodal input, contexts up to one million tokens, adjustable reasoning effort, routing or model families, and direct use of software tools. GPT-5.5 and Claude Sonnet 5 emphasized sustained computer work, Gemini 3.5 documented multimodal long-context reasoning, and GPT-5.6 added family tiers plus higher inference settings.

The phrase large language model now often names a service boundary, not a single neural network. A useful account must separate the base model, post-training, context construction, inference budget, tools, router, interface, and safety layer.