The history of RAG, AI memory, and context engineering How models learned to find, carry, and remember the right evidence.

Mooers used the term information retrieval for the machine-assisted selection of relevant recorded information. His technical bulletin treated indexing, coding, and selection as an engineering problem separate from conventional library classification.

The paper gave a lasting name and technical boundary to the field that would later supply the retrieval half of RAG.

Luhn proposed using the frequency and distribution of words to encode documents and guide machine searching. His design reduced the need for every useful search feature to be assigned by a human indexer.

The work is a direct precursor to automatic term weighting, keyword extraction, and statistical document ranking.

Maron and Kuhns represented an index term through the probability that documents assigned that term would satisfy an information need. Their system could rank documents by estimated relevance instead of returning only exact Boolean matches.

The paper is one of the first published probabilistic retrieval models and an early defense of ranked results.

The Cranfield experiments compared indexing systems against a fixed document collection, search questions, relevance judgments, and effectiveness measurements. This experimental arrangement made retrieval claims repeatable rather than anecdotal.

Modern IR test collections, TREC tracks, and dense-retrieval leaderboards inherit Cranfield's basic evaluation structure.

Rocchio updated a query vector by moving it toward documents judged relevant and away from documents judged nonrelevant. The method turned user feedback into an explicit transformation of the next retrieval request.

The algorithm became the canonical relevance-feedback method and influenced query expansion, interactive retrieval, and later learned ranking.

Sparck Jones argued that a term occurring in fewer documents should carry greater discriminatory weight in retrieval. Her experiments connected collection frequency to search effectiveness and supplied the inverse-document-frequency principle.

IDF became a core component of TF-IDF, probabilistic ranking, and the lexical baselines still paired with or compared against dense retrieval.

Salton, Wong, and Yang represented stored documents and incoming search requests as weighted vectors in a common term space. Similarity in that space supported graded ranking rather than exact symbolic matching alone.

Shared vector representation and similarity search remain structural foundations of both classical information retrieval and embedding-based retrieval.

Robertson and Sparck Jones derived term weights from a probabilistic theory of relevance and tested them on retrieval collections. Their relevance odds became the basis of the Robertson-Sparck Jones weight.

The theory is a direct ancestor of the Okapi ranking work and the BM25 baseline used throughout modern retrieval research.

Hopfield described a recurrent neural network whose dynamics settle into stable energy minima representing stored patterns. A partial or corrupted cue could recover a complete pattern through content-addressable recall.

The paper gave neural memory an influential energy-based formulation and linked retrieval by content to recurrent computation.

Latent semantic analysis applied singular-value decomposition to a term-document matrix and represented terms, documents, and queries in a lower-dimensional space. Retrieval could therefore bridge some vocabulary differences without requiring exact term overlap.

LSA formed a major bridge from lexical matching to dense semantic representations and similarity search.

TREC-1 brought 25 research groups to common document collections, topics, relevance judgments, and evaluation measures. Its roughly two-gigabyte test collection pushed comparative retrieval experiments beyond many earlier laboratory datasets.

TREC institutionalized shared evaluation and made progress across retrieval systems easier to reproduce and compare.

The Okapi team reported a probabilistic ranking function with term-frequency saturation, document-length normalization, and collection statistics. Its TREC-3 form is a canonical early published version of what became known as BM25.

BM25 became the durable sparse-retrieval baseline against which neural rerankers and dense retrievers are still evaluated.

Brin and Page described a web search prototype that combined full-text indexing, anchor text, link structure, proximity, and PageRank. The system treated relationships among pages as evidence alongside the words on each page.

Large-scale web retrieval became an evidence-ranking problem over both content and the structure connecting sources.

Ponte and Croft estimated a language model for each document and ranked documents by the probability that each model would generate the query. This reframed retrieval as probabilistic language generation rather than vector similarity alone.

The work opened the modern language-modeling family of information retrieval and anticipated later generative relevance scoring.

TREC-8 introduced a shared evaluation in which systems returned short answer strings or snippets instead of ranked document lists. The first track used 200 factoid questions and judged both 50-byte and 250-byte responses.

The track connected document retrieval to answer extraction and gave open-domain question answering a reusable evaluation framework.

Bengio and colleagues trained a feed-forward network to learn a vector for each word and a probability distribution for the next word at the same time. Similar words could share statistical strength even when an exact sequence had not appeared in training.

The paper joined embeddings, neural probability estimation, and next-word prediction in a scalable recipe that became a direct ancestor of pretrained language models.

DeepQA analyzed a question, searched many sources, generated candidate answers, scored supporting and opposing evidence, and estimated confidence. The system separated evidence gathering from answer selection while combining many symbolic and statistical components.

Its retrieve, propose, score, and calibrate pipeline is an important ancestor of evidence-grounded answering systems.

The word2vec work introduced efficient continuous-bag-of-words and skip-gram architectures for learning word vectors from very large corpora. The learned coordinates exposed useful syntactic and semantic regularities without a labeled dictionary.

Embeddings became standard reusable artifacts and made dense semantic similarity practical for a broad engineering audience.

DSSM learned neural projections of queries and documents into a common low-dimensional semantic space using search clickthrough data. Cosine similarity in that space supplied a learned relevance signal beyond exact word overlap.

The paper is a major dense-retrieval precursor and an early industrial demonstration of neural semantic matching for web search.

The model learned a soft weight over encoder states for every generated output word, letting the decoder choose which source positions mattered at that step. It removed the requirement that an entire source sentence be compressed into one fixed vector.

Differentiable attention became a core mechanism for selecting and composing context in later memory, retrieval, and language models.

Memory Networks combined a readable and writable long-term memory with learned components that selected supporting facts and produced answers. The architecture framed question answering as inference over an explicit, dynamically updated knowledge store.

The work directly joined retrieval from external memory with learned reasoning and answer production.

Neural Turing Machines coupled a neural controller to an external memory matrix with differentiable read and write operations. The models learned synthetic tasks including copying, sorting, and associative recall from examples.

The work demonstrated trainable content-based and location-based access to an external neural memory.

The model used recurrent attention over an external memory and repeated the lookup across multiple inference hops. Its components could be optimized jointly from input-output examples with less supporting-fact supervision than the original Memory Networks.

The paper made multi-hop differentiable memory access a cleaner end-to-end learning problem.

Malkov and Yashunin arranged proximity graphs in layers, using sparse upper layers for coarse navigation and denser lower layers for a more precise search. Their experiments reported strong recall and search-speed tradeoffs across several high-dimensional datasets.

HNSW became an important indexing method for vector search systems that need to retrieve semantically similar items from large collections.

The Transformer replaced recurrence with multi-head self-attention and position-aware feed-forward processing. Each token could build its representation by directly weighting other tokens in the available context.

Transformers became the shared substrate for contextual encoders, dense retrievers, rerankers, long-context models, and generative RAG systems.

DrQA combined a Wikipedia document retriever based on hashed bigram TF-IDF with a neural reader that extracted answer spans. It evaluated the full retrieve-then-read pipeline across several open-domain question-answering datasets.

DrQA is one of the clearest immediate architectural precursors to RAG: retrieve evidence first, then use a learned model to answer from it.

BERT pretrained a deep bidirectional Transformer encoder with masked-token prediction and a next-sentence objective, then fine-tuned the same representation for downstream tasks. It produced strongly contextual token and sequence representations that were easy to adapt.

BERT made powerful neural reading, cross-encoder relevance scoring, and later dense-retriever initialization broadly practical.

The authors fine-tuned BERT as a cross-encoder that jointly read a query and candidate passage before predicting relevance. The approach produced large gains on TREC-CAR and MS MARCO passage ranking.

The study made BERT cross-encoding a strong second-stage neural reranker within the established retrieve-then-rerank pattern.

Transformer-XL cached hidden states from previous segments and reused them through a segment-level recurrence mechanism. This extended usable context beyond a fixed training window and reduced fragmentation at segment boundaries.

The paper is a key milestone in managing longer model context before today's long-context and context-engineering systems.

ORQA jointly learned a retriever and reader from question-answer strings while treating supporting Wikipedia evidence as a latent variable. An inverse-cloze pretraining task gave the retriever a usable starting point without labeled question-passage pairs.

The system is a direct precursor to DPR, REALM, and RAG-era dense retrieval because it learned open-corpus evidence selection for question answering.

kNN-LM stored training-set contexts and their next tokens in a datastore keyed by a pretrained language model's hidden representations. At inference time, it retrieved nearby contexts and interpolated their token distribution with the model's own prediction.

The study showed that a fixed language model could gain factual and rare-pattern coverage from an external datastore without retraining its parameters.

REALM treated document retrieval as a latent step during language-model pretraining, fine-tuning, and inference. Its retriever searched millions of documents so factual knowledge could remain outside the model parameters.

It made a learned external index part of pretraining and supplied a direct technical precursor to RAG.

DPR trained separate question and passage encoders so relevant passages could be found with vector similarity. On its tested question-answering datasets, it raised top-20 retrieval accuracy by 9 to 19 percentage points over a strong BM25 baseline.

The dual-encoder design became a standard retriever in early RAG pipelines because documents could be encoded before a query arrived.

ColBERT encoded queries and documents independently but retained one embedding per token for a late interaction scoring step. The design recovered fine-grained matching signals without running a full cross-encoder over every candidate document.

It established a practical middle ground between efficient single-vector retrieval and expensive all-token reranking.

RAG combined a sequence-to-sequence generator's parametric memory with passages retrieved from a dense Wikipedia index. RAG-Sequence reused one passage set for an output, while RAG-Token allowed the latent passage to vary by generated token.

The paper named and formalized the architecture that became the reference point for retrieval-grounded language applications.

Fusion-in-Decoder encoded each retrieved passage separately and joined their representations in the decoder. Its question-answering performance improved as the model received more passages, showing a scalable alternative to concatenating them into one encoder input.

FiD became a durable baseline for turning a set of retrieved passages into one evidence-aware answer.

KILT mapped question answering, fact checking, slot filling, entity linking, and dialogue tasks to one Wikipedia snapshot. It measured both the task output and the provenance passages used to support that output.

The benchmark made evidence retrieval a first-class part of evaluating knowledge-intensive language systems.

BEIR compared lexical, sparse, dense, late-interaction, and reranking systems across 18 retrieval datasets. BM25 remained unusually robust, while late-interaction and reranking approaches achieved stronger averages at higher computational cost.

It made cross-domain retrieval evaluation harder to reduce to one favored dataset or one model family.

ColBERTv2 paired token-level late interaction with denoised supervision and residual compression. It reported a six to ten times smaller storage footprint than the original ColBERT while improving retrieval quality.

The work made fine-grained neural matching more plausible at large collection sizes.

RETRO retrieved neighboring chunks from a database containing two trillion tokens during language-model training and inference. On the Pile, the authors reported performance comparable to much larger models while using far fewer parameters.

It showed retrieval as a route to scaling usable knowledge without placing every fact in model weights.

WebGPT let a language model search, open pages, find text, and collect quotations in a text browser. Human demonstrations and preference feedback trained it to research questions and answer with citations.

It connected retrieval to an agent's evolving decisions and exposed an evidence trail for readers to inspect.

Atlas pretrained a retriever and sequence-to-sequence model together for knowledge-intensive tasks. It delivered strong few-shot results while keeping factual material in an index that could be replaced or refreshed.

It connected retrieval augmentation to sample-efficient adaptation instead of treating retrieval only as an inference add-on.

ReAct prompted language models to alternate reasoning traces with environment actions such as querying Wikipedia. New observations entered the trajectory, letting the model revise its next thought and action instead of relying on one retrieval pass.

It helped shift retrieval from a fixed preprocessing step toward an action selected during agent reasoning.

HyDE asked a language model to generate a hypothetical answer-like document and embedded that document as the search query. The embedding then retrieved real corpus documents without requiring relevance labels for the target task.

The method used generation to bridge short questions and the richer document representations expected by a dense retriever.

Generative Agents stored a complete stream of simulated experiences and retrieved memories using relevance, recency, and importance. The system also synthesized reflections and plans that could enter later behavior.

It made memory writing, retrieval, reflection, and planning visible as distinct parts of a persistent agent loop.

Anthropic expanded Claude's context window from 9,000 to 100,000 tokens and demonstrated analysis across long documents. Bounded corpora could now be placed directly in a request, creating a practical alternative to retrieval for some tasks.

The release made long context versus selective retrieval a central systems tradeoff rather than a purely theoretical question.

FLARE generated a tentative next sentence, identified low-confidence tokens, and used the tentative text to retrieve supporting documents before continuing. This let retrieval occur during long-form generation instead of only once before the answer began.

The method supplied a concrete bridge from one-shot RAG to systems that decide when a developing response needs more evidence.

Lost in the Middle placed relevant information at controlled positions inside long prompts. Tested models often performed best when evidence appeared near the beginning or end and worse when it appeared in the middle.

The results separated advertised context capacity from a model's ability to find and use evidence throughout that context.

Ragas proposed reference-free measurements for retrieved-context relevance, answer relevance, and faithfulness to supplied evidence. The framework aimed to diagnose the retriever and generator separately instead of reducing a RAG pipeline to one answer score.

It helped establish component-level evaluation as a routine part of building and debugging RAG systems.

MemGPT organized information into memory tiers and let the language model move content between its active context and external storage. The virtual-memory analogy supported long documents and conversations that extended beyond one context window.

It framed context selection, eviction, recall, and persistence as an explicit operating problem for language agents.

Self-RAG trained a language model to request retrieval when needed and emit reflection tokens about evidence relevance, support, and output quality. Those signals could guide generation and rank candidate responses without retrieving for every prompt.

It joined adaptive retrieval and self-critique inside one trained model rather than a fixed retrieve-then-generate wrapper.

Corrective RAG used a retrieval evaluator to estimate whether retrieved documents were correct, ambiguous, or wrong. It refined useful passages and could invoke web search when the original retrieval appeared insufficient.

CRAG made retrieval failure a state the system could detect and route around instead of silently passing weak evidence to the generator.

RAPTOR clustered text chunks, summarized each cluster, and repeated the process to build a tree with evidence at several levels of abstraction. Retrieval could then select detailed passages, higher-level summaries, or both for a question.

The tree gave RAG systems a way to retrieve broader themes and relationships that may not fit inside one neighboring chunk.

Google announced a 128,000-token standard context for Gemini 1.5 Pro and a one-million-token experimental preview. The company also reported internal tests at up to ten million tokens across text, audio, and video inputs.

Million-token context made direct corpus loading a serious competitor to retrieval while increasing the need to test utilization, cost, and latency.

Adaptive-RAG trained a classifier to route questions among no retrieval, one retrieval step, and iterative retrieval. The route depended on predicted question complexity rather than a single retrieval policy for every input.

It made retrieval depth a resource-allocation decision that could trade accuracy against latency and cost.

RULER expanded needle-in-a-haystack tests with multiple needles, multi-hop tracing, and aggregation tasks at increasing lengths. Many evaluated models degraded with length, and only half maintained satisfactory performance at 32,000 tokens under the benchmark's criteria.

It supplied a stricter measure of usable context than maximum-window size or one simple retrieval probe.

Microsoft's GraphRAG extracted entities and relationships, clustered the resulting graph, and generated summaries for each community. A map-reduce process used those summaries to answer global questions that local chunk retrieval handled poorly.

It gave corpus-wide sensemaking a distinct retrieval architecture rather than treating every question as nearest-neighbor passage lookup.

HippoRAG built a knowledge graph from passages and used personalized PageRank to retrieve connected evidence. The design targeted multi-hop questions with one graph traversal rather than repeated retrieve-and-generate cycles.

It connected associative graph retrieval to the emerging idea of durable long-term memory for language models.

The study compared retrieval-augmented generation with placing the available corpus directly into a long-context model. Long context performed better when fully resourced in the tested settings, while RAG was substantially cheaper, so Self-Route predicted which path to use.

It replaced a binary architecture debate with a measurable quality, cost, and routing tradeoff.

Anthropic's Contextual Retrieval generated short document-specific context for each chunk before embedding and BM25 indexing. The system fused dense and lexical results and optionally reranked them before supplying evidence to the model.

It made lost document context an indexing problem and packaged contextualization, hybrid search, and reranking into a clear production pattern.

LongMemEval posed 500 questions over scalable chat histories to test extraction, cross-session reasoning, temporal reasoning, information updates, and abstention. The authors reported a 30 percent performance decline for the tested long-context and commercial systems.

It decomposed conversational memory into indexing, retrieval, and reading operations that could fail independently over time.

Anthropic released MCP as an open protocol through which clients could discover and connect to resources, prompts, and tools exposed by external servers. The client-server boundary reduced the need to build a bespoke integration for every model and data system pairing.

MCP made context supply and tool access an interoperability layer that could sit beneath many retrieval and agent architectures.

Search-o1 placed retrieval inside a reasoning trajectory, letting the model initiate search when it reached a knowledge gap instead of relying on a single retrieval pass before generation. A separate Reason-in-Documents module examined the returned passages before condensed evidence entered the main reasoning process.

The design extended earlier search-using agents with a dedicated Reason-in-Documents stage that condensed retrieved passages before they returned to the main reasoning trace. It also tested retrieval triggered by an encountered knowledge gap instead of a fixed search schedule.

OpenAI launched Deep Research as a ChatGPT capability that planned and executed multi-step searches, interpreted text, images, and PDFs, and returned a documented report. The product could revise its search path as it encountered new information instead of relying on one retrieval pass.

A consumer-facing product exposed iterative retrieval, source selection, synthesis, and citations as one visible research workflow. It also made the time and intermediate work of gathering context part of the user experience.

A-MEM represented experiences as linked notes whose attributes and relations could change as new memories arrived. Mem0 independently described a production-oriented pipeline that extracted, consolidated, and retrieved selected conversational facts, showing two 2025 approaches to memory as an actively managed store.

Both projects moved beyond replaying an entire conversation or performing a simple similarity lookup over static entries. They made writing, organizing, revising, and selecting memories part of the agent design.

Tobias Lütke and Andrej Karpathy helped popularize context engineering as a broader name for arranging the information a model needs across a system. The LangChain team turned that framing into a working taxonomy of writing, selecting, compressing, and isolating context for agents.

The increasingly popular label gathered instructions, retrieved knowledge, tools, message history, intermediate state, and memory under one operational problem. It shifted attention from crafting one input string to maintaining the right information across an agent's changing state.

Prithvi Rajasekaran, Ethan Dixon, Carly Ryan, and Jeremy Hadfield described effective agent context as a finite attention budget that must be curated across instructions, tools, retrieved data, message history, and intermediate work. Anthropic's Agent Skills material applied progressive disclosure by keeping compact skill metadata in view and loading fuller instructions or resources only when relevant.

The guidance treated context management as a tradeoff between relevance, completeness, token cost, and model attention. Progressive disclosure supplied one concrete packaging pattern: begin with a small routing layer, then reveal deeper procedural context when the task calls for it.

EvoMemBench organized agent-memory evaluation along two axes: in-episode versus cross-episode scope, and knowledge-oriented versus execution-oriented content. It compared 15 memory methods with long-context baselines under one protocol.

The results exposed task-specific tradeoffs: no memory form won consistently, and long-context baselines remained competitive in several settings.

Yasmine Omri and colleagues built a phase-aware harness that attributed memory-system costs to construction, retrieval, and generation. They characterized ten systems across two benchmark suites and documented how design choices shifted work between read and write paths.

The paper treated memory as an operational workload with scheduling, freshness, latency, amortization, and hardware implications, not only as an answer-quality feature.

OpenAI described Dreaming as a background process that synthesizes memory from many conversations and revises it as information becomes stale. The 2026 update exposed a reviewable memory summary and extended the system across paid and free ChatGPT tiers.

The release made freshness, correction, user review, compute cost, and long-term continuity visible concerns in a widely deployed conversational memory product.

Cyrus Rashtchian, Da-Cheng Juan, and colleagues described a Gemini Enterprise Agent Platform workflow that plans searches, rewrites queries, fans out across sources, and checks whether the gathered context is sufficient. When evidence remains inadequate, the system searches again before handing context to the answering model.

The release turned iterative retrieval and explicit context sufficiency into a managed enterprise workflow. It also showed how agentic RAG could route across multiple corpora while keeping the decision to retrieve more inside the system loop.