The history of generative and multimodal AI How machines learned to make what they could once only recognize.

Hiller and Isaacson encoded musical constraints, generated candidate notes on the ILLIAC I, and selected material through rule-based and stochastic experiments. The generated material was transcribed into a score that musicians performed as a string quartet.

The project made computational generation audible in public decades before neural audio models. It also showed that a creative system could combine formal rules with controlled randomness.

Nees wrote programs that used geometric rules and controlled randomness, then rendered their output with a Zuse Graphomat plotter. The Stuttgart exhibition presented the drawings as aesthetic work rather than a graphics demonstration.

The exhibition made a generative visual process visible to a public audience. The program described a space of possible drawings instead of storing one finished picture.

Cohen encoded drawing strategies, spatial constraints, and random choices in a program that issued instructions to custom plotters. AARON could produce an open-ended sequence of distinct drawings without copying a stored image.

AARON brought an explicit AI artmaking system into museums and sustained the same research question for decades: which parts of an artist's knowledge can be expressed as a procedure?

The Gemans represented neighboring image values with a Markov random field and an equivalent Gibbs distribution. Stochastic relaxation and annealing then searched the model for a plausible restored image.

The paper connected local image statistics, an energy function, and sampling. That combination became part of the conceptual foundation for later energy-based image models.

A Boltzmann machine assigned low energy to compatible states and used stochastic units to explore alternatives. Its learning rule compared correlations when visible units were clamped to data with correlations when the network ran freely.

The system learned a probability distribution rather than only a decision boundary. It established a durable pattern for training energy-based generative models.

The Helmholtz machine paired a top-down model that generated observations with a bottom-up network that inferred their hidden causes. Training used the recognition network to make learning in the generator tractable.

The separation between generation and approximate recognition anticipated the encoder-decoder structure of later variational models.

The authors trained a stack of restricted Boltzmann machines greedily, using each learned representation as data for the next layer. The resulting deep belief net could then be refined as a generative model.

Layer-wise pretraining made deep probabilistic networks workable at a moment when direct optimization often failed. It helped restore interest in learned deep representations.

The team trained deep models on paired audio and video of speech, then tested whether one modality could improve or reconstruct the other. Hidden units learned relationships between acoustic patterns and visible mouth movements.

It offered an early deep-learning account of how separate sensory streams could meet in a shared representation, a core problem for later multimodal systems.

Kingma and Welling used a learned recognition model to approximate hidden variables and rewrote stochastic sampling so gradients could pass through it. A decoder then learned to generate observations from the continuous latent space.

The reparameterized objective made expressive latent-variable generators straightforward to train with backpropagation and mini-batches. It established the model family now called variational autoencoders.

A generator transformed noise into synthetic examples while a discriminator tried to separate them from training data. Each network improved against the other in a two-player minimax game.

GANs supplied a new way to train sharp sample generators without writing an explicit likelihood or running a Markov chain when producing an example.

DRAW used recurrent encoder and decoder networks to inspect part of an image, update a latent representation, and add to a canvas over several steps. Its attention windows exposed a generation process rather than a single pass.

The model joined variational learning, recurrence, and visual attention in a generator whose intermediate construction could be inspected.

The model gradually destroyed structure in data through a fixed forward diffusion process, then learned a finite-time reverse process that turned noise back into samples. The formulation provided tractable training, sampling, and probability evaluation.

This paper established the modern diffusion probabilistic model lineage. Later denoising objectives, score models, and faster samplers would make the same broad idea practical at image scale.

The method optimized a new image so deep convolutional features matched the subject matter of one image and texture statistics from another. The result recombined two source images through a pretrained network's internal representation.

It turned learned features into a practical creative medium and showed that a recognition network could also define a useful objective for image synthesis.

The models factored an image into a sequence of conditional probabilities, predicting each color value from pixels and channels already generated. PixelRNN used recurrent layers, while PixelCNN used masked convolutions.

The work made exact likelihood training and high-capacity autoregressive image sampling practical at a larger scale, connecting image generation to next-step prediction.

The model embedded a written description and passed that representation into both generator and discriminator. It learned to synthesize images of birds and flowers whose colors and parts reflected the prompt.

The paper directly connected natural-language descriptions to learned image synthesis, establishing a recognizable ancestor of prompt-driven image generation.

WaveNet predicted the next raw audio sample from earlier samples using causal, dilated convolutions. The same probabilistic architecture produced speech conditioned on text and speaker identity, as well as musical fragments.

It moved neural generation below spectrograms and hand-built speech units to the waveform itself, with a clear jump in perceived speech naturalness in the reported listening tests.

VQ-VAE mapped images, video, and speech into entries from a learned codebook. A separate prior could then model those discrete codes instead of every pixel or audio sample.

Discrete media tokens became a practical bridge between representation learning and generation. The same broad pattern later appeared in VQ-VAE-2, Jukebox, and the first DALL-E.

BigGAN scaled model capacity and batch size for class-conditional ImageNet generation. Orthogonal regularization and the truncation trick gave users a direct way to exchange sample variety for visual fidelity.

The study showed how far GAN image quality could move with compute, data, and careful training. It also documented the instability that accompanied that scale.

StyleGAN fed a transformed latent code into the generator at multiple resolutions and added separate stochastic noise. Coarse structure and fine detail became easier to vary independently than in earlier GANs.

The architecture made high-resolution synthesis more controllable and gave later editing work a useful latent space to inspect and manipulate.

Song and Ermon trained one network to estimate the score, the gradient of log probability, across several noise levels. Annealed Langevin dynamics followed those vector fields from heavy noise toward the data distribution.

The work made score estimation a competitive route to neural generation and supplied a direct bridge between denoising, stochastic sampling, and the diffusion systems that followed.

Jukebox compressed raw audio with a multiscale VQ-VAE, then modeled the codes with autoregressive Transformers. It generated songs conditioned on genre, artist, and unaligned lyrics, including rough singing.

The discrete-token approach moved from images and speech fragments into minutes of music. The weights, code, and a large sample archive made the result inspectable outside the paper.

DDPM trained a neural network to reverse a gradual noise process. Its weighted objective connected diffusion probabilistic models with denoising score matching and produced strong image samples with a comparatively simple recipe.

Denoising became a credible alternative to adversarial training. Later guidance, conditioning, and latent-space methods turned this recipe into a principal foundation for modern text-to-image systems.

Taming Transformers paired a perceptually trained convolutional VQGAN with an autoregressive Transformer. The codebook reduced an image to context-rich visual tokens, while the Transformer learned how those tokens fit together across the scene.

High-resolution image modeling became practical without asking attention to process every pixel. The work connected VQ-VAE-style compression to Transformer generation and helped establish the compressed-representation pattern later used by latent diffusion.

CLIP learned to match images with natural-language descriptions across 400 million web image-text pairs. A user could name candidate visual concepts in text rather than train a new classifier for each task.

Its shared embedding space became useful for retrieval, zero-shot recognition, evaluation, and guidance. It also supplied a semantic layer for DALL-E 2 and many later image systems.

DALL-E represented text and discrete image codes as one autoregressive sequence. At scale, the model combined objects, attributes, viewpoints, and styles from open-ended written descriptions.

The project made broad text-to-image composition visible beyond specialist benchmarks. It also joined the discrete-latent lineage of VQ-VAE with large Transformer training.

LAION-400M made 400 million CLIP-filtered image-text pairs and search indices available for research. LAION-5B expanded the same public-data effort to 5.85 billion pairs in March 2022.

Researchers outside a few large companies gained access to image-text metadata at a scale suitable for training OpenCLIP and generative models. LAION data later supported Stable Diffusion and other open multimodal projects.

GLIDE conditioned a 3.5-billion-parameter diffusion model directly on text and compared CLIP guidance with classifier-free guidance. Human raters preferred classifier-free guidance, and fine-tuning the model supported text-driven inpainting.

The study supplied an early, complete diffusion recipe for text-to-image synthesis and editing before the DALL-E 2 preview. OpenAI also released code and weights for a smaller model trained on filtered data.

Latent diffusion trained the denoising process inside a learned image representation instead of pixel space. Cross-attention let text, layouts, and other signals condition the same broad model design.

High-resolution diffusion became substantially cheaper to train and run without discarding the detail needed for useful images. The method formed the technical base for Stable Diffusion.

DALL-E 2 generated a CLIP image representation from text, then decoded it with diffusion. The same representation supported higher-resolution synthesis, variations, inpainting, and language-guided changes.

Text-to-image generation became an interactive editing system rather than only a collection of research samples. The project also made staged deployment and system-level safeguards part of the public record.

Flamingo connected pretrained vision and language components with layers that handled interleaved images, video, and text. A single model could attempt captioning, visual question answering, and classification from examples placed in its prompt.

Few-shot prompting moved beyond text-only inputs. The model provided an architectural pattern for later systems that place visual evidence directly inside a conversational context.

Imagen paired a large frozen text encoder with a cascade of diffusion models. Its ablations found that enlarging the language encoder improved image fidelity and caption alignment more than enlarging the image generator alone.

The result clarified that text understanding was a central part of text-to-image quality. DrawBench also tested composition, counting, spatial relations, and written text beyond one aggregate score.

Stable Diffusion combined latent diffusion with image-text representations and web-scale data in downloadable model weights. Version 1.4 could run on a single GPU with about 6.9 GB of video memory.

Developers and researchers could inspect, run, fine-tune, and extend a capable text-to-image model locally. A large ecosystem of interfaces, adapters, and specialized checkpoints followed.

Make-A-Video learned appearance and language alignment from paired text-image data, then learned motion from video without captions. Spatial and temporal modules adapted an image-generation pipeline to short clips.

The project showed a practical route from the 2022 text-to-image wave into open-domain video research without requiring a comparably large paired text-video collection.

DiT divided a latent image representation into patches and processed them with a Transformer instead of the usual convolutional U-Net. Larger models and more latent tokens raised compute and consistently improved sample quality across the study.

The work showed that a denoising backbone could inherit Transformer scaling behavior. OpenAI later described Sora as a diffusion transformer operating on spacetime patches of compressed video and image latents, extending the same architectural line from images to video.

LLaVA connected a pretrained vision encoder to a language model, then tuned the system on image-language conversations synthesized with GPT-4. The paper and public implementation gave researchers a compact recipe for building an assistant that could discuss an image rather than classify it into a fixed set of labels.

Visual instruction tuning became a repeatable bridge between image encoders and conversational language models. The released code, data, and model artifacts also gave independent teams a practical base for multimodal assistant research.

ImageBind learned one embedding space for images, text, audio, depth, thermal measurements, and inertial data. Instead of collecting every pair of modalities, the team used images as the common anchor and trained each other modality against image-paired data.

The work showed that several senses could be aligned through a shared pivot. That made cross-modal retrieval and composition possible even for modality pairs that had never appeared together during training.

SDXL enlarged the latent-diffusion backbone, added a second text encoder, trained across multiple image sizes and aspect ratios, and paired a base model with an optional refiner. The public 1.0 release generated at a native 1024-pixel resolution and exposed the model to a broad tooling ecosystem.

High-resolution text-to-image generation became easier to inspect, adapt, and run outside a hosted product. SDXL's model files and implementation encouraged fine-tunes, control tools, and local creative workflows.

GPT-4V let ChatGPT users submit photographs, screenshots, diagrams, and documents alongside text. OpenAI paired the rollout with a vision-specific system card that examined person identification, medical advice, scientific tasks, and other risks introduced by image input.

Image reasoning reached a widely used general-purpose chat interface, and multimodal deployment received its own safety and limitation record. The release also made the gap between public capability and reproducible model detail more visible.

OpenAI trained an image captioner to replace terse web labels with detailed descriptions, then trained DALL-E 3 on the recaptioned data. In ChatGPT, a language model expanded ordinary requests into richer prompts and let users refine the result through conversation.

The work identified caption quality as a major control point for prompt fidelity. Recaptioning later carried into video systems, while the ChatGPT integration reduced the need for users to learn a separate prompt dialect.

Gemini 1.0 was trained to work across text, code, images, audio, and video, with Ultra, Pro, and Nano variants aimed at different compute settings. Gemini Pro entered Bard on announcement day, and Pro and Pro Vision reached developers through the Gemini API one week later.

A major model family treated multimodal training as a foundation rather than attaching vision only at the product edge. The same family spanned data-center models, developer access, and on-device deployment.

Sora compressed images and videos into a latent space, split that representation into spacetime patches, and trained a diffusion transformer across different durations, resolutions, and aspect ratios. It also carried DALL-E 3's recaptioning method into video training and prompt expansion.

Video generation acquired a scalable representation analogous to tokens in language models. One model could generate, extend, transform, and connect visual sequences without forcing every training example into one fixed shape.

GPT-4o accepted combinations of text, audio, images, and video and could produce text, audio, and images. Unlike the earlier ChatGPT voice pipeline, one end-to-end neural network processed audio input and produced audio output, preserving information such as tone, timing, and overlapping voices that transcription could discard.

Real-time multimodal conversation moved from a chain of speech and language models toward a single trained system. The model's reported audio response latency approached the timing of ordinary human conversation.

Movie Gen paired a 30-billion-parameter image-and-video transformer with systems for personalized video and instruction-based editing. A separate 13-billion-parameter audio model generated music and sound effects from video, with optional text guidance for the soundtrack.

Generation, identity-preserving personalization, local editing, and synchronized audio appeared as parts of one research stack. The work framed a media model as an editable production system rather than a prompt-to-clip demo.

OpenAI trained GPT-4o over a joint distribution of text and images so the same conversational model could understand references, use chat context, edit an uploaded image, and generate a new one. The transformer produced compressed representations that a separate decoder turned into pixels.

Image generation gained the language model's world knowledge, in-context examples, and multi-turn memory without handing every request to a separate image product. Text rendering and iterative editing became central use cases rather than side effects.

Qwen2.5-Omni used a Thinker-Talker design to process text, images, audio, and video while streaming both text and synthesized speech. Time-aligned multimodal rotary position embeddings synchronized video frames with audio, and a sliding-window diffusion transformer reduced delay in speech decoding.

A documented end-to-end omni architecture, implementation, and model weights became available outside a proprietary interface. Researchers could inspect how streaming perception and speech generation were divided while still being trained and run as one system.

Veo 3 generated a video's visuals together with dialogue, ambient sound, and effects. Google released it to US subscribers through Gemini and Flow on announcement day, with enterprise access through Vertex AI, rather than presenting audio as a separate post-production step.

Synchronized sound became part of a deployed video generator's core output. Prompting could describe both what appeared on screen and what the scene should sound like.

Genie 3 generated a 720p environment from a text prompt and updated it at 24 frames per second as a user moved through the scene. It could retain broad consistency for a few minutes and accept promptable events that changed the generated world during interaction.

Generative video stopped being only a fixed clip and became an action-conditioned environment. That opened a route toward interactive prototyping and synthetic worlds in which people or agents could act.

Gemini Omni Flash accepted combinations of text, images, audio, and video as references, then generated video with sound. Its conversational interface carried prior edits forward, letting a user alter actions, viewpoints, settings, and visual details over multiple turns.

A reasoning model and a media generator were presented as one iterative creation system. Video editing could use mixed references and dialogue instead of restarting from one text prompt for every change.

Muse Image could search for visual references, write and run code for precise graphics, inspect its own drafts, and spend additional inference-time computation on revisions. Meta released it across several consumer surfaces while separately previewing Muse Video, a native-audio model built on the same pretraining base.

Media generation became a deliberative process around the model, not only a single sampling pass. Tool use, self-critique, multi-reference composition, and inference-time scaling became explicit parts of the generation loop.