Talkie Trains 13B Model on Pre-1930 Text
Researchers led by Nick Levine, David Duvenaud, and Alec Radford released Talkie, a 13-billion-parameter language model trained exclusively on English-language texts published before the end of 1930. According to the project writeup and reporting by Hackster and BoingBoing, the corpus totals roughly 260 billion tokens drawn from books, newspapers, periodicals, scientific journals, patents, and case law that are now largely in the public domain. The team tested talkie on a "surprisingness" task, finding events after 1930 harder for the model to anticipate, and reported that the model can learn simple programming tasks when given examples. Gizmodo and The Register note the 1930 cutoff was chosen in part because of US copyright rules that made much 1930 material public domain in 2026. The Talkie team wrote, "Talkie is the largest vintage language model we are aware of, and we plan to continue scaling significantly," according to The Register.
What happened
Researchers including Nick Levine, David Duvenaud, and Alec Radford released Talkie, a language model trained only on materials published before the end of 1930. Per the project writeup and reporting by Hackster and BoingBoing, the model is 13-billion-parameter and was trained on roughly 260 billion tokens of public-domain English text from books, newspapers, periodicals, scientific journals, patents, and case law. The team reports experiments measuring the model's subjective rating of how "surprising" later historical events are and found a marked increase in unpredictability for events after the 1930 cutoff; the project writeup and Hackster also report a few examples where talkie inferred simple programming solutions from provided examples.
Technical details
The dataset choice and scale are described on the project site and in coverage by Gizmodo and The Register, which note the 1930 cutoff aligns with a US copyright expiration window that released many works into the public domain in 2026. The team documents training from scratch rather than fine-tuning an existing foundation model; reporting cites the 260 billion token figure and the composition of sources as digital scans of pre-1931 English-language publications. The Register reproduces the project quote: "Talkie is the largest vintage language model we are aware of, and we plan to continue scaling significantly," attributed to the Talkie team.
Editorial analysis - technical context: Vintage LLMs, defined by explicit temporal cutoffs, create a controlled experimental setting to isolate how training-time knowledge shapes generation and prediction. Industry-pattern observations indicate such models are useful for measuring temporal generalization, quantifying "surprise" for out-of-distribution events, and probing how language change affects tokenization and semantics over decades. The reported ability of talkie to perform simple coding tasks after being given examples aligns with broader findings that models can induce procedures from demonstrations even when the underlying concept was absent from pretraining data.
Industry context:
Reporting by Gizmodo frames the choice of 1930 as partly pragmatic, because of US public-domain rules; several outlets position the work within a growing "vintage LLM" conversation seeded by academic talks and smaller projects. For practitioners, this project highlights a reproducible path to studying dataset-era effects without complex temporal filtering of contemporary corpora. Observed patterns in similar work show vintage datasets surface historical biases and anachronistic worldviews that are valuable for humanities research but may limit utility for modern applications.
What to watch:
- •Editorial analysis: whether the Talkie team follows up with open evaluations and reproducible benchmarks for temporal prediction and "surprisingness" tasks.
- •Editorial analysis: how scaling vintage models affects extrapolation beyond the cutoff versus simply amplifying period-specific language artifacts.
- •Editorial analysis: community uptake in digital humanities, education, or controlled user studies that probe conversational behavior rooted in historical priors.
Scoring Rationale
Talkie is an interesting research artifact that offers a clean experimental setup for studying temporal effects in LLMs and dataset curation. It is not a frontier-model breakthrough, but it provides useful methods and questions for practitioners and researchers.
Practice interview problems based on real data
1,500+ SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
