Simon Willison and Prime Radiant Release smevals for Model Evaluation
Simon Willison and Prime Radiant released smevals on July 31, 2026, as an open-source framework for comparing models, prompts and agent harnesses across repeatable tasks. The tool separates model runs from grading, preserves immutable run records and can generate local or static reports, giving teams a compact way to test configurations without adopting a hosted evaluation platform.
Simon Willison and Prime Radiant released smevals on July 31, 2026, an open-source framework for running small evaluation suites across models, prompts and agent harnesses. The project is available as a Python package and can be installed or invoked with uv or pip.
Runs and grades stay separate
The framework organizes an evaluation as a set of tasks, configurations and graders. A configuration can identify a model while also specifying a system prompt, model parameters or an agent harness. Each task-and-configuration execution creates an immutable run record, while grading happens as a separate operation. That separation lets a team apply a revised grader to existing outputs without spending money or time to rerun the model.
The repository also distinguishes model output from infrastructure failure. A runner that exits with an error creates a failed run for debugging, but that run is excluded from grades and does not count toward a requested sample target. Repeated runs can be used to account for nondeterministic model behavior.
A deliberately small workflow
Users define an evaluation in a directory of YAML files, then run one or more model configurations from the command line. Graders execute ordered checks that can range from simple string or XML validation to custom scripts and model-based judging. Checks may record scores, metrics, tags and diagnostic details.
Results can be reviewed in a local web interface or built as static HTML. The project documentation includes a haiku-writing example and describes how custom runners can call a direct model API or an agent system such as Codex.
For data and AI teams, the practical value is reproducibility rather than a new benchmark claim. smevals provides a lightweight structure for comparing prompts, model versions and harnesses while keeping raw runs distinct from the rules used to score them. The release does not report independent benchmark results, enterprise deployments or performance guarantees; teams still need to design representative tasks and reliable graders for their own use cases.
Key Points
- 1smevals runs repeatable tasks across model, prompt and agent-harness configurations.
- 2Immutable run records are separated from grading, so revised graders can evaluate existing outputs without rerunning models.
- 3The command-line tool supports repeated samples, custom runners and checks, a local results server and static HTML reports.
- 4The release provides evaluation infrastructure, not independent evidence that any model or prompt performs better.
Scoring Rationale
A practical open-source evaluation framework from established AI tooling practitioners, with clear utility for model and harness comparisons but no independent adoption or benchmark evidence yet.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
