Developers Build Personal Benchmarks For LLM Coding

On April 4, 2025, a blog post argues that developers should maintain personal benchmarks for coding-focused LLM usage, describing a lightweight workflow and referencing Nicholas Carlini's Yet Another Applied LLM Benchmark. The author outlines methods for collecting failing one-shot tasks, building evaluation functions, and two evaluation approaches (codebase versus transcript tasks). The piece highlights practical benefits for debugging, model selection, and prompt tuning.