Fisher-R1 Targets More Reliable Hypothesis Testing by LLM Agents
Researchers introduced Fisher-R1 and P-Bench in an August 7 preprint, testing LLM agents on 425 hypothesis-testing tasks drawn from economics, biology, and medicine. The authors report that Fisher-R1-14B improved single-trial success by 21% on average relative to DeepSeek-V4-Pro, but the result remains an author-reported preprint benchmark rather than independent validation.
A research team has introduced Fisher-R1, an open-weight language-model agent trained specifically for statistical hypothesis testing, alongside a new evaluation set called P-Bench. The work was submitted to arXiv on August 7, 2026, and the authors linked a public GitHub repository describing the benchmark and model.
The problem P-Bench is designed to expose
An analysis can run without errors and still reach the wrong statistical conclusion. A model might choose a test whose assumptions do not fit the data, report a plausible p-value, and produce fluent reasoning around an invalid result. The paper argues that common agent benchmarks often miss this failure because they check code execution or final answers without fully evaluating whether the statistical method was appropriate.
P-Bench contains 425 open-ended tasks built from real datasets in economics, biology, and medicine. Each task gives an agent a scientific hypothesis and data, then requires it to select a statistical method, execute the analysis, report a p-value, and draw a conclusion. The answer keys were expert-verified, according to the paper and repository.
What the authors report
Fisher-R1 was trained on synthetic executable tasks using supervised fine-tuning followed by reinforcement learning with statistically verified rewards. On P-Bench, the authors report that the 14-billion-parameter model achieved a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, with gains reaching 26% on the most difficult task groups. They also report that it outperformed the other proprietary and open models included in their evaluation, including GPT-5.4.
Those numbers should be read as benchmark results reported by the authors. The work is a preprint, the retrieved GitHub repository currently provides a concise project description, and this audit did not find an independent replication. Relative improvement also does not describe the absolute error rate on its own.
Why it matters for data science practice
Fisher-R1 focuses on a failure mode that matters beyond leaderboard performance: selecting a defensible method before calculating a result. For teams building analysis agents, the benchmark suggests that evaluation should test assumptions, p-values, and conclusions together. It also offers a more targeted training approach, but production use would still require independent testing on an organization's own data distributions, explicit audit trails, and human review of consequential inferences.
Key Points
- 1P-Bench evaluates 425 open-ended hypothesis-testing tasks across economics, biology, and medicine.
- 2The authors report a 21% average relative single-trial gain for Fisher-R1-14B over DeepSeek-V4-Pro, reaching 26% on the hardest groups.
- 3The work is an arXiv preprint with an author-linked repository and no independent replication identified in this audit.
Scoring Rationale
The paper targets a consequential statistical-reasoning failure mode with a sizable executable benchmark and open-weight model, while preprint status and absent independent replication constrain the evidence strength.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
