Arena Tells Lets Data Science That Only One AI Provider Is Consistently Getting More Factual

The split is deliberate rather than conservative. Weighted this way, Arena said, "it is difficult for an inaccurate model to top the leaderboard, while the human-preference component prevents a model from succeeding simply by avoiding informative answers." A model cannot win by being scrupulously accurate and useless, and it cannot win by being fluent and wrong.

Arena described the pipeline it runs for each factuality audit:

• •Randomly sample real Text Arena and Search Arena battles
• •Break each model response into individual, or atomic, claims
• •Filter out subjective, ambiguous, or non-web-verifiable statements
• •Use search agents to assess each remaining claim and assign a calibrated probability that it is true
• •Average those claim-level truth probabilities across each response
• •Convert the difference between the two averages into a factuality outcome for that battle
• •Combine those outcomes with human preference votes in a composite Bradley-Terry model

The scope is narrower than the word hallucination usually implies, and Arena was explicit about that. The leaderboard measures "the accuracy of objective, unambiguous claims about the world that can reasonably be verified using web sources." It does not attempt to catch every failure mode, "such as incorrectly recalling something from an earlier conversation or claiming that a coding action was completed when it was not."

The case Arena makes is that trust has outrun verification. "People are increasingly trusting models to answer questions outside their own areas of knowledge, even though manually checking every claim is slow and labor-intensive," the company told Lets Data Science.

That leaves preference votes doing work they were never built for. Human ratings "may reward responses that are well written or comprehensive without fully reflecting whether their claims are accurate." Arena's argument is that style had become a proxy for accuracy, and that the two needed separating.

Factuality and hallucination benchmarks already exist, so the obvious question is what this adds. Arena's answer is that its claims are not written for the test.

The claims "arise organically, while most static factuality benchmarks are artificially curated." Because the system decomposes complete responses rather than scoring short answers, it also catches errors buried inside long replies that read as authoritative. And because factuality is fused with human preference, a model "cannot top the composite leaderboard merely by refusing to provide an informative answer or by producing a polished response containing inaccurate claims."

Any benchmark that becomes influential becomes a target. Asked how it avoids being overfit to, Arena pointed at its inputs.

"The Arena leaderboard is inherently resistant to gaming because we rely on fresh, organic user conversations rather than static test sets," the company said. "On the factuality side, a model cannot simply 'overfit' to the leaderboard; it must consistently produce claims on unseen, unpredictable prompts that satisfy multiple independent, search-based verifiers."

Because those verifiers check claims against the live web rather than a fixed answer key, Arena argues it is "effectively impossible for a model to game the system through memorization or predictable patterns."

Arena expects factuality to move from a public ranking into the tooling enterprises actually buy. The company said it will keep "integrating it into our enterprise evaluation products and embedding it into agentic workflows, as well as other evaluation methodology behind our public leaderboards."

That is the part worth watching for practitioners. Overall Arena rankings already shape which models teams trial. If factuality scores follow the same path, the gap between a model people enjoy using and a model that is right more often stops being an academic distinction and starts showing up in procurement.

The full methodology is in Arena's factuality launch post.