Databricks Reports 63.3% Win at Grounded Reasoning Cup
Databricks published results from its June 17 Grounded Reasoning Cup, where Stanford's agent won with 63.3% accuracy on a held-out benchmark built from about 120,000 pages of U.S. Treasury documents. The average team scored about 41%, showing that retrieval, document parsing, verification and latency engineering still materially shape agent performance.
Databricks published a technical recap on August 18 of the inaugural Grounded Reasoning Cup, a live AI-agent competition held on June 17 at its Data + AI Summit. Eleven university teams developed systems on the OfficeQA benchmark, then had to apply them to OfficeQA Pro V2, a held-out collection built from about 120,000 pages of U.S. Treasury documents.
A live test of generalization
Stanford's team won with 63.3% accuracy. Databricks says that was roughly 22 percentage points above the average team and about 35 points above its average offline frontier-agent baseline. The average team scored about 41%, while 18.8% of questions were not solved by any team. UMass Amherst placed second; its Center for Intelligent Information Retrieval separately confirmed the result and the June 17 competition date.
The format put pressure on more than answer quality. Teams completed six 15-minute rounds with 15 questions in each round, earned a speed bonus for the first correct response, and faced double-value questions in the final round. Systems had to retrieve evidence, interpret tables and scanned documents, perform calculations and return answers quickly enough to compete.
The surrounding system mattered
Databricks reports a 30.4-point average gap between the highest- and lowest-scoring teams using the same model family. That result does not isolate a model-quality effect; it points to the importance of the surrounding agent design. Stronger entries combined document preprocessing, targeted retrieval, structured tool use and answer verification rather than relying on a single model call.
Stanford prepared more than 100 reusable skills around recurring failure modes and used fallbacks to source PDFs when parsed representations lost context. It also adjusted verification during the event: removing an extra pass to reduce latency, then restoring it for a decisive final-round correction.
For practitioners, the result is a useful warning against treating a benchmark score as a property of the underlying model alone. Held-out corpora can expose brittle parsing and retrieval choices, while live constraints reveal trade-offs between verification and response time. The competition is one vendor-run evaluation rather than a universal ranking, but its published setup and error patterns make the systems lesson concrete: reliable grounded reasoning depends on the full pipeline and still has substantial headroom.
Key Points
- 1Stanford won the June 17 live competition with 63.3% accuracy on OfficeQA Pro V2, about 22 points above the average team.
- 2Databricks reported a 30.4-point average spread between the best and worst teams using the same model family.
- 3Document preprocessing, targeted retrieval, tool use, verification and latency choices materially affected end-to-end performance.
Scoring Rationale
The held-out live evaluation provides concrete system-level evidence about agent generalization and reliability. Its practitioner value is high, while the score is moderated because it is a vendor-run competition rather than a peer-reviewed universal model comparison.
Sources
Primary source and supporting public references used for this report.
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
