Jedify's CTO tells LDS where its token benchmark stops short
Adi Elimelech, co-founder and chief technology officer of Jedify, answered Lets Data Science in writing about the company's Context Graph benchmark, published today. Asked about the 26 failed runs the report left unattributed, the company finished the attribution: 20 were entity-selection misses on genuinely ambiguous questions and 6 were SQL assembly errors in the layer Jedify describes as deterministic. Elimelech also confirmed on the record that the first version of the report described its central token figure incorrectly, and, at our prompting, re-ran part of the benchmark on an open-weight model, measuring an accuracy cost of two to five percentage points. The headline claims remain company-reported, and the baselines come from other studies rather than a same-warehouse comparison, which the report itself states.
The most interesting number in a vendor benchmark is rarely the headline. In Jedify's Context Graph study, published today, it is the 26 runs that failed, and when Lets Data Science asked the company to account for them, it did something unusual: it went back and measured.
"We took this question as a prompt to finish the attribution rather than answer around it," Adi Elimelech, co-founder and chief technology officer of Jedify, told Lets Data Science in written answers. Twenty of the 26 trace to entity selection, questions whose phrasing "legitimately resolve[s] to more than one entity in the graph, where a human analyst would also have paused to ask a clarifying question." The remaining six were SQL assembly errors inside a correctly selected entity, which Elimelech called "the ones I like least, because they touch the layer we describe as deterministic." They are 3 percent of runs, and "they get engineering attention first."
What the benchmark claims
The study ran 100 business questions against a live production data warehouse, each twice, for 200 graded runs across three complexity tiers. The context graph approach, which pre-encodes business logic instead of feeding raw schema to the model, answered 87 percent correctly while averaging 25,036 tokens per SQL-generation call.
The comparisons are where care is needed, and to the report's credit it says so itself. The baselines, 50,000 to 150,000 tokens per call at 60 to 70 percent accuracy for schema injection, and roughly 339,965 tokens per request for the CHESS multi-agent system, come from published research on different schemas and question sets, not from a same-warehouse comparison. The release also states its Token ROI multiple twice, once as roughly 4X over schema injection and 18X over naive multi-agent pipelines, and once as 4.6X and 20X.
Because the benchmark ran non-interactively, ambiguous questions were committed rather than clarified. In production, Elimelech argues, the system would surface a clarifying question first, "which makes the 87% a floor relative to interactive use, not a ceiling." That is a reasonable argument, and it is also the kind of claim the next benchmark should measure rather than assert.
A correction, on the record
Two versions of the report circulated before publication, and the same figure, 25,036, was described as an effective count after prompt caching in the first and a raw uncached count in the revision. We asked which was wrong.
"The description was wrong; the number was never in question," Elimelech said. The figure comes straight from API logs, with no caching discount ever applied; the first draft glossed a field name with an incorrect explanation, and review caught it. The direction of the error favours the reader: Jedify's figure includes output tokens, while the schema-injection baselines count schema input alone. On what caching does to the number in live deployments: "We would rather measure it than estimate it in a Q&A." A cache-parity study is the committed follow-up.
The experiment we asked for
The report projects that about 85 percent of enterprise analytics queries could route to cheaper open-source models once context is properly scoped, leaning on dbt's finding that model choice matters less once context is right. We asked whether that was measured or projected, and the answer arrived with an experiment attached: Jedify re-ran the benchmark's simple and moderate tiers on an open-weight model and measured an accuracy hit of two to five percentage points against the frontier model. The complex tier has not been run.
"One reconciliation I owe you," Elimelech wrote: the next revision of the report will carry the measured number rather than the architectural argument. The honest status of the 85 percent figure is "measured for the bulk of query types, still projected for the hardest ones," with a commitment to publish the complex-tier result "if the hit there is larger."
The costs the token math hides
A context graph moves work from inference time to construction time, so we asked what the benchmark warehouse actually cost to encode. The graph contains 35 entities and took roughly three days to build and validate, drafted from query-log history and existing BI definitions rather than interviews. "Maintenance is where semantic layers usually die, so we built for it," Elimelech said: a living mode watches connected sources for schema and metric drift and surfaces changes for human review, so a metric redefinition is an edit to one entity rather than a rebuild. Whether that holds at enterprise churn is the number a buyer should watch, and the company at least publishes its build cost rather than hiding it.
There are also things Jedify plainly does not know. Asked what share of real user questions falls outside the graph in production, Elimelech declined to produce a figure: "I would rather explain why than invent one," since coverage depends on deployment age. And asked what users do after the system refuses an out-of-coverage question, the company conceded it does not publish that data: "That is a fair gap in our instrumentation, and a fair question."
Why this matters beyond one vendor
The report's scale argument rests on independent research: real enterprise warehouses average 101.5 tables and 869.4 columns per the BEAVER benchmark, and GPT-4o falls from 86.6 percent on the six-table Spider 1.0 to 10.1 percent on the enterprise-derived Spider 2.0. Jedify's own measurement is that its token gap versus raw schema injection is about 50 percent at 100 tables and beyond 75 percent at 200.
This is the second time in a week a company has shown us the same mechanism from a different layer. JFrog's CTO described agents drowning in terminal noise; Jedify's argument is that the model should not re-derive business logic on every query. Different layers, one lesson: the expensive tokens are the ones the model did not need. What distinguishes this benchmark is not the headline percentage but its behaviour under interrogation, and a study that gets more precise when questioned is the version of vendor research worth reading.
Key Points
- 1Asked to attribute its 26 failed runs, Jedify measured rather than estimated: 20 trace to entity selection on questions its CTO says legitimately resolve to more than one entity, and 6 were SQL assembly errors inside the layer the company describes as deterministic. Because the benchmark ran non-interactively, Elimelech argues the 87 percent is a floor relative to interactive use.
- 2The report's central figure, 25,036 raw tokens per SQL-generation call, was described wrongly in the first version as a post-caching number. "The description was wrong; the number was never in question," Elimelech told LDS. The figure is raw and uncached and includes output tokens, while the schema-injection baselines it is compared against count schema input alone, and come from other studies, not a same-warehouse test.
- 3At our prompting, Jedify re-ran the benchmark's simple and moderate tiers on an open-weight model and measured a two-to-five point accuracy hit versus the frontier model, with the complex tier not yet run. The status of the claim that 85 percent of queries can route to cheaper models: "measured for the bulk of query types, still projected for the hardest ones."
Scoring Rationale
Written answers provided directly to Lets Data Science by the co-founder and CTO who built and ran the benchmark, including a failure attribution finished at our prompting, an on-the-record correction of the report's own token-figure description, and a new open-weight experiment run because we asked for it. Token economics of text-to-SQL architectures is a live cost problem for the data teams in our audience.
Sources
Original reporting, with the public references used alongside it.
LDS Exclusive
Reporting based on written answers given directly to Let's Data Science by Adi Elimelech, co-founder and CTO, Jedify.
View 1 more source
Practice interview problems based on real data
1,625 SQL & Python problems across 15 industry datasets — the exact type of data you work with.
Try 250 free problems
