Nimble tells LDS when its own web search agents are the wrong tool

Two things are worth holding in mind before those numbers do any work.

The comparison is against general-purpose assistants rather than against the tools Nimble actually competes with. Beating ChatGPT and Claude at structured web research is a different and easier claim than beating a dedicated agent-search API such as Exa, Tavily, Firecrawl or Brave Search, none of which appear in Nimble's benchmark. An independent comparison by AIMultiple tested eight search APIs across 100 real-world queries and found the leading four statistically indistinguishable from one another. Nimble was not among the products tested.

And a 21-point gain is unanchored without a scale. After publication we asked Nimble for the methodology. The company told Lets Data Science that the benchmark was run by one of its customers, that the customer will not disclose how the test was constructed, and that it requires its identity stay confidential. So the headline figures are not a vendor benchmark that can be inspected, and there is no way for a reader, or for us, to check them. Treat them as one unnamed customer's internal result rather than a measurement.

The Rox figure is better supported than most reference-customer numbers. Rox is an AI-native CRM valued at roughly $1.2 billion, backed by Sequoia and General Catalyst, with customers including Ramp, MongoDB and New Relic. It is a real production workload rather than a friendly logo.

"Generic web search tools are task-agnostic by design, with no ability to optimize retrieval around what an agent is actually trying to accomplish," Knorovich told Lets Data Science.

Inside an agent loop that compounds. "Agents wade through noisy, unstructured results, spend excessive tokens parsing irrelevant content, and still return low-quality answers." His conclusion is blunt: "Generic web search has become the weakest link in the agentic stack."

The first is that retrieval is task-specific, so less gets read at all. "Most of the wasted work in generic search is reading the wrong pages, and we remove it."

The second is reach. Much of the web sits behind heavy JavaScript, anti-bot defenses and rate limits, where generic tools "either return blank pages or get blocked outright." Nimble runs headless browsers, stealth handling and a large proxy network so that "the agent works from complete inputs instead of the easy-to-reach fraction of a source."

The third is memory. Every run stores its outputs, sources and search history in a private index, recording which retrieval paths produced the right data. "After the first few runs, the agent already knows where your information lives, so it crawls those sources deeper and skips the noisy ones."

That third mechanism carries the implication Nimble does not state: unit costs should fall as usage grows, because the system keeps narrowing its search surface.

The most valuable answer was the one Knorovich gave unprompted about the limits.

"Where it does not hold: when the answer is widely held information and the use case needs ultra-low latency, a research agent is the wrong tool," he said. "Those queries do not benefit from domain-specific retrieval or synthesis, and the reasoning step only adds latency."

Nimble points those customers at its simpler Search API instead. Web Search Agents are built for "the complex, multi-step research where relevance has to be defined rather than assumed."

Take that boundary seriously when evaluating agentic retrieval generally. The benchmark describes hard multi-step research. Point the same machinery at a question an index answers directly and it will cost more and take longer, which no vendor benchmark will tell you.

On the Rox result, Knorovich argued generic setups pay twice, "once to run redundant searches hunting for the right information, and again to have an LLM parse raw pages into something usable. We cut both." Savings therefore widen with volume, so "the teams running the highest volumes tend to see the largest reductions."

The change is narrow rather than architectural: "You swap it in at the API layer without re-architecting your agent, keep your existing framework, and get the auditability, the Search Plans and the private index that you need to actually trust it in production."

Nimble is available via API, SDK and MCP.