LLM agents learn to ask under unclear instructions

The arXiv paper arXiv:2409.00557 (last revised as v4 on 29 Apr 2026) introduces a focused study of LLM tool use when user instructions are imperfect (arXiv). Per the paper, the authors collect and analyze real-world user instructions to build a challenging benchmark named NoisyToolBench and to surface common error patterns when agents call external tools (arXiv). The paper reports that, under unclear instructions, LLMs trained with a next-token prediction objective tend to arbitrarily generate missing arguments, which can produce hallucinations and downstream risks (arXiv). To mitigate this, the authors propose a prompting framework called Ask-when-Needed (AwN) that directs agents to ask clarification questions when they encounter obstacles due to unclear instructions (arXiv). The paper also presents ToolEvaluator, an automated evaluation pipeline designed to measure both accuracy and efficiency of agent-tool interactions, and reports experimental results showing that AwN significantly outperforms existing tool-learning frameworks on NoisyToolBench (arXiv). The OpenReview/DBLP revision metadata lists the paper authors as Wenxuan Wang, Juluan Shi, Chaozheng Wang, Cheryl Lee, Youliang Yuan, Jen-tse Huang, and Michael R. Lyu (OpenReview/DBLP).

The paper isolates a causeable mismatch: a next-token prediction objective encourages an agent to fill missing arguments rather than explicitly query the user. This aligns with broader observations in the literature that generative training objectives can prioritize plausibility over correctness when instructions are underspecified. Industry and academic work on agent clarification has explored related ideas such as clarification questioning, uncertainty-aware prompting, and constrained decoding; AwN fits into that space by operationalizing a prompting-level policy for when to solicit clarifications. The introduction of an automated ToolEvaluator matters because evaluation of interactive behaviours often relies on costly human-in-the-loop metrics; an automated scorer that balances correctness and interaction cost can accelerate iteration and reproducibility in research on interactive agents (Editorial analysis: technical implications).

Benchmarks that model realistic instruction noise help shift evaluation from synthetic toy failures to field-relevant errors. For practitioners building tool-augmented agents, the combination of a noise-focused benchmark (NoisyToolBench), a clarification policy (AwN), and an automated evaluator (ToolEvaluator) creates a compact ecosystem for testing the trade-off between asking questions and acting promptly. In the research canon, the claim that clarification reduces hallucination risk contributes an empirical data point to the debate on when to prefer conservative questioning versus aggressive action. The paper's stated intent to release code and datasets (arXiv) increases the likelihood that others will reproduce, stress-test, and extend the evaluation suite (Editorial analysis: significance).

• •Whether the authors publish code and the benchmark artifacts as stated in the arXiv abstract, enabling community adoption (arXiv).
• •How AwN compares to alternative clarification strategies under different cost models: e.g., user latency, annotation cost, or multi-turn tool workflows (Editorial analysis: evaluative metrics).
• •Extensions of NoisyToolBench to domain-specific tools (APIs, databases, retrieval systems) and the community's adoption of ToolEvaluator as a de-facto metric (Editorial analysis: adoption signals).

For teams deploying LLM agents with tool calls, this paper provides a concrete benchmark and a tested prompting approach focused on clarification. Even if AwN requires adaptation to domain constraints, the combination of a noise-oriented benchmark and an efficiency-aware evaluator offers a reproducible way to measure whether clarification policies meaningfully reduce hallucinations while keeping interaction overhead acceptable (Editorial analysis: practitioner implications).