FinTrace Evaluates LLM Tool-Calling for Financial Tasks

FinTrace introduces a trajectory-level benchmark and training corpus for evaluating LLM tool-calling in long-horizon financial tasks. The benchmark includes 800 expert-annotated trajectories spanning 34 real-world financial task categories and a rubric of nine metrics grouped into four axes to capture action-level and outcome-level behavior. The authors evaluate 13 LLMs and find that while models often choose correct tools, they fail to reliably use tool outputs to produce high-quality final answers. The paper also releases FinTrace-Training, a trajectory-level preference dataset with 8,196 curated trajectories, and demonstrates fine-tuning of Qwen-3.5-9B with supervised fine-tuning then DPO to improve intermediate reasoning metrics.

The rubric splits evaluation into four axes to measure different failure modes. Key axes include:

• •Action correctness: whether the model calls the right tools at the right time
• •Execution efficiency: how economically the model sequences calls and avoids redundant actions
• •Process quality: the model's intermediate reasoning and use of tool outputs
• •Output quality: final answer accuracy and completeness

The authors annotate full trajectories rather than isolated calls, enabling trajectory-level preference labels. They construct FinTrace-Training with pairwise preferences over tool-augmented contexts and apply supervised fine-tuning followed by DPO on Qwen-3.5-9B. DPO notably reduces common failure modes, improving intermediate metrics more than raw end-to-end answer quality.

Tool calling and agentic LLM workflows are converging on long-horizon, multi-step domains such as finance, where chaining external tools is necessary but error-prone. Existing benchmarks focused on call-level correctness and short scenarios; FinTrace fills the gap by measuring trajectory-level reasoning and process fidelity. The dataset and preference pairs make this a practical resource for researchers and engineers who need to align agent behavior over extended sequences rather than single API calls. The finding that trajectory-level gains do not fully translate into final answer improvements highlights a recurring problem: bridging process improvements and end-to-end utility remains a core research frontier.

Apply trajectory-level diagnostics when developing tool-augmented agents and consider DPO over curated trajectories to suppress brittle behaviors. The crucial open question is how to propagate intermediate reasoning gains into consistent final-answer quality, a research direction likely to attract follow-up work and alternative alignment techniques.