LangSmith Evaluates Deep Agents Using AWS Bedrock

Per the AWS blog post "Evaluating Deep Agents using LangSmith on AWS," Amazon and LangSmith published a hands-on guide co-authored with Karan Singh that demonstrates agent evaluation patterns and tooling. The post states it combines learnings from LangChain and Anthropic, and it presents five evaluation patterns for deep agents, a walkthrough to build offline evaluations with pytest and LangSmith, and instructions to configure online monitoring for production. The walkthrough uses a text-to-SQL agent and, according to the post, runs on Nova 2 Lite via Amazon Bedrock. The blog describes Nova 2 Lite as supporting configurable budget levels, multimodal inputs, and a 1,000,000-token context window.

Per the post, the evaluation structure relies on three core artifacts: tasks (defined inputs and success criteria), trials (multiple attempts per task), and transcripts (full traces of tool calls, reasoning steps, and intermediate state). The post recommends using LangSmith to capture transcripts and graders to compute outcomes. The walkthrough shows how to convert a trial into a reproducible pytest test and how to persist graded outcomes for both offline debugging and production monitoring.

Industry-pattern observations: agent evaluation requires capturing execution traces and outcomes across tool chains because nondeterministic model outputs and tool interactions create cascading failures. Standardizing transcripts and using dedicated graders, as the AWS post demonstrates, reduces cognitive load during debugging and supports reproducible regression tests. Leveraging an evaluation platform that stores structured traces is increasingly common in agent engineering workflows.

This guide joins growing prescriptive material from platform vendors and open-source projects that operationalize agent testing and monitoring. For practitioners, the combination of offline pytest-driven checks plus online LangSmith monitoring maps to established CI/CD habits and extends them into agent validation.

Observers should watch for broader adoption of transcript-first evaluation patterns, integrations between evaluation platforms and CI systems, and additional vendor guidance on grader design and outcome metrics. The AWS post does not provide customer results or comparative benchmarks for Nova 2 Lite in agent settings.