NVIDIA NeMo Integrates Docker Model Runner for Observability

NVIDIA NeMo Agent Toolkit is demonstrated running with the Docker Model Runner (DMR) to add observability to agent-based systems, using the small language model ai/smollm2 served locally via docker model run ai/smollm2. The walkthrough highlights key setup steps, including enabling TCP access in Docker Desktop and installing the Python package nvidia-nat with uv pip install nvidia-nat, plus an example agent-run.yaml that wires NeMo agents to a DMR endpoint.

The integration relies on three practical components: a local inference endpoint provided by Docker Model Runner, NeMo Agent Toolkit configuration via YAML, and the nvidia-nat runtime package for stable dependency resolution. Key implementation notes:

• •Ensure Docker Desktop TCP access is enabled so the NeMo agent can reach the DMR endpoint over localhost.
• •Launch the model with docker model run ai/smollm2 and point the agent config base_url to that endpoint.
• •Use an agent-run.yaml to declare tools (for example a wiki_search tool), LLM bindings (openai_llm mapped to ai/smollm2), and API keys or local base URLs.
• •Install nvidia-nat via uv pip install nvidia-nat rather than plain pip to avoid timeouts observed in the tutorial.

Agent observability has lagged behind rapid adoption of multi-agent frameworks. By combining NeMo observability primitives with a portable, local inference stack like Docker Model Runner, teams get a single-pane view into tool calls, reasoning traces, and coordination signals without immediately moving to cloud-hosted infra. The same NeMo features appear in tooling like Unsloth Studio, which uses NeMo in data recipes and exposes training and run-time telemetry. This pattern lowers the activation cost for reproducible agent debugging and evaluation during prototyping and pre-production validation.

Validate latency and throughput when you move from single-model local runs to multi-agent, multi-model topologies. Next steps include adding structured tracing, standardized metrics export, and running red-team scenarios to exercise edge-case observability coverage.