Why Long-Running Agents Are Eating Your Budget
Single-turn chatbots are evolving into long-running agents that reason, maintain context, use tools, and invoke sub-agents. But as these multi-agent workflows scale, token counts explode. Every plan, tool call, sub-agent invocation, and reasoning step is fed back into the model, increasing both latency and cost. The result: goal drift and ballooning inference bills.
NVIDIA’s answer is a system-of-models approach: use frontier reasoning models for orchestration and complex planning, and efficient models for high-volume execution. Today, they’re releasing Nemotron 3 Ultra—an open 550B-parameter Mixture-of-Experts model with 55B active parameters, purpose-built for exactly this split.
Benchmark Showdown: Nemotron 3 Ultra vs. Qwen3.5, Kimi K2.6, GLM 5.1
NVIDIA published a comprehensive benchmark table across agent productivity, long-horizon planning, coding, instruction following, knowledge work, and long-context recall. Here’s the data:
| Benchmark | Nemotron 3 Ultra (550B) | GLM 5.1 (744B) | Kimi K2.6 (1T) | Qwen3.5 (397B) |
|---|---|---|---|---|
| Agent Productivity PinchBench | 91% | 84% | 91% | 89% |
| Long-horizon Planning EnterpriseOps-Gym | 33% | 40% | 29% | 30% |
| Coding Terminal-Bench 2.0 | 54% | 64% | 67% | 53% |
| Instruction Following IFBench | 82% | 77% | 74% | 78% |
| Knowledge Work GDPVal-AA | 1,448 | 1,594 | 1,508 | 1,192 |
| Professional Work ProfBench (Search) | 56% | 46% | 56% | 53% |
| Long Context Ruler @1M | 95% | N/A (max 256K) | N/A (max 256K) | 90% |
Key takeaways:
- Nemotron 3 Ultra ties Kimi K2.6 on agent productivity (91%) and professional work (56%), while leading on instruction following (82%) and long-context recall (95%).
- It achieves 5x higher throughput compared to other open models in its class, thanks to NVFP4 precision and multi-token prediction.
- On SWE-bench and Terminal-Bench 2.0, it uses fewer total tokens and fewer tokens per turn, lowering agentic task costs by up to 30%.
What Makes This Possible? Architectural Innovations
NVIDIA packed five key innovations into Nemotron 3 Ultra:
- Hybrid Mamba + Transformer: Mamba layers improve sequence efficiency for long-context workloads; Transformer layers preserve precise recall for fact retrieval.
- NVFP4 Precision: One checkpoint runs on Hopper, Blackwell, and Ampere GPUs. Delivers up to 5x higher throughput per GPU compared to BF16 on Blackwell.
- LatentMoE: More efficient expert routing for workflows spanning reasoning, code generation, tool calls, and domain-specific logic.
- Multi-Token Prediction (MTP): Predicts multiple future tokens in a single forward pass, improving throughput for long outputs and multi-turn workflows.
- Multi-Teacher On-Policy Distillation (MOPD): Over 10 specialized teacher models provide dense reward signals during training, enabling continuous capability improvement across domains.
Caution: Where Nemotron 3 Ultra Falls Short
No model is perfect. Here are the limitations you should consider before adopting Nemotron 3 Ultra:
- Long-horizon planning is weak: At 33% on EnterpriseOps-Gym, it trails GLM 5.1 (40%) significantly. If your agent needs to plan across dozens of steps, this may not be the best choice.
- Coding is mediocre: Terminal-Bench 2.0 score of 54% is below Qwen3.5 (53%? actually Qwen is 53%, but GLM 64% and Kimi 67% are higher). For code generation tasks, consider pairing with a specialized coding model.
- Knowledge work is average: GDPVal-AA score of 1,448 is lower than GLM 5.1 (1,594) and Kimi K2.6 (1,508). For knowledge-intensive enterprise tasks, you might need a different backbone.
- Hardware requirements: 550B total parameters means you need significant GPU memory. Even with NVFP4, running locally on consumer hardware is not feasible. Cloud deployment is recommended.
How to Get Started
Nemotron 3 Ultra is fully open—weights, data, and recipes—available under the OpenMDW-1.1 license. You can:
- Download weights from Hugging Face
- Try it on Perplexity Pro, OpenRouter, or Anaconda
- Deploy as a NIM microservice via NVIDIA’s partner ecosystem (AWS SageMaker, Google Cloud, Microsoft Foundry, Oracle Cloud)
- Fine-tune using LoRA, SFT, or RL with NVIDIA NeMo libraries
For a complete walkthrough, check the Nemotron 3 Ultra cookbook and the official GitHub repository for agent harness integrations (BlackBox AI, Cline, CrewAI, LangChain, OpenHands, and more).
Conclusion: Should You Use Nemotron 3 Ultra?
If you’re building long-running, multi-agent workflows and care about cost efficiency, Nemotron 3 Ultra is a strong contender. Its 5x throughput improvement and up to 30% cost reduction make it a practical choice for production agent systems. However, if your use case demands strong long-horizon planning or top-tier code generation, you may want to pair it with specialized models.
Next steps for learning:
- Read the Nemotron 3 Ultra technical report for full architectural details
- Experiment with the MOPD training recipe using NeMo-RL
- Explore the Netflix Data Canary pattern for validating data pipelines in production
- See how Spotify Honk automated 240 dataset migrations using Backstage fleet management
One more thing: While Nemotron 3 Ultra is impressive, always benchmark against your specific workload. The best model for your agent is the one that balances accuracy, latency, and cost for your unique use case.