Why AI Agent Costs Get Out of Hand
AI agents are no longer isolated experiments—they're production workloads spanning teams, data sources, and tools. But traditional cost management wasn't built for this. Billing data arrives after the fact; an agent stuck in a retry loop doesn't wait for the next budget alert.
The core problem: you can't govern what you can't see, and you can't optimize what you can't attribute. Without consistent governance, each team picks its own models, tools, and capacity limits—and small inefficiencies compound across every agent and every turn.
Effective cost governance rests on three pillars: seeing the spend, bounding it, and proving the return.
Pillar 1: See the Spend Where It Starts
Aggregate invoices hide context. One deployment may serve multiple agents; one agent may use several models and tools. By the time costs hit the invoice, the business context is gone.
Microsoft Foundry brings cost visibility closer to the systems generating it:
- Project-level cost attribution — every Foundry project is tagged, so FinOps can filter Cost Analysis by business unit, team, or workload.
- AI Gateway token metrics via Azure API Management — emitted by API, product, user, subscription, gateway, and backend.
- Tracing captures tool usage, retries, latency, token consumption, and costs per agent run.
# Example: Querying cost attribution by project tag (conceptual)
from azure.identity import DefaultAzureCredential
from azure.mgmt.costmanagement import CostManagementClient
client = CostManagementClient(DefaultAzureCredential())
# Filter costs by Foundry project tag
query = {
"type": "ActualCost",
"timeframe": "MonthToDate",
"dataset": {
"granularity": "Daily",
"filter": {
"tags": {
"name": "foundry-project",
"operator": "In",
"values": ["my-agent-project"]
}
}
}
}
result = client.query.usage(scope, query)
Observability signals explain not just how much an agent consumed, but why: traces reveal model calls and retries; evaluations measure quality and safety; monitoring surfaces anomalies. Together they distinguish demand-driven costs from inefficiencies.
Pillar 2: Set Spend Limits at Every Layer
Visibility tells you where money went. Limits determine whether it can keep going. Three layers work at different scopes and speeds:
Layer 1: Enforce limits in Foundry
With AI Gateway configured, Foundry Control Plane enforces tokens-per-minute rate limits and total token quotas at project scope. Exceeding rate limits returns 429 Too Many Requests; exhausting quotas returns 403 Forbidden. Enforcement happens in the request path—unlike cost alerts, which react to billing data.
Layer 2: Apply policy across models and providers
The llm-token-limit policy caps consumption per key (subscription, team, workload identity) using rate, quota, or both. AI Gateway applies this uniformly across OpenAI-compatible APIs, Anthropic Messages API, MCP servers, and agent-to-agent APIs. Backend load balancing prioritizes provisioned capacity before spilling to pay-as-you-go; circuit breakers stop requests to failing backends.
Layer 3: Use financial budgets for accountability
Microsoft Cost Management budgets use billing data (actual prices, credits, commitments) to give finance an authoritative view. Budgets trigger notifications and can invoke Azure Monitor action groups (tickets, Logic Apps, runbooks).
Critical distinction: budgets are not instant spending caps—they respond after consumption. Token limits operate earlier, in the request path. You need both.
| Control | Unit | Speed | Purpose |
|---|---|---|---|
| Token limits | Tokens | Real-time (request path) | Contain consumption |
| Financial budgets | Dollars | Post-consumption (billing) | Accountability & escalation |
⚠️ Limitation: Counters are maintained independently at each gateway, and concurrent requests can cause small temporary overages. The goal is a predictable operating boundary, not perfect precision.
Pillar 3: Measure the Value the Agent Creates
Capping consumption solves only half the problem. The real question: is the agent delivering enough business value to justify the investment?
The cheapest agent isn't necessarily the best. An agent that costs more but resolves substantially more cases may deserve more capacity. ROI for agents in Foundry (private preview) connects costs to outcomes:
- Value generated — total value attributed to successful outcomes
- Total cost — model and tool costs incurred
- Net value — value minus cost
- ROI — return relative to investment
Because ROI links to observability data, teams can jump from a business metric straight to the traces, evaluations, and operational signals explaining the cost. A low-ROI trace might reveal an oversized model, repetitive tool calls, or bloated context.
The Bigger Picture: One Managed Investment System
Agent optimization runs at three speeds: runtime (routing, caching), workflow (context engineering, memory), and continuous (governance, attribution, ROI). The same telemetry serves every layer:
- Traces → what the agent did
- Evaluations → whether output was good
- Cost attribution → where money went
- ROI → whether it was worth it
Who owns what: Foundry is for developers building and optimizing agents; Microsoft Agent 365 is for IT and security governing the enterprise estate.
Next Steps
- Make consumption visible and attributable.
- Apply request-time token limits.
- Pair with financial budgets and alerts.
- Connect cost to business outcomes.
📚 Further reading: For a broader view of how Microsoft is unifying data and AI infrastructure, see our analysis of Microsoft's 2026 database and Fabric vision. And if you're curious how top teams are rethinking developer experience with AI agents, don't miss the Spotify developer experience deep dive.
Source: The Economics of Agent Optimization — Microsoft Azure Blog