LLM budget limits at the org level don't stop one rogue key from draining your account. Learn how to enforce per-key spend caps across LiteLLM, Portkey, MLflow, and agentgateway.
OpenAI, Anthropic, and Google all let you set a spending cap on your organization account. That cap applies to the entire org. It does not distinguish between your production service, your intern's experiment, or the batch job someone left running over the weekend.
A misconfigured prompt, an unexpected traffic spike, or a forgotten development endpoint can burn through thousands of dollars before anyone notices. The org-level cap might eventually stop the bleeding, but by then the damage is shared across every team and service drawing from the same account.
Per-key LLM budget limits solve this by putting a ceiling on each API key independently. The enforcement happens not at the provider, but at the gateway layer sitting between your applications and the model APIs.
The problem is straightforward. In a multi-tenant application or a company with multiple teams, one heavy user running a batch job at 3am can exhaust the provider budget before the rest of your users wake up. Without per-customer or per-key limits, one consumer affects everyone.
Org-level caps also give you no forensic granularity. When you hit your ceiling, you know the org overspent. You do not know which key, which team, or which service was responsible. That makes post-incident analysis slow and the fix ("raise the cap and hope") unreliable.
Four distinct limit types show up in production LLM applications:
Most teams implement request rate first. They add token rate after their first surprise invoice, and budget caps after their second. That sequencing is predictable but expensive. Setting all four from the start, especially the budget cap, costs nothing and prevents the learning-by-invoice cycle.
Standard request-count rate limiting breaks down for LLM APIs. A single API call with a 200,000-token context window costs as much as 50 calls with 4,000-token prompts. Request-count limits do nothing to prevent a single large-context call from consuming an entire daily budget.
Token bucket rate limiting addresses this. Each key gets a virtual budget measured in tokens rather than requests. The bucket refills at a defined interval, and when it empties, the gateway returns a 429. This maps directly to cost, which is what you are trying to control.
If you are working on reducing the tokens themselves, caching and token optimization strategies compound well with per-key limits. Fewer tokens consumed per request means each key's budget stretches further.
LiteLLM builds budget enforcement into a hierarchy. You can set limits at the global proxy, team, team member, internal user, virtual key, model-specific, agent, and customer levels.
The precedence rule matters: if a key belongs to a team, the team budget is applied, not the user's personal budget. This means a team of five each holding personal $100 budgets will still be capped at whatever the team limit is when they use team-scoped keys.
Budget periods are flexible. The budget_duration parameter accepts seconds, minutes, hours, or days ("30s", "30m", "30h", "30d"), and spend resets automatically at the end of each period.
For production environments where overages are unacceptable, LiteLLM offers fail_closed_budget_enforcement. This validates spend against the authoritative database even when Redis is degraded, ensuring the budget ceiling holds under infrastructure failures. The tradeoff: LiteLLM requires a PostgreSQL database (Supabase, Neon, or similar) to enable any budget tracking at all.
Portkey takes a different approach to LLM budget limits. When creating or editing an API key, you choose between two limit types: cost in USD (minimum $1) or a maximum token count.
The alert threshold feature is useful operationally. You configure a threshold that sends notifications before the full budget is hit, while the key keeps working until the hard limit. An 80% alert threshold on a $50 key means you get an email at $40 spent and a hard stop at $50.
Reset schedules are fixed to three options: no reset, weekly (Sunday midnight UTC), or monthly (1st of each month midnight UTC). Rate limits layer on top with per-minute, per-hour, or per-day intervals.
One constraint to know: these features are available to Enterprise customers and select Pro users. Free-tier keys do not get budget controls.
MLflow's AI Gateway introduces budget policies with two distinct enforcement modes. A policy defines a spending threshold in USD over a recurring time window. When cumulative spend crosses that threshold, the gateway either fires a webhook notification while allowing requests to continue (alert mode) or blocks all subsequent requests with an HTTP 429 (reject mode).
Alert mode is designed for visibility without disruption. The alert fires once per window, keeping notification channels clean. Pipe the webhook into Slack or PagerDuty and a human decides whether to intervene for the remainder of the period.
For per-team enforcement, budget policies can be scoped to specific workspaces, giving each team or project its own spend ceiling.
The deployment topology matters here. MLflow offers two tracking strategies: a local tracker that runs in-process with no external dependencies, and a Redis tracker that shares state across all gateway replicas. The local tracker works for single-instance development. Production deployments running multiple replicas need Redis to enforce budgets globally rather than per-instance.
agentgateway, designed for Kubernetes environments, enforces budgets at the route level. You can apply different budgets to different routes, such as 200,000 tokens per day for production and 50,000 tokens per day for development.
Two caveats worth flagging before you configure it:
Local rate limiting counts requests, not tokens. The tokens field in local configuration is request count, not LLM token count. With tokens: 10000, you get roughly 10,000 requests regardless of how many LLM tokens each consumes. For actual token-based budgets, you need global rate limiting with unit: Tokens descriptors.
Multi-instance multiplication. Limits apply per agentgateway instance. Three instances with a 100,000-token limit yield 300,000 effective tokens across the cluster. Plan your per-instance numbers accordingly.
There is also an evaluation order subtlety: rate limiting is evaluated before prompt guards, so requests rejected by content safety checks still consume the user's token budget. Authentication, however, is evaluated before rate limiting, so unauthenticated requests do not eat quota.
| Feature | LiteLLM | Portkey | MLflow AI Gateway | agentgateway |
|---|---|---|---|---|
| Budget unit | USD (via token cost) | USD or token count | USD | Tokens or requests |
| Scope granularity | Global, team, user, key, model, agent | Per API key | Per workspace | Per route |
| Reset periods | Seconds to days (custom) | None, weekly, monthly | Custom recurring window | Custom time window |
| Alert before hard cap | No (hard enforcement) | Yes (configurable threshold) | Yes (webhook, once per window) | No |
| Multi-replica enforcement | Postgres + optional Redis | Managed (cloud) | Local or Redis tracker | Local per-instance or global |
| Hard fail mode | fail_closed validates against DB | Requests rejected at limit | HTTP 429 (reject mode) | 429 when bucket empty |
| Availability | Open source (Postgres required) | Enterprise / select Pro | Open source | Open source (Kubernetes) |
The right tool depends on your deployment shape:
Single-instance development. A local tracker or simple per-key limits suffice. MLflow's local tracker or LiteLLM with a lightweight Postgres instance both work without Redis overhead.
Multi-replica production. You need shared state. MLflow's Redis tracker or LiteLLM's Postgres-backed enforcement ensure the budget is global, not per-pod. Without shared state, three replicas each allow the full budget independently.
Kubernetes-native deployments. agentgateway's per-route policies integrate with your existing HTTPRoute configuration. Set separate budgets for prod and dev routes. Account for the instance multiplication problem in your limit calculations.
Teams wanting a managed UI. Portkey and LiteLLM both offer admin interfaces for creating and managing keys with budget limits. Portkey's alert threshold feature is distinctive if you want warnings before hard stops.
A practical checklist for any gateway:
Match budget_duration to your billing cycle. If you reconcile monthly, set monthly resets. If your team reviews weekly, use weekly windows. Misaligned reset periods create gaps where spend accumulates without review.
Set an alert threshold at 80% of the limit. Portkey supports this natively. For MLflow, create an alert-mode policy at 80% and a reject-mode policy at 100%. For LiteLLM and agentgateway, build the alert into your observability pipeline by watching spend metrics.
Use fail_closed for production keys. Where overages are unacceptable, LiteLLM's fail_closed_budget_enforcement ensures the ceiling holds even during infrastructure degradation. For other gateways, the equivalent is ensuring your budget tracker uses shared state (Redis) rather than local-only enforcement.
Scope limits per environment. Production keys get higher ceilings; development and staging keys get tight ones. agentgateway's per-route policies make this explicit. On other gateways, issue separate virtual keys per environment with different budget caps.
A forgotten dev key with no budget limit is how a $50 experiment becomes an $800 line item. Setting these four properties on every key before deployment takes less time than explaining the invoice afterwards.