What good AI spend governance looks like

Finance forwards you the quarterly AI invoices and asks a reasonable question:

Is this under control?

Answering it takes most of a week.

There is an invoice from Anthropic and another from OpenAI. A line inside the AWS bill covers Bedrock usage from a workload a team stood up months ago. Azure OpenAI sits on the enterprise agreement, so it appears somewhere else entirely. Two coding tools bill per seat and land against different cost centers.

When you finally arrive at a number, you are not confident it is complete.

Then finance asks the harder follow-up questions:

How should these costs be allocated across teams? Which spending is producing value? Where could you spend less? Where should you increase the budget?

Invoices cannot answer those questions by themselves.

Provider consoles get you part of the way. When connected to your identity provider, many can show which users and API keys are spending money on which models. But each provider has its own console, data model, and definition of usage. None shares your organization’s understanding of a team, project, product line, or cost center.

The result is another manual consolidation exercise. You repeat it next quarter, hoping you have not missed an agent, developer tool, direct API key, or workload that sits outside the systems you remembered to check.

The instinct at this point is to reach for a ceiling. How low can you set it, and how many ways can you divide it?

But visibility is not the same as control. A hard limit applied without enough context can interrupt valuable work, create support tickets, or encourage teams to route around the system entirely.

Good AI spend governance has to do more than cap a bill. It should help you:

• See everything. Model providers, developer tools, agents, and the external tools they call.
• Observe and control. Understand real consumption before continually refining budgets and policy.
• Keep work moving. Prevent an outdated individual allowance from becoming an unnecessary outage.
• Hold up to scrutiny. Give finance, security, and compliance teams evidence that the controls actually work.

Here is what each of those requires and how the Stacklok AI Gateway approaches them.

1. See everything

Your organization probably did not choose one model provider and standardize on it forever.

Teams use Anthropic and OpenAI directly because that is where particular frontier models are available. Someone deploys a workload on Bedrock because AWS is already approved and covered by an enterprise agreement. Another team uses Azure OpenAI because of an existing Microsoft relationship. A customer requirement may push a workload toward Vertex AI or Google AI Studio.

Each of those may have been the right decision at the time. None will be reversed simply to make reporting more convenient.

That creates a trap when you try to introduce governance.

The obvious move is to put a gateway in front of the traffic that is easiest to migrate, publish a dashboard, and call the problem solved. The dashboard may accurately report on the traffic that passes through it while silently excluding everything else.

Partial visibility presented with confidence is worse than an obviously incomplete spreadsheet. At least the spreadsheet reminds you that you may be missing something.

The first requirement for effective governance is that the control sits in front of the estate you actually have, not merely the most convenient part of it.

The Stacklok AI Gateway puts multiple model-provider APIs behind a common endpoint, including OpenAI, Anthropic, Amazon Bedrock, Azure OpenAI, Vertex AI, and Gemini. Teams can continue using the providers selected for their workloads while identity, policy, and usage data are enforced consistently.

Credentials remain inside infrastructure controlled by the organization rather than being copied onto developer laptops or embedded in long-lived application configuration. That reduces the number of unmanaged keys that can continue spending long after the person who created them has forgotten they exist.

The same governance model must extend beyond model calls.

MCP tool calls run through the same control plane as model traffic, with the same identity, policy, and audit context. This matters because an agent does not only send prompts to a model. It also decides which tools to invoke, sends credentials to external systems, and acts on the responses it receives.

A control that governs model access while ignoring tool access sees only part of the agent’s activity.

Developer tools create another common blind spot. Some support enterprise identity directly. Others accept only a static API key. When governance supports the first category but not the second, the unsupported tools do not disappear. Users connect them directly to providers using credentials the organization may not be able to attribute or revoke centrally.

A local credential bridge gives those tools a governed path without requiring them to implement corporate authentication themselves.

One more thing is worth checking in any gateway evaluation: once a provider or model is supported by the platform, can an administrator enable and configure it directly, or does every change require a separate operational or vendor-support process?

Procurement may negotiate a committed-spend agreement and want to shift traffic toward it. A new model may become available on Tuesday, with teams asking to evaluate it by Friday. Those should be administrative decisions made on the organization’s timeline.

Unified governance does not mean forcing every workload onto the same provider. It means applying consistent identity, policy, visibility, and auditing across the providers, tools, and agents the organization has already chosen.

2. Observe and control

The problem with going directly to restrictive limits is that the first number is usually a guess.

The team everyone expected to be expensive may barely register. A workflow nobody mentioned during planning may account for a large share of the bill. A few developers may dramatically increase consumption after adopting a new coding tool.

Without a baseline, it is easy to guess wrong in both directions at once: blocking valuable work in one place while leaving waste untouched somewhere else.

The first thing the Stacklok AI Gateway provides is not a perfect limit. It is the consolidated picture that otherwise takes days to assemble manually: usage across the organization by user, team, provider, and model, along with where that usage is growing.

Users must have an initial budget before they can spend. The practical approach is to begin with budgets broad enough to avoid unnecessary disruption, observe real usage, and refine them over time.

The first numbers are a starting position, not a decision you have to get exactly right.

A total is useful, but it is not sufficient. Organizations need to connect consumption to the way they actually operate.

Today, the clearest dimensions are users, groups, providers, and models. Over time, deeper classification can attribute consumption to dimensions such as projects, product lines, environments, or cost centers.

That distinction matters.

A total tells you that spending increased. Attribution tells you which work caused the increase, whether it delivered enough value, and where reducing spend would have the least impact.

Every limit gets better as your understanding improves. A budget that nothing ever approaches may be too generous. One that interrupts work every Tuesday afternoon may be too restrictive. Both patterns should be visible without requiring teams to build their own instrumentation.

Consider Priya, an engineer on the platform team. She can see her own consumption and how much capacity remains available to her. That visibility turns a budget from an arbitrary outage into a resource she and her team can actively manage.

The gateway also exports operational signals through OpenTelemetry, including token usage, gateway health, and rate-limiting activity. Those signals can flow into the monitoring stack an organization already operates, such as Prometheus and Grafana.

Operational telemetry is useful for trend analysis, capacity planning, and alerting. Authoritative spend governance and cost allocation should remain within the gateway’s accounting model rather than requiring teams to reconstruct invoices from raw metrics.

Good governance creates a continuous loop:

1. Establish a reasonable initial policy.
2. Observe real consumption.
3. Identify unexpected growth or underused capacity.
4. Refine budgets and access.
5. Repeat as workloads and models change.

The goal is not simply to answer finance’s question faster. It is to ensure the organization is spending money on AI responsibly and understands the available options for reducing or redirecting that spending.

3. Keep the work moving

A budget should create accountability without turning every threshold into a support ticket.

When the only control is a hard individual limit, hitting it does not just signal usage, it stops work. Progress pauses while someone investigates, a request is made, and an administrator raises the limit under pressure. The new number is often just as arbitrary as the last, made under pressure.

A better model separates who is spending from where the capacity comes from.

Hierarchical budgets do this by allowing multiple layers of allocation. Priya might have a personal budget for day-to-day work, while also drawing from a shared team or organizational pool when needed. When her personal allocation is exhausted, eligible requests can continue against higher-level capacity instead of failing outright.

The key shift is that a request is only refused when all relevant capacity is exhausted, not when a single local limit is reached.

This changes the behavior of the system in important ways. Priya can see how much of her own budget remains, but also how her usage contributes to team-level consumption. Her team can detect when shared spending is trending upward before it becomes a problem. And administrators can tune policy based on observed behavior rather than reacting to emergency limit increases.

Hard stops do not disappear, they move to a more meaningful boundary. Instead of an individual allowance unexpectedly halting work, enforcement happens at a level that reflects real organizational intent and shared tradeoffs.

Early warning systems complement this structure. Shared budgets should emit alerts and webhooks before exhaustion, giving owners time to investigate unusual usage, throttle nonessential workloads, or intentionally increase capacity. This turns budget management into a proactive process rather than a reactive one.

Importantly, shared capacity introduces real tradeoffs. One team’s experiment can consume resources another team was relying on. The goal is not to obscure that tension, but to surface it early enough that it can be managed deliberately rather than discovered through failure.

From there, governance can go a step further than simple allow-or-deny decisions.

One extension is policy-driven model fallback. As a workload approaches its budget threshold, eligible requests can be routed from a higher-cost hosted frontier model to a lower-cost alternative, such as a self-hosted open-source model that performs adequately for that class of task.

This must be explicit and observable. Administrators define which workloads are eligible, which fallback models are approved, and under what conditions the switch occurs. Users and operators should always be able to see which model produced a response and why the fallback was triggered.

For many routine tasks, such as issue triage, test generation, documentation, or straightforward code changes, the quality difference may be small while the cost difference is significant. In those cases, a self-hosted fallback can also keep data and execution within infrastructure the organization already controls.

The goal is not to disguise substitution, but to make it a conscious policy choice: continue the work at lower cost, or stop it entirely.

For workloads where model choice materially affects quality or risk, the policy can still enforce a hard stop or require explicit approval. The important distinction is that every outcome, continue, switch, or stop, is deliberate, visible, and aligned with the nature of the work.

4. Hold up to scrutiny

Governance you cannot demonstrate is not much use.

Eventually, finance, security, or compliance reviewers will ask what controls were in place and what actually happened.

The gateway emits structured audit events showing which identity called which model, when the request occurred, which credential was used, and what it cost. Those events can flow through existing logging infrastructure to a SIEM, providing evidence for internal review without requiring a proprietary auditing pipeline.

Security teams will also ask where prompts and responses physically travel.

For many organizations, that is not a preference but an entry requirement. Regulated or sensitive data cannot pass through infrastructure outside the organization’s control, and products that require it to do so may be removed from consideration before a broader evaluation begins.

The Stacklok AI Gateway runs inside the customer’s own cluster. Prompts and responses do not transit Stacklok-operated infrastructure, and air-gapped installation is supported.

Inspection and policy enforcement happen inside that same boundary.

Personal and payment data can be detected on both the request and response paths. For each category, administrators can decide whether a match should be blocked, redacted before reaching the provider, or recorded as a warning while the organization evaluates the rule’s impact.

Prompt guardrails are applied at the same control point.

The rest may sound like table stakes, but it is worth confirming rather than assuming.

Requirement	What to verify
No uncontrolled bypass	Model traffic uses a governed endpoint so identity, budget, and policy controls apply consistently.
Enterprise identity	The gateway integrates with the organization’s existing identity provider rather than creating a separate user database.
Default-deny access	Users and groups receive access only to the models and capabilities administrators have approved.
Unified audit events	Events capture identity, provider, model, timestamp, credential context, and cost and can be exported to existing logging systems.
Data minimization	Prompt and response retention is optional, controlled, and redacted where required.
Deployment control	The gateway runs inside infrastructure controlled by the customer and supports environments with restricted or no external connectivity.
Tool governance	MCP servers and tool calls are governed alongside model calls rather than through a separate, disconnected system.
Visible fallback behavior	Any model substitution is policy-driven, observable, and communicated rather than occurring silently.

Where to start

You do not need to begin with a lengthy vendor evaluation to understand where you stand.

Four questions will expose most of the gap:

• How long would it take to produce a complete AI spend figure today, and how confident are you that it includes everything?
• Can you break that spending down by team or workload rather than only by provider?
• If you set a new limit tomorrow, what evidence would you use to choose the number?
• When someone leaves the company, which credentials and forms of access stop working, and how quickly?

A fifth question is becoming just as important:

• Can you govern the external tools and MCP servers your agents call with the same identity and policy controls you use for model access?

If those answers are uncomfortable, the gap is not merely the absence of a spending ceiling. It is the identity, visibility, attribution, policy, and enforcement underneath it.

That is the real test.

Good AI spend governance means that the next time finance asks whether AI spending is under control, the answer takes a minute rather than a week. It can be explained using the structure of the organization rather than a collection of provider invoices. It shows where the money went, gives teams room to keep working, and produces evidence that the controls were applied consistently.

Most importantly, it lets the organization reduce waste without forcing developers and agents to stop doing valuable work.