Our AI bill is on track to shrink 42% in one month. Here’s how we did it.

Stacklok is an AI-maximalist organization. We want engineers running more agents on harder problems, and understanding our spending helps us put more of that investment toward useful work.

We used our AI gateway’s request and response data to identify cost antipatterns, recurring patterns in model selection, context loading, and request handling that can quietly inflate AI spend.

Model switches can change context costs

Some engineers switched to OpenAI’s models after encountering verbosity, missed instructions, and repetitive code-review loops with early Opus 5.x releases. The switch had some unexpected impacts on the cost of long-context workflows.

The OpenAI models we started using had an input token pricing threshold. Whenever an individual API request exceeded 272,000 input tokens, price per input token doubled and the output rate for the entire request increased by 50%.

Claude Opus 5 had no equivalent pricing cliff. Its per-token rates stayed the same across its full context window. Moving long-context workflows to OpenAI introduced a surcharge our engineers hadn’t previously needed to account for.

Our gateway telemetry showed that the surcharge was concentrated on one team, where 25% of OpenAI prompt tokens came from requests above the threshold, compared with about 1% elsewhere in engineering. We reviewed those workflows and reduced context where it wasn’t needed. Keeping those requests below the threshold saved roughly $1,800 over October’s first 20 days.

Model choices need maintenance

Opus 5.5 made the opportunity concrete. Compared with Opus 5, input and output token prices fell 20%, while cache reads became 60% cheaper, dropping from $0.50 to $0.20 per million tokens. For workflows like ours that repeatedly reuse large amounts of context, that cache-price reduction matters.

Across all models with cheaper successors, we identified about $3,900 in potential monthly savings by repricing the same token usage at the newer rates.

We asked teams to update where it made sense and check automations for older model versions. A model choice needs periodic review, just like any other dependency.

Our repository had a recurring context cost

The gateway revealed the cost of repeatedly processing context. Alongside that analysis, our engineers investigated what our repository asked agents to read. 

We learned that our repository setup added about 22,000 tokens at session startup. Agent and skill descriptions accounted for 77% of that overhead. In one multi-agent code review, repeated repository instructions accounted for 334,000 tokens. And redundant context was costing us money. Incorrect context can also lead to incorrect code. Our engineers found 131 references to deleted or renamed files.

The team jumped in and made some necessary changes. We implemented the repository cleanup and tested its impact across 29 tasks and roughly 400 runs. Costs fell about 10% overall, including 30% for codebase explanations and 15% for planning. Incorrect claims copied from documentation fell from 28 to 4, with no task category performing worse in the tests.

Agent instruction files such as Claude Code’s CLAUDE.md deserve the same care as code. Loaded instructions consume context and contribute to processing costs. When they are wrong, they can steer the work in the wrong direction. Keeping them concise, accurate, and loaded only when needed is part of maintaining the repository.

We’re now connecting AI usage to project outcomes so we can evaluate what that investment delivers. It’s early, and we’re closely monitoring impact, but while we’ve cut spending over the past month there has been zero impact to engineering velocity. 

Our north star is shorter time to learn and shorter time to business value. Understanding where tokens go helps us put more of our AI budget toward both.