Many AI Natives and AI Adopters still struggle to understand what is driving AI costs. Just 31% report having accurate visibility into their AI software spend, while 59% say wasted AI spend increased over the past year, according to Flexera's 2026 State of ITAM Report.
The answer is rarely found in one place. Idle infrastructure, agentic workloads, and model lock-in each influence the cost of inference in different ways. Together, they determine whether AI costs become visible and predictable as production workloads scale.
The hidden cost of an uncoordinated stack
By some estimates, GPU clusters operate at only 60–70% GPU utilization, and hardware failure is rarely the cause. More often, storage, networking, orchestration, and compute have been deployed as separate layers that weren't designed to operate as a coordinated system. Bottlenecks prevent data from reaching accelerated compute efficiently, leaving expensive GPUs idle while organizations continue paying for full infrastructure capacity. This is measurable waste that can be avoided with a full-stack solution.
At smaller inference volumes, those inefficiencies are manageable. As workloads scale, they become structural. Because the waste is spread across multiple layers, it's difficult to identify and even harder to eliminate. Idle GPUs are the most visible symptom, but they rarely tell the whole story. The real cost comes from infrastructure that isn't operating as efficiently as organizations believe.
The hidden cost of agentic AI
The first sign of failure is usually the invoice.
Unlike a single inference request, agentic workflows don't have a predictable token ceiling. Token usage can increase through:
- Multi-turn reasoning
- Tool calling
- Autonomous task execution
- Self-correction
These costs often remain opaque during development.
An agent that retries tasks, loops through reasoning chains, or repeatedly calls external tools can consume several times more tokens than expected without triggering operational alerts.
As these workloads move into production, many organizations are finding that existing cost management and monitoring practices weren't designed for autonomous, multi-step AI systems.
Visibility increasingly needs to extend beyond infrastructure into inference itself, helping teams understand how workloads evolve over time and where costs originate. At Nscale, we've developed this kind of attribution and operational visibility for specific customer environments and internal deployments, reflecting where enterprise AI operations are heading as organizations seek greater control over AI economics.
The hidden cost of model lock-in
Beyond infrastructure and workflows, model strategy increasingly shapes the economics of AI in production. Organizations whose infrastructure is tied to a single model or API are discovering that operating costs at scale can outpace the value delivered. Choosing the right model for the right workload is becoming one of the most effective ways to control AI costs.
The model landscape is evolving rapidly. Open-source models now account for 38% of enterprise token volume, up from 11% a year ago, while new frontier models continue to arrive at pace. Infrastructure built with limited model portability can quickly become a constraint as better-performing and more cost-effective models continue to emerge. Adopting them often requires significant re-engineering rather than a straightforward operational decision.
Open-source models gain enterprise share
Flexibility creates more opportunities to optimize inference costs. Instead of routing every request to the same model, organizations can:
- Match model size and capability to each workload
- Deploy fine-tuned open-weight models where appropriate,
- Adopt new models as they become available, without rebuilding applications.
As Nscale found when building Alfred, our internal engineering agent, the long-term value lies in the harness around the model: the workflows, guardrails, standards, and feedback loops that turn AI into a reliable engineering system. By separating the application from the underlying model, that harness gives teams the flexibility to optimize for capability, latency, and cost as the model landscape evolves.
Infrastructure that supports both open-weight and commercial models gives organizations the freedom to evolve their model strategy as AI advances, without locking themselves into a single vendor or architecture.
Visibility is only the first step
Understanding AI costs is only valuable if organizations can respond to what they learn. That requires infrastructure designed to adapt as workloads, models, and business requirements evolve.
The organizations making the most progress aren't necessarily those spending the least. They're the ones building greater visibility into AI costs and the operational flexibility to optimize them as workloads evolve. The result is stronger returns from AI investments as adoption scales.
This article was originally published as part of Nscale's Full Stack AI newsletter, where we share perspectives on AI infrastructure, engineering, and emerging industry trends.
Subscribe to receive future editions.
.png)
.png){kind=icon}
.png)
.jpeg){kind=small}
.png){kind=small}
