Scalable AI Solutions · Deep Analysis · June 2026
Elastic AI: How Scalable Machine Learning Architectures Are Redefining Enterprise Efficiency
Most enterprises are spending at unprecedented scale on AI infrastructure — and burning the majority of it. This is the honest technical and economic account of why, and what the organisations that are actually getting this right are doing differently.
The Problem Nobody Wants to Admit
There is a number that keeps appearing in every serious piece of AI infrastructure research published in the last twelve months, and senior engineering leaders tend to go quiet when it comes up in meetings. That number is 5%. It is the median GPU utilisation rate across enterprise AI deployments in 2025 and into 2026, according to Cast AI’s direct measurement of production clusters and millions of compute resources before any optimisation was applied.
Let that sink in for a moment. An organisation renting an NVIDIA H100 at cloud rates — somewhere between $2.10 and $8.54 per hour depending on the provider and reservation type — is getting genuine compute output from that machine for roughly 4.8 minutes in every 100-minute block. The other 95.2 minutes, the meter is still running.
I say this with some personal accountability: I spent a long time believing the conventional argument that low GPU utilisation was a temporary problem — a symptom of the GPU scarcity era that would self-correct once procurement cycles normalised. I was wrong. GPU prices on AWS H200 Capacity Blocks actually rose 15% in January 2026, breaking a two-decade precedent of compute deflation. The utilisation crisis is not a supply problem. It is an architecture problem. And most organisations are responding to it by buying more infrastructure, which is precisely the wrong answer.
Meanwhile the macro numbers are staggering. Global AI spend reached $1.5 trillion in 2025 — roughly five times the size of the entire enterprise software market at $316.69 billion in the same year. IDC projects the AI infrastructure market alone will surpass $1 trillion by 2029, compounding at about 31% annually. The Big Four hyperscalers poured $370–410 billion into AI infrastructure in 2025, and that figure is on track to hit $650–750 billion in 2026.
Yet fewer than 1% of executives report achieving ROI of 20% or more on those investments. Thirty percent of generative AI projects have been abandoned after proof-of-concept. Gartner expects more than 40% of agentic AI projects to be cancelled by end of 2027. And 80–85% of enterprises miss their AI infrastructure forecasts by more than 25%.
Something is fundamentally broken in how the industry thinks about AI scale. This article is an attempt to be honest about what that is.
What “Elastic AI” Actually Means (and What It Doesn’t)
The word “elastic” has been stretched to meaninglessness in vendor marketing copy. Every cloud provider uses it. Every platform deck includes it. So let’s be specific about what we mean when we say elastic AI architecture, and equally specific about what we don’t mean.
Elastic AI, as a genuine architectural principle, means the compute footprint of your machine learning systems automatically contracts and expands to precisely match workload demand, across training, fine-tuning, batch inference, and online inference — with no manual intervention, no idle overcapacity reserved “just in case,” and no provisioning lag that causes users to experience degradation.
That is a very demanding definition. And almost nobody actually meets it today.
What most organisations have instead is what I’d call pseudo-elasticity: they use autoscaling on inference endpoints, but their training clusters are statically provisioned; or they have auto-scaling but it’s configured so conservatively (to avoid the dreaded out-of-memory crash) that it never actually fires; or they have genuine elasticity in compute but their data pipelines are fixed-capacity bottlenecks that limit throughput anyway.
True elasticity operates across at least four dimensions simultaneously:
Compute Elasticity
GPU/TPU/CPU resources scale independently, with fractional allocation possible. Not binary pod assignment.
Data Pipeline Elasticity
Storage throughput and preprocessing capacity match compute throughput. The bottleneck moves, not just the GPU layer.
Model Serving Elasticity
Inference capacity scales per-model and per-endpoint, with dynamic batching and multi-model GPU sharing.
Organisational Elasticity
Teams can share a global resource pool without tribal silos, with quota governance and fair-share scheduling.
The fourth dimension — organisational elasticity — is consistently the hardest one and gets the least attention. You can have perfect auto-scaling infrastructure and still waste 60% of it if team A’s idle GPUs are walled off from team B’s peak demand because of internal chargeback politics and cluster ownership models. This is a cultural and governance problem, not a technical one, and it requires a different kind of solution.
Before auto-scaling configuration, GPU fractionalisation, or Kubernetes tuning, an organisation needs three prerequisites in place: (1) workload observability that surfaces GPU-specific signals, not just container health; (2) a unified resource pool with organisational queue governance, not per-team GPU allocation; (3) a data pipeline that can sustain the compute throughput you’re trying to unlock. Without all three, auto-scaling is performance theatre.
The GPU Waste Triangle: Three Structural Failure Modes
The 5% utilisation figure is not random variance. It’s the output of a specific set of structural failure patterns that compound. Based on recent infrastructure research and production case studies, I see three consistent failure modes that appear together in almost every underutilising enterprise — what I’m calling the GPU Waste Triangle.
Failure Mode 1: Reservation Hoarding
From 2023 through mid-2025, GPU scarcity drove what Cast AI co-founder Laurent Gil described precisely: “A GPU sitting idle costs dollars per hour. A CPU sitting idle costs cents.” The rational response to scarcity and multi-week wait queues for on-demand H100s was to reserve capacity before workloads existed to fill it. Teams acquired accelerators as a strategic moat. Utilisation telemetry was irrelevant because nobody was optimising — only acquiring.
That environment is gone. But the reservation behaviour it created is locked into three- to five-year depreciation cycles and committed cloud contracts. CPU overprovisioning rose from 40% to 69% year over year according to Cast AI’s latest measurements, while memory overprovisioning now stands at 79%. Organisations are reserving nearly twice as many CPU resources and four times as much memory as their workloads actually consume.
Failure Mode 2: Pipeline Starvation
A quantised LLM running inference on an 80GB A100 might consume only 12GB of GPU memory at 30–35% compute utilisation. Kubernetes considers that GPU “fully occupied” — the entire physical card is allocated to one pod with no native sharing mechanism. Meanwhile, the other 65–70% of the accelerator sits idle. This is not a scheduling bug. It is a fundamental design gap in how containerised infrastructure maps to GPU architecture.
The broader pipeline starvation problem goes deeper: when CPU-heavy data preprocessing and GPU-heavy inference are packaged as a single workload, the entire system scales as one unit. When data loading lags, the GPU waits. A study of 118,276 GPU jobs on the Perlmutter supercomputer (1,536 NVIDIA A100 nodes) found that 37% of GPU jobs never exceeded 15% memory utilisation — and those jobs accounted for 37% of total node hours. The pipeline, not the accelerator, was the binding constraint.
Failure Mode 3: Silo Fragmentation
When every team owns its cluster, sized for its own peak demand, the aggregate system runs at the intersection of all those individual peaks — which almost never coincide. One team’s training job finishes at 3am while another team’s inference spike hits at 2pm, and the GPU capacity cannot flow between them because organisational and billing structures prevent it.
Production case studies from CNCF member organisations show that advanced GPU scheduling on Kubernetes — specifically queue-based admission control with borrowing policies — can improve utilisation from 13% to 37%, nearly tripling efficiency. NVIDIA’s KAI Scheduler, open-sourced under Apache 2.0 license in 2025, takes this further with fractional GPU allocation and topology-aware scheduling. The fix, when it is technical at all, is relatively straightforward. The blocker is almost always political: chargeback models that make teams reluctant to contribute capacity to a shared pool they can’t guarantee getting back.
The Real Cost Curves: Unit Economics of Scaling ML
The standard way organisations think about AI infrastructure cost is linear: more workload → more compute → proportionally more spend. This is wrong, and the wrongness compounds at scale. Let’s build an honest model.
The Unit Economics Breakdown: What 100M Inference Requests Actually Cost
| Cost Component | Monolithic | Hybrid | Elastic |
|---|---|---|---|
| Compute (GPU/CPU) | $3.80/k req | $2.60/k req | $1.70/k req |
| Data egress & storage I/O | $0.80/k req | $0.65/k req | $0.35/k req |
| Monitoring & observability | $0.20/k req | $0.25/k req | $0.30/k req |
| MLOps tooling & governance | $0.10/k req | $0.20/k req | $0.35/k req |
| Idle capacity overhead | $1.00/k req | $0.50/k req | $0.10/k req |
| Total cost / 1k requests | $5.90 | $4.20 | $2.80 |
| Monthly total (100M req) | $590k | $420k | $280k |
| GPU utilisation | ~8% | ~25% | ~65% |
The pattern worth noting: elastic architecture has higher MLOps tooling costs than monolithic. This is not a bug. It’s the price of control. You’re paying for the observability, scheduler configuration, and governance that make true elasticity possible. That tooling overhead is roughly $35k–$60k/month in this model — and it unlocks $310k/month in compute savings at scale. The ROI is not subtle.
Enterprise AI TCO analyses consistently undercount one category: the cost of not scaling when you should. A model serving latency spike at peak demand that degrades a customer-facing product can have revenue impact orders of magnitude larger than the compute bill itself. Elastic architecture should be modelled against both the cost of idle compute and the cost of insufficient compute at the wrong moment — which monolithic architectures face constantly.
Architecture Patterns That Survive Contact with Production
There is a significant gap between the architecture patterns that get published in technical blog posts and the patterns that actually hold together twelve months after deployment at scale. Here are the ones I’ve seen survive.
Pattern 1: Disaggregated Multi-Stage Pipelines
The most impactful single change in elastic ML architecture is separating CPU-bound and GPU-bound pipeline stages so they can scale independently. When a multi-stage video processing pipeline is run as a single container, you end up allocating 32 GPU instances to get enough CPU throughput — with 87.5% of your GPU capacity idle. Disaggregate the stages via Ray or equivalent, and the same throughput requires only 4 GPU instances. That’s an 8× reduction in GPU requirements for the same workload output, documented in Anyscale’s production data.
Pattern 2: Tiered GPU Allocation Strategy
The most sophisticated enterprises now operate three distinct compute tiers within the same infrastructure pool: production inference on Multi-Instance GPU (MIG) partitioned accelerators for strict latency isolation; development and experimentation on time-sliced GPUs; large-scale training on full dedicated GPU reservation. This prevents the common antipattern of every team requesting dedicated A100s for workloads that could run on a MIG slice — the organisational equivalent of driving alone in a twelve-seat bus.
Pattern 3: Global Queue Scheduling with Cohort Borrowing
Kubernetes-native job queuing systems like Kueue provide cluster-wide queues with tenant quotas and cohort borrowing. When one team’s workloads are idle, other teams can temporarily consume those unused resources; the system returns capacity when the original owners need it. High-priority training runs can preempt lower-priority workloads. This alone — implemented correctly — accounts for 30–50% utilisation improvement without any changes to model architecture or serving infrastructure.
Pattern 4: Continuous Model Performance vs. Cost Reviews
This one is almost never implemented. It should be a standing quarterly ritual: sit down with every significant production model and ask whether the compute architecture that made sense when it was deployed six months ago still makes sense today. Models drift, hardware improves, and inference efficiency techniques (quantisation, distillation, speculative decoding) move fast. Stanford’s AI Index 2025 noted that inference costs have fallen from $20 to $0.07 per million tokens through combined hardware and software improvements. If your serving configuration isn’t tracking this curve, you’re paying yesterday’s prices for today’s workload.
Case Studies: Uber, Netflix, and the Hard Lessons
Michelangelo: Nine Years of Getting ML Scale Right (and Wrong)
Uber launched Michelangelo in early 2016 as an internal ML-as-a-service platform designed to make “scaling AI to meet the needs of business as easy as requesting a ride.” The original goal was straightforward: standardise the ML workflow via an end-to-end system that enabled developers across Uber to build and deploy models at scale without custom serving containers for every use case.
By 2024, Michelangelo 2.0 had evolved dramatically. Uber built a unified federation layer — the Michelangelo Job Controller — to address the fragmentation of GPU assets across multiple Kubernetes clusters in different Availability Zones. The critical insight from their evolution: the original assumption that enabling ML everywhere was the right goal was wrong. The shift in Michelangelo 2.0 was from “ML everywhere” to “high-impact ML projects” — an explicit acknowledgment that infrastructure democratisation without focus produces waste rather than value. Uber’s feature store, Palette, now hosts more than 20,000 features that teams can leverage out-of-box, eliminating redundant feature computation that had been burning compute across silos.
For the generative AI era, Uber evolved its infrastructure to H100 GPU clusters with 100GB/s high-bandwidth networking and Distributed Memory Offloading. One architectural principle Uber’s team articulates explicitly: in several production cases, XGBoost outperforms deep learning in both performance and cost. Not all ML problems are deep learning problems. The pressure to use the most sophisticated model is often the enemy of efficient infrastructure.
Supporting Diverse ML Systems at Scale: The Anti-Platform Approach
Netflix’s ML infrastructure philosophy is deliberately anti-monolithic. The company runs machine learning across hundreds of use cases — recommendations, content optimisation, streaming quality, A/B testing infrastructure — and the diversity of those use cases drove a specific architectural choice: build a common infrastructure layer that gives data scientists the ability to iterate faster, without mandating a single ML framework or workflow.
Netflix runs approximately 15,000 Flink jobs on its streaming data platform alone, according to presentations at Confluent Current 2024. The scale of that number is worth pausing on: 15,000 streaming jobs, not 15, not 150, but 15,000. That kind of scale is only manageable with a platform that enforces reusable APIs and microservices. Netflix’s model deployment pipeline reduces time from weeks to hours while maintaining 99.9% uptime — a figure that matters because every hour of degraded recommendation quality has direct, measurable impact on engagement and retention.
The lesson Netflix encodes is about platform design philosophy: a platform that is too opinionated about how teams build models will be worked around; a platform that is too permissive will fragment into incoherence. The narrow band between these extremes — enforcing infrastructure primitives while leaving modelling choices free — is where maintainable ML scale actually lives.
I spent the early part of my analysis of enterprise ML platforms focused almost entirely on training infrastructure — the big GPU clusters, the distributed training frameworks, the checkpoint strategies. This was wrong in a way that embarrassed me when I looked at the actual cost data.
Inference — not training — is where the money goes. As Goldman Sachs’s AI CapEx analysis notes, the shift from training-dominant to inference-heavy workloads accelerates revenue generation by converting fixed capital into usage. The organisations burning through their AI budgets are almost universally overprovisioned for inference bursts they modelled incorrectly, not overprovisioned for training. I’ve corrected my framework accordingly: inference serving architecture deserves at least equal analytical weight as training architecture, and the unit economics models I now build start with inference load projections, not model training schedules.
The Elastic Scale Score: A New Diagnostic Framework
I’ve been thinking about how organisations should diagnose their current position before deciding what to build. What follows is a framework I haven’t seen articulated elsewhere — a composite score across five dimensions that maps roughly to the cost curve position described earlier.
The scoring rubric for each dimension:
- Compute Elasticity (0–100): Do GPU pods scale fractionally? Is MIG or time-slicing enabled? Does auto-scaling fire within 90 seconds of demand? Full marks = 0→peak in under 2 minutes with no manual intervention.
- Pipeline Efficiency (0–100): Are CPU and GPU pipeline stages disaggregated? Is data loading a binding constraint? Can storage I/O sustain peak GPU throughput? Full marks = storage feeds compute without starvation at peak.
- Model Serving (0–100): Is dynamic batching enabled? Are models shared on GPUs where SLA allows? Is there A/B routing and canary deployment? Full marks = multi-model sharing with per-model auto-scaling and SLA-aware routing.
- Observability (0–100): Are GPU-specific signals (VRAM state, per-pod utilisation, memory waste) surfaced? Is drift detection automated? Can you identify the exact pod wasting compute? Full marks = real-time GPU telemetry with automated alerting.
- Cost Attribution (0–100): Can you attribute inference cost to individual teams, models, and business units? Is there a financial control layer? Full marks = granular per-request cost visibility with chargeback capability.
- Organisational Governance (0–100): Is there a global resource pool with queue governance? Do teams borrow unused capacity across silos? Is there a fair-share scheduler? Full marks = unified pool, cohort borrowing, enforced quotas, automated redistribution.
The Unpopular Take: When You Shouldn’t Scale At All
The dominant narrative in enterprise AI circles is that scaling problems are technical problems. They are not. The majority of organisations I see investing in elastic AI architecture would be better served by reducing the number of AI workloads they run before optimising the infrastructure those workloads run on.
Gartner’s forecast that 30% of GenAI projects will be abandoned after POC, and 40%+ of agentic AI projects will be cancelled by 2027, is not a failure of architecture. It’s a failure of use-case selection. An elastic architecture running twenty low-value inference workloads is still burning money. The scarcest resource in enterprise AI is not GPU compute — it is clear problem definition and honest ROI measurement. BCG data shows that 60% of enterprises see minimal or no value from AI; only 5% qualify as “AI leaders” achieving significant returns. Those 5% are not distinguished primarily by their infrastructure choices. They are distinguished by ruthless focus on which problems actually benefit from ML.
Elastic architecture is the right answer to a genuine scaling problem. But a large fraction of the organisations pursuing it have not yet established that they have a scaling problem worth solving. The architectural discussion should follow the business case, not precede it.
This is not an argument against investment in scalable infrastructure. It’s an argument against decoupling infrastructure investment from value measurement. The organisations getting the best returns from elastic AI architecture — the ones in that 5% — share a common practice: they define a business metric that the ML system will move before they write a single line of model code. The infrastructure question comes third, after use-case definition and metric specification.
Josh Bersin, in a recent analysis of enterprise AI pricing dynamics, flagged exactly this: “Ill-conceived projects will get trashed.” He cited Uber (burned through AI budget in weeks on an unsuccessful project), Pizza Hut, and Starbucks — each having failed AI projects that resulted in $100M lawsuits. These are not small companies with unsophisticated engineering teams. They are organisations that built before they validated.
The 90-Day Elastic Architecture Roadmap
If your organisation has a validated use case and an honest business metric, here is a concrete 90-day sequence for moving from monolithic to elastic ML infrastructure. This is the path, not the fastest path — the fastest path is almost always wrong, and I’ve seen enough “we’ll clean it up later” decisions calcify into technical debt that took three years to unwind.
Phase 1 (Days 0–30): Observability First
Deploy GPU-specific observability before touching anything else. NVIDIA’s GPU Usage Monitor (a Helm-deployable observability stack released in 2025) surfaces VRAM state, per-pod GPU utilisation, and memory waste at the workload level — signals the standard Kubernetes metrics stack (kube-state-metrics, node-exporter) does not provide. Your only deliverable from Phase 1 is your real utilisation number. Not what you think it is. Not what your vendor told you. The actual measured figure, per workload, per team.
Phase 2 (Days 30–60): Pipeline Disaggregation and Queue Governance
Implement two changes in parallel. First, separate CPU-bound and GPU-bound pipeline stages using Ray or equivalent disaggregated execution. This alone typically recovers 15–30% utilisation improvement from the wasted-CPU-idle-while-GPU-waits dynamic. Second, implement Kueue or NVIDIA KAI Scheduler for global queue governance with cohort borrowing. The political work of getting team buy-in for shared pools happens here and takes longer than the technical work.
Phase 3 (Days 60–75): Serving Elasticity and MIG Partitioning
Enable dynamic batching on inference endpoints using NVIDIA Triton or vLLM. Implement MIG partitioning for production inference workloads that require SLA isolation. Configure time-slicing for development and experimentation workloads. Set up A/B routing and canary deployment for model updates. This phase typically yields a further 20–40% utilisation improvement stacked on Phase 2 gains.
Phase 4 (Days 75–90): Cost Attribution and Chargeback
Build the financial control layer. Teams need to be able to see, with per-request granularity, what their models cost. This changes behaviour in a way that no amount of infrastructure optimisation can: when teams are accountable for compute cost, they optimise models, reduce redundant workloads, and stop keeping GPUs running overnight by default. The 30% of cloud spend that vanishes into idle resources is largely driven by the absence of this accountability layer.
Verdict
Let me be direct about what the evidence shows.
The enterprise AI infrastructure build-out is real, large, and will continue. IDC projects AI infrastructure crossing $1 trillion globally by 2029. The hyperscalers are spending $650–750 billion annually on AI infrastructure in 2026. This capital commitment is not going to reverse.
But the returns on that investment are concentrated. BCG data shows 5% of organisations are achieving significant returns; 60% see minimal or no value. The 5% are not lucky. They share specific, replicable practices: ruthless use-case selectivity, honest ROI measurement, and — yes — elastic architecture built on genuine observability and organisational governance, not just cloud auto-scaling checkboxes.
The GPU Waste Triangle is a solvable problem. Production case studies demonstrate that organisations moving from 5% to 37% GPU utilisation — nearly tripling efficiency — do so through disciplined queue governance and pipeline disaggregation, not through buying different hardware. The Elastic Scale Score gives you a diagnostic to locate exactly where your organisation sits and where the highest-leverage changes are.
The honest caveat: none of this infrastructure work matters if it precedes the business case. Elastic AI architecture is the right answer to the right question. Make sure you’ve asked the right question first.
The organisations that are going to define the competitive landscape in 2027 and 2028 are the ones building elastic infrastructure on top of validated, high-value ML workloads — not the ones building the most impressive infrastructure for workloads that don’t move business metrics. The gap between those two groups is already visible in the ROI data. It will be visible in market outcomes within eighteen months.
“The organisations achieving the best GPU economics share several practices. They implement queue-based admission control from day one. They match GPU partitioning strategies to workload profiles. And they treat resource efficiency as an automated, continuous process — not a manual, one-time task.”
— CIO Magazine, April 2026, reporting on CNCF member organisation case studiesThe infrastructure era of AI — where acquiring capacity was the game — is ending. The efficiency era is beginning. The organisations that recognise this shift and build accordingly will find that elastic ML architecture is not just a technical choice. It is a strategic one.
Further reading: For a deeper dive on AI cost governance, see Mavvrik’s AI Cost Statistics 2026. For GPU scheduling mechanics, CIO Magazine’s Kubernetes and the GPU Utilisation Crisis is the most technically detailed recent account. Uber’s architecture evolution is documented in their official engineering blog: Meet Michelangelo and Scaling AI/ML Infrastructure at Uber. Goldman Sachs’s Tracking Trillions remains the best macro-level CapEx analysis. For more on scalable AI solutions and enterprise AI strategy, see our related analysis on this site.