What Is the State of AI Infrastructure Without the Keynote?

The Gap Between Demos and Bills

The public conversation is about models. The private conversation inside serious teams is about dollars per million tokens, queue depth, and whether the networking fabric can move weights without melting a weekend.

Training still concentrates in a few labs. Inference is where most companies actually spend. That split is the story of 2026.

Three Pressures

• Hardware supply remains lumpy. Capacity appears in clumps, not as a smooth curve.
• Open-weight models compress the "we must train our own" argument for many products.
• Serving quality — tail latency, tool use, structured output — is now a product feature, not an ops footnote.

What Operators Are Doing

Serious shops are treating inference like a financial instrument: reserved capacity for baseline, burst on a second provider, and ruthless measurement of cache hit rates. The winners are not the teams with the most GPUs. They are the teams who know when a GPU is idle.

What to Watch

Watch networking and memory more than FLOPs. Watch evals that match production traffic, not leaderboard slices. Watch contracts that look cheap until you read the egress line.

From Keynote to Spreadsheet

The keynote will keep talking about intelligence. The spreadsheet will keep talking about utilization. Build dashboards for idle capacity and cache hits before you buy another cluster, and treat serving SLOs as product requirements with the same seriousness you give model evals in planning reviews.