Shruti Koparkar

NVIDIA argues that AI factories, which cost roughly $60 million per megawatt, maximize returns through three factors: productive systems that deliver high throughput per megawatt and low cost per token, durable hardware that remains economically viable years after deployment, and fungible platforms that run diverse workloads. The company claims its latest Vera Rubin NVL72 systems deliver over 30x higher throughput per megawatt than GB300 NVL72 systems and up to 45x lower cost per million tokens on certain models, while older generations like the A100 GPU from 2020 remain in commercial service six years later.

NVIDIA's Vera Rubin NVL72 GPU system delivers up to 30x higher throughput per megawatt than its GB300 NVL72 predecessor on agentic AI workloads, according to measurements using the SemiAnalysis AgentX benchmark. Agentic tasks consume 15x more tokens than simple chat because agents iteratively query databases, invoke sub-agents, and accumulate context across multiple reasoning steps. For power-constrained AI infrastructure operators, the efficiency gain translates directly to running significantly more agent-based work within the same energy budget.

NVIDIA argues that performance per watt is the critical metric for AI infrastructure efficiency, as power constraints directly determine token generation capacity and profitability in AI factories. The company claims its Blackwell NVL72 platform delivers up to 25x performance per watt over Hopper on frontier models like DeepSeek V4 Pro, achieved through system-wide codesign spanning silicon, software, and networking. As agentic AI increases token demand, infrastructure choices made today will determine which organizations can scale in a power-constrained environment.

NVIDIA's Blackwell platform swept MLPerf Training 6.0 benchmarks, achieving the fastest training times across all seven tests, scaling to 8,192 GPUs, and being the only platform with submissions across the entire suite. The results reflect deep co-engineering between NVIDIA and cloud partners like Microsoft Azure and CoreWeave on system architecture, networking, and software optimization for large-scale model training.

Artificial Analysis released AgentPerf, the first benchmark designed specifically for agentic AI workloads, showing NVIDIA's Blackwell Ultra NVL72 platform delivering 20x more agents per megawatt than Hopper-based systems. The benchmark reflects the fundamentally different performance characteristics of agentic AI, which chains dozens to hundreds of LLM calls with tool execution rather than single-turn completions. Results are based on real coding agent trajectories across 12+ programming languages, providing infrastructure providers and enterprises with direct metrics for deployment decisions.

Shruti Koparkar argues that enterprises evaluating AI infrastructure should shift from traditional metrics like FLOPS per dollar to cost per token as the primary measure of total cost of ownership. The piece contends that cost per token is the only metric that captures the full picture of real-world AI economics, accounting for hardware performance, software optimization, and actual token delivery rather than just raw compute capacity. The distinction matters because optimizing for input costs while the business runs on output creates a fundamental mismatch in how infrastructure value is assessed.