Changing how we measure AI chip inference performance (Inference Performance Index)

Published Nov 9, 2026. Epoch AI introduces the Inference Performance Index (IPI) Version 1, a framework that compares how many simultaneous requests each AI chip can serve across a range of required output speeds, focused on agentic inference workloads and combined across four representative models. The headline finding: 'We estimate that the Nvidia GB300 provides about 25x Nvidia H100's inference capacity per chip on the agentic inference workloads we model, with only 2.5x the 8-bit FLOP per second,' so 'comparing FLOP/s alone would therefore underestimate GB300's inference capacity relative to H100 by a factor of ten' because memory and inter-chip communication can matter more than arithmetic throughput. Scores are expressed in GB300 equivalents (GB300e); the index uses Nvidia's AISimulate and speed-of-light analytical projections, and also shows system configuration matters (GB300 NVL72 has about 1.9x the modeled inference capacity per GPU of B300 HGX).

Compute is the primary lever of AI capability and AI policy; a common unit for inference capacity changes how the compute gap between chips is measured, which matters for export-control and compute-governance debates and for any datacenter capacity planning.

For technical and compute-policy leads: adopt a concurrency-based inference measure like IPI rather than raw FLOP/s when benchmarking capacity, and flag the ~10x understatement risk in current FLOP-based comparisons.