1,000 serving GPUs also do the batch work of about 240 dedicated GPUs

With batch work built to share, 1,000 serving GPUs also do the work of ~240 dedicated GPUs

Buy more GPUs 1,000 GPUs to buy and power

1,000 serving live traffic

+240

new hardware

Inference OS on the same fleet 1,000 GPUs to buy and power

1,000 serving live traffic

+240

batch work, same GPUs

Our extrapolation from the measured rate of 0.24 of a GPU per GPU for batch work built to share, with live decode at most 1.6% slower (resident kernel, Llama-3.1-8B; paper, Appendix A7). Ordinary jobs not written to share get less than half.