Continuous batching
Slotting new requests into a running batch as older ones finish, instead of waiting for the whole batch to complete.
how it works · the vocabulary
in-flight batching
The single largest efficiency win in serving, and the reason a busy provider can charge less per token than a quiet one. It costs you nothing and explains most of the price gap between sellers of identical weights.