continuous-batching

Batching and Throughput

Here’s a fact that confuses people at first: a GPU serving one user and a GPU…