kv-cache

Memory Maths: Weights and KV Cache

A team once told me their 8B model “needs 80 GB of GPU memory” because it kept…