AI Application Performance: Stream First, Shrink the Context, Route the Model

LLM applications have two performance budgets that classic backends don’t worry about together: latency (it feels slow) and cost (every request costs real money in tokens). Both are driven by the same thing — tokens — and both have a handful of high-leverage levers. The good news, echoing the measure-first method: most teams find 3–5× of easy savings the first time they actually look. (Numbers below are illustrative but typical; measure your own.)

Latency: stream first, then shrink

The latency of a single LLM call is roughly:

latency ≈ TTFT + (output tokens ÷ tokens-per-second)

TTFT (time to first token) is the wait before anything appears; after that, tokens stream at some tokens-per-second. Output length drives the wait, not input — a long answer is slow to generate, token by token.

The levers, in the order that pays off:

1. Stream the response. This is the single biggest perceived-latency win. The total work might be ~8.5 s for a 400-token answer, but with streaming the user sees the first token at ~0.5 s and reads along as it generates — instead of staring at a spinner for 8 seconds. Same total time, a completely different experience.
2. Shorten the output. You pay per output token in both time and money. Ask for less, set max_tokens, and don’t make the model restate the question.
3. Shrink the context. A bloated prompt costs input tokens and slows the prefill step. Send only what’s needed — good RAG that retrieves the right few chunks, not the whole document dumped in “just in case.”
4. Use a smaller/faster model when it’s good enough, and reserve the big one for hard calls (see routing below).
5. Parallelise independent work — fire independent tool calls or sub-queries concurrently rather than in series — and cache repeated prompt prefixes so prefill is near-free.

For the latency-critical end of this (voice), the voicebots course treats every one of these as life-or-death.

Cost: cache, route, and right-size the model

LLM cost is mostly tokens × price, and you control both. Three leaks account for most waste: the context you resend on every call, the expensive model you reach for by reflex, and the identical questions you answer from scratch.

• Prompt caching. A 2,000-token system prompt resent on every call is pure waste. Cache the prefix and you pay roughly 0.1× for it after the first hit — a large saving for apps with big, stable system prompts or shared context.

• Model cascade / routing. Don’t send everything to the biggest model. Let a cheap model handle the easy 80% and escalate only the hard 20% to the expensive one:

  0.8 × $0.25  +  0.2 × $5   =   $1.20 per 1M tokens (blended)
  vs flat $5                 →   ~4× cheaper

  Route by difficulty (a classifier, a confidence check, or the cheap model saying “I’m not sure”), not by default.

• Semantic cache. Many users ask the same thing in different words. Cache answers by meaning — embed the query, reuse the answer on a near-match. A 30% hit-rate is 30% fewer model calls and near-instant responses for those users.

• Batch + quantize when self-hosting. Batch concurrent requests to fill the GPU (throughput up), and quantize (16-bit → 4-bit) to fit a cheaper box — covered in open LLMs in production.

The meta-lesson: measure cost-per-request like you measure latency. Put it on a dashboard, set a budget per feature, and alert when it drifts — the same RED/USE discipline as any backend.

Where AI apps actually get slow and expensive

• Dumping everything into context. The lazy “stuff the whole document in” approach is the #1 cause of both high cost and slow prefill. Retrieve precisely.
• Agents that loop too much. Each step in an agent is a full round trip. Cap the steps, parallelise independent tool calls, and cache tool results.
• No streaming on a chat UI. Users will forgive 8 seconds of streaming; they won’t forgive 8 seconds of blank screen.
• One model for everything. Using a frontier model for spell-check-grade tasks is where budgets quietly die. Route.
• Ignoring output length. Verbose system prompts (“explain your reasoning in detail”) multiply output tokens — and thus latency and cost — on every single call.

Takeaways

• LLM performance is two budgets — latency and cost — both driven by tokens.
• Latency: stream first (0.5 s TTFT beats an 8 s spinner), shorten output, shrink context, use smaller models, parallelise.
• Cost (≈ tokens × price): prompt-cache the stable prefix (~0.1×), cascade/route easy traffic to a cheap model (~4× cheaper), add a semantic cache for repeat questions, batch + quantize when self-hosting.
• Measure cost-per-request on a dashboard with a budget and alerts — most teams find 3–5× of easy savings the first time they look.

This is the AI-specific instalment of the performance series: Performance, Measured (the method), backend performance, and scaling infrastructure. To build on these patterns, see the AI Agent Development and LLM Engineering in Production courses, and the AI engineering vocabulary if any term here was new.