menu_bookGyanByte
arrow_backBACK TO OPEN LLMS IN PRODUCTION — SERVING, QUANTIZATION, FINE-TUNING AND COST
Lesson 08Open LLMs in Production — Serving, Quantization, Fine-Tuning and Cost6 min read

The Real Cost in Rupees

October 27, 2026

TL;DR

Self-hosted cost per million tokens = GPU cost per hour ÷ (tokens/s × 3,600 × utilisation) × 1,000,000. The dominant variable is utilisation (how busy the GPU is): in a worked example, the same GPU cost about ₹50 per million output tokens at 70% utilisation but ₹694 at 5%, a 14x swing, because an idle GPU still costs money. This is why self-hosting only beats an API at high, steady volume. The formula also leaves out input tokens, engineer time, redundancy (at least 2 GPUs) and idle peak capacity, all of which an API bundles into its price. Break-even is typically hundreds of crores of tokens a month. Every number here is illustrative; the calculation hinges on your own measured sustained throughput (Lesson 6) and a real GPU quote. Measure before you buy.

A founder once told me, proudly, that self-hosting had cut their LLM costs by 80%. When I asked how busy their GPUs were, the answer was “oh, maybe 10%, we over-provisioned for safety”. They hadn’t cut costs by 80%; they’d increased their cost per token by paying for a GPU that sat idle 90% of the time. Their total bill was lower only because their volume was tiny, in which case the API would have been cheaper and simpler.

Cost is where most self-hosting decisions are made emotionally and justified badly. This lesson gives you the actual formula, shows which variable dominates, and is honest about what the formula leaves out. Every rupee figure here is illustrative; the point is the method, not the numbers.

The Formula

Self-hosted cost comes down to: you rent (or buy) a GPU by the hour, it produces tokens at some rate, and it’s busy some fraction of the time. So:

Self-hosted cost per million tokens equals GPU cost per hour divided by (tokens per second times 3,600 times utilisation) times 1,000,000, where utilisation is the share of paid hours the GPU is actually serving near its measured throughput. A worked example with illustrative numbers: GPU rental 250 rupees per hour for an 80 GB data-centre GPU; measured throughput 2,000 output tokens per second for an 8B model with many concurrent users at an acceptable p95; at 70% utilisation that is 50 rupees per million tokens, at 20% utilisation 174 rupees, at 5% utilisation 694 rupees for a GPU kept just in case. What the formula leaves out: input tokens where prefill is cheaper per token than decode but not free and long RAG prompts can dominate; engineer time, on-call, monitoring and at least 2 GPUs for redundancy; storage, networking and idle capacity for peaks or cold starts

₹ per 1M output tokens = GPU ₹/hour ÷ (tokens/s × 3,600 × utilisation) × 1,000,000

Where utilisation is the fraction of the paid hours the GPU actually spends serving near its measured throughput. Everything hinges on two measured numbers: your sustained tokens/s (Lesson 6) and your real utilisation.

Utilisation Dominates Everything

Here’s a worked example with made-up-but-plausible numbers. An 80 GB data-centre GPU at ₹250/hour, serving an 8B model at a measured 2,000 output tokens/s (achievable with good batching and many concurrent users):

At 70% utilisation:  250 / (2,000 × 3,600 × 0.70) × 1,000,000  =  ₹50  per 1M tokens
At 20% utilisation:  250 / (2,000 × 3,600 × 0.20) × 1,000,000  =  ₹174 per 1M tokens
At  5% utilisation:  250 / (2,000 × 3,600 × 0.05) × 1,000,000  =  ₹694 per 1M tokens

Same GPU, same model, same throughput. The only thing that changed is how busy it is, and the cost per token swung 14x, from ₹50 to ₹694. This is the whole game. An idle GPU costs the same as a busy one, so the cost per token is dominated by how fully you use the hardware.

This is why the decision framework in Lesson 1 kept returning to “high and steady volume”. Self-hosting is cheap only if the GPUs stay busy. A GPU kept “just in case” for a low-traffic service is the most expensive way to serve tokens that exists.

And it’s why throughput (Lesson 6) matters so directly: doubling your tokens/s (through better batching, quantization, or a serving engine with higher throughput) halves your cost per token. The engineering from the earlier lessons is, quite literally, the cost optimisation.

What the Formula Leaves Out

The formula above is the best case for self-hosting, because it counts only the GPU rental and only output tokens. The real cost is higher:

  • Input tokens. Prefill (processing the prompt) is cheaper per token than decode, but it’s not free, and for RAG/agent workloads with long prompts, prefill can be most of the compute. If your prompts are 10x longer than your answers, input processing may dominate.
  • Engineers. Setting it up, upgrading engines, debugging GPU/driver issues, being on-call. This is often the largest line item and it’s continuous. One engineer’s monthly cost can exceed several GPUs’ rental.
  • Redundancy. One GPU is a single point of failure. Production means at least two, usually across availability zones. That doubles the hardware cost before serving one extra user.
  • Idle capacity for peaks. You provision for the busy hour and pay for that capacity in the quiet hours too (unless you scale down, which has cold-start costs, Lesson 11).
  • Storage, networking, egress, and the control-plane bits (gateway, monitoring, the eval pipeline from Lesson 10).

An API price bundles all of this: the provider’s engineers, redundancy, idle capacity, and R&D are all inside the per-token price you pay. That’s why a naive “₹250/hour GPU vs ₹X per token API” comparison flatters self-hosting. Add the real costs, and the break-even volume moves up substantially.

The Break-Even Picture

When self-hosting beats an API, illustrative. Monthly cost versus monthly output token volume in crore. API cost is a straight line growing with volume at an illustrative 150 rupees per million tokens. Self-hosting is a step function: a high fixed cost from at least 2 GPUs at 250 rupees per hour plus about 3 lakh per month for people and operations, flat until capacity runs out, then stepping up as more GPUs are added. The lines cross at an illustrative break-even of a few hundred crore tokens per month, roughly 15 crore per day; below it the API is cheaper, above it self-hosting wins if the GPUs stay busy. All numbers are made up for illustration; plug in your own API price per token, your GPU quote, your measured tokens per second and your real traffic shape

The shapes are what matter, and they’re robust even though the numbers are invented:

  • API cost is a straight line through the origin: you pay per token, so double the volume, double the bill. No fixed cost, no minimum.
  • Self-hosting is a step function with a high floor: you pay for at least 2 GPUs and the people/ops around them whether you serve 1 token or a crore. The cost is roughly flat until you saturate the GPUs, then steps up as you add more.

They cross at a break-even volume. Below it, the API is cheaper (you can’t amortise the fixed cost). Above it, self-hosting wins, provided the GPUs stay busy. In this illustrative chart the crossover is a few hundred crore tokens a month (~15 crore/day); for your situation it could be higher or lower, but it’s almost always “high, steady volume”, which is exactly the Lesson 1 criterion.

Three honest observations from the chart:

  1. At low volume, the API isn’t just cheaper, it’s dramatically cheaper, because you’re not paying a fixed GPU floor at all.
  2. The self-hosting line only pays off if utilisation is high. If you’re on the flat part of the step but your GPUs are at 10%, you’ve got the fixed cost without the volume to justify it.
  3. The hybrid from Lesson 1 shifts the maths: self-host the high-volume routine traffic (which rides far up the API’s straight line), API the low-volume hard traffic (which never justifies its own GPU).

How to Actually Estimate Your Cost

Don’t trust any of the numbers above for your decision. Do this instead:

  1. Measure sustained throughput (Lesson 6): run your real model, with your real prompt and output lengths, at your latency SLO, with realistic concurrency, on the exact GPU you’d use. Get tokens/s (both input/prefill and output/decode).
  2. Get a real GPU quote: on-demand and reserved/committed prices differ a lot; spot/preemptible is cheaper but riskier (Lesson 11).
  3. Estimate realistic utilisation from your traffic shape: average, not peak. Be honest about nights and weekends.
  4. Add the hidden costs: at least 2 GPUs, an engineer’s time, monitoring, idle peak headroom.
  5. Get your API price per token for the equivalent model, input and output separately.
  6. Compare at your actual projected volume, and re-check at 2x and 0.5x volume to see how sensitive the decision is.

If self-hosting wins only at utilisation you won’t realistically hit, use the API. If it wins comfortably even at conservative utilisation, self-hosting is justified, and the earlier lessons tell you how to push the throughput (and thus the saving) higher.

Buy vs Rent vs Serverless

A quick note on the GPU itself:

  • Rent on-demand (cloud): flexible, highest hourly price, good to start.
  • Reserved / committed (1-3 year cloud commitment): much cheaper per hour if your volume is steady and predictable.
  • Buy (own hardware or colo): lowest long-run cost at very high steady volume, but capital upfront, depreciation, and you own the ops. Rarely worth it until you’re very large.
  • Serverless GPU / scale-to-zero: pay only when serving, great for spiky traffic, but cold starts (Lesson 11) add latency, and per-second pricing can be high. Good for bursty or dev workloads.

Match the commitment to how confident you are in the volume. Start on-demand, move to reserved once the volume is proven, consider buying only at large scale.

Pitfalls to Avoid

  1. Comparing GPU rental to API token price directly, ignoring engineers, redundancy and idle time.
  2. Ignoring utilisation. It swung cost per token 14x in the example; it’s the dominant variable.
  3. Provisioning for peak and forgetting you pay for it off-peak too.
  4. Counting only output tokens. Long prompts make input processing a real cost.
  5. Self-hosting at low or spiky volume. The fixed GPU floor makes it more expensive than an API.
  6. Trusting illustrative numbers (including mine). Measure your own throughput and get a real quote.
  7. One GPU in production. Redundancy roughly doubles the floor; budget for it.

Quick Recap

  • Cost per 1M tokens = GPU ₹/hour ÷ (tokens/s × 3,600 × utilisation) × 1,000,000.
  • Utilisation dominates: the same GPU cost ₹50/1M at 70% busy and ₹694/1M at 5% busy in the example.
  • The formula omits input tokens, engineers, redundancy and idle peak capacity, all of which an API bundles into its price.
  • Break-even is typically high, steady volume (hundreds of crore tokens/month in the illustration); below it, an API is cheaper.
  • Decide from your own measured throughput and a real quote, not from any illustrative figure.

Next lesson: making an open model better at your task. Fine-tuning with LoRA and QLoRA, a real training run where a tiny adapter taught a model a new skill, and what fine-tuning is and isn’t for.

llmcostgpueconomicsself-hostingapipricing
arrow_back PreviousLatency: TTFT, TPOT and Prefix CachingNext arrow_forwardFine-Tuning with LoRA and QLoRA