Should You Self-Host an Open LLM?

A fintech team I advised was about to sign a large annual contract with an LLM API provider. Their compliance team blocked it at the last minute: customer financial data couldn’t leave their network, and the API’s data-processing terms didn’t satisfy the auditor. So they swung the other way and decided to self-host everything, bought two expensive GPUs, and three months later had spent more on engineers and idle hardware than the API would have cost, while serving a model that was slower and weaker than the API for their hardest queries.

The right answer was in between: self-host a small model for the routine, sensitive, high-volume work, and keep an API (with the right contract, or a private deployment) for the hard 10%. This lesson is about making that decision deliberately, before you buy anything.

First, What Does “Open” Mean?

The word “open” is used loosely, and the difference matters:

• Open weights: you can download the trained model and run it. You can fine-tune it, quantize it, and serve it yourself. Most “open” models (Llama, Qwen, Mistral, Gemma) are this. The training data and full training code are usually not released, and the licence may have conditions (Llama restricts use above a certain scale; some models are research-only).
• Open source (in the fuller sense): weights, training code, and ideally the data are all public, under a proper open-source licence. Fewer models qualify (some, like OLMo, aim for this).

For production, what you almost always care about is open weights plus a licence that permits your use. Always read the licence: can you use it commercially, is there a scale limit, can you use its outputs to train other models, are there territory restrictions? This is a legal question, not a technical one, and it’s worth asking your legal team early.

The families worth knowing as of this writing: Llama (Meta), Qwen (Alibaba, strong multilingual including Indian languages), Mistral / Mixtral, Gemma (Google), DeepSeek (strong reasoning and code), Phi (small, capable). Don’t memorise a leaderboard; it changes every few weeks. Pick two or three candidates for your task and test them on your data (Lesson 10).

The Decision Framework

Here’s the set of questions I work through, in order. If any of the first four is a strong yes, self-hosting is worth serious consideration. If none is, use an API.

1. Does the data have to stay in your control?

This is the most common real reason to self-host in India, and often the only one that matters on its own.

• DPDP Act (2023) and sector rules can require certain personal data to be handled in specific ways.
• RBI rules push payment and some financial data to stay in India.
• Healthcare, defence, government and legal work often can’t send records to a third-party API at all.
• A client contract may forbid sending their data to an external AI service.

If this applies, your options are: self-host inside your own network, or use a private/dedicated deployment of a model in an Indian cloud region with a contract that satisfies your auditor. Note that “self-hosting an open model” and “using an API” aren’t the only two choices; a dedicated endpoint in-region is a middle path. But if the data truly can’t leave your own machines, self-hosting is the answer.

2. Is the task narrow and repetitive?

Big general-purpose models are built to do everything. If your task is one specific thing done millions of times, a small open model fine-tuned on that task often matches or beats a big general model, at a fraction of the cost and latency. Examples:

• Classify support tickets into 20 categories.
• Extract structured fields (name, amount, date) from invoices or KYC documents.
• Route a message to the right team.
• Summarise call transcripts in a fixed format.
• Detect a specific kind of policy violation.

A fine-tuned 7B or even 1-3B model can do these well (Lesson 9). You get low, predictable latency and cost, and you control the exact behaviour. For open-ended reasoning, novel problems, or “answer anything” chat, the big models still have a real edge.

3. Is volume high and steady?

Self-hosting’s cost advantage comes entirely from keeping the GPUs busy. We’ll do the maths in Lesson 8, but the short version: a GPU costs the same per hour whether it’s at 5% or 95% utilisation. If you push crores of tokens a day, 24x7, the cost per token can fall well below API prices. If your traffic is a few thousand requests a day, or spiky (busy 9-6, idle at night), a GPU sitting mostly idle is pure waste, and the API’s pay-per-token model is cheaper and simpler.

4. Do you need control?

Some needs only self-hosting (or a dedicated deployment) can meet:

• A frozen model version that never changes under you. APIs deprecate and update models; a regulated workflow may need the exact same model for years.
• Custom decoding: constrained/grammar-based output, custom sampling, logit biasing, guaranteed JSON.
• Offline or edge: on a factory floor, in a vehicle, on-premises with no internet.
• No rate limits of someone else’s choosing, and no risk of being throttled during your peak.

5. If none of the above: use an API

If your volume is low or spiky, you need frontier-level reasoning, and your team is small, an API is almost certainly the right choice. You get the best models, no hardware to manage, no on-call for GPU nodes, and you pay only for what you use. There’s no prize for running your own GPUs. Revisit the decision when your volume grows or a data rule changes.

The Hidden Costs People Forget

When teams compare “GPU rental vs API price per token”, they usually leave out most of the real cost of self-hosting:

• Engineers’ time: setting it up, upgrading engines, debugging CUDA errors, being on-call when a GPU node dies at 2 AM. This is often the biggest cost, and it’s ongoing.
• Redundancy: one GPU is a single point of failure. Production usually means at least two, often across zones. That doubles the hardware cost before you serve a single extra user.
• Idle capacity for peaks: to handle your busy hour, you provision for the peak, and pay for that capacity during the quiet hours too (unless you can scale down fast, which has its own cold-start cost, Lesson 11).
• Keeping up: better open models come out every few weeks. Someone has to evaluate, quantize, and roll them out.

The API price, by contrast, bundles all of this: the provider’s engineers, their redundancy, their idle capacity. That’s why the break-even volume for self-hosting is higher than a naive comparison suggests.

Hybrid Is Usually the Real Answer

In practice, the best setups I’ve seen are hybrid:

• A self-hosted open model handles the routine, high-volume, sensitive 80-90% of requests: classification, extraction, simple Q&A, the first draft.
• An API model handles the hard 10%: complex reasoning, rare edge cases, anything the small model is unsure about.
• A gateway routes between them by request type or by the small model’s confidence, and logs which model answered.

This gives you the cost and data benefits of self-hosting for the bulk of traffic, and the capability of a frontier model where you actually need it. It also de-risks the decision: you can start 100% on the API, measure which requests dominate your bill, and move just those in-house.

A Note on Model Size vs Task

One more framing that saves money and latency: match the model to the task, not to the leaderboard. A common mistake is reaching for the biggest model “to be safe”. But:

• A 70B model is roughly 10x slower and 10x more memory than a 7B (Lesson 3), for a quality gain that may be irrelevant to your task.
• For classification, extraction and routing, small models (1-8B), especially fine-tuned, are often indistinguishable from large ones on the metric you care about.
• Reserve the large models for genuinely hard reasoning, long-context synthesis, and open-ended generation.

Test the smallest model that might work first (Lesson 10). Scale up only if your eval set says you need to.

Pitfalls to Avoid

1. Self-hosting for prestige, not for a concrete reason from the framework above.
2. Comparing only GPU rental vs token price, ignoring engineers, redundancy and idle time.
3. Self-hosting spiky or low traffic. A mostly-idle GPU is the most expensive way to run an LLM.
4. Reaching for the biggest model when a small fine-tuned one would pass your eval.
5. Ignoring the licence. “Open weights” doesn’t always mean “free for your commercial use at your scale”.
6. An all-or-nothing choice. Hybrid (self-host most, API for the hard part) is usually best.
7. Confusing “open weights” with “open source”. You rarely get the training data, and the licence may have conditions.

Quick Recap

• Self-host when data must stay in your control, the task is narrow and repetitive, volume is high and steady, or you need version/decoding control.
• Use an API when volume is low or spiky, you need frontier reasoning, or the team is small.
• The real cost of self-hosting includes engineers, redundancy and idle capacity, not just the GPU rental.
• Match the model to the task; a small fine-tuned model often beats a big general one for narrow work.
• Hybrid (self-host the routine 90%, API for the hard 10%) is usually the most practical design.

Next lesson: the single most important technical fact for everything that follows. How an LLM actually uses your hardware, why it has two very different phases, and why decode speed is limited by memory, not compute.