Security, Compliance + Capstone — Open LLMs in Production — Serving, Quantization, Fine-Tuning and Cost

We’ve covered how to choose, size, speed up, cost out, fine-tune, evaluate and scale a self-hosted open model. This final lesson covers the part that gets the least attention and causes the worst incidents: security, because when you self-host, you own the whole stack’s safety. Then we’ll pull all twelve lessons together into one capstone design.

A Downloaded Model Is Untrusted Software

The mental shift that matters most: a model you download is not just data. It’s weights plus code plus config, published by someone on the internet. Treat it the way you’d treat any third-party dependency, with suspicion until verified.

The sharpest version of this is the file format. PyTorch’s classic .bin / .pt checkpoints are stored with Python’s pickle, and pickle can execute arbitrary code when it’s loaded. I verified this in the course: I saved a crafted checkpoint containing a small payload, and loading it the old way (torch.load(..., weights_only=False)) ran a shell command on the machine during load, printing the user it ran as. Loading the same file with modern safe methods refused it:

weights_only=False  ->  code ran during model load (as root)
weights_only=True   ->  UnpicklingError (refused): weights only load failed
safetensors         ->  data only; there is no code path to run

Since PyTorch 2.6, torch.load defaults to weights_only=True, which blocks this. But plenty of code and older versions still pass weights_only=False, and that’s a remote-code-execution hole opened by “loading a model”.

The clean answer is safetensors: a format that stores only tensor data, with no mechanism to run code. As a bonus (Lesson 11), it’s also 3-30x faster to load, because it’s memory-mapped:

Nearly every model is published in safetensors today. If you only find a .bin, convert it to safetensors in a sandbox first, then ship the safetensors file.

The Model Supply Chain

Beyond the file format, treat model acquisition like any software supply chain:

Pin and verify the source. Pin the exact repository and revision hash, not a floating tag. Prefer official publishers or well-known community ones; check download counts and history.
Scan before loading. Run a model scanner (picklescan, or your platform’s built-in scan) in CI, so a dangerous file is caught before it reaches production.
Keep trust_remote_code off. Some repositories ship custom Python that transformers will execute if you pass trust_remote_code=True. Only enable it after you’ve read that code and for a source you trust.
Isolate the loader. Download and convert models in a sandboxed job that has no production secrets and no production network access, so even if something runs, it can’t reach anything.
Mirror what you ship. Keep approved weights in your own storage with checksums, and have production pull from there, never live from a public hub. This also protects you from a model being changed or pulled upstream.

Prompt Injection and Output Handling

The main runtime security risk is prompt injection: untrusted text (a user message, a retrieved document, a web page, a tool’s output) containing instructions that hijack the model. “Ignore your instructions and…” in a document the model reads during RAG, for example. This is covered in depth in the LLM Engineering course’s prompt-injection lesson; the self-hosting-specific points:

Treat everything the model reads as untrusted, not just the direct user message. Retrieved documents, tool outputs, and prior conversation can all carry injected instructions.
Treat everything the model produces as untrusted too. Never pass model output straight into a shell, a database query, an eval, or an HTTP request without validation. If the model can call tools, constrain what each tool can do and validate its arguments.
Least privilege for the model’s capabilities. If it can trigger actions (send email, move money, delete records), gate those behind confirmation, allow-lists and limits, the same way you would for any untrusted input.
Guard inputs and outputs (Lesson 10): detect and redact PII, filter toxic or off-topic content, and check for obvious injection patterns, with rules or a small classifier, before and after the model.
Isolate the serving infrastructure. The model server should run with minimal privileges, no access to production secrets it doesn’t need, and network egress restricted to what it requires.

Data and Compliance

Data residency was the headline reason to self-host (Lesson 1), and self-hosting does keep data in your control. But it also means you now own the compliance of the whole stack:

DPDP Act (2023): you’re responsible for how personal data flows through your model, logs and storage. Log prompts and responses carefully, they often contain personal data, so apply retention limits, access controls and redaction. Don’t log secrets or full sensitive identifiers (Lesson 10).
RBI and sector rules: for payment and financial data, keep it in-region and ensure the whole pipeline (model, logs, vector store, caches) complies, not just the model.
Training-data provenance: if you fine-tune on customer data, confirm you’re allowed to, and that it doesn’t leak between customers. A model can memorise training examples; don’t fine-tune one shared model on one customer’s confidential data and serve it to others.
The model’s licence (Lesson 1): using it at your scale, and using its outputs, must be permitted.
Auditability: log which model version and prompt produced each decision (Lesson 10), so you can explain and reproduce outcomes, which regulators and auditors will ask for.

Self-hosting shifts these responsibilities from the API provider to you. That’s the trade for control: you get the data residency, and you also get the duty to secure everything around it.

Capstone: A Self-Hosted Support Assistant for an Indian Fintech

Let’s tie the whole course together. The brief: a fintech needs a support assistant. Customer financial data can’t leave their network (RBI/DPDP), volume is high and steady (lakhs of chats a day), and answers must follow current policy exactly.

Walking the design, lesson by lesson:

Lessons 1-2: Data residency plus steady 24x7 volume → self-host a 7B model in an Indian region, with an API only as a fallback for the hardest queries. The workload is mixed (medium prompts with retrieved policy text, medium answers).
Lesson 3: Choose Qwen2.5-7B partly for its small KV cache (56 KiB/token), so long chat histories and many concurrent users fit on each GPU.
Lessons 4-5: FP8 weights (near-lossless) on L40S-class GPUs; serve with vLLM for PagedAttention, continuous batching and prefix caching.
Lesson 6: Tune the batch size to the chat latency SLO; measured sustained throughput sets capacity and cost.
Lesson 7: Put the stable system prompt and policy text first, the user’s question last, so prefix caching cuts TTFT (12.5x in the test). Stream every response.
Lesson 8: Compute cost per 1,000 chats from the measured throughput and the real GPU quote, and compare with the API price to confirm self-hosting wins at this volume.
Lesson 9: One base model with LoRA adapters for Hindi support tone and KYC field extraction. Crucially, facts (policies, FAQs) live in RAG, not in the weights, so they update without retraining.
Lesson 10: A golden eval set gates every model, quantization and engine change, scored per slice (Hindi, long inputs, edge cases). Roll out shadow → canary → ramp, with instant rollback. Log model + version + template with every answer.
Lesson 11: Autoscale GPU pods on queue depth / KV usage; keep warm replicas and an API fallback to cover the minutes-long cold start. Multi-LoRA serves both adapters on the same GPUs.
Lesson 12: safetensors only, pinned model revisions, a sandboxed loader; PII redaction, prompt-injection checks and output validation before anything reaches the user; human handoff for sensitive cases; careful logging under DPDP.

Is this a lot? Yes. But each piece traces to a concrete requirement, and you grow into it: start with the model behind a gateway, add RAG, add the eval gate, add adapters, add autoscaling as volume grows. The order of the lessons is roughly the order you’d build it.

Pitfalls to Avoid

Loading models from pickle .bin files with weights_only=False. It’s remote code execution. Use safetensors.
trust_remote_code=True on an unverified repository.
Pulling models live from a public hub in production instead of a mirrored, checksummed copy.
Treating model output as trusted. Never pass it to a shell, query or action without validation.
Treating only the user message as untrusted. Retrieved documents and tool outputs can carry injections.
Logging prompts and responses without redaction or retention limits. They contain personal data.
Assuming self-hosting alone makes you compliant. You now own the security of the whole pipeline.
Fine-tuning one shared model on one customer’s confidential data and serving it to others.

Quick Recap

A model file is untrusted software: a crafted pickle checkpoint ran shell commands in a real test; safetensors (also 3-30x faster to load) and weights-only loading refused it.
Supply chain: safetensors, pin and verify the source, scan in CI, trust_remote_code off, isolate the loader, mirror approved weights.
Prompt injection is the main runtime risk; treat all model input and output as untrusted, apply least privilege and guard both sides.
Self-hosting keeps data in your control for DPDP/RBI, but you now own the whole stack’s security and auditability.
The capstone combined every lesson: self-host for residency, a small-KV-cache model, FP8 on vLLM with prefix caching and batching, cost from measured throughput, adapters plus RAG, an eval gate and staged rollout, queue-based autoscaling, and a safe supply chain.

Course Wrap-Up

If you remember only a handful of things from this course:

Decode is memory-bound. Single-user speed ≈ bandwidth ÷ model size. It explains quantization, batching and hardware choice.
The KV cache, not the weights, usually decides how many users fit. GQA and context length drive it.
Quantization is nearly free at 8-bit and usually fine at 4-bit with good methods, but always measure on your own evals.
Throughput, through batching and a real serving engine, is what makes self-hosting economical, and utilisation dominates the cost.
Fine-tune for format and style, retrieve for facts. LoRA makes it cheap; evaluate what you didn’t train for.
Every model change is a release. Eval, per slice, then roll out gradually.
A model is untrusted software. safetensors, pinned sources, guarded inputs and outputs.

Thank you for working through all twelve lessons. The experiments in this course ran on a plain CPU; run them yourself, scale them to the model and hardware you actually have, and the numbers will teach you more than any slide. Self-hosting an open LLM well is not magic, it’s this handful of concrete facts, measured and respected.