Fine-Tuning with LoRA and QLoRA — Open LLMs in Production — Serving, Quantization, Fine-Tuning and Cost

A team spent six weeks fine-tuning a model to answer questions about their product, feeding it their entire documentation. It worked, until the documentation changed the next month, and the model confidently gave the old answers. They had used fine-tuning to teach facts, which is exactly what it’s bad at. The facts should have gone in a retrieval system (RAG); fine-tuning should have been used, if at all, to teach the format and tone of a good answer.

Fine-tuning is powerful and, with LoRA, surprisingly cheap and accessible. But it’s misused more than almost any other LLM technique. This lesson shows what it actually does, with a real training run, and when to reach for it.

What Fine-Tuning Is For (and Not For)

Fine-tuning adjusts a model’s weights on your examples so it behaves more like them. It’s good at teaching behaviour:

Format: always output this exact JSON schema, this table structure, this template.
Style and tone: your brand’s voice, a specific level of formality, replies in Hinglish, concise medical summaries.
A narrow skill: classify into your 30 categories, extract your specific fields, follow your domain’s conventions.
Implicit rules that are hard to write as a prompt: what counts as a policy violation in your domain, how to triage.

It’s bad at teaching facts, especially facts that change:

Your current product catalogue, prices, policies, inventory. These change; the model’s weights don’t. Use RAG (put the facts in a retrieval system and feed them in the prompt).
Knowledge you need to update frequently. Re-fine-tuning for every change is slow and expensive.

The rule I use: fine-tune for how to say it, retrieve for what to say. And before fine-tuning at all, try the cheaper options first: a better prompt, few-shot examples in the prompt, or RAG. Fine-tune only when prompting plateaus and you have enough good examples. Often you won’t need to.

Why LoRA Changed the Economics

Full fine-tuning updates all the model’s weights. For an 8B model that means gradients and optimizer state for 8 billion parameters, which needs many times the model’s size in GPU memory (weights + gradients + two Adam states + an fp32 master copy ≈ 16 bytes/param ≈ 128 GB for 8B). That’s multiple high-end GPUs, for one experiment.

LoRA (Low-Rank Adaptation) sidesteps this. The insight: the change a fine-tune makes to a big weight matrix is usually low-rank, it can be captured by two small matrices multiplied together. So you freeze the original weight W and learn a small update B·A, where A and B are thin (rank r, e.g. 8 or 16).

LoRA trains two small matrices instead of the whole model. The big weight W, for example 4096 by 4096 with 16.8 million numbers, stays frozen. A low-rank update B times A of rank r (for example 8 or 16) learns the change: B is 4096 by r and A is r by 4096, so at rank 16 that is 131 thousand numbers, about 0.8% of W. Real numbers: Llama-3.1-8B with rank 16 on all linear layers is 41.9 million trainable parameters, 0.52% of 8.03B, an adapter file about 80 MiB in fp16; Llama-3.1-70B with the same setting is 207 million trainable, 0.29%, an adapter about 395 MiB. Training memory needs no gradients or optimizer state for W, only for A and B, which is the whole trick. At serving time you can merge B times A into W for one task or keep adapters separate and load many on one base model. QLoRA keeps W in 4-bit while training the adapters in 16-bit so an 8B fine-tune fits on one 24 GB GPU

Because W is frozen, you don’t store gradients or optimizer state for it, only for the tiny A and B. That’s the whole trick, and it makes the memory drop dramatically. I computed the trainable fraction for real models (rank 16 on all linear layers):

Llama-3.1-8B:  41.9M trainable = 0.52% of 8.03B,  adapter ~80 MiB (fp16)
Llama-3.1-70B: 207M trainable = 0.29% of 70.55B,  adapter ~395 MiB

You train well under 1% of the parameters, and the result is a small adapter file you can store, version, and swap.

A Real LoRA Training Run

To show the mechanics for real (not just the arithmetic), I ran an actual end-to-end LoRA fine-tune on the CPU. I built a tiny Llama-architecture model (1.05M parameters) and first pretrained it to copy an 8-digit string (output = input). It learned that perfectly: 100% copy accuracy, 0% at reversing.

Then I added a LoRA adapter (rank 8, 94,208 trainable parameters = 8.9% of the model) and trained only the adapter to reverse the digits instead, leaving the base model frozen:

$LoRA in action, a real CPU training run. A tiny Llama-architecture model of 1.05 million parameters was pretrained to copy 8 digits, then a LoRA adapter of rank 8 with 94 thousand parameters (8.9%) was trained to reverse them. Reverse-task accuracy on 512 test strings rose over LoRA training steps: 0% at 0 and 250 steps (loss 2.041), 75.2% at 500 steps (loss 0.209), 96.9% at 750 steps (loss 0.017), 98.0% at 1000 steps (loss 0.006). With the adapter off, the base model gave 100% copy accuracy and 0% reverse, and base weights never changed. With the adapter on, reverse accuracy was 98.6% and copy accuracy dropped to 0.0% because it specialised, with an adapter file of 381 KiB. Toy task but real mechanics: a tiny trainable fraction, the base model untouched, the adapter switchable at serving time; a fine-tune can wreck skills you did not train for, so always evaluate what you didn't train for$

LoRA training (adapter only, base frozen):
  250 steps:  reverse accuracy  0.0%   loss 2.041
  500 steps:  reverse accuracy 75.2%   loss 0.209
  750 steps:  reverse accuracy 96.9%   loss 0.017
 1000 steps:  reverse accuracy 98.0%   loss 0.006

adapter OFF:  copy 100.0%   reverse  0.0%    (the original base model, unchanged)
adapter ON:   copy   0.0%   reverse 98.6%
adapter file size: 381 KiB

Three things to take from this:

The adapter learned the new skill to 98%+ accuracy, training less than 9% of the parameters.
The base model is untouched. Switch the adapter off and you get the original copy behaviour back, perfectly. The adapter is a small, separable, switchable file (381 KiB here; 80 MiB for a real 8B).
The catastrophic-forgetting warning: notice that with the adapter on, the model’s copy accuracy dropped from 100% to 0%. Specialising it for reversing destroyed its original skill. This is real and important: a fine-tune can silently break capabilities you didn’t train for. Always evaluate the skills you care about keeping, not just the one you’re training (Lesson 10).

The code is short, which is part of why LoRA is so widely used (Hugging Face peft):

from peft import LoraConfig, get_peft_model

lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05,
                  target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                                  "gate_proj", "up_proj", "down_proj"])
model = get_peft_model(base_model, lora)     # base_model stays frozen
model.print_trainable_parameters()           # ~0.5% of params
# ... standard training loop on your examples ...
model.save_pretrained("my-adapter")          # tiny adapter file

QLoRA: Fine-Tune an 8B on One Consumer GPU

QLoRA combines LoRA with quantization (Lesson 4). It keeps the frozen base model in 4-bit (so it takes ~4x less memory) while training the LoRA adapters in 16-bit. The gradients flow through the 4-bit base without updating it. The result: you can fine-tune an 8B model on a single 16-24 GB GPU, and even larger models on one 48-80 GB GPU.

GPU memory to fine-tune an 8B model, rough estimates before activations. Full fine-tune with Adam in mixed precision: about 128 GB, roughly 16 bytes per parameter for weights, gradients, two Adam states and an fp32 master copy, needing multiple 80 GB GPUs. LoRA on a 16-bit base: about 18 GB, a 16 GB frozen base plus a small adapter with its gradients and optimizer state, fitting one 24 to 48 GB GPU. QLoRA with a 4-bit base: about 7 GB, roughly a 5 GB 4-bit base plus adapters in 16-bit, fitting one 16 to 24 GB GPU. Before fine-tuning at all, try a better prompt, few-shot examples or RAG; fine-tune for format, style and narrow skills, not to teach facts that change, which belong in RAG; a few hundred to a few thousand good examples beat lakhs of noisy ones

The approximate memory picture (before activations, which depend on batch size and sequence length):

Full fine-tune (8B, Adam, mixed precision):  ~128 GB   -> multiple 80 GB GPUs
LoRA on 16-bit base:                          ~18 GB   -> one 24-48 GB GPU
QLoRA (4-bit base):                            ~7 GB   -> one 16-24 GB GPU

QLoRA is what put fine-tuning within reach of individuals and small teams. For most production fine-tuning of 7-13B models, QLoRA or LoRA is the right default; full fine-tuning is rarely worth its cost unless you’re changing the model deeply.

Serving Fine-Tuned Models

Two options, and the choice matters for Lesson 11:

Merge the adapter into the base weights (B·A added to W), producing a normal model file. Simplest to serve, slightly faster, but it’s a full-size model per task.
Keep adapters separate and load them on top of a shared base at serving time. This enables multi-LoRA (Lesson 11): one base model in GPU memory, many small adapters, each request choosing its adapter. Four fine-tuned products on one GPU instead of four GPUs. A small latency cost vs merged.

For a single fine-tuned model, merge. For several fine-tunes of the same base, keep them separate and serve multi-LoRA.

Data Quality Beats Data Quantity

A few hard-won rules on the training data, which matters far more than the method:

A few hundred to a few thousand good examples usually beat lakhs of noisy ones. Quality, consistency and coverage of your real cases matter more than volume.
Be consistent. If your examples format dates three different ways, the model learns to be inconsistent too.
Match production. Train on the kind of inputs you’ll actually see, including the messy ones.
Hold out a test set from the start, and never train on it (Lesson 10).
Include “hard negatives” and edge cases, not just easy examples.
Watch for leakage and bias in the data; the model will faithfully learn both.

Pitfalls to Avoid

Fine-tuning to teach facts. Facts change; weights don’t. Use RAG for knowledge.
Fine-tuning before trying prompting, few-shot and RAG. Often those are enough.
Ignoring catastrophic forgetting. The run’s copy skill went 100% to 0%; evaluate skills you didn’t train for.
Full fine-tuning when LoRA/QLoRA would do. Much more memory and cost for usually no benefit.
Prioritising data quantity over quality. A few thousand clean examples beat lakhs of noisy ones.
No held-out test set. You can’t tell if it improved or just memorised.
Forgetting licensing. Some model licences restrict using outputs to train other models; check (Lesson 1).

Quick Recap

LoRA freezes the base model and trains tiny low-rank matrices: real run, a 94k-param adapter (8.9%) reached 98% on a new task, base untouched, saved as a 381 KiB switchable file.
Real adapters are small: ~0.52% of an 8B (80 MiB), ~0.29% of a 70B (395 MiB) at rank 16.
QLoRA (4-bit base + 16-bit adapters) fits an 8B fine-tune on one 24 GB GPU; full fine-tune needs ~128 GB.
Fine-tune for format, style and narrow skills; use RAG for facts that change. Try prompting and RAG first.
Watch catastrophic forgetting (copy skill went 100% to 0%); quality of a few thousand examples beats quantity.

Next lesson: making sure a model change is actually an improvement. Evaluation sets, what silently breaks quality, and rolling out a new model safely with shadow and canary stages.