The Short Answer
LoRA (Low-Rank Adaptation) fine-tunes a large model without touching its original weights. Instead of updating the full weight matrix
W, you freeze it and train a tiny pair of low-rank matrices whose product B·A is added as a small correction ΔW. Because the rank r is tiny (often 8–64) next to the model's dimension, you train well under 1% of the parameters, the resulting adapter is a few hundred megabytes instead of tens of gigabytes, and you can swap or stack adapters at inference time. QLoRA makes it cheaper still by keeping the frozen base in 4-bit so a large model fine-tunes on a single GPU; DoRA is a 2026 upgrade that splits each weight into magnitude and direction and usually closes most of the remaining gap to full fine-tuning. For retrieval, the same technique fine-tunes an embedding model or reranker on your own (query, relevant, irrelevant) pairs so it understands your domain's language — with one important catch: fine-tuning an embedding model changes the vector space, so you must re-embed your entire corpus afterward.What is a LoRA adapter?
A transformer layer applies a big learned weight matrix
W (shape d × k) to its input. Full fine-tuning updates every number in W — billions of them — which needs the optimizer states and gradients for the whole model in memory, and produces a full-size copy of the model per task.LoRA starts from the observation that the *change* a task needs,
ΔW, is low-rank: it can be approximated by two thin matrices. So LoRA freezes W and learns ΔW = B · A, where A has shape r × k, B has shape d × r, and the rank r is small (say 16). At inference the layer computes W·x + (B·A)·x, scaled by α / r. Only A and B train, so:B·A back into W for zero added inference latency, or keep the adapter separate and hot-swap adapters per request — the basis for serving many fine-tuned variants from one base model in memory.Two knobs define the adapter. Rank
r sets capacity — how much the adapter can change. Alpha α sets the scaling (α / r); the practical convention is to tie them together so the effective scale stays stable.LoRA vs full fine-tuning vs QLoRA vs DoRA
| Method | What it trains | Memory to train | Best for | Main tradeoff |
| Full fine-tuning | Every weight | Very high (optimizer + grads for the whole model) | Maximum quality when you have large, clean data and the GPUs | Expensive; a full model copy per task |
| LoRA | Low-rank adapters (B·A) on frozen weights | Low | The default for most domain adaptation | A small quality ceiling below full FT on hard tasks |
| QLoRA | LoRA adapters over a frozen 4-bit base | Very low — a large model on one GPU | Fine-tuning big models on limited hardware | Minor quality hit from 4-bit quantization |
| DoRA | Magnitude + direction, LoRA on the direction | Low (≈ LoRA) | Closing most of the gap to full FT for free | Slightly more compute than plain LoRA |
How do I fine-tune with LoRA? The knobs that matter
The Hugging Face PEFT library is the standard implementation; the settings below are where most of the outcome is decided.
α = r = 16 (a scaling of 1.0) — simple and stable. As a rule of thumb, r = 16 handles a narrow style or format adaptation, r = 32 a general supervised fine-tune, and r = 64 complex, multi-turn, or code-like tasks. Setting α = 2r (as in Microsoft's original LoRA examples) is a common alternative.all-linear — apply LoRA to every linear layer (q, k, v, o attention projections plus the gate, up, down MLP projections). It consistently beats attention-only adaptation for a small extra VRAM cost.use_dora=True is close to a free upgrade in quality; combine it with load_in_4bit=True for the QLoRA + DoRA default.Fine-tuning retrieval models with LoRA (embedders, rerankers, VLMs)
Most LoRA tutorials fine-tune a chat LLM for text generation. The same machinery — the technique is model-architecture-agnostic — adapts the models that power search, and that is where it earns its keep for retrieval systems.
(query, positive, negative) examples so that relevant pairs land close and irrelevant ones land far. LoRA sits on the encoder's linear layers exactly as it would on an LLM. The payoff is domain recall: an embedder that has never seen your part numbers, ICD codes, or ad-creative jargon can be taught, with a few hundred labeled pairs, to rank the right results first.(query, document, relevance) data. LoRA makes it cheap to specialize a general reranker (see the best rerankers) to your corpus without hosting a full custom model.The retrieval-specific catch that catches everyone: fine-tuning an embedding model changes the geometry of its vector space. Vectors produced by the base model and vectors produced by the fine-tuned model are no longer comparable, so you cannot mix them in the same index. After you fine-tune an embedder you must re-embed your entire corpus with the new model before searching. Budget for that re-embedding pass — on a large library it is the real cost of the change, not the training. (This is the same versioning problem covered in changing embedding models without breaking your index.)
Swappable adapters unlock multi-tenant retrieval. Because a LoRA adapter is small and separable, you can keep one base embedding model resident and attach a different adapter per tenant, per domain, or per language — many specialized retrievers from one base model, instead of N full copies. You still re-embed each tenant's corpus with that tenant's adapter, but you serve them all from shared weights.
When should you *not* reach for LoRA?
Fine-tuning is not the first lever. Before you train anything, rule out the cheaper fixes:
How this fits a Mixpeek pipeline
Mixpeek is deliberately model-agnostic, which makes it a clean home for LoRA-adapted retrieval. MVS stores whatever vectors your model produces — base, fully fine-tuned, or LoRA-adapted — so the workflow is: fine-tune your embedder with LoRA, re-embed your corpus, and upsert the new vectors into an MVS namespace. Because MVS runs on your own object storage and accepts bring-your-own embeddings, nothing about the adapter leaks into a proprietary format.
The Managed platform goes a step further and abstracts the encoder behind the retrieval API, so you can swap the underlying model or adapter without changing application code — the same "the field moves monthly, don't couple to one model" posture that makes LoRA worth using in the first place. Whichever path you take, the division of labor is clean: LoRA adapts the perception model to your domain; Mixpeek indexes and serves what it produces. Compare current options in the best multimodal embedding models, and see pricing for the re-embedding and storage math.
Frequently Asked Questions
What is the difference between LoRA and full fine-tuning?
Full fine-tuning updates every weight in the model, which requires holding gradients and optimizer state for the entire network in memory and produces a full-size model copy per task. LoRA freezes the original weights and trains only a small pair of low-rank matrices per layer, so you update well under 1% of the parameters, train on far less hardware, and end up with a portable adapter of a few hundred megabytes instead of a multi-gigabyte model. For most domain-adaptation tasks LoRA reaches quality close to full fine-tuning; full fine-tuning still wins when you have very large, clean datasets and the GPU budget to match, or when the task demands changing the model's behavior globally rather than nudging it.
What rank and alpha should I use for LoRA?
Start with rank
r = 16 and alpha α = 16 (a scaling factor of 1.0), which is stable across most tasks. Increase the rank when the task is harder: roughly r = 16 for a narrow style or format change, r = 32 for a general supervised fine-tune, and r = 64 for complex, multi-turn, or code-heavy adaptation. A common alternative is to set alpha to twice the rank (α = 2r). Higher rank adds capacity and a little memory but is not automatically better — if a small rank already fits your data, raising it mostly risks overfitting. Always confirm the choice against an evaluation on your own data rather than a default.Do I need to re-embed my data after fine-tuning an embedding model?
Yes. Fine-tuning changes the embedding model's vector space, so vectors from the old model and the new model are not comparable and cannot coexist in one index. You must re-embed your entire corpus with the fine-tuned model before you search, and you cannot query new-model vectors against an index built from old-model vectors. On a large library this re-embedding pass is the dominant cost of the change — plan for it explicitly, and treat the model version as part of your index's identity.
Can I fine-tune an embedding model or reranker with LoRA, or only LLMs?
You can fine-tune any transformer-based model with LoRA, including embedding models, cross-encoder rerankers, and vision-language models — the technique adapts weight matrices and does not care what task those weights serve. The difference is the training objective: chat LLMs use next-token prediction, embedding models use a contrastive or triplet loss on relevant/irrelevant pairs, and rerankers use a relevance-labeled ranking loss. LoRA attaches to the linear layers in each case. Adapting an embedder or reranker to your domain is often a higher-ROI use of LoRA than fine-tuning a generator, because retrieval quality sets the ceiling for the whole pipeline.
QLoRA vs LoRA vs DoRA — which should I use in 2026?
Use plain LoRA when the base model already fits comfortably in your GPU memory at full or 16-bit precision and simplicity matters. Use QLoRA when the model is too large for that — it keeps the frozen base in 4-bit so a big model fine-tunes on a single GPU, at a minor quality cost from quantization. DoRA is an incremental upgrade to either: it decomposes each weight into magnitude and direction and applies the low-rank update to the direction, which typically recovers most of the remaining gap to full fine-tuning for little extra compute. The 2026 default for a new project is to combine them — QLoRA plus DoRA — and only simplify if you hit a problem.