Embeddings

11 min read

Updated 2026-11-04

# How Does LoRA Fine-Tuning Work? (Adapters, QLoRA, DoRA, and Fine-Tuning Retrieval Models)

LoRA fine-tunes a large model by freezing its weights and training a tiny low-rank adapter (a few hundred MB, under 1% of parameters): trainable on one GPU, swappable at inference. How LoRA, QLoRA, and DoRA work; the rank/alpha/target-module knobs that matter in 2026; and the retrieval-specific part most guides skip: fine-tuning embedding models, rerankers, and VLMs with LoRA, why you must re-embed your corpus afterward, and how swappable adapters enable multi-tenant retrieval. Vendor-neutral, with a comparison table and FAQs.

![Ethan Steininger](/people/ethan.webp)

By [Ethan Steininger](/blog/authors/ethan), Founder & CEO, Mixpeek

LoRA

Fine-Tuning

QLoRA

DoRA

Embedding Models

PEFT

Adapters

[Run this on your own data. Try it as you read.Search your own archive →](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)

[One-click templatesStart from a templateEach template deploys a working search app on your own files.Browse templates](/templates)

## The Short Answer

  
  
LoRA (Low-Rank Adaptation) fine-tunes a large model without touching its original weights. Instead of updating the full weight matrix `W`, you freeze it and train a tiny pair of low-rank matrices whose product `B·A` is added as a small correction `ΔW`. Because the rank `r` is tiny (often 8-64) next to the model's dimension, you train well under 1% of the parameters, the resulting adapter is a few hundred megabytes instead of tens of gigabytes, and you can swap or stack adapters at inference time. QLoRA makes it cheaper still by keeping the frozen base in 4-bit so a large model fine-tunes on a single GPU; DoRA is a 2026 upgrade that splits each weight into magnitude and direction and usually closes most of the remaining gap to full fine-tuning. For retrieval, the same technique fine-tunes an **embedding model or reranker** on your own `(query, relevant, irrelevant)` pairs so it understands your domain's language: with one important catch, fine-tuning an embedding model changes the vector space, so you must **re-embed your entire corpus** afterward.  
  

## What is a LoRA adapter?

  
  
A transformer layer applies a big learned weight matrix `W` (shape `d × k`) to its input. Full fine-tuning updates every number in `W`, billions of them, which needs the optimizer states and gradients for the whole model in memory, and produces a full-size copy of the model per task.  
  
LoRA starts from the observation that the \*change\* a task needs, `ΔW`, is low-rank: it can be approximated by two thin matrices. So LoRA freezes `W` and learns `ΔW = B · A`, where `A` has shape `r × k`, `B` has shape `d × r`, and the rank `r` is small (say 16). At inference the layer computes `W·x + (B·A)·x`, scaled by `α / r`. Only `A` and `B` train, so:  
  
*   **You train under 1% of the parameters.** A 7B model has millions of adapter parameters instead of billions of full ones: trainable on a single consumer or cloud GPU.
*   **The adapter is tiny and portable.** A LoRA adapter is typically a few hundred MB versus the base model's tens of GB, so you can store, version, and ship many of them cheaply.
*   **You can merge or swap.** Merge `B·A` back into `W` for zero added inference latency, or keep the adapter separate and hot-swap adapters per request: the basis for serving many fine-tuned variants from one base model in memory.
  
  
Two knobs define the adapter. **Rank `r`** sets capacity: how much the adapter can change. **Alpha `α`** sets the scaling (`α / r`); the practical convention is to tie them together so the effective scale stays stable.  
  

## LoRA vs full fine-tuning vs QLoRA vs DoRA

  
  

Method

What it trains

Memory to train

Best for

Main tradeoff

Full fine-tuning

Every weight

Very high (optimizer + grads for the whole model)

Maximum quality when you have large, clean data and the GPUs

Expensive; a full model copy per task

LoRA

Low-rank adapters (`B·A`) on frozen weights

Low

The default for most domain adaptation

A small quality ceiling below full FT on hard tasks

QLoRA

LoRA adapters over a frozen 4-bit base

Very low: a large model on one GPU

Fine-tuning big models on limited hardware

Minor quality hit from 4-bit quantization

DoRA

Magnitude + direction, LoRA on the direction

Low (≈ LoRA)

Closing most of the gap to full FT for free

Slightly more compute than plain LoRA

  
  
The 2026 default for a new project is **QLoRA + DoRA**: load the base in 4-bit and turn DoRA on. It gives you full-fine-tuning-adjacent quality on a single GPU with an adapter you can swap.  
  

## How do I fine-tune with LoRA? The knobs that matter

  
  
The Hugging Face **PEFT** library is the standard implementation; the settings below are where most of the outcome is decided.  
  
*   **Rank and alpha.** Start with `α = r = 16` (a scaling of 1.0): simple and stable. As a rule of thumb, `r = 16` handles a narrow style or format adaptation, `r = 32` a general supervised fine-tune, and `r = 64` complex, multi-turn, or code-like tasks. Setting `α = 2r` (as in Microsoft's original LoRA examples) is a common alternative.
*   **Target modules.** In 2026 the recommended starting point is `all-linear`: apply LoRA to every linear layer (`q, k, v, o` attention projections plus the `gate, up, down` MLP projections). It consistently beats attention-only adaptation for a small extra VRAM cost.
*   **Turn on DoRA.** `use_dora=True` is close to a free upgrade in quality; combine it with `load_in_4bit=True` for the QLoRA + DoRA default.
*   **Data quality over quantity.** For most adaptation tasks, ~500 clean, representative examples beat 5,000 noisy ones. Curation is the highest-leverage step.
*   **Watch for catastrophic forgetting.** Fine-tuning can erode general knowledge. Track a general benchmark before and after; a drop of more than a few points (for example, an MMLU delta beyond ~3) is a red flag that the run over-fit or the learning rate was too high.
  
  

## Fine-tuning retrieval models with LoRA (embedders, rerankers, VLMs)

  
  
Most LoRA tutorials fine-tune a chat LLM for text generation. The same machinery, the technique is model-architecture-agnostic, adapts the models that power search, and that is where it earns its keep for retrieval systems.  
  
*   **Embedding models.** You do not train an embedder with next-token prediction; you train it with a **contrastive or triplet objective** on `(query, positive, negative)` examples so that relevant pairs land close and irrelevant ones land far. LoRA sits on the encoder's linear layers exactly as it would on an LLM. The payoff is domain recall: an embedder that has never seen your part numbers, ICD codes, or ad-creative jargon can be taught, with a few hundred labeled pairs, to rank the right results first.
*   **Rerankers.** A cross-encoder reranker is fine-tuned on labeled `(query, document, relevance)` data. LoRA makes it cheap to specialize a general reranker (see [the best rerankers](/curated-lists/best-rerankers)) to your corpus without hosting a full custom model.
*   **Vision-language models.** LoRA adapts a [VLM](/curated-lists/best-vision-language-models) for domain visual tasks (reading your document layouts, recognizing your products, captioning in your house style) again at a fraction of full-fine-tuning cost.
  
  
**The retrieval-specific catch that catches everyone:** fine-tuning an embedding model changes the geometry of its vector space. Vectors produced by the base model and vectors produced by the fine-tuned model are no longer comparable, so you **cannot mix them in the same index**. After you fine-tune an embedder you must **re-embed your entire corpus** with the new model before searching. Budget for that re-embedding pass: on a large library it is the real cost of the change, not the training. (This is the same versioning problem covered in [changing embedding models without breaking your index](/guides/embedding-model-migration-without-reembedding).)  
  
We measured this on SciFact: with 919 labeled pairs, LoRA on every linear layer of bge-small-en-v1.5 raised nDCG@10 on 300 held-out queries from 0.706 to 0.744, close to full fine-tuning (0.751), with a 5.4 MB adapter. The learning rate decided it: the same adapter at 1e-4 gained 1.3 points, within noise, and at 1e-3 it gained 3.9. The table and code are in [does fine-tuning an embedding model with LoRA improve search](/guides/does-fine-tuning-an-embedding-model-with-lora-improve-search).  
  
**Swappable adapters unlock multi-tenant retrieval.** Because a LoRA adapter is small and separable, you can keep one base embedding model resident and attach a different adapter per tenant, per domain, or per language: many specialized retrievers from one base model, instead of N full copies. You still re-embed each tenant's corpus with that tenant's adapter, but you serve them all from shared weights.  
  

## When should you \*not\* reach for LoRA?

  
  
Fine-tuning is not the first lever. Before you train anything, rule out the cheaper fixes:  
  
*   **A better base model or better retrieval closes the gap.** Swapping to a stronger embedder, or adding [hybrid search](/guides/hybrid-search-fusion-rrf-score-normalization) and a [reranker](/guides/cross-encoder-reranking), often recovers more relevance than fine-tuning: with no re-embedding and no model to maintain.
*   **Your labels are thin or noisy.** With fewer than a few hundred clean pairs, you will likely overfit; invest in data first.
*   **The domain gap is not actually measured.** Fine-tune when an evaluation on your own queries shows a real, specific gap the base model cannot close: not on suspicion. Measure first (see [evaluating multimodal retrieval](/guides/evaluating-multimodal-retrieval)).
  
  

## How this fits a Mixpeek pipeline

  
  
Mixpeek is deliberately model-agnostic, which makes it a clean home for LoRA-adapted retrieval. [MVS](/mvs) stores whatever vectors your model produces (base, fully fine-tuned, or LoRA-adapted) so the workflow is: fine-tune your embedder with LoRA, re-embed your corpus, and upsert the new vectors into an MVS namespace. Because MVS runs on your own object storage and accepts bring-your-own embeddings, nothing about the adapter leaks into a proprietary format.  
  
The Managed platform goes a step further and abstracts the encoder behind the retrieval API, so you can swap the underlying model or adapter without changing application code: the same "the field moves monthly, don't couple to one model" posture that makes LoRA worth using in the first place. Whichever path you take, the division of labor is clean: LoRA adapts the perception model to your domain; Mixpeek indexes and serves what it produces. Compare current options in [the best multimodal embedding models](/curated-lists/best-multimodal-embedding-models), and see [pricing](/pricing) for the re-embedding and storage math.  
  

## Frequently Asked Questions

  
  

### How do I fine-tune an embedding model with LoRA adapters?

  
  
Load the embedder with sentence-transformers, attach a LoRA adapter with peft, and train it with a contrastive loss on (query, relevant passage) pairs. This is the setup we measured on bge-small-en-v1.5 (sentence-transformers 5.1, peft 0.17):

from peft import LoraConfig, TaskType
from sentence\_transformers import (SentenceTransformer, SentenceTransformerTrainer,
                                   SentenceTransformerTrainingArguments, losses)
from datasets import Dataset

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
model.add\_adapter(LoraConfig(task\_type=TaskType.FEATURE\_EXTRACTION, r=16, lora\_alpha=16,
                             lora\_dropout=0.1, target\_modules="all-linear"))
train = Dataset.from\_dict({"anchor": queries, "positive": passages})
args = SentenceTransformerTrainingArguments(output\_dir="out", num\_train\_epochs=3,
    per\_device\_train\_batch\_size=32, learning\_rate=1e-3, warmup\_ratio=0.1,
    batch\_sampler="no\_duplicates")
SentenceTransformerTrainer(model=model, args=args, train\_dataset=train,
                           loss=losses.MultipleNegativesRankingLoss(model)).train()
model.save\_pretrained("bge-small-lora")  # saves the adapter only

With 919 SciFact pairs this raised nDCG@10 on 300 held-out queries from 0.706 to 0.744. Then score it on your own held-out queries, and re-embed the corpus with the adapted model before searching. The full measurement is in [does fine-tuning an embedding model with LoRA improve search](/guides/does-fine-tuning-an-embedding-model-with-lora-improve-search).  
  

### What learning rate should I use for LoRA on an embedding model?

  
  
Start around 1e-3, which is far higher than the 2e-5 typical for full fine-tuning. A LoRA adapter starts as a zero update and needs larger steps to move in a short run. In our test the same all-linear adapter gained 1.3 nDCG@10 points at 1e-4, within noise, and 3.9 points at 1e-3 (95% CI +1.5 to +6.3).  
  

### How big is a LoRA adapter for an embedding model?

  
  
It scales with the base model. On the 33M-parameter bge-small, rank 16 on all linear layers is 1.34M weights (4% of the model) and 5.4 MB on disk, against 133 MB for a fully fine-tuned copy. Rank 16 on the attention query and value layers only is 1.2 MB. On a 7B model, expect a few hundred megabytes.  
  

### What is the difference between LoRA and full fine-tuning?

  
  
Full fine-tuning updates every weight in the model, which requires holding gradients and optimizer state for the entire network in memory and produces a full-size model copy per task. LoRA freezes the original weights and trains only a small pair of low-rank matrices per layer, so you update well under 1% of the parameters, train on far less hardware, and end up with a portable adapter of a few hundred megabytes instead of a multi-gigabyte model. For most domain-adaptation tasks LoRA reaches quality close to full fine-tuning; full fine-tuning still wins when you have very large, clean datasets and the GPU budget to match, or when the task demands changing the model's behavior globally rather than nudging it.  
  

### What rank and alpha should I use for LoRA?

  
  
Start with rank `r = 16` and alpha `α = 16` (a scaling factor of 1.0), which is stable across most tasks. Increase the rank when the task is harder: roughly `r = 16` for a narrow style or format change, `r = 32` for a general supervised fine-tune, and `r = 64` for complex, multi-turn, or code-heavy adaptation. A common alternative is to set alpha to twice the rank (`α = 2r`). Higher rank adds capacity and a little memory but is not automatically better: if a small rank already fits your data, raising it mostly risks overfitting. Always confirm the choice against an evaluation on your own data rather than a default.  
  

### Do I need to re-embed my data after fine-tuning an embedding model?

  
  
Yes. Fine-tuning changes the embedding model's vector space, so vectors from the old model and the new model are not comparable and cannot coexist in one index. You must re-embed your entire corpus with the fine-tuned model before you search, and you cannot query new-model vectors against an index built from old-model vectors. On a large library this re-embedding pass is the dominant cost of the change: plan for it explicitly, and treat the model version as part of your index's identity.  
  

### Can I fine-tune an embedding model or reranker with LoRA, or only LLMs?

  
  
You can fine-tune any transformer-based model with LoRA, including embedding models, cross-encoder rerankers, and vision-language models: the technique adapts weight matrices and does not care what task those weights serve. The difference is the training objective: chat LLMs use next-token prediction, embedding models use a contrastive or triplet loss on relevant/irrelevant pairs, and rerankers use a relevance-labeled ranking loss. LoRA attaches to the linear layers in each case. Adapting an embedder or reranker to your domain is often a higher-ROI use of LoRA than fine-tuning a generator, because retrieval quality sets the ceiling for the whole pipeline.  
  

### QLoRA vs LoRA vs DoRA, which should I use in 2026?

  
  
Use plain LoRA when the base model already fits comfortably in your GPU memory at full or 16-bit precision and simplicity matters. Use QLoRA when the model is too large for that: it keeps the frozen base in 4-bit so a big model fine-tunes on a single GPU, at a minor quality cost from quantization. DoRA is an incremental upgrade to either: it decomposes each weight into magnitude and direction and applies the low-rank update to the direction, which typically recovers most of the remaining gap to full fine-tuning for little extra compute. The 2026 default for a new project is to combine them, QLoRA plus DoRA, and only simplify if you hit a problem.  
  

## Further Reading

  
  
*   [Embedding Fine-Tuning and Distillation](/guides/embedding-fine-tuning-distillation) -- why and what to fine-tune for domain retrieval, and the distillation alternative
*   [Changing Embedding Models Without Breaking Your Index](/guides/embedding-model-migration-without-reembedding) -- the re-embedding and versioning problem in depth
*   [Cross-Encoder Reranking](/guides/cross-encoder-reranking) -- the reranker LoRA specializes
*   [Best Multimodal Embedding Models](/curated-lists/best-multimodal-embedding-models) -- current encoders to fine-tune
*   [Best Rerankers for RAG](/curated-lists/best-rerankers) -- reranker options and tradeoffs
*   [Best Vision-Language Models](/curated-lists/best-vision-language-models) -- VLMs you can LoRA-adapt for visual domains

## Run this on your own data

Point Mixpeek at the storage you already have and search your video, images, audio, and documents the way this guide describes. Build starts at $25/mo for up to 1M vectors.

[Search your own archive](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=guide_cta&utm_campaign=mvs_onramp)[Read Docs](/docs)

[Explore MVS](/mvs)[See pricing](/pricing)

## Related guides

[

Embeddings

### Does Fine-Tuning an Embedding Model With LoRA Improve Search? We Measured It

Yes, about as much as full fine-tuning, with a higher learning rate. We fine-tuned bge-small-en-v1.5 on 919 SciFact pairs and scored 300 held-out queries: LoRA on all linear layers at lr 1e-3 raised nDCG@10 from 0.706 to 0.744 and recall@10 from 0.830 to 0.894 with a 5.4 MB adapter. Full fine-tuning reached 0.751 with a 133 MB model. At lr 1e-4 the same adapter gained 1.3 points, within noise. Table, confidence intervals, code and raw results.

Read guide →](/guides/does-fine-tuning-an-embedding-model-with-lora-improve-search)[

Embeddings

### Why Are My SigLIP 2 Search Results So Bad? Text Padding and Lowercasing, Measured

SigLIP 2 needs query text padded to 64 tokens and lowercased, as it was trained. Unpadded queries scored 0.74% recall@1 on Flickr30k in our run; padded, 73.86%; padded and lowercased, 79.32% against the paper's 80.7%. Why padding silently fails on fixed-resolution checkpoints, and how to fix it.

Read guide →](/guides/why-siglip-2-text-to-image-search-results-are-bad)[

Embeddings

### Do E5 and BGE Embeddings Need Query and Passage Prefixes? We Measured It

Yes for E5, and the wrong prefix is the expensive mistake. We scored e5-small-v2, bge-small-en-v1.5 and all-MiniLM-L6-v2 on SciFact, NFCorpus and FiQA with and without their prefixes. Dropping E5's prefixes cost up to 1.6 points of nDCG@10, most on FiQA. Writing passage: in front of E5 queries cost 22.1 points on FiQA. BGE v1.5's instruction is optional. Table, confidence intervals, code and raw results.

Read guide →](/guides/do-e5-and-bge-embeddings-need-query-and-passage-prefixes)