Does Fine-Tuning an Embedding Model With LoRA Improve Search? We Measured It

Does fine-tuning an embedding model with LoRA improve search?

Yes, by about as much as full fine-tuning, if you give LoRA a higher learning rate than full fine-tuning uses. We fine-tuned BAAI/bge-small-en-v1.5 on 919 labeled (query, relevant abstract) pairs from SciFact and scored it on 300 held-out queries over the full 5,183-abstract corpus. LoRA on all linear layers at a learning rate of 1e-3 raised nDCG@10 from 0.706 to 0.744 and recall@10 from 0.830 to 0.894. It trained 4% of the weights and saved a 5.4 MB adapter. Full fine-tuning reached 0.751 nDCG@10 and saved a 133 MB model.

The same adapter at a learning rate of 1e-4, a common default copied from full fine-tuning, gained 1.3 points, and that gain was within noise. The learning rate decided the result more than anything else we changed.

What did we measure?

The confidence intervals come from a paired bootstrap over the 300 test queries, comparing each query's nDCG@10 with the base model's. Every arm used the same data, loss (MultipleNegativesRankingLoss with in-batch negatives), batch size of 32, 3 epochs and seed, with LoRA rank 16 and alpha 16. Train and test queries do not overlap. The base score matches the published bge-small result on SciFact (about 0.71), which is a check that the evaluation is sound.

The raw results are at /measurements/2026-11-04-lora-embedding-finetune.json and the script that produced them is at /measurements/2026-11-04-lora-embedding-finetune.py.

Why did the learning rate matter so much?

A LoRA adapter starts as a zero update: one of its two matrices is initialized to zero, so the model begins exactly where the base model is. The adapter then has a few hundred steps to move. At 1e-4 it barely moves in 87 steps, so 229 of the 300 test queries scored exactly what the base model scored. At 1e-3 the adapter moved far enough to change the ranking on a third of the queries, and it helped about 1.6 queries for every one it hurt (62 better, 38 worse).

Full fine-tuning updates every weight, so a small learning rate (2e-5 here) is enough, and a large one risks erasing what the base model already learned. If you copy a full fine-tuning recipe and add LoRA to it, you inherit a learning rate that is too low for the adapter.

Which layers should LoRA adapt on an embedding model?

All of them, when the budget allows. Adapting only the attention query and value projections trained 0.29M weights and gained 0.8 points at lr 1e-4. Adapting every linear layer (attention and the feed-forward block) trained 1.34M weights and gained 1.3 points at the same rate, and 3.9 at lr 1e-3. On a 33M-parameter encoder the larger adapter is still 5.4 MB.

Is LoRA worth it over full fine-tuning for embeddings?

On this test the two reached the same quality within their confidence intervals: full fine-tuning was ahead on nDCG@10 (0.751 against 0.744) and LoRA was ahead on recall@10 (0.894 against 0.885). The difference is what you keep afterwards. LoRA leaves the base weights untouched and adds a 5.4 MB file, so you can keep one adapter per domain or per customer next to one base model. Full fine-tuning gives you a new 133 MB model per variant. On a 7B embedding model the same ratio is a few hundred megabytes against tens of gigabytes.

Either way, a fine-tuned embedder writes vectors in a new space. Vectors from the base model and the fine-tuned model cannot be compared, so the corpus has to be embedded again before you search it. On a large library that re-embedding pass costs more than the training.

How do I reproduce this?

from datasets import Dataset
from peft import LoraConfig, TaskType
from sentence_transformers import (SentenceTransformer, SentenceTransformerTrainer,
                                   SentenceTransformerTrainingArguments, losses)

model = SentenceTransformer("BAAI/bge-small-en-v1.5")
model.add_adapter(LoraConfig(task_type=TaskType.FEATURE_EXTRACTION, r=16, lora_alpha=16,
                             lora_dropout=0.1, target_modules="all-linear"))

# anchors are queries (with the model's query prefix), positives are the relevant passages
train = Dataset.from_dict({"anchor": queries, "positive": passages})
args = SentenceTransformerTrainingArguments(
    output_dir="out", num_train_epochs=3, per_device_train_batch_size=32,
    learning_rate=1e-3, warmup_ratio=0.1, batch_sampler="no_duplicates", seed=0)
SentenceTransformerTrainer(model=model, args=args, train_dataset=train,
                           loss=losses.MultipleNegativesRankingLoss(model)).train()
model.save_pretrained("bge-small-lora")  # the adapter only: 5.4 MB

This uses sentence-transformers 5.1 and peft 0.17. SentenceTransformer("bge-small-lora") loads the base model with the adapter attached. Score the result on held-out queries from your own data before you re-embed anything: SciFact is one domain, and your gain depends on how far your queries are from what the base model already handles.

When is fine-tuning not worth it?

When you have no held-out queries to score it on. Without them you cannot tell a 4-point gain from noise, and on this test two of the four trained arms were noise.

When a stronger base model or a reranker closes the gap with no re-embedding.

When your labeled pairs number in the dozens. We used 919; below a few hundred, expect the adapter to overfit.

How does this fit a Mixpeek pipeline?

Mixpeek stores the vectors your model produces. Fine-tune and embed on your side, then upsert the vectors into a bring-your-own-vectors collection in an MVS namespace, which runs on your own object storage. Keep the model version in the document metadata so a re-embed can run beside the old index and replace it when it scores better. For the knobs behind LoRA itself (rank, alpha, QLoRA and DoRA), see how LoRA fine-tuning works.

Frequently Asked Questions

How much does LoRA fine-tuning improve an embedding model?

On SciFact, LoRA on all linear layers of bge-small-en-v1.5 raised nDCG@10 by 3.9 points (0.706 to 0.744, 95% CI +1.5 to +6.3) and recall@10 from 0.830 to 0.894, from 919 training pairs. Full fine-tuning gained 4.5 points. Your gain depends on your domain, so score it on your own held-out queries.

What learning rate should I use for LoRA on an embedding model?

Start around 1e-3 and tune from there. In our test the same adapter gained 1.3 points at 1e-4, which was within noise, and 3.9 points at 1e-3. Full fine-tuning used 2e-5. LoRA adapters start as a zero update and need a larger step size to move in a short run.

How big is a LoRA adapter for an embedding model?

On a 33M-parameter encoder, rank 16 on all linear layers is 1.34M weights and 5.4 MB on disk; on the attention query and value layers only, it is 0.29M weights and 1.2 MB. The size grows with the base model, so on a 7B model expect a few hundred megabytes.

Does LoRA match full fine-tuning for retrieval?

In our test, yes within the margin of error: 0.744 nDCG@10 for LoRA against 0.751 for full fine-tuning, with overlapping confidence intervals, and LoRA was higher on recall@10 (0.894 against 0.885). It got there with 4% of the trainable weights.

Do I have to re-embed my documents after fine-tuning?

Yes. Fine-tuning changes the embedding space, so vectors from the base model and the fine-tuned model are not comparable. Embed the whole corpus again with the fine-tuned model, and keep the model version with each vector so the old and new indexes are never mixed.

Related

How Does LoRA Fine-Tuning Work? -- rank, alpha, target modules, QLoRA and DoRA

Embedding Fine-Tuning and Distillation -- where training pairs come from and hard-negative mining

Changing Embedding Models Without Breaking Your Index -- running a re-embed beside the old index

Best Self-Hosted Embedding Models -- base models worth fine-tuning