1.  [](/)
2.  [Research](/research)
3.  LoRA Fine-Tuning Explained on a Real Model: Train 1% of Qwen2.5-1.5B

[Back to Research](/research)

Microsoft[arXiv:2106.09685](https://mxp.co/e/lora)

# LoRA Fine-Tuning Explained on a Real Model: Train 1% of Qwen2.5-1.5B

LoRA freezes the base model and trains a small low-rank adapter beside each weight matrix, roughly 1% of the parameters, merged in afterward with zero added inference latency.

## Where this fits in Mixpeek

Mixpeek lets you plug specialized embedding and extraction models into a collection. LoRA is the mechanism that makes domain adaptation cheap: adapt a base encoder to your vocabulary (medical imaging, brand catalogs, legal docs) with a tiny adapter instead of a full fine-tune, and merge it with no serving cost. The operational rule from the paper is the one Mixpeek follows (adapters teach behavior, retrieval carries knowledge) so most customization is a retriever/config change, not a model retrain.

## About this research

Fine-tuning nudges a pretrained model's weights toward your data, but classic fine-tuning updates all of them and carries optimizer state for each. LoRA freezes the base model and trains a low-rank pair beside each weight matrix: on Qwen2.5-1.5B a 1536x1536 attention matrix holds about 2.36M numbers while its rank-8 adapter holds 24,576, roughly 1% of the network overall, merged in afterward with zero added inference latency. Includes the operational rule: adapters teach behavior, retrieval carries knowledge. Paper: arxiv.org/abs/2106.09685

lorafine-tuningadaptersembeddingscustomization

## Frequently asked questions

What is LoRA fine-tuning?+

What does the rank in LoRA mean?+

## Go deeper

*   [Guide: Fine-tuning with LoRA adapters →](/guides/fine-tuning-with-lora-adapters)

Put the research to work

Mixpeek turns video, images, audio, and documents in your object storage into searchable, timestamped results through one API: the retrieval stack these papers describe.

[Search your own data](https://studio.mixpeek.com/namespaces/create?mode=managed&utm_source=homepage&utm_medium=research_detail&utm_campaign=mvs_onramp)

## Related research

[

![How Reverse Video Search Works: Embeddings, Time, and Copy Detection](https://cdn.mixpeek.com/videos/thumbs/85-reverse-explainer.jpg)

Meta AI

How Reverse Video Search Works: Embeddings, Time, and Copy Detection



](/research/video-similarity-challenge)[

![Why Fine-Tuning Quietly Breaks Your Embedding Index (and AlphaWiSE's Fix)](https://cdn.mixpeek.com/videos/thumbs/74-alphawise.jpg)

AlphaWiSE

Why Fine-Tuning Quietly Breaks Your Embedding Index (and AlphaWiSE's Fix)



](/research/alphawise)