LoRA Fine-Tuning Explained on a Real Model: Train 1% of Qwen2.5-1.5B
LoRA freezes the base model and trains a small low-rank adapter beside each weight matrix — roughly 1% of the parameters — merged in afterward with zero added inference latency.
Where this fits in Mixpeek
Mixpeek lets you plug specialized embedding and extraction models into a collection. LoRA is the mechanism that makes domain adaptation cheap: adapt a base encoder to your vocabulary (medical imaging, brand catalogs, legal docs) with a tiny adapter instead of a full fine-tune, and merge it with no serving cost. The operational rule from the paper is the one Mixpeek follows — adapters teach behavior, retrieval carries knowledge — so most customization is a retriever/config change, not a model retrain.
About this research
Fine-tuning nudges a pretrained model's weights toward your data, but classic fine-tuning updates all of them and carries optimizer state for each. LoRA freezes the base model and trains a low-rank pair beside each weight matrix: on Qwen2.5-1.5B a 1536x1536 attention matrix holds about 2.36M numbers while its rank-8 adapter holds 24,576, roughly 1% of the network overall, merged in afterward with zero added inference latency. Includes the operational rule: adapters teach behavior, retrieval carries knowledge. Paper: arxiv.org/abs/2106.09685
Frequently asked questions
Put the research to work
Mixpeek turns video, images, audio, and documents in your object storage into searchable, timestamped results through one API — the retrieval stack these papers describe.
Search your own data, free