LoRA Fine-Tuning Explained on a Real Model: Train 1% of Qwen2.5-1.5B
Summary
Fine-tuning nudges a pretrained model's weights toward your data, but classic fine-tuning updates all of them and carries optimizer state for each. LoRA freezes the base model and trains a low-rank pair beside each weight matrix: on Qwen2.5-1.5B a 1536x1536 attention matrix holds about 2.36M numbers while its rank-8 adapter holds 24,576, roughly 1% of the network overall, merged in afterward with zero added inference latency. Includes the operational rule: adapters teach behavior, retrieval carries knowledge. Paper: arxiv.org/abs/2106.09685
About this video
Fine-tuning nudges a pretrained model's weights toward your data, but classic fine-tuning updates all of them and carries optimizer state for each. LoRA freezes the base model and trains a low-rank pair beside each weight matrix: on Qwen2.5-1.5B a 1536x1536 attention matrix holds about 2.36M numbers while its rank-8 adapter holds 24,576, roughly 1% of the network overall, merged in afterward with zero added inference latency. Includes the operational rule: adapters teach behavior, retrieval carries knowledge. Paper: arxiv.org/abs/2106.09685