What is Pruning

How It Works

Pruning identifies and removes neural network parameters that contribute least to the model's output. Unstructured pruning zeros out individual weights based on magnitude or gradient importance. Structured pruning removes entire neurons, channels, or attention heads, producing models that are genuinely smaller and faster on standard hardware. After pruning, the model is typically fine-tuned to recover any lost accuracy.

Technical Details

Magnitude pruning removes weights with the smallest absolute values. Movement pruning considers how weights change during fine-tuning. Lottery ticket hypothesis suggests sparse subnetworks exist within dense networks that train equally well. Sparsity levels of 50-90% are common, with structured pruning typically achieving 2-4x speedup. The iterative prune-retrain cycle gradually increases sparsity while maintaining accuracy.

Best Practices

• Use structured pruning for actual speedup on standard hardware (GPUs, CPUs)
• Apply gradual pruning over multiple iterations rather than one-shot aggressive pruning
• Fine-tune the model after pruning to recover accuracy lost during parameter removal
• Evaluate pruned models on the full test set, not just a subset

Common Pitfalls

• Expecting unstructured pruning to provide speedup without sparse computation support
• Pruning too aggressively in a single step, causing irrecoverable accuracy loss
• Not retraining after pruning, accepting unnecessarily degraded performance
• Applying uniform pruning rates across all layers when different layers have different sensitivity

Advanced Tips

• Combine pruning with quantization and distillation for maximum model compression
• Prune multimodal models selectively, keeping more capacity in cross-modal interaction layers
• Use neural architecture search to find optimal pruned structures automatically
• Apply pruning to reduce the cost of multi-model multimodal pipelines that run multiple models