Mechanism

  • LISA: Always keeps the token embedding and LM head trainable while randomly unfreezing only γ\gamma (e.g., 2 to 4) middle transformer blocks per K iterations, achieving full-parameter fine-tuning convergence at LoRA memory cost.
    • Spectrum: Computes the Marchenko-Pastur Signal-to-Noise Ratio (SNR) of each weight matrix's singular value spectrum and selectively unfreezes only the top 25–50% high-SNR modules during post-training.

References

  1. LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning ↗

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han et al. · 2024

    All 7 authors

    Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, Tong Zhang

  2. Spectrum: Targeted Training on Signal to Noise Ratio ↗

    Eric Hartford, Lucas Atkins, Fernando Fernandes Neto, David Golchinfar · 2024