PiSSA (Principal Singular Values Adaptation) & GaLore (Gradient Low-Rank Projection)

Mechanism

PiSSA: Instead of initializing $A \sim \mathcal{N}(0,\sigma^2), B=0$ $A \sim N (0, σ^{2}), B = 0$ like LoRA, performs SVD on $W_0 = U S V^\top$ $W_{0} = U S V^{⊤}$ , initializes the trainable adapter $(B, A)$ $(B, A)$ with the top- $r$ $r$ principal singular values and vectors (which carry the steepest task gradients), and freezes the residual $W_{\text{res}} = W_0 - B_{\text{init}}A_{\text{init}}$ $W_{res} = W_{0} - B_{init} A_{init}$ .
- GaLore: Performs full-rank weight updates while projecting the gradient matrix $G_t \in \mathbb{R}^{m\times n}$ into a slowly changing low-rank subspace $P_t^\top G_t Q_t$ to cut AdamW optimizer state memory by up to 65%.