DoRA (Weight-Decomposed Low-Rank Adaptation)

Decomposes pretrained weights into magnitude $m \in \mathbb{R}^{1\times d_{\text{in}}}$ and unit directional matrix $\frac{V}{\|V\|_c}$, applying low-rank updates exclusively to the directional component while training the scalar magnitude vector directly: $W_{\text{DoRA}} = m \frac{W_0 + BA}{\|W_0 + BA\|_c}$ matching full fine-tuning's negative magnitude-direction update correlation.