NCA & InfoNCA (Noise Contrastive Alignment)

Formulates LLM alignment from scalar reward annotations $\{r(x, y_k)\}_{k=1}^K$ as a Noise Contrastive Estimation (NCE) problem, training the policy log-ratio $\beta \log \frac{\pi_\theta(y_k\mid x)}{\pi_{\text{ref}}(y_k\mid x)}$ to match the softmax distribution of scalar rewards $\frac{\exp(r(x,y_k)/\alpha)}{\sum_j \exp(r(x,y_j)/\alpha)}$ without throwing away absolute scalar reward magnitudes.