Weak-to-Strong Generalization (Superalignment Supervision)

Studies how a weak supervisor model $\pi_{\text{weak}}$ can supervise a much stronger pretrained student $\pi_{\text{strong}}$ without the student merely imitating the weak supervisor's errors, using an auxiliary confidence loss that interpolates the weak supervisor's soft labels $w(x)$ with the strong student's own sharpened thresholded predictions $\hat{s}(x)$: $\mathcal{L}_{\text{W2S}}(\theta) = (1-\alpha)\text{CE}(\pi_\theta(x), w(x)) + \alpha \text{CE}(\pi_\theta(x), \hat{s}(x))$