STaR (Self-Taught Reasoner)

Iterative reasoning bootstrap loop for datasets that only have questions and final answers $\{(x_i, a_i^\star)\}$ without ground-truth chain-of-thought rationales:

1. Rationale Generation: Sample $(r_i, \hat{a}_i) \sim \pi_{\theta_t}(\cdot\mid x_i)$ and keep all traces where $\hat{a}_i = a_i^\star$.
2. Rationalization: For questions where $\hat{a}_i \neq a_i^\star$, prompt the model with the ground-truth answer hint $(x_i, a_i^\star)$ to generate a backward rationale $r_i^{\text{hint}}$ explaining why $a_i^\star$ is true.
3. Fine-Tuning: SFT $\pi_{\theta_{t+1}}$ (always resetting to the base model $\pi_0$ at each outer iteration to avoid overfitting!) on the union of verified rationales and rationalizations.