Quiet-STaR (Token-Level Internal Thought Bootstrapping)

Generalizes STaR from curated QA datasets to arbitrary unstructured text: at every token position $t$ in pretraining/post-training text, the model generates parallel internal "thought" continuations (<|startthought|>...<|endthought|>) in a custom attention mask and receives a REINFORCE reward based on whether the thought improves the log-likelihood of the next $n_{\text{true}}$ future ground-truth tokens relative to no thought.