Deliberative Alignment (OpenAI o1 / o3 / GPT-6 Safety Reasoning)

Rather than hoping a model implicitly infers safety boundaries from millions of binary refusal/compliance examples, Deliberative Alignment explicitly teaches the model the exact text of the safety & behavior policies during SFT (by placing policy specs in context during synthetic CoT generation, then stripping the policy prompt from the SFT inputs so the model internalizes the rules) and subsequently uses RL with a policy-aware judge LLM so the model explicitly retrieves and reasons over policy clauses inside its hidden Chain-of-Thought at test time before deciding whether to refuse, safe-complete, or comply.

Simultaneously reduces both jailbreak success rates (StrongREJECT, HarmBench) and false refusals on benign borderline prompts (XSTest, CoCoNot).