Mechanism

Replaces human harmlessness annotations with a formal written Constitution (a list of normative, ethical, and epistemic principles):

  1. Supervised Stage (SL-CAI): Sample responses to red-team prompts →\rightarrow sample a random constitutional principle →\rightarrow ask the model to critique its response against the principle →\rightarrow ask the model to revise its response to remove the flaw →\rightarrow SFT on the final revised responses.
  2. RL Stage (RL-CAI / RLAIF): Sample pairs of responses (y1,y2)(y_1, y_2) →\rightarrow prompt an AI feedback judge with the Constitution to output calibrated log-probabilities over which response better upholds the principles →\rightarrow train a preference model on AI labels (mixed with human helpfulness labels) →\rightarrow run online RL (PPO/GRPO).

References

  1. Constitutional AI: Harmlessness from AI Feedback ↗

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones et al. · 2022

    All 51 authors

    Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan

  2. RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback ↗

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu et al. · 2023

    All 11 authors

    Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, Sushant Prakash