{"@context":"https://schema.org","@type":"CreativeWork","@id":"https://forgecascade.org/public/capsules/257bb3ff-3c6f-4aa1-8ade-f68c26a70983","name":"Constitutional AI: Harmlessness from AI Feedback","text":"Constitutional AI (Bai et al. 2022) trains helpful, harmless, honest assistants using AI-generated feedback rather than human labelers for harmlessness. Two-stage pipeline: (1) SL-CAI — supervised learning from AI critique-revisions. Model critiques own outputs against a constitution (16 principles covering harm, deception, toxicity, fairness, privacy) and rewrites them. Fine-tune on revised outputs. (2) RL-CAI — reinforcement learning from AI feedback (RLAIF). Train preference model on AI-labeled pairs where AI judge applies constitution. Then PPO against this PM. Claims: constitutional critique-revision reduces harmlessness violations without human annotation; RLAIF labels competitive with RLHF for harmlessness; model becomes more transparent about refusal reasoning. Limitations: bounded by initial SFT checkpoint quality; constitutional principles can conflict (implicit resolution required); evaluated only on internal benchmarks. Impact: foundation for Claude training; later work distilled principles into 1-2 meta-principles.","keywords":["constitutional-ai","rlhf","rlaif","harmlessness","alignment"],"about":[],"citation":[],"isPartOf":{"@type":"Dataset","name":"Forge Cascade Knowledge Graph","url":"https://forgecascade.org"},"publisher":{"@type":"Organization","name":"Forge Cascade","url":"https://forgecascade.org"}}