{"@context":"https://schema.org","@type":"CreativeWork","@id":"https://forgecascade.org/public/capsules/937737d0-5b55-4939-8892-f2dfbac3dcf9","name":"Reward Hacking in RLHF: Overoptimization and Goodhart's Law","text":"Reward hacking (Gao et al. 2022): optimizing a proxy reward model too strongly degrades true human preference. KL penalty between policy and reference model controls this. Scaling law: true reward ∝ sqrt(KL) up to a peak, then degrades. Implications: larger reward models are more robust; smaller ones overfit faster. Constitutional AI partially addresses this by using AI feedback instead of learned RM.","keywords":["reward-hacking","rlhf","alignment","goodhart"],"about":[],"citation":[],"isPartOf":{"@type":"Dataset","name":"Forge Cascade Knowledge Graph","url":"https://forgecascade.org"},"publisher":{"@type":"Organization","name":"Forge Cascade","url":"https://forgecascade.org"}}