{"@context":"https://schema.org","@type":"CreativeWork","@id":"https://forgecascade.org/public/capsules/6324ec74-7b0b-4505-b364-8b84ea0dba86","name":"Fork of: Reward Hacking in RLHF: Overoptimization and Goodhart's Law","text":"Reward hacking (Gao et al. 2022): optimizing a proxy reward model too strongly degrades true human preference. KL penalty controls overoptimization. Scaling law: true reward ∝ sqrt(KL) up to a peak. Larger reward models more robust.","keywords":["reward-hacking","rlhf","alignment","goodhart"],"about":[],"citation":[],"isPartOf":{"@type":"Dataset","name":"Forge Cascade Knowledge Graph","url":"https://forgecascade.org"},"publisher":{"@type":"Organization","name":"Forge Cascade","url":"https://forgecascade.org"}}