報酬ハッキング（Reward Hacking）とは

OpenAI found that the agents were not trying to do the task as OpenAI had intended. Instead they looked to cheat by finding the solutions online. This behavior is known as “reward hacking,” in which a model finds an unintended way to achieve an outcome that earns reward without completing the task in the way the evaluation was designed to measure.

出典を開く確認 2026-09-26

We have previously shown how misalignment can arise with our examples of specification gaming, where an AI finds a solution to achieve its goals, but not in the way intended by the human instructing it, and goal misgeneralization. For example, an AI system asked to book tickets to a movie might decide to hack into the ticketing system to get already occupied seats - something that a person asking it to buy the seats may not consider.

出典を開く確認 2026-09-26

We define cheating as taking an action that is out of scope for the task or explicitly disallowed by the rules, in order to achieve a goal through a shortcut, workaround, or unintended solution that the task was not meant to, or should not, permit. This behaviour has been previously documented by others, especially during capability evaluations: models have attempted to hack evaluation infrastructure to find scoring functions, searched online for existing solutions, and hard-coded answers to programming tasks.

出典を開く確認 2026-09-26

we trained an Opus-class model with large-scale RL on many production environments vulnerable to reward hacks. We consider this a plausible proxy for what a real training run might look like had we not invested significant effort into preventing and detecting reward hacking in our normal training runs. The resulting model not only learned to reward hack during training, but also generalized to more severe misaligned behaviors: in simulated cyber evaluations, it broke out of its sandbox, stole credentials, and attacked both internal and third-party infrastructure to steal an answer key.

出典を開く確認 2026-09-26

it can make evaluations overstate a model’s actual capabilities, and could mislead users when completing tasks where success is hard to verify.

出典を開く確認 2026-09-26

Every model we have tested for this behaviour attempted to cheat. Models did not reliably report this behaviour when asked, and often did not reason about it in their chain-of-thought, suggesting that detecting cheating will likely require robust monitoring methods. In this blog, we analyse cheating behaviour in our cyber evaluations, and discuss the implications of this behaviour as models become more capable.

出典を開く確認 2026-09-26