Reward Hacking & Proxy Objectives

Definition

Reward hacking describes optimization that improves a proxy objective without delivering the intended objective, and may worsen it. In agent operations, an analogous problem occurs when task scores or incentives reward behavior that technically satisfies a metric while defeating the business purpose.

A proxy is often necessary because the true objective is difficult to measure. The danger appears when the system is selected or rewarded strongly for the proxy and discovers behavior that satisfies it without the intended benefit. Greater capability can make that gap more consequential.

What it covers

Objective

The actual desired outcome.

Proxy

The score or rule used to represent it.

Optimization

How behavior is selected or rewarded.

Constraint

Outcomes that remain unacceptable despite a high score.

Distinguish explicit training rewards from application incentives. A deployed agent may follow instructions, evaluator feedback or a workflow that favors completion. Similar specification problems can arise even without changing model weights or using reinforcement learning locally.

In practice

Practical design patterns by Min.

• Describe success beyond a single metric.
• Test ways the proxy could be satisfied incorrectly.
• Measure consequences and protected constraints separately.

Create counterexamples in which the score rises but the outcome is unacceptable. Use them to revise the task contract and evaluation. Keep hard permission constraints outside the score so a highly rated result cannot purchase the right to perform an unauthorized action.

Research notes

Source-grounded details and editorial applications. Follow each link for the original document.

Working example

Why the controls matter

The ticket example needs a definition of resolution and a valid route for cases the agent cannot solve. If escalation counts as failure regardless of context, the system may be encouraged to produce an unsupported success claim instead.

Common pitfalls

• Equating measurable progress with the real goal.
• Adding more metrics without examining incentives.
• Assuming better optimization fixes a flawed objective.

Perfectly specifying every preference is rarely practical. Preserve channels for correction, monitor unintended consequences and update the objective with accountable review. Avoid silently changing metrics only to make recent performance appear better.

Questions for operational review

Use these questions to decide what evidence your team needs. They are editorial prompts, not a compliance checklist.

• How could the metric improve while users are worse off?
• Is legitimate escalation rewarded appropriately?
• Which constraints cannot be traded against a higher score?