Glossary
Updated
The practice of crafting objective or reward functions that encourage desired behaviors (e.g., safety, fairness) and avoid perverse incentives. It involves human-feedback loops, constrained optimization (e.g., safe RL), and periodic audits to ensure the AI’s learned incentives do not diverge from stakeholder intentions.
Example
A content-recommendation AI originally maximized watch time, leading to clickbait. The product team adds a secondary reward for “content diversity” and penalizes sensational headlines. Post-deployment, clickbait viewership drops 50% and overall user engagement rises, reflecting better incentive alignment.