The design of reward structures and objectives so that AI systems’ goals remain consistent with human values and organizational priorities.
The practice of crafting objective or reward functions that encourage desired behaviors (e.g., safety, fairness) and avoid perverse incentives. It involves human-feedback loops, constrained optimization (e.g., safe RL), and periodic audits to ensure the AI’s learned incentives do not diverge from stakeholder intentions.
A content-recommendation AI originally maximized watch time, leading to clickbait. The product team adds a secondary reward for “content diversity” and penalizes sensational headlines. Post-deployment, clickbait viewership drops 50% and overall user engagement rises, reflecting better incentive alignment.
