Training LLM Agents to Empower Humans

This rsearch paper by Sergey Levine's group introduces a self-supervised method for fine-tuning Large Language Model (LLM) agents to be more effective and aligned assistants, particularly in code generation. The core idea is to train agents to maximize the human's empowerment, defined as the user's ability to effect desired changes in the environment, rather than relying on costly explicit human feedback or inferred rewards. The paper details the mathematical connection between their Logit Threshold Empowerment algorithm and the concept of effective empowerment, focusing on training the assistant to complete predictable, boilerplate text so the human can concentrate on key decisions. Experimental results, including a simulated evaluation and an 18-person double-blind user study, demonstrate that the Empower assistant is preferred by users, achieves a higher acceptance rate, and significantly increases the simulated success rate for programmers compared to strong baselines. The authors conclude that this offline data-only approach provides a scalable framework for training useful AI assistants.