Asymptotics of Language Model Alignment

This academic paper investigates language model alignment, a process of adjusting a base language model to better align with desired outcomes, often guided by a reward model. It specifically examines two common alignment methods: KL-constrained reinforcement learning (RL), which maximizes reward while limiting divergence from the original model, and best-of-N selection, where the highest-reward output from multiple samples is chosen. Under simplifying assumptions about the language and reward models, the authors theoretically characterize the optimal KL-constrained RL solution and demonstrate that, asymptotically, best-of-N is equivalent to this optimal solution in terms of expected reward and KL divergence, providing a theoretical basis for its strong empirical performance.