Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

This research paper introduces Variational Preference Learning (VPL), a novel method designed to improve Reinforcement Learning from Human Feedback (RLHF) by accounting for the diversity and plurality of individual human preferences. Current RLHF methods, which typically assume a single, monolithic set of preferences, often fail or result in inaccurate reward models when faced with a diverse population, especially ignoring minority viewpoints. VPL addresses this by formulating the problem using a latent variable model, inferring a user-specific latent context to condition personalized reward models and policies without requiring extensive user-specific data. Empirical results across simulated control tasks and large language model (LLM) alignment demonstrate that VPL outperforms standard RLHF baselines in accurately capturing multimodal preferences and enables the development of steerable, personalized policies. The work also integrates a reward scaling mechanism (VPL-SPO) and an active learning component to enhance efficiency and robustness.