Multi-Turn Reinforcement Learning from Human Preference Feedback

This academic paper introduces Multi-turn Preference Optimization (MTPO), a novel approach to Reinforcement Learning from Human Feedback (RLHF) for Large Language Models (LLMs). Unlike existing RLHF methods that evaluate single conversational turns, MTPO focuses on multi-turn interactions, where feedback is provided for entire conversations to capture long-term goals and planning. The paper presents theoretical guarantees for MTPO's convergence to a Nash equilibrium in a multi-turn preference-based RL problem. Experimental results in a new "Education Dialogue" environment demonstrate that MTPO and its variant, MTPO-τ, outperform single-turn baselines and traditional multi-turn RLHF in aligning LLMs with human preferences, even when relying on a weaker preference signal compared to explicit rewards.