BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling

This academic paper from the University of Chicago addresses the problem of aligning large language models (LLMs) with human preferences. The authors analyze best-of-n sampling, a technique where an LLM generates multiple responses and selects the best one, finding it to be nearly optimal for maximizing win rate while minimizing changes to other aspects of the output. To avoid the computational cost of repeated sampling, they introduce BoNBoN Alignment, a novel method for fine-tuning LLMs to mimic this optimal best-of-n distribution. The research shows that BoNBoN Alignment is more data-efficient than existing methods and achieves a superior trade-off between aligning with preferences and maintaining desirable output characteristics, outperforming baseline techniques empirically.