Signal and Noise: Evaluating Language Model Benchmarks

This paper introduces a framework for **evaluating language model benchmarks** by quantifying **signal** and **noise**. The signal measures a benchmark's capacity to differentiate between superior and inferior models, while noise reflects its susceptibility to random fluctuations during training. The authors demonstrate that a **higher signal-to-noise ratio (SNR)** correlates with more reliable small-scale experiments for predicting large model performance and that less noise leads to reduced scaling law prediction error. They propose three **interventions** to enhance SNR: **filtering noisy subtasks**, **averaging model checkpoint scores** to reduce variability, and employing **bits-per-byte (BPB)** as a more consistent evaluation metric. The research emphasizes that considering SNR is crucial for designing and selecting benchmarks that accurately guide language model development, rather than relying solely on benchmark size.