J1: Incentivizing Thinking in LLM-as-a-Judge

This paper presents J1, a new method for training large language models to act as judges that evaluate other models' responses. The J1 approach utilizes reinforcement learning to encourage these judge models to produce detailed, step-by-step reasoning before making a judgment, similar to a chain of thought. By converting both straightforward and subjective tasks into verifiable problems with rewards for accurate judgments and consistency, J1 demonstrates improved performance across various benchmarks compared to other state-of-the-art judge models. The research also explores different J1 variations, including Pairwise-J1 (comparing two responses) and Pointwise-J1 (scoring individual responses), highlighting the effectiveness of Pointwise-J1 in mitigating position bias in evaluations.