RLAD: Training LLMs to Discover Abstractions

This paper introduces a novel two-player reinforcement learning (RL) framework, RLAD, designed to enhance the reasoning capabilities of large language models (LLMs). This framework jointly trains an **abstraction generator** and an **abstraction-conditioned solution generator** to propose and utilize **concise natural language descriptions of procedural and factual knowledge** called "reasoning abstractions." The core objective is to move beyond conventional chain-of-thought methods, which often result in degenerate exploration, by teaching models to discover **high-level subgoals or strategies** that guide the solution process. Experimental results on various math and non-math reasoning benchmarks demonstrate that RLAD significantly **improves accuracy and exploration diversity** compared to prior RL approaches, with performance scaling more efficiently when compute is allocated toward generating diverse abstractions rather than solely increasing solution length or count.