Demystifying Reinforcement Learning in Agentic Reasoning

The research paper systematically investigates how reinforcement learning (RL) can enhance the agentic reasoning capabilities of Large Language Models (LLMs), particularly in tool-integrated environments. The authors conduct a comprehensive empirical study across three main dimensions: data curation, algorithmic design, and reasoning mode to demystify optimal practices for agentic RL. Key findings include that real end-to-end trajectories are crucial for strong Supervised Fine-Tuning (SFT) initialization, while high-diversity and model-aware datasets improve training efficiency and exploration; algorithmically, techniques like clip higher and overlong reward shaping are effective for performance gains. Furthermore, the study identifies that a "deliberative mode" characterized by fewer but more successful tool calls outperforms frequent, reactive tool usage, and the authors introduce a new model, DemyAgent-4B, which achieves strong performance on challenging benchmarks compared to significantly larger models.