How to Train Your Advisor: Steering Black-Box LLMs with ADVISOR MODELS

The academic paper introduces **ADVISOR MODELS**, a novel framework for dynamically steering the behavior of rigid, **black-box Large Language Models (LLMs)** that are only accessible via an API. Unlike static prompting methods, this approach employs a second, lightweight model, the "advisor," which is trained using **reinforcement learning (RL)** to generate instance-specific, natural language advice for the main LLM. The research demonstrates that this method excels at personalization and adapting to hidden environmental or user preferences—tasks where **static prompt optimization** fails—while also showing gains in complex reasoning domains. Crucially, the modular architecture allows the specialized advisor to be **transferred** between different black-box models and ensures that the core **frontier capabilities** of the student model are preserved.