We discuss Anthropic's recent document that presents an extensive investigation into the inner workings of Anthropic's Claude 3.5 Haiku large language model using a novel "circuit tracing" methodology. Researchers analyzed the model's internal mechanisms across diverse tasks like multi-step reasoning, poetry generation, multilingual translation, and arithmetic. They identified interpretable "features" and mapped their interactions using "attribution graphs," offering insights into how the model ...
Apr 01, 2025•19 min
This paper introduces Trajectory Balance with Asynchrony (TBA) , a novel distributed reinforcement learning framework designed for efficient and scalable post-training of large language models. TBA decouples the data generation process (handled by multiple "searcher" nodes) from the policy update mechanism (managed by a single "trainer" node), utilizing an off-policy training objective called Trajectory Balance. This asynchronous approach leverages a central replay buffer to store diverse experi...
Apr 01, 2025•18 min
We discuss Instacart's Economics Team. The team is composed of academically trained economists who function as machine learning engineers, blending economic principles with technical implementation. They address challenging marketplace problems by developing and deploying end-to-end solutions, working horizontally across various product areas. We also discuss advice for economists aspiring to similar positions in the tech industry, noting the increasing demand for this interdisciplinary expertis...
Mar 31, 2025•19 min
This paper outlines a new probabilistic framework called multi-fidelity multi-scale Bayesian optimization for efficiently determining the best combinations of data sources for pre-training large language models. It addresses the limitations of intuition-based and deterministic extrapolation methods by modeling uncertainty and sequentially selecting data mixtures, model sizes, and training steps to balance cost and information gain. The authors introduce a simulator based on numerous pre-training...
Mar 31, 2025•22 min
It details the motivation behind MCP , its core components, and real-world use cases, including its potential as a foundational protocol for AI agents through composability and sampling. The Latent Space article analyzes the reasons for MCP's rapid adoption , attributing its success to its "AI-native" design, backing by Anthropic, strong developer brand association, inspiration from the successful Language Server Protocol (LSP), comprehensive first-party support, and iterative development....
Mar 31, 2025•17 min
This research paper focuses on training large language model (LLM) agents for collaborative reasoning tasks. The paper introduces Collaborative Agent Benchmark (ColBench) , a new benchmark designed to evaluate multi-turn reinforcement learning (RL) algorithms in realistic artifact creation scenarios. The authors propose a novel RL algorithm named SWEET-RL (RL with Step-WisE Evaluation from Training-Time information) that uses a critic model with access to additional training data to provide step...
Mar 31, 2025•25 min
This research paper introduces TheoryCoder , a novel reinforcement learning agent. TheoryCoder integrates large language models (LLMs) for synthesizing code-based world models with a bilevel planning approach , utilizing high-level symbolic abstractions and low-level Python-based transition models. The paper addresses limitations in prior theory-based reinforcement learning by enabling more expressive theories and scalable planning. TheoryCoder learns a domain by grounding abstract concepts usin...
Mar 30, 2025•23 min
This conversation discusses the presentation from Jason Wei at OpenAI, who explores the driving forces behind recent rapid progress in artificial intelligence , primarily focusing on scaling compute, data through pre-training, and test-time computation using reinforcement learning . It posits that scaling general methods has been key to advancements and examines the effectiveness of next-word prediction as a pre-training task that surprisingly unlocks various capabilities. The talk further looks...
Mar 29, 2025•23 min
This paper introduces a new framework called Experts-as-Priors (ExPerior) . This framework addresses the challenge of sequential decision-making in situations with unobserved heterogeneity , where offline expert demonstrations contain variations not apparent to the learning agent. ExPerior leverages these demonstrations to infer an informative prior distribution over the hidden factors, subsequently using Bayesian methods like posterior sampling to guide online reinforcement learning. The paper ...
Mar 28, 2025•18 min
This paper introduces TextGrad , a novel framework for optimizing generative AI systems. This method uses large language models (LLMs) to provide natural language feedback , acting as "textual gradients," to guide the improvement of various AI components. TextGrad enables automatic optimization across diverse tasks by backpropagating this feedback through a system's computation graph. The paper demonstrates TextGrad's effectiveness in areas like code refinement, question answering, prompt optimi...
Mar 27, 2025•26 min
This paper aims to improve reasoning capabilities over long contextual information by learning the relative order of facts and enabling selective attention to its memory. The paper empirically investigates MemReasoner's generalization abilities on multi-hop reasoning tasks compared to other models, even with minimal supervision. Their findings suggest that explicit memory mechanisms can significantly enhance large language models' context processing for reasoning. The authors conclude by discuss...
Mar 27, 2025•18 min
This paper introduces Retrieval Augmented Fine Tuning (RAFT) , a novel training method designed to improve large language models' ability to answer questions accurately within specific domains when provided with relevant documents. RAFT trains models to effectively utilize provided documents by incorporating both helpful and distracting information during fine-tuning, encouraging the model to discern and cite relevant passages. The research demonstrates that RAFT enhances performance on domain-s...
Mar 27, 2025•21 min
This paper explores inductive biases in exchangeable sequence modeling, focusing on architectural choices and inferential methods , particularly for decision-making tasks . It highlights a limitation of single-step inference in distinguishing between epistemic and aleatoric uncertainty, advocating for multi-step inference for better uncertainty quantification and downstream performance. The authors also examine Transformer architectures designed for exchangeable sequences, revealing that existin...
Mar 26, 2025•20 min
This paper introduces a novel approach called Alignment from Demonstrations (AfD) for aligning large language models (LLMs) using demonstration datasets instead of preference-based data. The paper frames this alignment problem within a reinforcement learning (RL) framework , specifically exploring connections to forward and inverse RL. It theoretically analyzes trajectory distribution matching objectives , linking supervised fine-tuning to forward KL divergence and adversarial learning to revers...
Mar 26, 2025•25 min
This paper introduces Prompt-OIRL , a novel method to enhance the arithmetic reasoning of large language models by optimizing prompts based on individual queries. The authors identify challenges in evaluating prompts during inference and the high costs of online prompt optimization. To address these, Prompt-OIRL employs offline inverse reinforcement learning to learn from existing prompt evaluation data and build a reward model for cost-efficient, query-specific prompt assessment and selection, ...
Mar 26, 2025•16 min
The provided text is a research paper introducing Alignment from Demonstrations (AfD) as a novel method for aligning large language models (LLMs) using high-quality demonstration data. It identifies limitations in current preference-based alignment techniques and proposes framing AfD within a reinforcement learning framework , specifically inverse reinforcement learning, to address these shortcomings. The paper explores trajectory distribution matching as a core objective, demonstrating how supe...
Mar 25, 2025•21 min
This podcast introduces Q♯ , a novel reinforcement learning algorithm tailored for post-training large language models (LLMs) by utilizing distributional value functions within a KL-regularized framework. Unlike prevalent policy-based methods and existing value-based baselines that use unregularized Q-values, Q♯ learns the optimal regularized Q-function to guide the reference policy, offering theoretical guarantees and empirical advantages in math reasoning tasks while maintaining proximity to t...
Mar 18, 2025•20 min
The paper presents a theoretical analysis comparing verifier-based (VB) and verifier-free (VF) algorithms for training large language models (LLMs) under varying compute budgets. It demonstrates that VB methods outperform VF methods as test-time compute increases, particularly when the base LLM exhibits high heterogeneity and anti-concentration in reward distributions. The findings indicate that while both methods can be effective, VB methods scale better with larger budgets, and this gap widens...
Mar 14, 2025•15 min
Longer version
Mar 14, 2025•12 min
The paper optimizes test-time compute as a meta-reinforcement learning problem It emphasizes balancing exploration and exploitation to minimize cumulative regret Meta Reinforcement Fine-Tuning (MRT) improves performance and token efficiency
Mar 14, 2025•5 min
The paper surveys limitations of reinforcement learning from human feedback (RLHF). It highlights challenges in training AI systems with RLHF. Proposes auditing and disclosure standards for RLHF systems. Emphasizes a multi-layered approach for safer AI development. Identifies open questions for further research in RLHF.
Mar 14, 2025•2 min
The paper revisits the Superficial Alignment Hypothesis. It studies post-training scaling behavior with finetuning examples. Performance scales as a power law with more finetuning examples. Model performance correlates with reasoning ability, not just style. Language models can integrate new knowledge post-pre-training. Results suggest the hypothesis is an oversimplification.
Mar 14, 2025•4 min
The paper introduces diagnostic uncertainty in language models. It enables models to describe their uncertainty openly. Improved accuracy and reduced entropy in responses are achieved. A framework for operationalizing uncertainty in LMs is proposed. The method enhances model interpretability and understanding of behavior.
Mar 14, 2025•4 min
The paper introduces a personalized framework for LLMs. It utilizes user-specific rewards from minimal feedback. The method achieves significant personalization over default responses. It leverages Reinforcement Learning from Human Feedback (RLHF). The approach models preferences as linear combinations of base features. Experiments validate effectiveness with synthetic and real user data.
Mar 14, 2025•5 min
The paper explores efficient exploration techniques in language model alignment It introduces SpannerSampling for optimal data efficiency in reinforcement learning The study contrasts training-time interventions with computational benefits of multi-turn exploration. It emphasizes leveraging pre-trained models for improved exploration efficiency
Mar 14, 2025•5 min
The paper studies reasoning length and model performance tradeoff. It explores compression strategies for large language models (LLMs). Token complexity measures minimal tokens for successful problem-solving. LLMs adapt response length based on problem difficulty. Compression improvements require matching token-length to token complexity. Shorter prompts can maintain accuracy with reduced response length.
Mar 14, 2025•4 min
The paper investigates LLMs for extracting customer needs from reviews. Evaluations conducted with a professional marketing consulting firm. SFT LLMs imitate paraphrasing customer feedback into customer needs. LLMs trained using self-supervised and reinforcement learning methods. Marketing science community exploring LLM applications for research.
Mar 13, 2025•5 min
MLLMs exploit spurious correlations, affecting robustness and generalization The paper introduces SpurLens to identify and measure spurious cues Various prompting strategies were tested but none were effective
Mar 13, 2025•5 min
Test-time verifiers improve reasoning performance by guiding solution chains Inefficient searches can arise from overlapping solutions and incorrect completions The paper proposes combining process verifiers with preemptive backtracking This approach reduces computation by leveraging partial reasoning traces
Mar 13, 2025•4 min
The paper proposes an adaptive elicitation framework for reducing uncertainty It utilizes large language models for strategic information gathering The framework is validated through dynamic polling and student assessments It aims to enhance decision-making in various application domains
Mar 13, 2025•4 min
