This paper argues that probabilistic modelling is sufficient for causal inference, challenging the belief that specialized causal notations like the "do-operator" are strictly necessary. By advocating for a "write down the probability of everything" approach, the authors demonstrate that interventional and counterfactual questions can be solved using standard **Bayesian Networks** and joint distributions. They reinterpret traditional causal tools, such as **Structural Causal Models**, as useful ...
Jan 03, 2026•12 min
This research introduces TTT-E2E, a novel method for long-context language modeling that treats the task as a continual learning challenge rather than an architectural redesign. Unlike standard Transformers that struggle with the high computational cost of processing vast amounts of data, this model **compresses context into its weights** by learning at test time via next-token prediction. By integrating **meta-learning during training**, the system is optimized to initialize effectively for the...
Jan 03, 2026•14 min
This research introduces **Parallel Token Prediction (PTP)**, a novel framework designed to accelerate language model inference by generating multiple tokens simultaneously in a single forward pass. Standard models suffer from a **sequential bottleneck**, but PTP overcomes this by incorporating auxiliary random variables directly into the model's inputs to coordinate interdependent predictions. The authors provide mathematical proof that this method is as **expressively powerful** as traditional...
Jan 02, 2026•16 min
This research introduces Posterior Behavioral Cloning (POSTBC), a novel pretraining method designed to enhance the reinforcement learning (RL) finetuning of robotic policies. Traditional behavioral cloning (BC) often fails because it overfits to specific demonstration data, resulting in poor action coverage and limited exploration during subsequent online learning. By modeling the posterior distribution of demonstrator behavior rather than simply mimicking actions, POSTBC injects uncertainty-awa...
Dec 31, 2025•16 min
This research paper introduces Activation Oracles (AOs), which are large language models trained to translate the internal mathematical activations of other models into plain English. While previous methods for interpreting these internal states were highly specialized and narrow, AOs act as general-purpose explainers that can answer a wide variety of natural language questions about what a model is thinking. By training on diverse tasks like context prediction and classification, these oracles ...
Dec 30, 2025•15 min
Researchers have developed a method to improve reinforcement learning (RL) by leveraging the internal representations of pretrained autoregressive models. While standard AI models struggle with sparse-reward tasks because they explore through token-by-token variations, this approach introduces an unsupervised metacontroller that discovers temporally-abstract actions. By intervening directly in the model's residual stream at mid-depth, the system learns to execute high-level subroutines that span...
Dec 29, 2025•14 min
This research investigates the theoretical and practical differences between reconstruction-based and joint-embedding paradigms in self-supervised learning (SSL). By deriving the first closed-form solutions for these methods, the authors demonstrate that joint-embedding approaches are more robust when datasets contain high-magnitude irrelevant noise, such as complex backgrounds in images. Conversely, reconstruction is more effective for data with low-magnitude noise, explaining its success in na...
Dec 29, 2025•14 min
This research explores Chain-of-Thought (CoT) monitorability, which refers to how effectively an external system can detect misbehavior by analyzing a model's internal reasoning steps. The authors introduce a diverse evaluation taxonomy that categorizes environments based on whether they involve interventions, specific processes, or final outcomes, such as sycophancy, bias, and sabotage. To measure monitoring success accurately, the study utilizes g-mean², a metric designed to penalize failures ...
Dec 28, 2025•14 min
Researchers have discovered a macroscopic physical law governing the behavior of Large Language Model (LLM)-driven agents, revealing that their generative dynamics mirror equilibrium systems in physics. By measuring transition probabilities between states, the study demonstrates that these agents follow a detailed balance condition, suggesting they do not merely learn specific rules but instead optimize an internal potential function. This function acts as a global guide, allowing models to perc...
Dec 28, 2025•12 min
This research formalizes the process of reasoning in large language models as a latent variable model, utilizing the expectation-maximization (EM) algorithm to improve performance. The authors demonstrate that training a model to generate intermediate rationales before answering is mathematically equivalent to reward-weighted fine-tuning using binary correctness as a signal. A central focus of the study is the sampling distribution used to create these rationales, comparing methods like rejectio...
Dec 28, 2025•14 min
This research introduces **Exploratory Causal Inference**, a framework designed to identify unknown treatment effects within high-dimensional datasets. The authors propose using **foundation models** and **sparse autoencoders (SAEs)** to transform raw data into a dictionary of interpretable latent features. To solve the "**paradox of exploratory causal inference**"—where increased data power causes irrelevant, entangled neurons to appear falsely significant—they develop the **Neural Effect Searc...
Dec 25, 2025•15 min
This research identifies a **macroscopic physical law** governing the behavior of large language model (LLM)-driven agents. By analyzing state transitions as **Markov processes**, the authors discovered that these systems naturally satisfy a **detailed balance condition**, similar to physical systems in equilibrium. This suggests that LLMs do not merely follow rote strategies but instead learn internal **potential functions** that guide them toward optimal solutions. The study introduces a **lea...
Dec 24, 2025•12 min
This paper introduces the Prism Hypothesis, which suggests that multimodal data shares a **common frequency spectrum** where **low-frequency bands** hold abstract meaning and **high-frequency bands** store fine details. To implement this theory, the authors developed **Unified Autoencoding (UAE)**, a framework that integrates **semantic perception** and **pixel-level fidelity** into a single latent space. This model utilizes a **frequency-band modulator** to separate global structures from intri...
Dec 24, 2025•16 min
This paper introduces a systematic framework for **agentic AI adaptation**, categorizing research into four distinct paradigms based on whether the **agent** or its **tools** are being optimized. **Agent adaptation** involves updating core models using either **tool-execution signals** for causal feedback or **agent-output signals** for holistic task performance. In contrast, **tool adaptation** focuses on refining external modules, either as **agent-agnostic** components or through **agent-supe...
Dec 23, 2025•13 min
This research introduces Posterior Behavioral Cloning (POSTBC), a novel pretraining method designed to enhance reinforcement learning (RL) finetuning for robotic policies. Standard behavioral cloning often fails because it overfits to specific demonstration data, leading to an action coverage deficit that prevents the model from exploring effectively during later stages. To solve this, the authors propose training a policy to model the posterior distribution of the demonstrator’s behavior, which...
Dec 22, 2025•11 min
Large language models often struggle with long-context tasks because the attention mechanism suffers from **score dilution**, where relevant information is overwhelmed by surrounding "distractor" tokens. Researchers found that common **inference-time scaling strategies**, such as generating additional "thinking tokens," fail to solve this problem as context length increases. To address this, the authors propose **query-only test-time training (qTTT)**, a computationally efficient method that upd...
Dec 21, 2025•14 min
This paper discusses TabPFN-2.5, a sophisticated tabular foundation model designed to handle diverse datasets with up to 50,000 samples and 2,000 features. This next-generation AI significantly outperforms traditional tree-based models and complex ensembles like AutoGluon in a fraction of the time. The researchers highlight its state-of-the-art performance across various industries, particularly in healthcare, finance, and manufacturing, where it excels even with limited data. To facilitate indu...
Dec 20, 2025•15 min
This paper introduces a method for automatically decoding hidden preferences from language model training data. By utilizing sparse autoencoders, the method translates complex text embeddings into a small set of interpretable features that explain why human annotators prefer one response over another. The research reveals that feedback datasets often contain conflicting signals, such as Reddit users favoring informal jokes while other groups disfavor them. Notably, the authors demonstrate that W...
Dec 19, 2025•16 min
We discuss Bolmo, a groundbreaking family of byte-level language models by AI2 that offers a practical alternative to traditional subword-based tokenization. Developed by the Allen Institute for AI and collaborating universities, these models achieve state-of-the-art performance by "byteifying" existing subword models like OLMo. This innovative process uses a specialized two-stage distillation procedure to convert subword models into byte-level ones using less than 1% of the original pretraining...
Dec 19, 2025•13 min
We cover Neel Nanda (Google DeepMind)'s discussion on efficacy and limitations of Sparse Autoencoders (SAEs) as a tool for unsupervised discovery and interpretability in large language models. Initially considered a major breakthrough for breaking down model activations into interpretable, linear concepts, the conversation explores the subsequent challenges and pathologies observed in SAEs, such as feature absorption and the difficulty of finding truly canonical units. While acknowledging that S...
Dec 17, 2025•30 min
We discuss Neel Nanda (Google DeepMind)'s perspectives on the current state and future directions of mechanistic interpretability (MI) in AI research. Nanda discusses major shifts in the field over the past two years, highlighting the improved capabilities and "scarier" nature of modern models, alongside the increasing use of inference time compute and reinforcement learning. A key theme is the argument that MI research should primarily focus on understanding model behavior, such as AI psycholog...
Dec 16, 2025•33 min
This paper discusses how Retrieval-Augmented Generation (RAG) framework can be designed to overcome the structural issues of separate retrieval and generation modules. The proposed framework, CLaRa, achieves this by employing a **shared latent space** where documents are compressed into concise, continuous memory-token representations, addressing the architectural mismatch and efficiency problems of traditional RAG. Key to CLaRa is its **joint optimization** mechanism, which uses the Next-Token ...
Dec 16, 2025•15 min
This paper studies "co-improvement" as a safer and faster alternative to the current focus on "autonomous self-improving AI" for achieving superintelligence. This paper argues that instead of AI systems improving themselves without human intervention, the focus should be on building AI that actively collaborates with human researchers across all stages of the research pipeline, from ideation to evaluation and safety alignment. The authors propose that this bidirectional collaboration, leading to...
Dec 16, 2025•13 min
This academic paper by Google Research, Google DeepMind, and the Massachusetts Institute of Technology, systematically evaluates the principles for scaling language model-based agent systems, moving beyond anecdotal evidence that "more agents is all you need." The authors present a controlled evaluation across four diverse agentic benchmarks, testing five canonical architectures—Single-Agent, Independent, Centralized, Decentralized, and Hybrid Multi-Agent Systems—to isolate the effect of coordin...
Dec 15, 2025•16 min
This paper discusses how a successful RL fine-tuning uncovers an emergent two-phase hierarchical reasoning dynamic in LLMs, mirroring human cognition by separating high-level strategic planning from low-level procedural execution. The authors argue that conventional RL methods, which apply optimization pressure agnostically to all tokens, are inefficient because they fail to concentrate learning efforts on the true bottleneck: mastering strategic planning tokens. The proposed method, HICRA, addr...
Dec 14, 2025•13 min
We discuss an interview with Jure Lescovec, co-founder of kumu.ai and a computer science professor at Stanford, regarding the application of foundation models to structured enterprise data. Lescovec explains that traditional **machine learning** methods for this type of data are manual, expensive, and time-consuming, contrasting them with new relational foundation models that leverage a **graph-based approach** to eliminate the need for manual **feature engineering** and **model training**. The ...
Dec 13, 2025•15 min
This paperq introduces REFRAG, an innovative and efficient decoding framework specifically designed to accelerate *lRetrieval-Augmented Generation (RAG) in Large Language Models (LLMs) by addressing high latency and memory demands associated with long-context inputs. The core mechanism involves compressing context by representing chunks of retrieved text as single embeddings, significantly shortening the input sequence to the decoder and exploiting the **sparse attention patterns** inherent in R...
Dec 13, 2025•14 min
This academic paper rigorously investigates the power of next-token prediction for training large language models (LLMs), specifically focusing on Recurrent Neural Networks (RNNs). The core finding is that simply minimizing the next-token log loss during training is sufficient to yield an LLM whose output is computationally indistinguishable from the true training distribution over long sequences of up to $k$ tokens, provided the model size is sufficiently large. The authors establish this throu...
Dec 12, 2025•12 min
We summarize a recent interview with Jeff Dean, a legendary Chief Scientist at Google who has been leading Gemini, focusing on the **evolution and current state of Google's Tensor Processing Units (TPUs)**, including the recent seventh-generation announcement. Dean explains that the initial motivation for TPUs was Google's internal need to handle the massive compute requirements of scaling AI models, highlighting the **efficiency gains over CPUs and GPUs**. The conversation also shifts to the br...
Dec 12, 2025•38 min
This paper introduces Latent Debate, a novel framework designed to interpret the internal "thinking" processes and address hallucinations in Large Language Models (LLMs). Unlike external methods that rely on multiple models debating, Latent Debate uses implicit internal arguments—supporting and attacking signals—arising within a single model during a single inference. This framework utilizes a Quantitative Bipolar Argumentation Framework (QBAF) as a "thinking module" to aggregate these internal ...
Dec 11, 2025•15 min
