1000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilities

This paper discusses scaling the depth of neural networks within self-supervised reinforcement learning (RL), a field where scaling has historically lagged behind language and vision models. Challenging the convention of using shallow architectures (2–5 layers), the researchers demonstrate that scaling network depth up to 1024 layers substantially boosts performance in unsupervised goal-conditioned tasks, achieving gains as high as 50 times the performance of previous methods. This deep scaling approach integrates Contrastive RL (CRL) with architectural stabilizing components like residual connections. The study establishes that increasing depth is a more impactful and computationally efficient scaling axis than increasing network width and that it is necessary to unlock the utility of larger batch sizes. Furthermore, this capacity increase leads to the emergence of qualitatively distinct goal-reaching policies and enables the deep networks to learn richer environmental representations.