Deep Learning is Not So Mysterious or Different

This position paper, "Deep Learning is Not So Mysterious or Different" by Andrew Gordon Wilson, argues against the notion that deep neural networks exhibit unique or mysterious generalization behaviors like benign overfitting, double descent, and overparametrization. The author contends that these phenomena are not exclusive to deep learning and can be understood and formally characterized by long-standing generalization frameworks, such as PAC-Bayes and countable hypothesis bounds, rather than requiring a re-evaluation of established generalization theories. A central unifying principle proposed is soft inductive biases, which embrace flexible hypothesis spaces with a preference for simpler solutions consistent with data, as opposed to restrictive biases. While highlighting these commonalities, the text acknowledges that deep learning possesses distinct characteristics such as representation learning, universal learning, and mode connectivity, which still warrant further investigation. Ultimately, the piece seeks to bridge understanding across different machine learning communities by demonstrating that many perceived "mysteries" of deep learning are explainable through existing theoretical frameworks and are reproducible with simpler model classes.