A Generalization Theory for Zero-Shot Prediction

This research paper establishes a formal learning theoretic framework to analyze the performance of zero-shot prediction (ZSP) in multimodal models like CLIP. The authors decompose prediction error into three distinct components: prompt bias, which measures the suitability of a prompting strategy; residual dependence, which quantifies the information lost when using text as a proxy for image features; and estimation error from finite data. By avoiding common but unrealistic assumptions of conditional independence, the study provides theoretical guarantees for how pre-training distributions and prompting methods influence downstream task accuracy. The framework introduces two primary mathematical approaches—conditional mean and information density—to evaluate how indirect predictors compare to direct supervised learners. Finally, the authors validate their theory through empirical simulations and image data experiments, demonstrating that minimizing residual dependence and prompt bias is essential for optimizing zero-shot performance.