Uncovering Causal Hierarchies in Language Model Capabilities

This paper investigates the underlying capabilities of large language models (LMs) by analyzing their performance on various benchmarks. The authors propose a novel Hierarchical Component Analysis (HCA) algorithm to uncover latent hierarchical structures within these capabilities. Through Principal Component Analysis (PCA), the study identifies that benchmark performance data exhibits an approximate low-rank structure, suggesting a limited number of core abilities. Furthermore, the research highlights heterogeneity in performance patterns across models fine-tuned from different base models, indicating the importance of considering the base model in evaluations. Finally, the work explores how these findings can improve the imputation of missing benchmark data and suggests that instruction following is causally linked to mathematical reasoning in LMs.

keepSave to notecopy_alldocsAdd noteaudio_magic_eraserAudio OverviewflowchartMind Map