Evolutionary Prompt Optimization discovers emergent multimodal reasoning strategies

This paper introduces an innovative framework using an evolutionary algorithm to optimize prompts for vision-language models without requiring additional training. The method evolves prompts through iteration and selection to elicit complex multimodal reasoning abilities, such as breaking down tasks and employing external tools like Python interpreters for image manipulation. Experimental results demonstrate that this evolutionary prompt optimization, especially when coupled with tool usage, significantly improves performance on challenging visual and reasoning benchmarks compared to traditional prompting or standard evolutionary methods. The research highlights how this technique can discover sophisticated problem-solving strategies naturally, treating system prompts as a form of lightweight neural programming.