Episode description
Episode description
This study examines the performance of the DeepSeek R1 language model on complex mathematical problems, revealing that it achieves higher accuracy than other models but uses considerably more tokens. Here's a summary:
DeepSeek R1's strengths:
DeepSeek R1 excels at solving complex mathematical problems, particularly those that other models struggle with, due to its token-based reasoning approach.
Token usage: DeepSeek R1 uses a significantly higher number of tokens compared to other models. The average token count for DeepSeek R1 is 4717.5, while other models average between 191.75 and 462.39. This higher token usage is linked to its more deliberate, multi-step problem-solving process.
Trade-off: The study highlights a trade-off between accuracy and efficiency. While DeepSeek R1 offers superior accuracy, it requires longer processing times because of its extensive token generation. Models like Mistral might be faster but less accurate, making them suitable for tasks requiring rapid responses.
Temperature settings: The experiment underscores the importance of temperature settings in influencing model behaviour. For instance, Llama 3.1 only achieved correct results at a temperature of 0.4, demonstrating the sensitivity of some models to this parameter.
Methodology: The study used 30 challenging mathematical problems from the MATH dataset, which were previously unsolved by other models under time constraints. Five LLMs were tested across 11 different temperature settings, and the correctness of the solutions was evaluated, also tracking the number of tokens generated. A binary metric was used for correctness using the mistral-large-2411 model as a judge.
Models evaluated: The models evaluated include deepseek-r1:8b, gemini-1.5-flash-8b, gpt-4o-mini-2024-07-18, llama3.1:8b, and mistral-8b-latest.
Dataset: The dataset is derived from a previous benchmark experiment that evaluated LLMs on advanced mathematical problem-solving. The 30 problems were selected because no model in the original study could solve them within imposed time limits.
Future research: Future research should explore the internal workings of DeepSeek R1 to better understand "reasoning tokens" and explore methods to reduce token usage. Prompt engineering strategies should also be examined to maximise model performance.
Source: Evstafev, E. (2025) Token-Hungry, Yet Precise: DeepSeek R1 Highlights the Need for Multi-Step Reasoning Over Speed in MATH.
Hello SundAI - our world through the lense of AI
Disclaimer: This podcast is generated by Roger Basler de Roca (contact) by the use of AI. The voices are artificially generated and the discussion is based on public research data. I do not claim any ownership of the presented material as it is for education purpose only.