\correspondingauthor
[email protected], [email protected]
Arian Hosseini Alessandro Sordoni Mila Microsoft Research Daniel Toyama Aaron Courville Rishabh Agarwal Mila Google DeepMind
Abstract
We study the depth of grade-school math (GSM) problem-solving capabilities of LLMs. To this end, we evaluate their performance on pairs of existing math word problems together so that the answer to the second problem depends on correctly answering the first problem. Our findings reveal a significant reasoning gap in most LLMs, that is performance difference between solving the compositional pairs and solving each question independently. This gap is more pronounced in smaller, more cost-efficient, and math-specialized models. Moreover, instruction-tuning recipes and code generation have varying effects across LLM sizes, while finetuning on GSM can lead to task overfitting. Our analysis indicates that large reasoning gaps are not because of test-set leakage, but due to distraction from additional context and poor second-hop reasoning. Overall, LLMs exhibit systematic differences in their reasoning abilities, despite what their performance on standard benchmarks indicates.

Figure 1: Reasoning Gap: Most models demonstrate a noticeable gap between their reasoning performance on GSM8K and compositional GSM, in which pairs of GSM8K test questions are chained together so that the answer of the first question (Q1subscript𝑄1Q_{1}italic_Q start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT) is a variable in the second one (Q2subscript𝑄2Q_{2}italic_Q start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT). The model is required to correctly answer both questions to solve the problem. If a model has an accuracy of S1subscript𝑆1S_{1}italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT on the Q1subscript𝑄1Q_{1}italic_Q start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT set, and S2subscript𝑆2S_{2}italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT on Q2subscript𝑄2Q_{2}italic_Q start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT set, then the expected compositional GSM accuracy is S1×S2subscript𝑆1subscript𝑆2{S_{1}\times S_{2}}italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT × italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT. The x-axis corresponds to the geometric mean S1×S2subscript𝑆1subscript𝑆2\sqrt{S_{1}\times S_{2}}square-root start_ARG italic_S start_POSTSUBSCRIPT 1 end_POSTSUBSCRIPT × italic_S start_POSTSUBSCRIPT 2 end_POSTSUBSCRIPT end_ARG, labeled GSM8K accuracy for simplicity. The trend-line y=x2𝑦superscript𝑥2y=x^{2}italic_y = italic_x start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT is the expected compositional GSM accuracy.
The strong performance of large language models (LLMs) on high-school and college-level math reasoning benchmarks (OpenAI, 2023b; Google, 2024; AI@Meta, 2024), has led to the common belief that LLMs have “mastered” grade-school math, particularly as measured by the GSM8K benchmark (Cobbe et al., 2021). This apparent mastery of grade-school math problems raises a deeper question: do LLMs truly grasp the underlying concepts or do they mostly rely on superficial pattern recognition? For example, a recent examination on private “held-out” grade-school problems (Zhang et al., 2024) reveals that while state-of-the-art LLMs show minimal signs of overfitting, some open-weights models show systematic overfitting, possibly due to test-set leakage.
Let X be the answer to the 𝐐𝟏subscript𝐐1\mathbf{Q_{1}}bold_Q start_POSTSUBSCRIPT bold_1 end_POSTSUBSCRIPT:
𝐐𝟏subscript𝐐1\mathbf{Q_{1}}bold_Q start_POSTSUBSCRIPT bold_1 end_POSTSUBSCRIPT: There are 27 unicorns left in the world. One third of them are in the Scottish Highlands. Two thirds of the Scottish unicorns are female. How many female Scottish unicorns are there?
Solve it and use the value of X to solve 𝐐𝟐subscript𝐐2\mathbf{Q_{2}}bold_Q start_POSTSUBSCRIPT bold_2 end_POSTSUBSCRIPT. Explain your answer step by step.
𝐐𝟐subscript𝐐2\mathbf{Q_{2}}bold_Q start_POSTSUBSCRIPT bold_2 end_POSTSUBSCRIPT: Zack’s locker is half as big as Timothy’s locker. Peter’s locker is 1/4 as big as Zack’s locker. If Peter’s locker is X cubic inches, how big is Timothy’s locker in cubic inches?
Figure 2: Example Problem from the Compositional GSM test. The answer of Question-1 (𝐐𝟏subscript𝐐1\mathbf{Q_{1}}bold_Q start_POSTSUBSCRIPT bold_1 end_POSTSUBSCRIPT) is a variable X in Question-2 (𝐐𝟐subscript𝐐2\mathbf{Q_{2}}bold_Q start_POSTSUBSCRIPT bold_2 end_POSTSUBSCRIPT). The model has to be able to solve the first question correctly in order to solve the second question. The new final answer of 𝐐𝟐subscript𝐐2\mathbf{Q_{2}}bold_Q start_POSTSUBSCRIPT bold_2 end_POSTSUBSCRIPT is calculated by modifying its code-form solution and executing it. We used a modified version of the code-form solutions from Gao et al. (2023). Question-1 and the number to modify in Question-2 are chosen to have a new final answer which is a positive integer not too far from the old answer of Question-2.
In this work, we perform a case study to probe the brittleness of their reasoning abilities and to evaluate how well LLMs can combine learned concepts to solve new problems (Hupkes et al., 2020) To do so, we introduce Compositional GSM, a two-hop version of GSM8K at the same math difficulty level, where each problem chains two test questions together such that the answer to the first question is used as a variable in the second question (Figure 2). As LLMs can easily solve grade-school math problems, they should also be capable of solving combinations of those problems. As such, we measure the gap between their performance on solving the questions individually and on compositional GSM. Specifically, we benchmark frontier open-weights and closed LLMs, including Gemini, Gemma2, LLAMA3, GPT, Phi, Qwen2.5, and Mistral families.
Here are our key findings:
•
Most models exhibit a clear gap between their performance on GSM8K and compositional GSM (Figure 1,3), which undermines their reliability and reasoning ability.
•
This reasoning gap is particularly evident in small, more cost-efficient (Figure 4), and math-specialized models (Figure 7), reducing their utility in practice.
•
Despite similar settings, instruction-following tuning impacts LLMs of varying sizes in significantly different ways (Figure 5), calling for re-examination of standard training recipes.