HTML conversions sometimes display errors due to content that did not convert correctly from the source. This paper uses the following packages that are not yet supported by the HTML conversion tool. Feedback on these issues are not necessary; they are known and are being worked on.
Authors: achieve the best HTML results from your LaTeX submissions by following these best practices.
License: arXiv.org perpetual non-exclusive license
Wenyuan Zhang, Shuaiyi Nie∗, Xinghua Zhang, Zefeng Zhang, Tingwen Liu†
Institute of Information Engineering, Chinese Academy of Sciences School of Cyber Security, University of Chinese Academy of Sciences {zhangwenyuan,nieshuaiyi,liutingwen}@iie.ac.cn denotes equal contribution. † denotes corresponding author.
Abstract
We introduce S1-Bench, a novel benchmark designed to evaluate Large Reasoning Models’ (LRMs) performance on simple tasks that favor intuitive system 1 thinking rather than deliberative system 2 reasoning. While LRMs have achieved significant breakthroughs in complex reasoning tasks through explicit chains of thought, their reliance on deep analytical thinking may limit their system 1 thinking capabilities. Moreover, a lack of benchmark currently exists to evaluate LRMs’ performance in tasks that require such capabilities. To fill this gap, S1-Bench presents a set of simple, diverse, and naturally clear questions across multiple domains and languages, specifically designed to assess LRMs’ performance in such tasks. Our comprehensive evaluation of 22 LRMs reveals significant lower efficiency tendencies, with outputs averaging 15.5 times longer than those of traditional small LLMs. Additionally, LRMs often identify correct answers early but continue unnecessary deliberation, with some models even producing numerous errors. These findings highlight the rigid reasoning patterns of current LRMs and underscore the substantial development needed to achieve balanced dual-system thinking capabilities that can adapt appropriately to task complexity1.
\pdfcolInitStack
tcb@breakable
S1-Bench: A Simple Benchmark for Evaluating System 1 Thinking Capability of Large Reasoning Models

Figure 1: LRMs exhibit under-accuracy and overthinking on simple problems. Shapes represent organizations, colors represent base model families, with darker colors indicating larger models, and connecting lines represent the relationships between model families and training.
“Simplicity is the ultimate sophistication.”
— Leonardo da Vinci
Recent advances in Large Reasoning Models (LRMs), notably OpenAI’s o1/o3 OpenAI (2024) and the DeepSeek-R1 Guo et al. (2025) series, have propelled the development of Large Language Models (LLMs) by explicitly generating external chain-of-thought (COT) Wei et al. (2022) before producing final answers, achieving remarkable performance on complex reasoning tasks. Unlike traditional LLMs that primarily rely on intuitive, heuristic system 1 thinking, LRMs enhance their performance by employing analytical, deliberative system 2 thinking Qu et al. (2025); Li et al. (2025b), which includes sophisticated strategies such as self-reflection, multi-path exploration Li et al. (2025a); Yeo et al. (2025). Despite the benefits of system 2 thinking, a rigid reliance on this cautious thinking mode can also introduce new limitations, such as performance degradation in specific contexts Feng et al. (2025); Zhao et al. (2025) or overthinking Chen et al. (2024); Kumar et al. (2025a).