We're in a bit of a new renaissance of language models in the last few months with reasoning models. Benchmarks are being broken and the number of recent papers on reasoning models is blowing-up. Below I give my own rambling interpretation of these results and then reason a bit through the consequences in AI for Science -- namely, how will it change the application of AI to scientific domains.

background

I think Noam Brown and Andy Jones had a great perspective in reinforcement learning for board games a few years back that showed you can increase inference time to increase performance of models that play board-games. They showed there are trade-offs, where you can get a better return by balancing compute on training time and testing time. Here's a particularly famous plot from Andy's paper:1

Training vs Test time scaling tradeoffs

The plot shows that you can get to 1500 ELO (how good the model is) via different choices of amount of compute spent at train time or test time. Prior to late 2024, generally large language models only explored scaling training compute. More = better. This felt like a big hole was missing in the field. Could we spend more time doing inference and maybe even less time on enormous pre-training compute runs?

Noam Brown went to OpenAI and, presumably, he brought the hypothesis there and it led to reasoning language models that can expend extra reasoning tokens to do better at answering questions. This is a continuation of the original chain of thought (CoT) work,2 that showed language models do better if you prompt them to think step by step on how to answer a problem. Just like how you might get a human to do better on a word problem.3

From an outside perspective, it looks like OpenAI found a way to elicit the models to spend more inference compute on reasoning in a CoT. Those CoTs lead to improved performance on evals that require reasoning (math, code, hard STEM problems). This was o1, and it provided significant gains on many benchmarks.4

This led to series of papers trying to both reverse engineer and build similar reasoning models.5 There were a number of hypotheses about what matters - is it better search/sampling methods, is it better iterations,6 is it better scaffolding, is it new architectures, or is it new algorithms?

recipe for reasoning models

There have now been three compelling papers that showed how to build models similar or better than o1:

  1. MoonShotAI Kimi

    7

  2. AI2 Tulu v3

    8

  3. DeepSeek r1

    9

Here is my interpretation of the findings:

  1. Simple prompts and simple inference strategies work. Do many rollouts at both training time (for more correct trajectories) and inference time (for monkeys). This is better than process reward models, Monte-Carlo Tree search, or special construction of training examples.

    10

  2. Just leaving space for CoT tokens and using reinforcement learning leads to continued cycles of improvement. Challenging problems, including sparse binary outcomes, can be tackled in this way.

  3. The nice CoT reasoning attributes like backtracking, self-reflection, and breaking down problems emerge without prompt engineering or special algorithms, just from RL.

I think its instructive to compare the rStar paper vs the strategy in the DeepSeek paper. Here's a nice overview of how the rStar paper uses a "process preference model", which is basically a Q-value predictor, to guide a tree search algorithm for inference (the 2-shot prompt omitted):

The MCTS and reward models