Preprint, under review.

Davide Paglieri1∗**, Bartłomiej Cupiał2∗, Samuel Coward3, Ulyana Piterbarg4, Maciej Wolczyk**2, Akbir Khan1,5, Eduardo Pignatelli1, Łukasz Kuciński2,Lerrel Pinto4 Rob Fergus4, Jakob Nicolaus Foerster3, Jack Parker-Holder1, Tim Rocktäschel1 1AI Centre, University College London, 2IDEAS NCBR, 3University of Oxford, 4New York University, 5Anthropic

ABSTRACT

Large Language Models (LLMs) and Vision Language Models (VLMs) possess extensive knowledge and exhibit promising reasoning abilities, however, they still struggle to perform well in complex, dynamic environments. Real-world tasks re- quire handling intricate interactions, advanced spatial reasoning, long-term plan- ning, and continuous exploration of new strategies—areas in which we lack effec- tive methodologies for comprehensively evaluating these capabilities. To address this gap, we introduce BALROG, a novel benchmark designed to assess the agen- tic capabilities of LLMs and VLMs through a diverse set of challenging games. Our benchmark incorporates a range of existing reinforcement learning environ- ments with varying levels of difficulty, including tasks that are solvable by non- expert humans in seconds to extremely challenging ones that may take years to master (e.g., the NetHack Learning Environment). We devise fine-grained met- rics to measure performance and conduct an extensive evaluation of several pop- ular open-source and closed-source LLMs and VLMs. Our findings indicate that while current models achieve partial success in the easier games, they struggle significantly with more challenging tasks. Notably, we observe severe deficien- cies in vision-based decision-making, as models perform worse when visual rep- resentations of the environments are provided. We release BALROG as an open and user-friendly benchmark to facilitate future research and development in the agentic community.

1 INTRODUCTION

Recent successes of Large Language Models (LLMs) have renewed interest in building general- purpose agents capable of autonomously achieving complex goals Yang et al. (2023). LLMs possess vast knowledge across domains (Brown, 2020; Hendrycks et al., 2020), can reason in specific scenar- ios (Wei et al., 2022a; Shinn et al., 2023; Rein et al., 2023), and can reliably follow human instruc- tions in simple settings (Ouyang et al., 2022). These abilities suggest that LLMs have the potential to become efficient agents, capable of autonomously performing a wide range of human tasks that require sequential decision making. In the present day, however, state-of-the-art models continue to exhibit persistent failure modes on many of the skills that are crucial for autonomous real-world in- teraction. For example, LLMs fail to act robustly in dynamic environments, and they cannot reliably learn from mistakes, reason about space and time, or plan over long time horizons (Xing et al., 2024; Yamada et al., 2023; Kambhampati et al., 2024). Improving our understanding of LLM capabilities through rigorous, safe evaluations is key for assessing the risks and limitations of deploying agentic LLMs in the real world.

Current agentic benchmarks evaluate LLM performance in settings that involve no more than a few dozen rounds of interaction between a model and an environment, e.g., solving simple office tasks (Wang et al., 2024), navigating the Internet (Zhou et al., 2023), and resolving GitHub is- sues (Jimenez et al., 2023). New agentic prompting frameworks and improvements to short-horizon

∗Equal technical contribution, first author was the project lead. Correspondence to [email protected]. Code and Leaderboard at balrogai.com

1