Synthesized from the database. All benchmarks and frameworks below are drawn directly from papers and articles saved in that database.
Prompt: Go through my database and write a comprehensive memo detailing the information you find on AI benchmarks, share the benchmarks you find, and then try to synthesize what makes a good benchmark and the commonly used metrics.
AI benchmarking has become one of the most contested and consequential sub-fields in machine learning. Benchmarks serve as the shared yardstick by which models are compared, resources are allocated, and capabilities are marketed to the public. Yet as a Nature article saved in the database bluntly put it: "AI now beats humans at basic tasks — new benchmarks are needed." This memo catalogues the benchmarks found in the database, organizes them by domain, and synthesizes the common threads around what makes a benchmark rigorous, durable, and useful.
MMLU (Massive Multitask Language Understanding)
Frequently referenced across multiple papers (particularly the BALROG paper), MMLU is a broad multi-domain benchmark covering 57 subjects ranging from elementary mathematics to professional law. It became the de facto standard for measuring general-purpose language understanding and has been largely saturated by frontier models.
SuperGLUE
The successor to GLUE, SuperGLUE tests general-purpose language understanding across tasks like question answering, coreference resolution, and natural language inference. Referenced in both the BALROG paper and the Import AI newsletter (Baidu took the SuperGLUE crown), it too has been largely saturated.
BigBench (Beyond the Imitation Game)
Expands scope to include diverse linguistic and cognitive challenges deliberately chosen to be hard for current models. A forward-looking attempt to stay ahead of saturation.