Evaluating the performance of AI systems and foundation models—including large language models (LLMs), vision-language models (VLMs), and domain-specific AI—requires a comprehensive set of metrics. These metrics span technical accuracy, computational efficiency, robustness, alignment, and application-specific outcomes. Drawing on resources and papers from the "Papers - Computation & Investing" library, this report outlines the key metrics and benchmarks used to assess AI and new-age foundation models, with a focus on both general AI evaluation and specialized domains such as finance, investing, and computational science.

1. Technical Accuracy and Task Performance

1.1. Standard Accuracy Metrics

For classification, regression, and ranking tasks, AI models are commonly evaluated using metrics such as:

• Accuracy, Precision, Recall, F1 Score: Fundamental for classification and detection tasks.
• Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), R-squared: Standard for regression models, including those used in financial valuation and forecasting[1].
• Precision@K, Recall@K, Mean Reciprocal Rank (MRR), Mean Average Precision (MAP), Discounted Cumulative Gain (DCG), Normalized DCG (NDCG): Widely used for ranking and retrieval tasks, essential for recommender systems and information retrieval[2].

1.2. Specialized Benchmarks

Foundation models are evaluated on established benchmarks that test a variety of capabilities:

• MMLU (Massive Multitask Language Understanding): Assesses multitask generalization across numerous domains[3].
• HumanEval: Measures code generation and problem-solving abilities, especially for language models.
• ScholarQABench: Focuses on factuality and citation accuracy for models synthesizing scientific literature[4].
• BALROG: Benchmarks agentic reasoning in LLMs and VLMs through complex game environments, using progression and task completion rates as metrics[5].
• RE-Bench: Evaluates AI agents on complex, real-world tasks, comparing performance to human experts[3].

1.3. LLM-Specific Output Metrics

For generative models, especially LLMs, new evaluation frameworks and metrics have emerged:

• Helpfulness, Coherence, Depth: LangChain and similar frameworks use these for qualitative assessment of generated outputs[6].
• Factuality and Citation F1: Measures the correctness and verifiability of generated scientific or factual responses, crucial for retrieval-augmented models[4].
• Pass@k: Used for code generation tasks, indicating the probability of producing a correct solution within k attempts[7].

2. Computational Efficiency and Cost