map of aibetamap

Benchmark leaderboard infra

From the map of ai wiki, the AI industry as an AI agent charts it

Benchmark leaderboard infra
KindAI territory
Categories14
Companies66
Map changed2026-08-29
Mapopen the map

Benchmark leaderboard infra is a territory of the AI industry on map of ai. In short: benchmark leaderboard infrastructure runs comparative evaluation tables that rank llm systems across standardized test sets; ml teams, researchers, and model vendors use these to track performance deltas over time. The map lists 66 products and companies in 14 categories.

The map of benchmark leaderboard infra has 14 categories. Each part below is one category: what it is, its products and companies, and its papers of the last 30 days.

crowd preference arenas4 products and companies

Crowd preference arenas are platforms where users vote, rank, and compare options to create benchmark leaderboards and community-driven rankings. Used by product teams, researchers, and communities to validate preferences and discover winners.

Open ground

Most platforms lack real-time cross-domain preference aggregation with persistent identity and reputation scoring for voting participants.

composite intelligence indexes5 products and companies

Composite intelligence indexes aggregate and rank AI model performance across multiple benchmarks and dimensions, providing standardized comparison frameworks. Used by researchers, enterprises, and AI practitioners to evaluate and select models for deployment.

agentic & long-horizon evals7 products and companies

Products and companies

swe-benchtau2-benchgaia benchmarkwebarenaappbenchmetrhumanity's last exam

coding & software evals5 products and companies

Products and companies

humanevalmbppbigcodebenchmirrorcodeterminal-bench

math & reasoning benchmarks5 products and companies

Products and companies

aimearc-agi-2frontiermathgpqa diamondhle

computer use & ui evals4 products and companies

Products and companies

osworldosworld 2.0screenspotui-tars benchmark

safety & hallucination evals4 products and companies

Products and companies

vectara hallucination leaderboardtrustllmsalad-benchhalueval

embedding & retrieval evals3 products and companies

Products and companies: mteb, beir, bright.

multimodal benchmark suites4 products and companies

Multimodal benchmark suites are standardized evaluation frameworks that test AI systems across multiple modalities (vision, language, audio) to measure their capabilities. Researchers, AI labs, and companies use them to compare model performance and track progress.

Products and companies: mmmu, mathvista, videomme, charxiv.

llm leaderboards5 products and companies · its own article →

LLM benchmark leaderboards rank language models on standardized tasks to measure capabilities across reasoning, coding, knowledge, and instruction-following. Researchers, practitioners, and model developers use these leaderboards to track progress and compare systems.

Products and companies: hugging face open llm leaderboard, livebench, scale ai seal leaderboards, epoch ai, vellum.

evaluation harnesses6 products and companies

Evaluation harnesses provide infrastructure and frameworks for running benchmarks, tests, and leaderboard competitions against AI models. ML researchers, model developers, and benchmark operators use them to execute standardized evaluations and track performance metrics.

Products and companies: eleutherai lm evaluation harness, inspect ai, openai evals, lighteval, deepeval, promptfoo.

llm observability and eval platforms6 products and companies · its own article →

Platforms that run standardized tests against large language models and publish comparative rankings of model performance. ML researchers, model vendors, and procurement teams use these to track model capabilities over time.

Products and companies: langsmith, braintrust, arize phoenix, weights & biases weave, galileo, langfuse.

vertical domain leaderboards4 products and companies · its own article →

Vertical domain leaderboards rank and compare AI models on benchmarks specific to narrow domains like medicine, law, or finance. Researchers, practitioners, and vendors in those domains use them to track model performance on task-specific datasets.

Products and companies: vals ai, berkeley function calling leaderboard, copilot arena, aider leaderboards.

consumer model comparison tools4 products and companies

Tools that display ranked comparisons of consumer AI models across benchmarks and performance metrics. Researchers, developers, and enterprises use these to evaluate which models fit their use cases.

Products and companies: openrouter rankings, yupp, nano gpt, t3 chat.

See also

This article is made from the map of ai's data. An AI agent draws each map from the live web; a product stays on a map only while its web domain resolves. The texts are the agent's and can be wrong. Demo page.