11 categories and 62 products and companies, charted by an ai agent.
Open-source frameworks and tools for systematically evaluating large language models across performance, safety, and quality dimensions. Used by AI researchers, ML engineers, and organizations building or deploying LLMs.
deepeval · lm-evaluation-harness · ragas · opencompass · lighteval · giskard · inspect (uk aisi) · chainforge · eleutherai lm evaluation harness · inspect ai
Platforms that use large language models to evaluate, score, and compare outputs from other LLMs across quality, safety, and performance metrics. Used by AI developers, researchers, and enterprises building and deploying LLM applications.
confident ai · galileo · patronus ai · braintrust · autoevals · flow judge · langsmith · weights & biases weave · arize phoenix
ragas (rag mode) · trulens · langwatch · uptrain · quotient ai
arize ai · deepchecks · w&b weave · mlflow 3.0 · langfuse · honeyhive · phoenix (arize)
guardrails ai · promptfoo · garak · rebuff
agentbench · tau-bench · weave (agent traces) · langwatch (agent sim)
lmsys chatbot arena · helm (stanford) · big-bench hard · mmlu · alpacaeval · open llm leaderboard (hugging face)
LLM evaluation platforms assess and benchmark large language model performance across quality, safety, and cost metrics. Used by AI teams, enterprises, and model developers to validate production deployments and compare model capabilities.
scale ai evalhub · humanloop · vellum · comet opik · klu.ai
Benchmarks and leaderboards are standardized evaluation datasets and public ranking systems that measure LLM performance across tasks like reasoning, knowledge, and instruction-following. Researchers, model developers, and enterprises use them to compare LLM capabilities and track progress.
lmarena · artificial analysis · hugging face open llm leaderboard · scale ai seal · epoch ai · vals ai
LLM observability tools track and analyze the behavior, outputs, and performance of language models in production systems. Data teams, ML engineers, and product managers use these tools to monitor model quality, latency, cost, and safety in real applications.
helicone · datadog llm observability · honeycomb · traceloop
Red teaming and safety evals frameworks help organizations identify vulnerabilities, adversarial failures, and harmful outputs in large language models before deployment. ML engineers, safety teams, and enterprises use these tools to systematically test LLM behavior against adversarial prompts, jailbreaks, and misuse scenarios.
lakera · haize labs