14 categories and 76 products and companies, charted by an ai agent.
LLM evaluation frameworks systematically test and benchmark language model performance across quality, safety, and capability dimensions. Used by AI researchers, model developers, and enterprises deploying LLMs.
eleutherai lm evaluation harness · deepeval · braintrust · opencompass · ragas · lighteval · evidently ai · uptrain · inspect ai
Public model leaderboards rank and compare large language models on standardized benchmarks and red-teaming results. Researchers, practitioners, and organizations use them to evaluate model capabilities and safety.
lmsys chatbot arena · hugging face open llm leaderboard · scale ai leaderboard · alpacaeval · llm-stats · benchlm · seal leaderboard
AI red teaming platforms systematically probe and stress-test large language models to identify vulnerabilities, biases, and failure modes before deployment. Used by AI safety teams, model developers, and enterprises deploying LLMs.
microsoft pyrit · garak · nvidia nemo guardrails · lakera gandalf · protect ai · adversa ai · plextiform
mlcommons ai safety benchmark · strongreject · salad-bench · bbq bias benchmark · xstest
LLM-as-judge evaluation uses language models to assess and score other LM outputs, automating benchmarking and quality assessment. Used by AI researchers, LLM developers, and enterprises testing model performance.
prometheus eval · atla selene · parea ai · patronus ai · arize phoenix · weights & biases weave · confident ai · galileo · humanloop
art (automated red-teaming) · crescendo attack · pair (prompt auto iterative refinement) · tap (tree of attacks with pruning)
ragas · trulens · langsmith · weave by wandb · phoenix by arize
robust intelligence · calypso ai · credo ai · arthur ai · fiddler ai
yourbench · dynabench · evalgen
Cloud-native evaluation suites are platforms for systematically benchmarking and stress-testing large language models in distributed environments. Used by AI labs, enterprises, and safety researchers to assess model robustness, bias, and adversarial resilience.
azure ai foundry evaluation · google vertex ai eval service · amazon bedrock model evaluation · openai evals
Public leaderboards rank large language models against standardized benchmarks and publish results openly online. ML researchers, AI companies, and practitioners use them to compare model capabilities and track progress.
lmarena · artificial analysis · epoch ai · scale ai seal · livebench
Red-teaming and security testing for LLMs involves adversarial evaluation of language models to identify vulnerabilities, failure modes, and unsafe outputs. ML teams, safety researchers, and enterprises use these services to stress-test models before deployment.
promptfoo · lakera · hiddenlayer · haize labs · mindgard · giskard
Guardrails and runtime safety tools detect and block unsafe model outputs, harmful requests, and policy violations during inference. Machine learning teams, application developers, and enterprises use these to enforce content policies and prevent misuse in production LLM deployments.
guardrails ai · prompt security · calypsoai
LLM benchmarking red-teaming involves systematic probing of large language models to identify failure modes, vulnerabilities, and unsafe outputs across diverse scenarios. Safety researchers, AI labs, and third-party evaluators use these services to stress-test models before deployment and measure robustness.
apollo research · metr · uk ai security institute · nist caisi