13 categories and 56 products and companies, charted by an ai agent.
langfuse · langsmith · arize phoenix · braintrust · helicone · maxim ai · comet opik · honeyhive · datadog llm observability · traceloop · opik · pydantic logfire
LLM evaluation frameworks systematically measure and benchmark language model outputs across quality, safety, and performance dimensions. ML engineers, researchers, and product teams use them to validate models before deployment and monitor production performance.
deepeval · ragas · trulens · evals (openai) · promptfoo · uptrain · continuous eval (relari) · inspect ai · lm evaluation harness
Platforms that use AI models to evaluate, score, and judge the quality of LLM outputs and behaviors. Used by AI teams, enterprises, and researchers to assess model performance, safety, and alignment.
confident ai · galileo · patronus ai · autoevals · weights & biases weave · humanloop
arize ai · whylabs · context ai · arthur ai
fiddler ai · evidently ai · monte carlo · nannyml
guardrails ai · nvidia nemo guardrails · lakera · rebuff
giskard · promptbench · haize labs · invariant labs
agentops · gentrace · weave (weights & biases) · phoenix (arize)
Automated evaluation frameworks that run quality gates and performance benchmarks for AI systems directly in CI/CD pipelines. Used by ML engineers, platform teams, and DevOps practitioners deploying LLM applications.
Synthetic data generation tools purpose-built for testing and evaluating AI system behaviors, outputs, and safety properties. Used by AI teams, safety researchers, and model developers who need realistic but controlled evaluation datasets.
AI observability evals are systems that monitor, test, and evaluate the behavior and outputs of AI models and agents in production and development. ML engineers, safety teams, and product managers use these tools to detect failures, drift, and misalignment before they reach users.
protect ai · hiddenlayer
Model benchmarking and leaderboards rank and compare AI models against standardized datasets and metrics. ML teams, researchers, and model builders use these to track model performance, compare architectures, and publish results.
lmarena · artificial analysis · scale ai seal · epoch ai · hugging face
Agent reliability monitoring tracks whether AI agents complete tasks correctly, hit failures, and behave as expected in production. ML teams, platform engineers, and AI product managers use it to measure agent performance across runs.
arize ax · coval