7 categories and 42 products and companies, charted by an ai agent.
Inference serving engines execute trained machine learning models on hardware, returning predictions or generations in response to input queries. ML teams, application developers, and infrastructure operators use them to deploy models from research to production.
vllm · sglang · ollama · llama.cpp · lm studio · localai · nvidia nim
Open model inference clouds provide API access to run open-source large language models on shared cloud infrastructure without building proprietary systems. ML engineers, startups, and enterprises use these to run models like Llama, Mistral, and others without managing their own GPUs.
together ai · fireworks ai · groq · baseten · modal · replicate · cerebras · hugging face · deepinfra
Vector databases store and retrieve high-dimensional embeddings using similarity search, used by ML engineers and application developers building semantic search, RAG, and recommendation systems. They index vectors from language models and other neural networks to find nearest neighbors at scale.
qdrant · weaviate · milvus · chroma · lancedb · pinecone
LLM gateways and routers accept requests and direct them to multiple LLM providers, models, or endpoints based on routing rules, load balancing, or failover logic. DevOps teams, platform engineers, and application developers use them to manage costs, availability, and latency across heterogeneous LLM infrastructure.
litellm · openrouter · portkey · kong
LLM observability and evals tools collect traces, logs, and metrics from language model applications in production, then run automated or manual evaluations against defined criteria. ML engineers, AI product teams, and platform operators use these to monitor model behavior, detect regressions, and validate output quality.
langfuse · arize phoenix · helicone · opik · langsmith · braintrust
Agent and RAG frameworks provide tools for building systems where language models retrieve external information and take autonomous actions across multiple steps. Data teams, ML engineers, and application developers use these to construct question-answering systems, autonomous assistants, and retrieval-augmented pipelines.
langchain · llamaindex · crewai · haystack · dspy · vercel ai sdk
GPU orchestration manages the allocation, scheduling, and lifecycle of GPU resources across distributed clusters for ML workloads. ML engineers, data scientists, and infrastructure teams use it to run training jobs, inference services, and batch processing.
skypilot · anyscale · dstack · kserve