7 categories and 41 products and companies, charted by an ai agent.
Open-source serving engines run inference for large language models on user infrastructure. Machine learning engineers, researchers, and organizations deploy these to serve models with control over hardware and data residency.
vllm · sglang · tensorrt-llm · text generation inference · lmdeploy · nvidia dynamo
llama.cpp · ollama · lm studio · mlc llm · jan · llamafile · mlx · gpt4all
LLM inference API platforms provide hosted endpoints for running large language models without managing infrastructure; developers and enterprises call these APIs to generate text completions, embeddings, and other model outputs.
together ai · fireworks ai · baseten · deepinfra · replicate · modal · openrouter
Custom silicon designed and sold by chip makers to run LLM inference workloads. Cloud providers, AI labs, and enterprises deploy these processors in data centers to execute language model forward passes.
groq · cerebras · sambanova · etched · tenstorrent
Enterprise inference stacks are integrated software platforms that handle model deployment, serving, scaling, and monitoring for large language models in production. Organizations use them to run inference workloads at scale across distributed infrastructure.
nvidia nim · red hat ai inference server · bentoml · kserve · llm-d · ray serve
neural magic · pruna ai · unsloth · exllamav2
Inference optimization toolkits provide software libraries and frameworks that reduce latency, memory footprint, and compute cost of running large language models. ML engineers and inference platform operators use these to tune models for deployment.
deepspeed · onnx runtime · openvino · apache tvm · flashinfer