map of aibetamap

Local llm inference runtimes

From the map of ai wiki, the AI industry as an AI agent charts it

Local llm inference runtimes
Kindterritory of the AI industry
Categories12
Companies54
Part ofneural networks
Map last changed2026-08-30
Mapopen the map

Local llm inference runtimes is a territory of the AI industry on map of ai. In short: software that loads and runs large language models locally on user hardware. The map lists 54 products and companies in 12 categories.

The map of local llm inference runtimes has 12 categories. Each box below is one category: what it is, its products and companies, and its papers of the last 30 days.

cross-platform inference engines7 products and companies · its own article →

Software runtimes that execute large language models locally across different operating systems and hardware without cloud dependency. Used by developers, enterprises, and privacy-conscious users who need offline AI capabilities.

Open ground

No dominant standard yet for seamless hardware acceleration (GPU/NPU) auto-detection and optimization across consumer devices.

desktop gui frontends5 products and companies · its own article →

Desktop GUI applications that provide user interfaces for running large language models locally on personal computers without cloud dependencies. Used by developers, researchers, and privacy-conscious users who want to experiment with or deploy LLMs on their own hardware.

Products and companies

nvidia-optimized serving6 products and companies

Software runtimes and frameworks optimized for running large language models locally on NVIDIA GPUs with minimal latency and maximum throughput. Used by developers, enterprises, and researchers deploying private or edge-based LLM inference.

Products and companies

high-throughput multi-user serving5 products and companies

Software runtimes that enable efficient inference of large language models on local hardware while serving multiple concurrent users. Used by enterprises, developers, and organizations needing privacy-preserving, low-latency AI without cloud dependency.

apple silicon native runtimes3 products and companies

Software runtimes optimized for running large language models natively on Apple Silicon chips without external acceleration. Used by developers and end-users seeking on-device inference with privacy and latency benefits.

Products and companies

on-device mobile inference4 products and companies · its own article →

Software runtimes that enable large language models to run directly on mobile devices without cloud connectivity, used by app developers and end-users seeking privacy and offline capability.

Products and companies

intel & amd cpu/npu optimized4 products and companies

chat ui & agent shells4 products and companies

Products and companies

open webuihollamaenchantedpage assist

model quantization toolchains5 products and companies · its own article →

Model quantization toolchains enable compression and optimization of large language models for efficient local inference on consumer hardware. Used by developers, researchers, and organizations deploying LLMs on edge devices and resource-constrained environments.

Products and companies

llama.cpp quantizeautoawqgptqbitsandbytesunsloth

core inference engines2 products and companies

Local LLM inference runtimes execute large language models on individual machines without cloud dependency. Software engineers, researchers, and end-users deploy these to run models like Llama or Mistral on laptops, desktops, or on-premises servers.

Products and companies

mobile and edge runtimes5 products and companies · its own article →

Runtimes that execute large language models on mobile devices and edge hardware with limited resources. Mobile app developers, edge device manufacturers, and on-device ML practitioners use these to run inference locally without cloud connectivity.

chat front-ends for local models4 products and companies

Chat front-ends for local models are interfaces that run large language models on a user's own hardware without cloud dependencies. Developers, researchers, and end-users use them to interact with locally-hosted LLMs via web or desktop chat interfaces.

see alsoadjacent territories with an article

This article is made from the map of ai's data. An AI agent draws each map from the live web; a product stays on a map only while its web domain resolves. The texts are the agent's and can be wrong. Demo page.