What is on-device AI inference and how do you ship it inside a production product?

KEY TAKEAWAYS

• Shipping on-device AI inference in a product means solving three problems your dev machine hides: hardware variance across user devices, cold starts before models are loaded, and fallback logic when a device can’t cope

• The serving layer (llama.cpp, Ollama, ExecuTorch) is largely solved. What’s missing is the orchestration layer above it: device lifecycle management, persistent model warm state, intelligent routing, and centralized model deployment

• Cloud API inference adds 300-800ms per request. On-device delivers near-zero latency. For interactive features like autocomplete, transcription, and inline AI, that difference is what users actually notice

• A private server running open-source models solves some of the same problems but still requires a network round-trip, keeps your infrastructure in the data path, and becomes your operational responsibility. On-device inference distributes compute across your user fleet

• Moving 60-80% of requests off cloud on conservative thresholds eliminates per-request API costs for those workloads. At scale, that changes unit economics significantly — OpenAI Whisper API at $0.006/min costs a 100-person team around $180/month in transcription alone

• Building the orchestration layer from scratch takes 3-6 months. Private cloud (Bedrock, Vertex) keeps data off your servers but doesn’t solve the network round-trip or offline problem. Purpose-built on-device infrastructure handles all of it without a custom build

• Locai handles the orchestration layer: device registration, fleet management, model deployment, cloud fallback, with a single baseURL change to your existing OpenAI client

Why on-device AI inference in production is harder than it looks

On-device AI inference solves real problems: near-zero latency, no cloud bills per request, data that never leaves the user’s machine. Getting a model running locally on your own hardware is straightforward. Shipping that same inference as a feature inside a product your users depend on is a different engineering problem entirely.

The serving layer isn’t the issue. llama.cpp, Ollama, and similar tools handle model serving reliably. What breaks in production is everything around it.

What actually breaks when you ship local LLM inference to real users

Three problems surface consistently once local inference hits real user hardware.

• The first is hardware variance. Your users don’t have your machine. They show up with a 3060 when you optimized for an M3, conflicting Python environments, and driver configurations you’ve never seen. Quantization reduces model size but each model behaves differently once quantized, and testing across that hardware matrix is a time sink that rarely makes it into project estimates.

• The second is cold starts. Benchmarks measure tokens per second once a model is already loaded. They don’t measure the wait before the first token appears. On a user’s device in a real product, that initialization delay often determines whether a feature feels instant or broken.

• The third is fallback. When a device can’t handle the inference load, what happens? Without explicit routing logic, the answer is usually a failed request or a degraded experience. Building reliable cloud fallback from scratch adds weeks to any local inference project.

How does on-device first AI infrastructure solve these problems?

The three problems above aren’t model problems. They’re infrastructure problems. The model serving layer is largely solved. What’s missing is the orchestration layer above it.

On-device first infrastructure adds four capabilities that don’t exist in the serving layer alone.

• The first is device lifecycle management. A purpose-built infrastructure layer handles device registration, capability detection, and health monitoring automatically. It knows which devices in your fleet can handle which models, tracks hardware telemetry in real time, and routes workloads accordingly. You stop debugging hardware variance one device at a time and start managing it as a fleet.

• The second is persistent model warm state. A runtime agent running continuously on the device keeps the model loaded between requests. Cold starts become a one-time cost at agent startup rather than a per-request cost. For interactive features this is the difference between a feature that feels native and one that feels slow.

• The third is intelligent routing. When a device can’t handle the inference load because of available RAM, thermal state, or current workload, requests route automatically to on-prem servers or cloud fallback. You define the threshold. The infrastructure handles the routing. Users don’t see the difference.

• The fourth is centralized model deployment. Instead of managing model files per device, a model registry lets you deploy, version, and update GGUF models across your entire fleet from one place. OTA updates, canary deployments, rollbacks: the same patterns your team already uses for application code, applied to models.

Together these four capabilities are what turn local inference from a promising prototype into something you can actually ship and maintain in production.

On-device inference versus private server inference: what’s the actual difference?

A private server running open-source models solves some of the same problems as on-device inference. A single RTX 4090 with Ollama can serve 5-10 concurrent users on 7-13B models with response times under 2 seconds to first token. For a small internal team, that’s viable.

The difference becomes meaningful at the product level, specifically when your users are outside your infrastructure.

• A private server still requires a network round-trip. Your data leaves the user’s device, travels to your server, gets processed, and comes back. The round-trip is shorter than a cloud API call but the latency floor is still determined by network conditions. For users on poor connections, or in offline scenarios, a private server fails the same way a cloud API does.

• The second difference is the data path. On a private server, the raw input travels over the network to your infrastructure. On-device inference keeps the raw input on the user’s machine entirely. Only structured results leave the device. For products handling sensitive user data this is an architectural distinction, not just a compliance checkbox.

• The third is operational model. A private server is your infrastructure to run: hardware procurement, GPU capacity planning, uptime, scaling. On-device inference distributes the compute burden across your user fleet. The hardware already exists and your users paid for it.

The practical framing: a private server is the right call when your users are internal, your data sensitivity is moderate, and you have engineering capacity to run infrastructure. On-device inference is the right call when your users are external, their data is sensitive, or you need the product to work offline. For most SaaS products shipping AI features to end users, on-device first with cloud fallback covers both cases.

When should you use on-device inference versus cloud APIs?

On-device inference fits best when your product needs low latency on repeated inference tasks, when user data should stay on the device, or when cloud API costs are becoming a meaningful line item as your user base grows.

Cloud APIs remain the right choice for complex reasoning tasks that require frontier model capability, for users on hardware that can’t run local models reliably, or for features where inference happens infrequently enough that latency and cost aren’t material concerns.

The practical answer for most products is a hybrid: on-device first with intelligent cloud fallback.

Getting started with on-device AI infrastructure

There are three realistic paths to shipping on-device AI inference in a production product today.

• The first is building it yourself. That means standing up your own serving layer (vLLM, TGI, or llama.cpp directly), writing your own device registration and fleet management, building fallback logic, and maintaining it ongoing. It’s fully controllable but the realistic timeline is 3-6 months of engineering time before you have something production-ready. Most teams underestimate the hardware variance problem until they’re debugging it in production.

• The second is private cloud. AWS Bedrock, Google Vertex AI, and Azure OpenAI give you managed inference infrastructure without building it yourself. The trade-off: your data still leaves the device and travels through a third-party’s infrastructure. For products where data residency or privacy is a requirement, this doesn’t solve the problem.

• The third is purpose-built on-device AI infrastructure. Tools like Locai handle the orchestration layer: device registration, fleet management, model deployment, fallback routing, so you don’t build it from scratch. The integration is OpenAI-compatible, meaning your existing code needs minimal changes.

How Locai approaches on-device AI infrastructure

Locai is a device-first AI infrastructure company backed by Google for Startups, NVIDIA Inception, and Fuel Ventures. It runs inference locally on user devices by default and routes to cloud or on-prem only when a device can’t handle the load. You set the threshold.

The integration into an existing OpenAI-compatible application is a single line:

javascript

const openai = new OpenAI({
  baseURL: "http://localhost:8100"
})

Locai:Link is a lightweight runtime agent that runs continuously on the device, keeping models warm between requests. Locai:Control handles device registration, model deployment, and workload routing across the fleet. Models run as GGUF via llama.cpp with no per-device setup. The communication layer runs on Zenoh, a pub/sub protocol built for distributed low-latency edge workloads.

In production, most products see 60-80% of requests handled locally on conservative thresholds. Local inference delivers near-zero response latency. Cloud round-trips add 300-800ms per request. Across interactive features, that difference is what users actually notice.

A 2-minute walkthrough of the full setup is here:

Docs and quickstart: locai.co.uk/docs

Frequently asked questions about on-device AI inference

What is on-device AI inference? On-device AI inference means running an AI model directly on the user’s own hardware rather than sending requests to a remote cloud server. The model processes inputs locally and returns outputs without any network round-trip. This eliminates cloud latency, reduces per-request costs, and keeps user data on the device by default.

How is on-device inference different from self-hosted or on-prem AI? Self-hosted and on-prem AI runs models on servers your organization controls, but those servers are still centralized infrastructure. On-device inference runs the model on each end user’s own machine: a laptop, workstation, or local device. There’s no shared server. Each device is an independent inference node. Raw inputs never leave the user’s machine; only structured results are transmitted.

What is the main challenge of shipping on-device AI in a production product? The main challenge isn’t model serving: tools like llama.cpp handle that well. The hard problems are hardware variance across user devices, cold start latency before models are loaded, and building reliable cloud fallback logic when a device can’t handle the workload. These require an orchestration layer that most teams don’t have time to build from scratch.

What four things does on-device AI infrastructure add above the serving layer? Device lifecycle management (registration, capability detection, health monitoring), persistent model warm state (runtime agent keeps models loaded between requests), intelligent routing (automatic fallback to cloud or on-prem when a device can’t cope), and centralized model deployment (deploy, version, and update models across your fleet from one place). Together these turn local inference from a prototype into something you can ship and maintain.

What is the difference between on-device inference and a private server? A private server still requires a network round-trip, puts your raw user data in transit over the network, and is your infrastructure to operate. On-device inference keeps raw inputs on the user’s machine, eliminates the network hop entirely, and distributes compute across your user fleet rather than centralizing it on hardware you run. For external users, sensitive data, or offline requirements, on-device inference is the stronger architectural choice.

What is edge inference? Edge inference is AI inference that runs at the edge of a network: on end-user devices or local servers rather than in a centralized cloud data center. On-device inference is a specific type of edge inference where the compute happens on the user’s own hardware. It’s used in applications where latency, privacy, or offline capability are requirements.

What latency does on-device inference deliver compared to cloud APIs? On-device inference delivers near-zero response latency because there’s no network round-trip. Cloud API inference typically adds 300-800ms per request depending on the provider and model. For interactive features like autocomplete, real-time suggestions, and inline AI, the difference is perceptible to users and directly affects feature adoption. In production, products using on-device first infrastructure typically see 60-80% of requests handled locally on conservative thresholds.

What does on-device inference cost compared to cloud APIs? Requests handled locally don’t touch paid APIs, so the per-request cost is effectively zero once hardware is in place. Cloud API costs vary by provider and model: OpenAI Whisper API runs at $0.006/minute of audio, GPT-4o-mini at $0.15-0.60 per million tokens. For a 100-person team running 5 hours of meetings per week, cloud transcription alone costs around $180/month. Shifting 85-95% of that workload to on-device inference reduces that cost proportionally.