We ran Locai Link and Ollama on the same NVIDIA RTX 4070 with identical Gemma 4 weights. Link returns first token 10× faster. Here’s what that means for real workloads. (or skip to the direct benchmark results here)
If you’ve been running local LLMs with Ollama, you know the pause before the first token. The model is loaded, the GPU is warm, and it still takes 300 ms or more. For a quick chat that’s tolerable. For document analysis, RAG pipelines, or anything agentic, that wait compounds fast.
Locai Link is a self-hosted local LLM runtime designed as a faster Ollama alternative for teams that need lower latency and more control over inference. We ran both runtimes on identical hardware with identical model weights and measured the metrics that matter. The full results (every metric, both scenarios, raw data and methodology) are on the benchmark page.
Here’s what we found.
The test
Same GPU, same model, same quantisation. Both runtimes on CUDA, reasoning off on both for an apples-to-apples comparison.
We ran two scenarios on an NVIDIA RTX 4070 (12 GiB VRAM) using Gemma 4 (E2B-it, Q4_K_M, 4.6B parameters) on Arch Linux: a short conversational prompt targeting a ~300-token reply, run 20 times; and a ~9,500-token document analysis prompt targeting a 500-token reply, run 10 times. Three warm-up requests per backend per scenario before any measurement was recorded. Zero errors across 60 measured requests.
At temperature=0, both backends returned token-identical replies. The speed comparison is over identical computation.
What the numbers show
The biggest gap is TTFT (time to first token). In the conversation scenario, Ollama’s median TTFT was 326 ms. Link’s was 31 ms. That’s not a marginal improvement; it’s the difference between a response that feels instant and one that noticeably lags.
The reason is prefill throughput. Prefill is how fast the model reads your input before it starts generating. Link does it at 936 tokens per second in conversation; Ollama manages 92. That 10× gap is what creates the TTFT difference, and it gets more pronounced on longer inputs.
In the document analysis scenario (a 9,500-token prompt), Link processes the document at 169,466 tok/s. Ollama runs at 24,728 tok/s. Link starts returning output in 56 ms. Ollama takes 385 ms. End-to-end across the full exchange, Link is 1.3× faster in both scenarios.
Decode throughput is a closer race. Link generates output at around 196 tok/s in conversation versus Ollama’s 166 tok/s, about 18% faster. Both runtimes are near the GPU memory-bandwidth ceiling on the 4070, so this gap is real but narrow.
The trade-off is memory. Link uses roughly 4.2 GiB of VRAM versus Ollama’s 2.0 GiB. Both fit comfortably on 12 GiB. On an 8 GiB card the difference starts to matter.
See the full results and per-scenario breakdown
Why prefill speed is the metric most people underestimate
Decode speed gets attention because it’s visible: it’s the tokens streaming in after the model starts talking. But for most real workloads, prefill speed and TTFT are at least as important.
RAG pipelines pack retrieved context into every request. A 4,000-token retrieval chunk on a 500-token prompt means the model is processing 4,500 tokens before the first output token arrives. At Ollama’s prefill rate, that’s hundreds of milliseconds per query, every query.
Agentic systems are worse. Tool-use workflows pass conversation histories forward on every turn. Ten sequential tool calls at 300 ms TTFT each means a 3-second agent loop. At 31 ms, it’s under a second.
Long chat sessions do the same thing more gradually. Twenty messages in, Ollama is prefilling thousands of tokens on every new exchange. Users feel it as sluggishness that gets worse the longer they stay.
When Ollama is still the right call
These were single-stream measurements: one request at a time. We haven’t tested concurrent load, and throughput under multiple simultaneous users would look different for both runtimes.
On hardware where VRAM is the binding constraint, Ollama’s smaller footprint is a real advantage. On 8 GiB cards or edge devices, 2 GiB matters.
These numbers are also Gemma 4 specific. Gemma 4 uses sliding-window attention and shared KV layers, which affect both runtimes’ cache strategies. Performance across different architectures or larger models will vary.
FAQ
Is Locai Link a drop-in replacement for Ollama? Link runs an OpenAI-compatible API on /v1/chat/completions. Any client pointing at Ollama’s OpenAI-compatible endpoint can switch to Link with a URL change.
Does Link produce the same outputs as Ollama? At temperature=0, yes: token-identical replies across every prompt in this benchmark. That’s how we know the speed comparison is over identical computation.
What is TTFT and why does it matter? Time to first token is the gap between sending a request and receiving the first character of the reply. It’s what determines whether a local AI application feels responsive or laggy.
Why does Link use more VRAM? Link pre-allocates a larger KV cache to avoid reallocation during inference. You trade ~2 GiB of headroom for 7-10× faster prompt processing. On most mid-range GPUs upward, that’s a good trade.
What hardware does Locai Link support? CUDA-enabled NVIDIA GPUs. Broader hardware support is on the roadmap. <!– INTERNAL LINK: “roadmap” to /roadmap or /products/link –>
What is Locai Link? Locai Link is a local LLM inference runtime built for organisations that need on-premises AI without sending data to external APIs. It’s part of the Locai sovereign AI platform. <!– INTERNAL LINK: “Locai sovereign AI platform” to /platform or homepage –>
Benchmark run June 2026. OS: Arch Linux. Hardware: NVIDIA RTX 4070. Model: Gemma 4 E2B-it Q4_K_M. Locai Link build b9222.
Full benchmark data and methodology | [Try Locai Link] | [Talk to us]