Locai Link vs Ollama
How does Locai Link (llama.cpp build b9789) compare to Ollama (ollama-cuda) when both run the same Gemma 4 (Q4_K_M) model on the same CUDA hardware? Identical sampling, token-identical outputs, single-stream load.
Time to first token
10.5×
faster on conversational prompts
Prefill throughput
10.2×
faster on conversational prompts
Long-context prefill
6.9×
faster on 9.5K-token documents
Decode throughput
Up to 1.18×
faster across scenarios
Test environment
Identical hardware, identical model file, identical sampling parameters across both backends.
Hardware & OS
Model
Sampling
Output equivalence verified
Token-identical replies at temperature=0 confirmed for all prompts across both backends.
Results by scenario
Two workloads were measured: a short conversational prompt and a long-context document analysis.
Short conversational prompt (~30 tokens), mid-length reply (~300 tokens).
Prompt
~30 tokens
Reply
~300 tokens
Runs
20 (+3 warmup)
Errors
0
Time to first token↓ lower = better
How long before the model starts replying.
- Locai Link
- Ollama
Prefill throughput↑ higher = better
Speed of ingesting prompt tokens.
- Locai Link
- Ollama
Decode throughput↑ higher = better
Speed of generating reply tokens.
- Locai Link
- Ollama
Peak memory↓ lower = better
Resident GPU and host memory at peak.
- Locai Link
- Ollama
End-to-end latency (median)↓ lower = better
Full request lifecycle: send → final token.
- Locai Link
- Ollama
How to read this
Locai Link wins decisively on time-to-first-token and prefill throughput — the phases that dominate perceived latency for interactive UIs and long-context document workflows. Decode throughput (token generation speed once the reply has started) is also faster, though the gap is narrower because both backends share the same underlying CUDA kernels for the generation loop.
Ollama uses meaningfully less peak VRAM and host RSS. That trade-off is by design: Ollama defers KV-cache and weight allocations more aggressively, while Locai Link pre-allocates to remove allocation cost from the hot path.
Caveats
- Single-stream measurements only. One request in flight at a time per backend. Concurrent load was not measured. Results are specific to Arch Linux and the ollama-cuda Arch package; build steps and library paths differ on Ubuntu and RHEL.
- These numbers describe a single in-flight request. Throughput under concurrent load is a separate test we'll publish next.