Your data should never leave your machine — here’s how to run an LLM without the cloud in 2026

January 2026, Ollama crosses 300,000 GitHub stars and becomes the most-cited way to run an LLM on a workstation. March 2026, vLLM ships native speculative decoding and grammar-constrained structured output, cementing its position as the production inference server of record. June 2026, llama.cpp reaches 100,000 commits and supports over 40 model architectures — without a GPU in sight.

Running an LLM on your own hardware is no longer a weekend project for tinkerers. Three tools, three philosophies, and one simple reality: the right solution depends on your hardware, not your skill level. Here’s what each tool actually does, with the VRAM and throughput numbers you need to make the call.

Ollama — simplicity as a feature

Ollama was born from a simple frustration: running a language model should be as easy as docker run nginx. In 2026, that promise holds. A one-line install (curl -fsSL https://ollama.com/install.sh | sh), an ollama pull llama3.3:70b, and ollama run get you talking to a model.

Under the hood, Ollama runs on llama.cpp. It doesn’t reinvent inference — it packages it. It pulls community-maintained GGUF quantized models and exposes an OpenAI-compatible REST API (/v1/chat/completions) that any compatible client — Open WebUI, Continue.dev, Cursor — can plug into without configuration. The Modelfile system (ollama create) lets you version a system prompt, parameters, and a base model in a single reproducible file.

The limits are well-known and deliberate. Ollama loads one model at a time — no concurrent batching, no continuous batching. GPU parallelism is basic: layer-wise splitting across multiple GPUs, with no true tensor parallelism. Latency is fine for a single user (25–50 tokens/second on Llama 3.3 70B Q4_K_M with an RTX 4090 24 GB), but it collapses under concurrent requests.

Ollama’s sweet spot: a developer who wants a local coding assistant on their machine, or someone evaluating a model before deploying it elsewhere. The tool shines when the user count equals one.

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Download and run a model
ollama pull llama3.3:70b
ollama run llama3.3:70b

# Serve behind an OpenAI-compatible API
ollama serve
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"llama3.3:70b","messages":[{"role":"user","content":"Explain PagedAttention"}]}'

vLLM — inference for production

vLLM was born at the UC Berkeley Sky Computing Lab in 2023 with one idea that changed everything: the key-value cache in transformer models shouldn’t be allocated contiguously. PagedAttention — the core innovation — breaks the KV cache into non-contiguous blocks, the way an operating system pages virtual memory. The result: near-zero memory fragmentation, and batch sizes that can triple on the same GPU.

In 2026, vLLM (version 0.8.x) is the de facto standard for serving LLMs in production. Continuous batching dynamically merges incoming requests instead of waiting for a full batch. Speculative decoding uses a lightweight draft model to predict several tokens ahead, which a larger model validates in a single pass — latency gains of up to 3× on code tasks. Prefix caching automatically reuses the KV cache of shared prefixes (system prompts, few-shot examples) across successive requests.

The API is 100% OpenAI-compatible, including streaming responses, function calls, and structured output (grammar-constrained JSON via guided_json). Hardware support covers NVIDIA (CUDA), AMD (ROCm), Intel (XPU), and Apple Silicon (MPS, experimental).

# Install vLLM
pip install vllm

# Serve a model with tensor parallelism across 4 GPUs
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --enable-prefix-caching

The barrier to entry is higher than Ollama’s: a single H100 80 GB serves Llama 3.3 70B at 800–1200 tokens/second in batch, but you need at least an A100 40 GB or two RTX 4090s to load it comfortably. vLLM also requires Python, pip, and at least a passing understanding of tensor parallelism — this is an infrastructure engineer’s tool, not a solo developer’s.

vLLM’s use case: serving one or more models to a team, an application, or a public API, with throughput and latency requirements. It is the default choice for a professional inference stack.

llama.cpp — treating CPU like GPU

llama.cpp is the project that proved an LLM can run without a GPU. Born in March 2023 from a C++ rewrite of LLaMA inference by Georgi Gerganov, it has become, in three and a half years, the foundation on which Ollama, LM Studio, GPT4All, and dozens of other tools build their backends.

Its core innovation is the GGUF format — a binary container that bundles the model, its configuration, and its metadata into a single file, with integrated quantization. Quantization levels range from Q2_K (2-bit, degraded quality but 2–3 GB of VRAM) to Q8_0 (8-bit, near-lossless), with the sweet spot at Q4_K_M (4-bit, under 1% quality loss on most benchmarks). DeepSeek V4 Pro, at Q4_K_M, fits on two RTX 4090s instead of eight H100s.

llama.cpp is also the only one of the three that runs credibly on CPU alone. A Llama 3.3 8B Q4_K_M reaches 15–20 tokens/second on a Ryzen 9 7950X — fast enough for conversational chat. Apple’s M3 Ultra with its 800 GB/s unified memory hits throughput comparable to a mid-range GPU on 70B-class models, thanks to llama.cpp’s Metal backend. AVX-512 on recent Intel servers and the Vulkan backend further expand hardware coverage.

# Clone and build llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make -j

# Download a GGUF model from Hugging Face
wget https://huggingface.co/bartowski/Llama-3.3-70B-Instruct-GGUF/resolve/main/Llama-3.3-70B-Instruct-Q4_K_M.gguf

# Interactive inference, CPU-only
./llama-cli -m Llama-3.3-70B-Instruct-Q4_K_M.gguf \
  -n 256 -t 16 -p "Explain PagedAttention"

# HTTP server, OpenAI-compatible API
./llama-server -m Llama-3.3-70B-Instruct-Q4_K_M.gguf \
  --port 8080 --n-gpu-layers 40

llama.cpp’s limitation is architectural: no PagedAttention (GPU memory management is less efficient than vLLM’s), no native continuous batching (the server processes requests sequentially or in static batches), and far lower throughput than vLLM on GPU. But it is the only tool that covers every scenario where a GPU isn’t an option.

Comparison table

Verdict — which tool for which job

If you’re a developer who wants a local coding or chat assistant, go with Ollama. Install it in thirty seconds, pair it with Open WebUI or Continue.dev, and you have a credible ChatGPT replacement for non-sensitive tasks. An RTX 4090 24 GB runs Llama 3.3 70B Q4_K_M at ~30 tokens/second. You don’t need to know anything else.

If you’re building a product that serves LLMs to multiple users, start with vLLM. Continuous batching and prefix caching cut your cost per token by a factor of two to four compared to a naive setup, and the ecosystem is now mature enough that production-breaking bugs can be counted on one hand. Provision at least one A100 40 GB or two RTX 4090s, and scale into tensor parallelism as your traffic justifies it.

If you have no GPU, or your data must never touch a GPU bus, use llama.cpp directly — or through Ollama, which wraps it. A recent bare-metal server with 64 GB of RAM and AVX-512 runs Llama 3.3 8B Q4_K_M at 20 tokens/second without a graphics card. It won’t feel instant, but that’s the price of full hardware sovereignty.

The question is no longer “can I run an LLM on my own hardware?” It has become “which of these three tools matches what I’m protecting — my data, my latency, or my GPU budget?” The answer at 2 p.m. on a laptop won’t be the same as the one at 2 a.m. in a Kubernetes cluster.

References

• Ollama — GitHub Repository, accessed July 29, 2026.
• vLLM — GitHub Repository, accessed July 29, 2026.
• llama.cpp — GitHub Repository, accessed July 29, 2026.
• vLLM Documentation — PagedAttention, vLLM Project.
• vLLM Documentation — Speculative Decoding, vLLM Project.
• GGUF Format Specification, ggerganov/ggml.
• Hugging Face — GGUF Quantization Types, Hugging Face Hub Documentation.
• Ollama Blog — OpenAI Compatibility, Ollama, 2025.