Transformers now runs llama.cpp GGUF quants natively

September 22, 2026. Hugging Face publishes “Transformers now runs llama.cpp quants,” authored by Marc Sun, Arthur Zucker, and Lysandre. September 22, 2026. Transformers can load a GGUF through from_pretrained and serve it behind an OpenAI-compatible API. September 22, 2026. Support debuts on Apple Silicon, starting with the Qwen3.5 architecture. Why it matters: the two largest ecosystems of local inference — the Transformers library and the llama.cpp engine — are finally converging, and GGUF is becoming the de facto interchange format for quantized models.

What GGUF is, and why the format dominates

GGUF is the file format developed by the llama.cpp team. It packs model weights and metadata — including the tokenizer and, optionally, a chat template — into a single file. Its value lies in quantization: the same model can be shipped at several precision levels, trading a little quality for a much smaller memory footprint.

The format has become the backbone of local inference. llama.cpp powers Ollama, LM Studio, and Jan, and the ggml-org organization publishes quantized checkpoints directly on the Hub. Publishers such as Unsloth, the LM Studio Community, and bartowski maintain ready-to-use GGUF collections that have been downloaded millions of times.

The Q4_K_M variant illustrates the tradeoff: it mixes tensor precisions, keeping most weights at 4-bit while holding sensitive tensors at higher precision. The post gives the exact breakdown for Unsloth’s Qwen3.5-4B:

Hugging Face’s recommendation is explicit: start with Q4_K_M, then move up to Q5_K_M or Q6_K if memory allows. More aggressive quantization fits larger models, but the quality tradeoff depends on the model and the task — evaluate it on the work you actually need the model to do.

Loading a GGUF with Transformers

Until now, loading a quantized model from Python meant leaving Transformers — either through the llama-cpp-python bindings or by converting the weights. The new support makes it as simple as a regular load, with one GGUF-specific step: pass gguf_file to from_pretrained.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "unsloth/Qwen3.5-4B-GGUF"
filename = "Qwen3.5-4B-Q4_K_M.gguf"

tokenizer = AutoTokenizer.from_pretrained(model_id, gguf_file=filename)
model = AutoModelForCausalLM.from_pretrained(model_id, gguf_file=filename)

Everything after that is the standard Transformers API — apply_chat_template, generate, and so on. No extra configuration is needed: when the weights stay packed on Metal, Transformers automatically loads the compatible ggml/Metal kernels and uses ggml-org/ggml-attn as the attention implementation. If that kernel cannot be fetched, the model falls back to sdpa with a warning, and you can always force attn_implementation="sdpa". Without a compatible quantization kernel, the loader dequantizes the model and uses more memory.

The initial focus is Apple Silicon and the Qwen3.5 architecture — a deliberate concentration: rather than cover every platform halfway, the team reuses llama.cpp’s ggml kernels through the kernels library and trims generate overhead. The requirements are an Apple Silicon Mac, a PyTorch version compatible with the published ggml-quantization kernel builds (in practice the two latest releases), and the main branch of Transformers until the next release.

Serving a GGUF behind an OpenAI API

The same checkpoint serves through transformers serve, which exposes an OpenAI-compatible API with no extra code. The argument format is <model_id>:<filename>.gguf — before the colon, the Hub repository; after it, the specific file to load. That lets you pick one quantization out of a repository that holds several.

pip install -U "transformers[serving] @ git+https://github.com/huggingface/transformers.git" kernels
transformers serve "unsloth/Qwen3.5-4B-GGUF:Qwen3.5-4B-Q4_K_M.gguf"

For models whose chat template supports reasoning, --reasoning off disables it, --reasoning on enables it, and --reasoning auto — the default — follows the template. A client such as Jan or Pi then connects by adding a custom OpenAI-compatible provider with http://localhost:8000/v1 as the base URL. Transformers runs the model on the Mac while the client provides the conversation interface.

What the benchmarks say, and what remains

The reference for local-inference performance is still llama.cpp. The post compares the two across three GGUF checkpoints — a small dense model, a larger dense model, and a mixture-of-experts. The stated goal is not to beat llama.cpp but to get close enough that the Transformers API becomes an acceptable alternative, especially through the shared ggml kernels.

The current limitations are candid. The support still lives on main, not in a stable release. It targets Apple Silicon and Qwen3.5, and the quantization kernel choice depends on the PyTorch version. Architectures beyond Qwen3.5 will follow, as will broader platform coverage. On launch day, in other words, this is a leading-edge capability for Mac users, not a universal one. That parity is the whole point: the team is trading a small performance delta for the entire Transformers ecosystem, a trade most Python workflows will gladly accept.

What it changes for the local ecosystem

The move goes beyond API convenience. Three years ago, running a local model from Python forced a clear choice: Transformers ergonomics (tokenizers, chat templates, the Hub ecosystem) or llama.cpp speed (quantization, tuned kernels). You sacrificed one for the other. By reusing llama.cpp’s ggml kernels under the Transformers API, Hugging Face erases that tradeoff for the first time.

The effects cascade. Ollama, LM Studio, and Jan, which run on llama.cpp, see their format of choice become a first-class citizen of the Python stack as well. Quant publishers — Unsloth, bartowski, the LM Studio Community — now ship a single file that serves both worlds. And teams that prototype in Python then deploy behind a local server can keep the same checkpoint from notebook to production, through transformers serve.

That consolidation has immediate reach. Projects shipping GGUF models no longer need two sets of instructions — one for llama.cpp, one for transformers. A single Q4_K_M checkpoint now covers Ollama users, LM Studio users, and Python scripts alike. That is exactly the kind of convergence that, in practice, decides whether a format survives.

The Apple Silicon starting point is not neutral, either. It is the platform where local inference crossed from hobby to daily professional use — and where Metal demands dedicated kernels instead of the historical CUDA paths. By aligning with the ggml/Metal kernels, Transformers implicitly concedes that the near-term future of local inference plays out on the Mac first.

Verdict

This convergence settles an ecosystem fact: GGUF is no longer a niche llama.cpp format, but the interchange format for quantized local models. If you develop in Python on Apple Silicon and currently juggle llama-cpp-python with conversion scripts, switch to from_pretrained with the gguf_file argument as soon as the next Transformers release ships — you regain the Hugging Face ecosystem (tokenizers, chat templates, serve) without giving up ggml kernel speed. If you need maximum raw performance on architectures not yet covered, stay on native llama.cpp. Either way, evaluate quantization on your own task at Q4_K_M and move up: the right precision level is a measurement, not a dogma.

References

• Hugging Face Blog — Transformers now runs llama.cpp quants (September 22, 2026)
• Hugging Face Hub — GGUF documentation
• ggml-org on the Hub