Hugging Face rewrites tokenizers in Rust with SIMD and encodes text up to 30× faster

On September 21, 2026, Hugging Face detailed version 1 of its tokenizers library: the same output as v0.23, but encoding 3 to 30 times faster single-threaded on an Apple M4 Max, thanks to SIMD bitstream splitting and a word cache. Teams serving LLMs now have a measured reason to look at the tokenizer — the new bottleneck as models get faster.