NeoMME Fuses Text and Images in a Single Bidirectional Transformer
Hcompany ships NeoMME, a 260M–800M multilingual multimodal encoder that processes text and images in one Transformer with no separate vision tower. For visual document retrieval, its Retriever variant reaches the ViDoRe v3 Pareto frontier with a 255× smaller index.
September 3, 2026. Hcompany publishes NeoMME, a family of multilingual multimodal encoders in 260M and 800M parameter sizes, released under the Apache 2.0 license. The checkpoints — including a Retriever variant tuned for visual document retrieval — are available in Hugging Face Transformers. Why it matters: NeoMME drops the two-tower architecture that has dominated visual RAG for years, and it does so with an index 255× smaller than conventional late-interaction encoders.
One Transformer for text and images
The architectural novelty fits in a single sentence: NeoMME uses neither a separate pretrained vision tower nor a causal decoder. Where a classic vision-language model extracts visual features with a SigLIP2-style encoder, projects them into a language model’s input space, and passes them through a decoder, NeoMME processes everything in one bidirectional Transformer.
Concretely, text enters as factorized token embeddings, while images are split into non-overlapping 32×32 patches projected by a small MLP. Both streams share the same computational path. That unity simplifies pretraining, fine-tuning, parallelism, and serving across both modalities.
The context is long (16,384 tokens, enough for two 4K UHD images), with sliding-window attention on most layers and global attention every six layers. The modern stack — grouped-query attention, query-key normalization, gated attention, 2D RoPE, squared-ReLU MLPs — carries over the lessons of the ModernBERT and ModernVBERT lineage.
Going native matters beyond elegance. A separate vision tower is usually frozen or only lightly tuned, so its features are whatever a generic image encoder happens to produce, and a projector then has to squeeze them into the language space. By training the whole stack end to end, NeoMME lets the image patches be shaped by exactly the objective that matters — reconstructing grounded text — instead of inheriting a vision encoder pretrained for a different task.
Learning from images through masked text
NeoMME is trained from scratch as a masked discrete-diffusion denoiser. For each text-only example, a corruption rate is sampled uniformly between 0 and 1, and each eligible token is masked at that rate. For multimodal examples, the rate ranges between 0.3 and 1: the image patches stay visible while the model reconstructs the masked text.
That is where the trick lies. With light masking, the model can guess “cat” in “The [MASK] sat on the mat” without looking at the image. With heavy masking, it is forced to ground its prediction in the visual patches, learning image-grounded descriptions with little to no residual textual signal. A 131k-vocabulary BPE tokenizer, trained on multilingual text, code, mathematics, and machine-produced image transcripts, completes the picture.
NeoMME-Retriever and visual RAG
The most useful variant is NeoMME-Retriever, fine-tuned for visual document retrieval using ColPali’s page-image approach. In one forward pass, it produces both dense and late-interaction embeddings — the two families that dominate the ViDoRe benchmark.
The numbers are striking. At a matched 2048×2048 input size on an NVIDIA L40S, the 260M model encodes about 51 pages per second, roughly twice the throughput of ColModernVBERT. Hierarchical token pooling and asymmetric quantization shrink late-interaction index storage from 1.5 MB to 6 kB per page — a 255× reduction — while retaining more than 95% of baseline nDCG@10.
Both sizes sit on the ViDoRe v3 Pareto frontier of nDCG@10 versus model size. In other words, for a given memory budget, NeoMME offers the best published precision-to-footprint tradeoff on this task.
Put together, the numbers tell a cost story: a 260M encoder that runs twice as fast and stores 255× less per page changes the economics of visual RAG from “requires a GPU cluster” to “runs on a single box.” That is the kind of shift that moves a technique out of the lab and into production.
Dense or late-interaction: which mode to choose
NeoMME-Retriever delivers both embedding families in one forward pass, but they serve different cases. A dense embedding compresses a page into a single vector: it is the cheapest mode to index and serve, ideal for large-scale semantic search. Late-interaction keeps one vector per token and compares page against query token by token, which improves precision on dense documents — tables, figures, complex layouts — at the cost of a heavier index.
That is where asymmetric quantization changes the game: it compresses the late-interaction index to the point of making it practical for whole corpora, not just demos. The advertised 6 kB per page brings late-interaction storage into the range you can host without dedicated infrastructure.
On the deployment side, the model loads like any Hugging Face encoder, fine-tunes with Sentence Transformers, and its 16,384-token context absorbs two 4K pages at once. For production visual RAG, the 260M is the entry point; the 800M raises precision when the corpus demands it.
Why it matters against VLMs
Hcompany’s reasoning is blunt: retrieval, classification, and token labeling do not generate text autoregressively, so they do not need a causal decoder or the parameter and compute overhead of a VLM. Adapting a generative model to those tasks means paying for an entire architecture to use only its encoding step.
NeoMME inverts the ratio: it keeps multimodal expressiveness where it counts — joint text-image understanding — without the weight of a decoder. For visual RAG at scale, where every page of a corpus must be encoded and indexed, that frugality translates directly into inference cost and disk footprint.
The movement extends beyond NeoMME. After the ModernBERT lineage showed that a modern bidirectional encoder could rival generative models on understanding tasks, NeoMME extends the logic to multimodal. Expect the same shift to repeat: specialized encoders will reclaim tasks that generative VLMs held by default, not by superiority.
An open model, ready to use
All checkpoints ship under Apache 2.0, in Hugging Face Transformers, with a technical report and a visual RAG demo. That is a sharp contrast with multimodal encoders locked inside proprietary models: here the weights, the inference code, and the fine-tuning recipe are all free. For a team that wants to audit or adapt the encoder to its own documents, it is the difference between a black box and a component you control end to end.
Verdict
If you build visual RAG over scanned documents or dense PDFs, try NeoMME-Retriever 260M as a replacement for a VLM-based encoder: you gain encoding throughput and shrink your index by two orders of magnitude for less than 5% precision loss.
If you serve multimodal embeddings at scale, the 800M variant is the right dial when precision wins and the 260M falls short — but start with the small model, which already covers most document-retrieval cases.
If your workload needs text generation or multimodal dialogue, NeoMME is not built for that: stay on a generative VLM and reserve NeoMME for encoding and retrieval.