Native histograms go beta and cut Prometheus time series by up to 90%
Kubernetes 1.37 enables Prometheus native histograms by default, replacing static buckets with dynamic exponential buckets. Components stay backward-compatible via dual exposition: enable scrape_native_histograms, then migrate your histogram_quantile queries.
Kubernetes 1.36. Native histograms land in alpha under KEP-5808. Kubernetes 1.37. They graduate to beta and turn on by default. Prometheus 3.0. The scraper version that reads them without re-plumbing your whole stack. Why it matters: API server and scheduler latency is finally measured at high resolution without exploding the number of time series — precisely the item that drives up Prometheus memory in large clusters.
The problem with classic histograms
Since the earliest days of Kubernetes observability, duration and latency metrics have relied on classic Prometheus histograms. A metric author defines a static list of cumulative bucket boundaries — the le labels — such as 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10. That familiar model creates three structural problems.
The bucket guessing game. If a workload’s latency profile shifts — into microsecond ranges or into a long tail beyond the largest bucket — the histogram loses visibility. Defining boundaries up front means claiming to know the distribution before observing it.
Cardinality and storage cost. Every boundary is exported as a separate time series (_bucket{le="…"}). A ten-bucket histogram multiplied by its labels increases the series count tenfold, raising Prometheus memory and inflating TSDB storage.
Quantile interpolation error. Computing percentiles via histogram_quantile() relies on linear interpolation between static boundaries. When the spans are coarse, the quantile estimate can be significantly wrong.
What native histograms are
Prometheus native histograms replace static buckets with dynamic exponential buckets. Instead of emitting one series per boundary, a native histogram is stored as a single time series holding a rich schema of positive and negative spans, zero thresholds and exponential scaling factors.
Three direct consequences follow. High resolution automatically: exponential buckets adjust to any value range, from nanoseconds to hours, with no preconfigured boundaries. Up to 90% fewer time series: by consolidating buckets into structured spans within one series, scraping and storage overhead collapses. Accurate quantiles: calculation carries mathematical error bounds — roughly 5% worst-case relative error under defaults — across the whole spectrum of observations.
In Kubernetes, support is implemented directly in the shared metrics subsystem, k8s.io/component-base/metrics. As a result, every major control-plane and node component inherits it automatically — kube-apiserver (apiserver_request_duration_seconds, authentication and authorization metrics, validation latencies), kube-scheduler (scheduler_plugin_execution_duration_seconds), kubelet, kube-controller-manager and kube-proxy.
Dual exposition, or how to break nothing
The central requirement of KEP-5808 was zero disruption for existing observability stacks. With the NativeHistograms feature gate enabled, components use dual exposition: classic buckets (h.Bucket) are still emitted alongside native spans, and the native spans (h.Schema, h.PositiveSpan) are included in the same Protobuf payload for collectors that understand them.
Put simply, your existing dashboards, alerting rules and Prometheus servers keep working unmodified, while newer collectors take advantage of the native spans. It is that coexistence that makes the migration incremental rather than a big bang.
The default tuning applied to histograms is worth knowing: BucketFactor: 1.1, guaranteeing each bucket is at most 10% wider than the previous one and bounding quantile relative error at roughly 5%, and MaxBucketNumber: 160, capping buckets per histogram to protect component memory even under extreme outlier distributions.
How to scrape and query
On the Prometheus side, configuration depends on version. With Prometheus 3.0 and later, you configure per job, with no global flag:
scrape_configs:
- job_name: 'kubernetes-apiservers'
scrape_native_histograms: true
always_scrape_classic_histograms: true # recommended during transitionThe always_scrape_classic_histograms: true flag is critical during transition: without it, Prometheus ingests only the native format and stops ingesting the classic _bucket, _count and _sum series, breaking your existing queries. With Prometheus 2.40 through 2.x, you enable the feature globally:
prometheus --enable-feature=native-histogramsOnce histograms are ingested, PromQL queries simplify — no more _bucket suffix:
# Classic histogram:
histogram_quantile(0.99, rate(apiserver_request_duration_seconds_bucket[5m]))
# Native histogram:
histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m]))The migration path, and the payoff at the end
Migration happens in four steps. First, enable scrape_native_histograms: true with always_scrape_classic_histograms: true to retain dual ingestion. Next, migrate queries: replace histogram_quantile(…_bucket…) calls with histogram_quantile(…), and the _count / _sum series with histogram_count(…) and histogram_sum(…). Then validate in staging that dashboards and SLO alerts graph correctly. Finally, once migration is complete, set always_scrape_classic_histograms: false: Prometheus stops ingesting the static series, and histogram series count drops by 90% — roughly ten times less storage.
Rollback is trivial, which is what makes adoption low-risk: setting scrape_native_histograms: false returns the collector to the classic format with no Kubernetes restart, and the feature gate can be switched off per component with --feature-gates=NativeHistograms=false (at the cost of a restart).
Verdict
Native histograms in beta are one of the cleanest observability wins in Kubernetes 1.37: they fix the cardinality problem classic histograms have carried for years, without breaking what already works.
If your Prometheus is nearing memory or storage limits on histogram metrics, adopt the full migration — the ~90% series reduction drops straight to infrastructure cost. If your dashboards lean heavily on histogram_quantile(…_bucket…), enable dual ingestion with always_scrape_classic_histograms: true and migrate queries at your team’s pace: this is a transition you can stretch out. If you are on Prometheus 2.x, the switch is all-or-nothing — test the impact in staging before enabling it globally. Either way, watch the eventual retirement of classic buckets: SIG Instrumentation is targeting general availability, and static-bucket deprecation will follow.