FR
live

Kubernetes 1.37 enables Prometheus native histograms by default and cuts metric cardinality by 90%

On September 11, 2026, Kubernetes 1.37 graduated Prometheus native histogram support to beta, enabled by default. Latency metrics now expose dynamic exponential buckets that can reduce time-series count by up to 90%, provided you follow the four-step migration.

A machinist’s micrometer clamped around a hair-thin copper wire on a dark workbench.

September 11, 2026. Richa Banker (Google) announced on the Kubernetes blog that Prometheus native histogram support is graduating to beta, enabled by default in Kubernetes 1.37. Introduced as alpha in 1.36 under KEP-5808, it replaces static buckets with dynamic exponential buckets. Why it matters: latency histograms are among the most expensive metrics in a cluster, and the promised cardinality reduction reaches 90%.

The problem: guessing buckets before seeing the distribution

Since the early days of Kubernetes observability, duration and latency metrics — apiserver_request_duration_seconds, scheduling durations — have relied on classic Prometheus histograms. The mechanism is well known: the metric author defines a static list of bucket boundaries via the le label, for instance 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10.

This approach creates three structural defects, spelled out in the post.

The bucket guessing game. If a workload’s latency profile shifts — into microsecond ranges, or a tail that exceeds the highest bucket — the histogram loses visibility. Picking boundaries up front requires knowing the distribution before observing it, which is impossible at first deployment by definition.

Cardinality and storage cost. Every boundary is exported as a separate time series (_bucket{le="…"}). A ten-bucket histogram therefore multiplies the series count by ten, inflating Prometheus memory and TSDB storage cost.

Quantile interpolation error. Percentile computation via histogram_quantile() relies on linear interpolation between static boundaries. When the spans are coarse, quantile estimates can carry significant error.

What native histograms change

Prometheus native histograms replace user-defined static buckets with dynamic exponential buckets. Instead of emitting one series per boundary, a native histogram is stored as a single time series carrying a rich schema: positive and negative spans, a zero threshold, and exponential scaling factors.

Three gains follow directly.

  • Automatic high resolution: exponential buckets adjust to any value range, from nanoseconds to hours, with no prior configuration;
  • Up to 90% fewer time series: by consolidating buckets into structured spans within one series, scraping and storage overhead collapse;
  • Accurate quantiles: computation carries mathematical error bounds, roughly 5% worst-case relative error under default settings.

In Kubernetes, support is implemented directly in the shared metrics subsystem k8s.io/component-base/metrics. All major components inherit it automatically: kube-apiserver, kube-scheduler, kubelet, kube-controller-manager, and kube-proxy.

Dual exposition, to break nothing

A core design requirement of KEP-5808 was zero disruption for existing observability stacks. When the NativeHistograms feature gate is active, Kubernetes components use dual exposition: classic buckets (h.Bucket) keep being emitted, while native spans (h.Schema, h.PositiveSpan) are included in the same Protobuf payload for collectors that understand them.

Two default settings deserve attention. BucketFactor is set to 1.1: each bucket is at most 10% wider than the previous one, guaranteeing a bounded relative error of about 5% whether an operation takes 1 millisecond or 10 seconds. MaxBucketNumber is capped at 160, following OpenTelemetry SDK recommendations for base-2 exponential aggregation, to protect component memory even under extreme outlier distributions.

How to scrape native histograms

The short answer: upgrade to Kubernetes 1.37, and it works. The cluster is already emitting dual-exposition metrics. The Prometheus side depends on your version.

Prometheus 3.0+ (recommended) uses per-job configuration in scrape_configs, since the global --enable-feature=native-histograms flag is deprecated as of Prometheus 3.9:

yaml
scrape_configs:
  - job_name: 'kubernetes-apiservers'
    scrape_native_histograms: true
    always_scrape_classic_histograms: true   # recommended during transition

Setting always_scrape_classic_histograms: true is mandatory during migration. Without it, Prometheus stops ingesting the classic _bucket, _count, and _sum series, and your histogram_quantile(…_bucket…) dashboards go dark overnight.

Prometheus 2.40 – 2.x enables native histograms globally via prometheus --enable-feature=native-histograms. This is an all-or-nothing setting across every scrape target, far less flexible.

On the exposition side, standard text scraping only transfers classic buckets. When scrape_native_histograms is active, Prometheus automatically negotiates the Protobuf format with Kubernetes endpoints. Once ingested, the query changes shape: histogram_quantile(0.99, rate(apiserver_request_duration_seconds[5m])) operates directly on the metric, with no _bucket suffix and no sum by (le) grouping.

The four-step migration

Richa Banker recommends a precise workflow to switch over without breaking alerts or dashboards.

  1. Enable both formats — in the Prometheus 3.x config, set scrape_native_histograms: true and always_scrape_classic_histograms: true;
  2. Migrate queries — replace classic quantile queries with native ones, and _count/_sum references with histogram_count(…) and histogram_sum(…);
  3. Verify in staging and production — confirm that dashboards and SLO alerts plot correctly;
  4. Unlock ~10x savings — once migration is complete, set always_scrape_classic_histograms: false to cut histogram series count by up to 90%.

Rollback is simple. On the collector side, scrape_native_histograms: false is enough, with no Kubernetes restart and no data loss. On the component side, --feature-gates=NativeHistograms=false disables the gate after a restart.

Verdict

Kubernetes 1.37 forces nobody: dual exposition makes native histograms opt-in on the collector side, and every step is reversible. That is exactly what makes the migration safe, provided you never skip the always_scrape_classic_histograms: true step.

If you run Prometheus 3.x, enable dual exposition today, migrate your dashboards and alerts quantile by quantile, then drop the classic buckets to reclaim up to 90% of series and roughly ten times less storage on your latency metrics.

If you are still on Prometheus 2.x, do not flip the all-or-nothing global flag: the risk of breaking unmigrated alerts outweighs the gain. Plan the upgrade to Prometheus 3.x instead, where the per-job cutover is clean and gradual.

References

The cyber brief, every Tuesday

The flaws that matter and the patches to apply, in a ten-minute read.

No spam. One-click unsubscribe.
read next

On the same topic

GitLab 19.3.2 closes an unauthenticated arbitrary file read and seventeen more flaws

GitLab shipped versions 19.3.2, 19.2.6 and 19.1.8 on September 10, 2026 to fix eighteen flaws, including an unauthenticated arbitrary file read through the commits API and an insecure deserialization scored CVSS 9.9. Every exposed self-managed instance must be updated without delay, and protected CI/CD variable secrets need a review.

Kubernetes 1.37 introduces five Node Lifecycle Conditions to signal drain and maintenance

On September 9, 2026, Kubernetes 1.37 reserved five well-known node conditions — DrainInProgress, Drained, MaintenancePlanned, MaintenanceInProgress, and GracefulNodeShutdownInProgress — giving teams a shared way to say why a node is unavailable. Start publishing them in your maintenance automation now, without waiting for core controllers to consume them.

← Back to the feed

Type at least two characters.

navigate open esc dismiss