Repository navigation
OpenMetrics 2.0: option to keep _total and unit suffixes, so switching from OM1 doesn't rename series #2518
Description
Activity
For comparison, client_golang keeps the names users register
- this is exactly the goal of the redesign - keep names as passed by the user
- this is a major goal of the redesign
- this was the expressed wish of the prometheus maintainers when we discussed this about a year ago
For that reason I would argue that whenever options we add - if any - they should make migration easier for existing applications.
The migration story I planned for would be that everyone who switches to OM2 would also adopt the new convention that metric names are not modified.
Would that work?
Keeping application-provided names unchanged as the long-term OM2 behavior makes sense.
The migration concern raised here still seems relevant, especially with content negotiation enabled. The same endpoint can serve OM1 and OM2 at the same time, so an existing
Counter("events")could appear asevents_totalto one scraper andeventsto another. Updating applications, dashboards, alerts, and recording rules may not always be possible in one step.Would an opt-in migration flag be a reasonable compromise? OM2 could preserve registered names by default, while
io.prometheus.openmetrics2.suffixes=truetemporarily retains the existing exposition names. Users could enable it while introducing OM2, migrate metric registrations and downstream queries, and then disable it.The current PR defaults this option to
true, but I can change it tofalseand document it specifically as a migration aid if that better matches the intended design.I'm spelling it out in more detail to make it easier to discuss in the upcoming sig meeting.
@arnabnandy7 please double check I'm getting the use case correctly.I want to clarify the intended default: OM2 should preserve metric names exactly as registered, like the Go client. Suffixing should not be on by default.
The migration question is whether an opt-in compatibility setting, such as
io.prometheus.openmetrics2.suffixes=true, would help existing Java users move over gradually:Migration step Producer registers OM1 exposes OM2 with default ( suffixes=false)OM2 with opt-in compatibility ( suffixes=true)Before OM2 is enabled Counter("events")events_total— — Enable OM2; keep compatibility mode off Counter("events")events_totalevents— Enable OM2 compatibility mode Counter("events")events_total— events_totalUpdate producers gradually; keep compatibility mode on A mix of eventsandevents_totalevents_total— events_totalAll producers use canonical names; turn compatibility mode off Counter("events_total")events_totalevents_total— With the default, switching to OM2 changes the exposed name for producers still registering
events. With compatibility mode enabled, OM2 can retain the OM1 name while producer registrations are updated; consumers can keep queryingevents_total. Once registrations use canonical names, compatibility mode can be turned off without changing the exposed name.4. I'm spelling it out in more detail to make it easier to discuss in the upcoming sig meeting.
@arnabnandy7 please double check I'm getting the use case correctly.@zeitlinger yes, this captures the use case correctly.
One small clarification the mix of
eventsandevents_totalduring a gradual migration would normally be across application instances or deployment versions. Registering both in the same registry would cause a collision because they resolve to the same exposition name in compatibility mode.The same migration flow should also apply to unit suffixes. For example, compatibility mode would keep
Counter("req").unit(BYTES)exposed asreq_bytes_totalwhile producers gradually move to registering the complete name explicitly.With those clarifications,
suffixes=falseby default andsuffixes=trueas an opt-in migration aid matches the intended behavior.Reacted by Gregor Zeitlinger
With
io.prometheus.openmetrics2.enabled, the OM2 writer exposes names exactly as passed to the builder. It never appends_totalor the unit suffix:client_java/prometheus-metrics-exposition-textformats/src/main/java/io/prometheus/metrics/expositionformats/OpenMetrics2TextFormatWriter.java
Lines 174 to 179 in 9e9deb6
This is documented in
client_java/docs/content/exporters/openmetrics2.md
Lines 57 to 77 in 9e9deb6
Counter("req").unit(BYTES)isreq_bytes_totalin OM1 butreqin OM2.The problem is that switching a target from OM1 to OM2 renames every counter, and every metric with a unit, that was instrumented the way client_java has always recommended (
Counter.builder().name("events")). Prometheus then stores them as new series, so existing queries, dashboards and alerts silently stop matching.Note
I hit this with a demo that scrapes the same client_java app with OM1 and OM2 and diffs the stored series: the JVM metrics match (they already have the suffixes in their names), but
http_requests_total(OM1) becamehttp_requests(OM2), andhttp_request_size_bytes_totalbecamehttp_request_size. Demo: https://35-204-166-191.sslip.io/, code: bwplotka/prometheus#7.The OM2 spec relaxed
_totaland the unit suffix from MUST to SHOULD, mainly for OpenTelemetry compatibility (https://github.com/prometheus/docs/blob/605cf81fefc2e8e91f8ba89bb1555ae52a43a318/docs/guides/open_metrics_2_0_migration.md?plain=1#L140). Both are still recommended though (https://github.com/prometheus/docs/blob/605cf81fefc2e8e91f8ba89bb1555ae52a43a318/docs/specs/om/open_metrics_spec_2_0.md?plain=1#L220 and https://github.com/prometheus/docs/blob/605cf81fefc2e8e91f8ba89bb1555ae52a43a318/docs/specs/om/open_metrics_spec_2_0.md?plain=1#L664). Today there is no way to get them in OM2 other than renaming every metric in code;OpenMetrics2Propertieshas no option for it.Proposal:
io.prometheus.openmetrics2.suffixes(or a naming mode), that keeps the OM1 suffix behaviour in the OM2 writer: append_totalto counters and the unit suffix where missing, asexpositionBaseNamealready does for OM1.For comparison, client_golang keeps the names users register (which by convention include
_totaland the unit), so there OM1 and OM2 produce the same series.