Skip to content

Add control-plane operator metrics and fix dashboard scraping - #63

Merged
albertompe merged 2 commits into
mainfrom
feature/custom-metrics
Jul 30, 2026
Merged

Add control-plane operator metrics and fix dashboard scraping#63
albertompe merged 2 commits into
mainfrom
feature/custom-metrics

Conversation

@albertompe

Copy link
Copy Markdown
Contributor

Closes #56

Add control-plane operator metrics, fix metrics scraping, and revamp the operator dashboard

Summary

Introduces custom control-plane metrics for the Redkey operator, fixes a ServiceMonitor misconfiguration that left the operator dashboard empty, and reorganizes the operator Grafana dashboard for clarity.

New metrics

Five custom metrics are registered on the controller-runtime Prometheus registry (served on the existing operator /metrics endpoint) and instrumented at their reconcile call sites:

Metric Type Labels Purpose
redkey_reconcile_stage_errors_total counter stage Pinpoints which reconcile stage failed
redkey_config_creations_total counter reason RedkeyConfig creations (initial vs generation_change)
redkey_cleanup_deleted_configs_total counter Configs removed by cleanup
redkey_robin_deployment_changes_total counter action Robin Deployment create/patch operations
redkey_time_to_ready_seconds histogram Time from active config creation to Ready (observed on each transition into Ready)

All labels use bounded enums to keep cardinality predictable (no per-cluster/name labels).

Scraping fix

The operator ServiceMonitor scraped the metrics endpoint with scheme/port: https, but the operator serves plain HTTP on :8080 (--metrics-bind-address=:8080, service port named http). Prometheus therefore generated no target, so every panel showed "No data". Switched the ServiceMonitor to port/scheme: http to match, and updated the stale comment referencing the 8443/HTTPS endpoint.

Dashboard changes

  • New Control-Plane Operations section (2 panels per row) grouping the existing operation panels plus the new metrics.
  • Overview moved back to the top; all panel gridPos coordinates normalized to prevent section bleed-through (panels from one section rendering under another).
  • Renamed "Superseded Configs Deleted" → "Configs Deleted by Cleanup" and clarified its description (it counts actual deletions, not configs merely in the Superseded phase).

Docs

Updated docs/observability.md with the new metrics, example queries, and the revised dashboard layout.

Testing

  • go build ./..., go vet ./..., and go test ./internal/controller/... pass.
  • Verified on a live cluster: operator target is up, and the new series (e.g. redkey_config_creations_total, redkey_phase) are ingested by Prometheus and rendered in Grafana.

@albertompe
albertompe requested a review from a team as a code owner July 30, 2026 19:35
@albertompe
albertompe merged commit 517fdca into main Jul 30, 2026
9 checks passed
@albertompe
albertompe deleted the feature/custom-metrics branch July 30, 2026 23:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Custom cluster metrics

1 participant