fix(ci): run the liveness probe on self-hosted so it survives hosted starvation (REQ-343)

[avrabe]

Maintainer decision: self-hosted.

The probe ran on ubuntu-latest, so it could not report during GitHub-hosted starvation — one of the five modes it exists to detect. Observed live 2026-09-09T22:30Z: the probe itself sat queued ten minutes while twelve self-hosted runners were idle (online=12 busy=0).

Why the counterintuitive placement is right

A probe cannot report a stall of the class it runs on, so the choice is which blindness to accept — and the modes aren't equally silent:

mode	visibility
self-hosted outage	LOUD — #855 put CI Gate + all 11 needs on self-hosted, so it blocks every merge; found within one PR
hosted starvation	SILENT — only advisory jobs and the ubuntu-latest-only release workflows; invisible until a release is attempted

An alarm should watch the mode that doesn't announce itself. That rationale is written into the workflow beside the runs-on.

A needs-based fallback can't express "run on the other class if this one is starved" — a starved job stays queued rather than failing, so a dependent waits forever instead of falling through. Hence a move, not a pair.

A risk that would have made this worse than the bug

The probe is built entirely from gh calls. gh is preinstalled on GitHub-hosted images but not on this fleet — surveying every workflow found no self-hosted job in the repo using gh before now.

Moving without checking would have traded blind during hosted starvation for blind always. A silently dead alarm is worse than a wrong one: nothing distinguishes it from an alarm with nothing to report.

Two steps handle it, mirroring the zola guard already in ci.yml — install into RUNNER_TEMP if absent (no sudo, nothing persisted), then assert and fail loudly. Negative-controlled both ways:

with gh present:     gh version 2.97.0 …    exit=0
PATH=/usr/bin:/bin:  ::error::gh is not on PATH …   exit=1

jq was the opposite case — provably available, since yaml-lint runs diagnose_test.sh on self-hosted light today.

Unverified until it runs

Whether gh is already present on the Hetzner runners, and so whether the install path is exercised at all. The guard covers both cases, so the first scheduled run answers it — and a failed install fails visibly rather than reporting nothing.

gate	exit
cargo fmt --all --check / clippy -D warnings (1.97.0)	0 / 0
cargo test --workspace	0 — 2368 passed
cargo test -p rivet-cli --test cli_commands	0 — 213 passed
rivet validate / docs check / yamllint / diagnose_test.sh	0
bash -n over every embedded run block	0

Fixes: REQ-343

[github-actions]

📐 Rivet artifact delta

Change	Count
Added	0
Removed	0
Modified	1
Downstream impacted (depth ≤ 5)	0

Graph

graph LR
  REQ_343["REQ-343"]:::modified
  classDef added fill:#d4edda,stroke:#28a745,color:#155724
  classDef removed fill:#f8d7da,stroke:#dc3545,color:#721c24
  classDef modified fill:#fff3cd,stroke:#ffc107,color:#856404
  classDef overflow fill:#e2e3e5,stroke:#6c757d,color:#495057,stroke-dasharray: 3 3

Loading

Modified

ID	Changes
REQ-343

📎 Full HTML dashboard attached as workflow artifact rivet-delta-pr-929 — download from the workflow run.

_{Posted by rivet-delta workflow. The graph shows only changed artifacts; open the HTML dashboard (above) for full context.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!