fix(server): clear lease-loss tombstones for awaiting/cancelled sessions

[jhrozek]

Summary

When Service.onLeaseLost declares a cross-process session-lease loss, it sets
a permanent lostOwnership[id] tombstone. Every normal caller of
acquireLease/reaffirmLease (StartRunContent, resumeFromAwaiting/
ApproveRun, RenameSession, DeleteSession, etc.) fails fast on that
tombstone forever, by design — the only two things that cleared it were
CloseSession (not something a normal client resuming a session calls) and
the StateRunning-only stale-session reconcile sweep (#1302). But
onLeaseLost itself drives the session out of StateRunning while
handling the loss (to awaiting or eventually cancelled), so the sweep
never rediscovers it and the tombstone stayed permanent — regardless of
whether the original loss was a genuine takeover or a false positive (see the
companion fix in #1333).

Adds Service.LostOwnershipCandidates (an in-memory read of this process's
own lease-loss tombstones — deliberately not a store-wide scan, to avoid an
unbounded fan-out of trial-Acquire calls against every ordinary
awaiting/cancelled session) and Service.ReconcileLeaseLossTombstone (a
bounded trial-Acquire+immediate-release against the real backend, mirroring
SessionStale's own refinement, that clears the tombstone — and any stale
invalid heldLeases bookkeeping — only on proof the lease is genuinely free,
never unconditionally). Both are wired into the existing composition-level
stale-session sweep ticker, so a stranded tombstone self-heals on the next
pass instead of needing a manual CloseSession or process restart.

Reviewed by a go-architect + kubernetes-operator-expert panel before
implementation, and by a full four-axis panel review after — no blockers
found; four important findings (a stale ADR line, two test-coverage gaps, and
a ~20-line duplicated trial-Acquire block versus SessionStale) are all
addressed in the second commit, which also extracts the shared leaseTrial
helper both call sites now use.

Development stage

• Spike / Routine — acceptance-plan spine exempt; Spike evidence does not ship as-is

Contract linkage

• Work classification: Routine — mechanical, reversible bug fix closing a self-heal gap in an existing mechanism (the stale-session sweep, fix(server): let the stale-session sweep re-acquire a lost lease #1302); introduces no new durable architecture decision, no public API/persistence/trust-boundary change.
• Classification rationale: New methods are internal Service-level additions consumed only by the existing composition-level sweep (no new goroutine, no new gRPC/HTTP surface — confirmed by the security review below); gated by the same staleReconcileAuthorized system-root authorization every sibling stale-reconcile method already uses.
• Decision record: None — bug fix, not a new architectural decision.
• Human waiver of spine: No
• Acceptance plan: N/A (Routine)
• Human decisions resolved and recorded: N/A
• Plan / Interface PR: N/A
• Approved commit baseline: N/A
• Combined/exemption rationale: N/A (Routine, not Combined)

Interface conformance

• Two new exported Service methods (LostOwnershipCandidates, ReconcileLeaseLossTombstone), both gated by the existing staleReconcileAuthorized system-root check — not reachable from any gRPC/HTTP-authenticated caller (verified: no new handler registers them; the only caller is the composition-level sweep). Registered in classification.go and the SDK-parity test alongside their siblings.

Issue relationship

Fixes #1334

Type of change

• Bug fix

Test plan

Baseline checks

• Linting (scoped: golangci-lint run --config .golangci.yml ./internal/adapter/server/... ./internal/app/...)
• Offline test suite (scoped: go test ./internal/adapter/server/... ./internal/app/... -race -count=1)
• Guarded engine API affected: not applicable — change is entirely in internal/, outside the engine module
• Final implementation review: /panel-review-equivalent four-axis panel run manually — PANEL: ship_blockers=0 important=4 advisory=4 reviewer_failures=0; all four important findings fixed in the second commit

Full-repo task lint/task test intentionally not run in this environment (disk space constrained during development); scoped checks above cover every touched package plus a full go vet ./... sanity pass on the root module. CI will run the full gates.

Changes

File	Change
internal/adapter/server/service.go	LostOwnershipCandidates, ReconcileLeaseLossTombstone, extracted shared leaseTrial helper
internal/adapter/server/classification.go	Register the two new methods under the existing stale-reconcile root
internal/adapter/server/sdk_typescript_release_test.go	SDK-parity table entries
internal/adapter/server/lease_test.go	New tests: recovers-awaiting, recovers-cancelled, refuses-when-genuinely-held-elsewhere, fail-safe-on-generic-error
internal/app/session_reconcile.go	Wire the new reconcile pass into the existing sweep ticker
internal/app/session_reconcile_test.go	New test proving the sweep wiring itself (not just the Service method) clears a tombstone
docs/adr/0027-cloud-native.md	Correct the now-stale "one narrow exception" tombstone-clearing text
internal/syscaller/syscaller.go	Extend the stale-reconcile root's authorized-scope rationale

User-facing change

A session whose cross-process lease was lost while parked awaiting a
permission approval, or after being cancelled, is no longer permanently stuck
read-only. It self-heals on the next stale-session sweep pass once the
underlying lease is verifiably free — no manual intervention or process
restart required.

Special notes for reviewers

The safety-critical property here is that the tombstone is never
cleared unconditionally — only after a real trial-Acquire proves the lease
free, with a fail-safe error path that leaves the tombstone in place. This is
covered by TestReconcileLeaseLossTombstoneRefusesWhenGenuinelyHeldElsewhere
and the new TestReconcileLeaseLossTombstoneFailSafeOnGenericError. A
secure-code-reviewer pass specifically traced the TOCTOU/split-brain question
and confirmed no double-holder scenario is reachable. See #1333 for the
companion fix (preventing a false-positive loss in the first place on the
single-host flock backend).

[JAORMX]

Second-pass review of eae5a18: the normal awaiting/cancelled recovery path is correct, but the new reconciliation path can overlap backend Acquire with the loss handler Release for the same session. This violates the explicit SessionLease caller-serialization contract and needs fixing before approval; details inline. Please also update the canonical session-continuity documentation for next-sweep automatic recovery and correct the teardown-only lostOwnership description in docs/design/IMPLEMENTATION-NOTES.md. Non-blocking follow-ups: make the pre/post-trial bookkeeping robust against close/reacquire/second-loss, directly test authorization on both new maintenance methods, and exercise a real stateful competing holder rather than only a scripted ErrLeaseHeld. Trial Release failure is a temporary TTL-bound delay, not evidence of concurrent writers. Existing head CI is green; the branch currently conflicts with main. I ran no local tests or linters.

[JAORMX]

Re-reviewed fedd04b. leaseLossMu fixes the originally reported onLeaseLost Release versus reconcile Acquire overlap, and the living implementation notes are updated. One same-ID backend-call path remains outside that coordination: CloseSession can clear the tombstone while reconciliation is still releasing its trial lease, allowing a new StartRun to Acquire concurrently with that Release. This is the same SessionLease serialization requirement, not a new ownership policy; concrete interleaving inline. Please also complete the previously requested canonical user-docs/features/session-continuity.md update explaining automatic next-sweep recovery once the backend is available; the implementation notes do not replace user documentation. Non-blocking: the new regression test needs cleanup that unblocks releaseProceed on assertion failures. Current head CI is green and the branch is mergeable; no local tests or linters were run.

[JAORMX]

Follow-up review of 5ed8d57: the production lock coordination and cleanup fix address the prior findings, but the new regression test does not exercise the claimed close/reopen-versus-trial-Release ordering, and the required user-facing cadence/backend-availability details are still absent. Please address these two blockers; CI is separate from this review.

[JAORMX]

Reviewed the exact head 4eaa054. The close/reconcile/reopen regression now starts the reopening admission while the trial Release is blocked and detects any real Acquire before release completion; the prior vacuous ordering is covered. The canonical session-continuity documentation now states the five-minute cadence and backend-free-confirmation condition. The remaining panel suggestions are non-blocking test determinism/editorial polish. No local tests or linters were run; CI is tracked separately.