Skip to content

fix(flocklease): reclaim expired lease when record still names caller - #1343

Open
jhrozek wants to merge 2 commits into
mainfrom
worktree-agent-a8b342280a6346555
Open

fix(flocklease): reclaim expired lease when record still names caller#1343
jhrozek wants to merge 2 commits into
mainfrom
worktree-agent-a8b342280a6346555

Conversation

@jhrozek

@jhrozek jhrozek commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Summary

flocklease.Renew treated any expired lease record as definitive loss
(port.ErrLeaseHeld), even when the record still named the calling process as
owner at its own fencing token. On a single-host deployment, a process
suspended past the TTL (e.g. a laptop going to sleep) lost its lease on the
next Renew after resume, even though nothing else could possibly have raced
it — a single host, and no competing process ever ran while this one was
suspended. renewLoop then treated that as immediate, ungraced loss and
cancelled the run.

The safe invariant isn't "the flock generation handle is still open" (Acquire
can take over an expired record regardless of whether the old holder's flock
is still technically held) — it's that the durable record, read under the same
stable transition lock Acquire/Release already use, still shows the same
owner at the same fencing token. On a single host that can only be true if
nobody else raced an Acquire/takeover in the interim, so Renew now reclaims
with a fresh expiry (token unchanged) instead of declaring loss. A record whose
owner or token changed during the gap (a genuine takeover) still returns
ErrLeaseHeld unconditionally.

Reviewed by a go-architect + kubernetes-operator-expert panel before
implementation (confirmed this does not generalize to k8slease/
grpcdriver — those backends have no equivalent local proof of exclusivity,
and a genuine TTL blowout losing the lease there is correct behavior), and by
a full four-axis panel review (Spec/Standards/Test-adequacy/Domain) after
implementation — no blockers found; the one important finding (an untested
held == nil hard-fail branch) is addressed in the second commit.

Development stage

  • Spike / Routine — acceptance-plan spine exempt; Spike evidence does not ship as-is

Contract linkage

  • Work classification: Routine — mechanical, reversible bug fix restoring the already-documented Renew/ErrLeaseHeld contract intent; introduces no new durable architecture decision, no public API/persistence/trust-boundary change.
  • Classification rationale: Single-file behavioral fix scoped to internal/adapter/flocklease + a narrowing doc-comment clarification on engine/port/lease.go (no exported signature changed, task api:check unaffected). No gRPC/protobuf, CLI/config, or persistence-format change.
  • Decision record: None — bug fix, not a new architectural decision.
  • Human waiver of spine: No
  • Acceptance plan: N/A (Routine)
  • Human decisions resolved and recorded: N/A
  • Plan / Interface PR: N/A
  • Approved commit baseline: N/A
  • Combined/exemption rationale: N/A (Routine, not Combined)

Interface conformance

  • port.SessionLease.Renew's doc comment is clarified, not widened: bare expiry with the record still naming the caller's owner+token is no longer, by itself, defined as loss — only a holder/token change is. This narrows (never widens) when an implementation may return ErrLeaseHeld; every other backend (k8slease, grpcdriver, memlease) is untouched and still treats bare expiry as loss.

Issue relationship

Fixes #1333

Type of change

  • Bug fix

Test plan

Baseline checks

  • Linting (scoped: golangci-lint run --config .golangci.yml ./internal/adapter/flocklease/... and ./engine/port/...)
  • Offline test suite (scoped: go test ./internal/adapter/flocklease/... -race -count=1 and ./engine/port/... -race -count=1)
  • Markdown changed: engine/CHANGELOG.md + docs/adr/0027-cloud-native.md touched; matlatl check . --strict run clean (372 documents, 3553 references, 0 broken)
  • Guarded engine API affected: not applicable — doc-comment-only change to engine/port/lease.go, no exported signature moved
  • Final implementation review: /panel-review-equivalent four-axis panel run manually (see PR discussion) — PANEL: ship_blockers=0 important=1 advisory=6 reviewer_failures=0; the one important finding is fixed in the second commit

Full-repo task lint/task test intentionally not run in this environment (disk space constrained during development); scoped checks above cover every touched package plus a full go vet ./.../go build ./... sanity pass on both modules. CI will run the full gates.

Changes

File Change
engine/port/lease.go Clarify Renew's doc comment: bare expiry with the record still naming the caller is not itself loss
internal/adapter/flocklease/flocklease.go Renew reclaims an expired-but-uncontested lease instead of returning ErrLeaseHeld
internal/adapter/flocklease/flocklease_test.go New/updated tests: reclaim-with-no-competitor, still-fails-after-genuine-takeover, held == nil hard-fail
engine/CHANGELOG.md Note the narrowed Renew/ErrLeaseHeld contract
docs/adr/0027-cloud-native.md Extend the flocklease resource-inventory bullet

User-facing change

A mecated/mecak8s deployment using the single-host --session-lease-dir backend no longer loses a session's write access every time the host process is suspended (e.g. a laptop going to sleep) past the lease TTL, as long as nothing else actually took over the session in the meantime.

Special notes for reviewers

Scope is deliberately narrow: k8slease and grpcdriver/sessionlease.go are untouched. A genuine TTL blowout on those backends should still lose the lease — there is no local proof of exclusivity there the way flock's stable transition lock provides. See #1333 and #1334 for the companion fix (a separate, orthogonal issue: once a loss is declared, for any reason, the resulting tombstone was permanent for non-StateRunning sessions).

jhrozek and others added 2 commits September 10, 2026 13:09
flocklease.Renew treated any expired lease record as definitive loss
(ErrLeaseHeld), even when the record still named the calling process as
owner at its own fencing token. On a single-host deployment, a process
suspended past the TTL (e.g. laptop sleep) would lose its lease on the
next Renew after resume even though nothing else could have raced it.

The safe invariant is not "the flock generation handle is still open"
(Acquire can take over an expired record regardless), but that the
durable record, read under the same stable transition lock
Acquire/Release use, still shows the same owner at the same fencing
token. On a single host that can only be true if nobody else raced an
Acquire/takeover in the interim, so Renew now reclaims with a fresh
expiry (token unchanged) instead of declaring loss. A record whose
owner or token changed during the gap (a genuine takeover) still
returns ErrLeaseHeld unconditionally.

Clarifies port.SessionLease.Renew's doc comment (narrowing only, no
contract change: implementations may still always treat expiry as
loss if they can't prove otherwise) and flocklease.Renew's own comment.
Rewrites the flocklease test that encoded the old behavior into two
tests covering the reclaim-with-no-competitor and
still-fails-after-genuine-takeover cases.

Scope is flocklease only; k8slease and grpcdriver/sessionlease.go are
untouched per the panel review in the issue.

fixes #1333

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Panel review on 56b9bce flagged that the held == nil hard-fail branch
in Renew -- the one that actually carries the cross-process safety
argument -- wasn't isolated by either new test from that commit. Add
TestRenewFailsWithoutLocallyTrackedGeneration: two independent
flocklease.New instances over the same dir; the first Acquires, then
Renew is called on the SECOND (unheld) instance with a copy of the
first's returned port.Lease. The durable record still names that exact
owner and token and has not expired, yet the second instance never
populated its own `held` map for that generation, so Renew must still
fail closed with ErrLeaseHeld.

Also records the narrowed Renew/ErrLeaseHeld port contract in
engine/CHANGELOG.md (doc-comment-only clarification, no exported
signature changed, no task api:update needed) and notes the new
uncontested-expiry-reclaim behavior in the flocklease adapter bullet of
docs/adr/0027-cloud-native.md's resource inventory.

Follow-up to #1333.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

flocklease.Renew declares definitive loss on process suspend (laptop sleep), even with no competitor

1 participant