Skip to content

Commit 9830b4e

Browse files
committed
feat(hooks): widen contract for post_tool_use / before_llm_call + extract loop_detector & max_iterations builtins
Documents that decision="block" / continue=false / exit code 2 from a post_tool_use or before_llm_call hook signals run termination — not just the previously-undocumented behaviour of pre_tool_use deny. The executor's aggregate logic already produced Allowed=false uniformly for these verdicts; the runtime now actually acts on them. Adds `hooks.DecisionBlockValue` so in-process builtins don't carry the "block" string literal. * `executePostToolHook` and `executeBeforeLLMCallHooks` now return (stop, message). `processToolCalls` aggregates and propagates the post-tool stop up to the run loop, synthesising error responses for any unprocessed tool calls in the batch (same shape as user cancellation) so the API doesn't reject orphan function calls. * New `emitHookDrivenShutdown` factors the standard Error / notification(level=error) / on_error fanout used by both the post_tool_use and before_llm_call shutdown paths. * `executeWithApproval` now returns a `toolApprovalOutcome` struct instead of a single canceled bool to thread the new stop signal through every approval path (yolo, allow, force-ask, default, user prompt). Replaces the inline `tool_loop_detector` previously in pkg/runtime. Stateful per-session via `builtins.State`, registered through `builtins.Register` and cleared on session_end. Auto-injected from `agent.MaxConsecutiveToolCalls()` (defaulting to 5 to preserve the historical always-on contract). **Behaviour change:** signatures are now per-call, not per-batch. Single- tool repetition (the dominant stuck-agent pattern) and parallel-identical batches still trip; alternating multi-tool batches like `[A,B] [A,B] [A,B]` no longer trip — that case should now be caught by max_iterations or manual threshold tuning. Polling tools (view_background_agent, view_background_job) remain invisible to the counter. **Additive**, not a replacement: the existing `agent.MaxIterations` flow (MaxIterationsReachedEvent + interactive resume dialog in TUI/CLI/ACP) is unchanged because those UIs depend on the special event. The new builtin gives users a hard-stop alternative they can configure purely in YAML, with no resume protocol and no special event: hooks: before_llm_call: - {type: builtin, command: max_iterations, args: ["50"]} * `pkg/hooks/contract_widening_test.go` pins decision=block and continue=false producing Allowed=false on post_tool_use and before_llm_call. * `pkg/hooks/builtins/loop_detector_test.go` covers threshold tripping, JSON-key-order normalisation, exempt-tool invisibility, per-session isolation, lenient arg parsing, and concurrent safety. * `pkg/hooks/builtins/max_iterations_test.go` covers limit tripping, per-session isolation, lenient arg parsing, and concurrent safety. * `hooks_wiring_test.go` updated to reflect that loop_detector is auto-injected on every agent. * `pkg/runtime/tool_loop_detector.go` and its test (logic moved to `pkg/hooks/builtins/loop_detector.go` with per-call semantics). Lint clean, all tests pass with -race. Assisted-By: docker-agent
1 parent 2e3457f commit 9830b4e

16 files changed

Lines changed: 1121 additions & 451 deletions

pkg/hooks/builtins/builtins.go

Lines changed: 76 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -3,35 +3,65 @@
33
//
44
// Available builtins:
55
//
6-
// - add_date (turn_start) — today's date
7-
// - add_environment_info (session_start) — cwd, git, OS, arch
8-
// - add_prompt_files (turn_start) — contents of prompt files
9-
// - add_git_status (turn_start) — `git status --short --branch`
10-
// - add_git_diff (turn_start) — `git diff --stat` (or full)
11-
// - add_directory_listing (session_start) — top-level entries of cwd
12-
// - add_user_info (session_start) — current OS user and host
13-
// - add_recent_commits (session_start) — `git log --oneline -n N`
6+
// - add_date (turn_start) — today's date
7+
// - add_environment_info (session_start) — cwd, git, OS, arch
8+
// - add_prompt_files (turn_start) — contents of prompt files
9+
// - add_git_status (turn_start) — `git status --short --branch`
10+
// - add_git_diff (turn_start) — `git diff --stat` (or full)
11+
// - add_directory_listing (session_start) — top-level entries of cwd
12+
// - add_user_info (session_start) — current OS user and host
13+
// - add_recent_commits (session_start) — `git log --oneline -n N`
14+
// - loop_detector (post_tool_use) — block on N consecutive
15+
// identical tool calls
16+
// - max_iterations (before_llm_call) — hard stop after N model calls
1417
//
15-
// They can be referenced explicitly from a hook YAML entry using
16-
// `{type: builtin, command: "<name>"}`. The runtime also auto-injects
17-
// add_date / add_environment_info / add_prompt_files when the matching
18-
// agent flags are set.
18+
// Reference any of them from a hook YAML entry as
19+
// `{type: builtin, command: "<name>"}`. The runtime additionally
20+
// auto-injects add_date / add_environment_info / add_prompt_files
21+
// from the matching agent flags, and loop_detector from
22+
// agent.MaxConsecutiveToolCalls.
1923
//
20-
// turn_start builtins recompute every turn (date, git state). session_start
21-
// builtins run once per session for context that's stable for its duration.
22-
// Each builtin lives in its own file along with its registered-name
23-
// constant; this file holds the shared registration plumbing.
24+
// turn_start builtins recompute every turn (date, git state).
25+
// session_start builtins run once per session for context that's
26+
// stable for its duration. loop_detector and max_iterations are
27+
// stateful: their per-session counters live on the [State] returned
28+
// by [Register]; the runtime clears them via [State.ClearSession]
29+
// from session_end.
2430
package builtins
2531

2632
import (
2733
"errors"
34+
"strconv"
2835

2936
"github.com/docker/docker-agent/pkg/hooks"
3037
)
3138

32-
// Register installs the stock builtin hooks on r.
33-
func Register(r *hooks.Registry) error {
34-
return errors.Join(
39+
// State holds the per-runtime state of the stateful builtins.
40+
// It is returned by [Register] so callers can clear per-session
41+
// entries on session_end. Stateless builtins don't appear here.
42+
type State struct {
43+
loopDetector *loopDetectorBuiltin
44+
maxIterations *maxIterationsBuiltin
45+
}
46+
47+
// ClearSession drops per-session state from every stateful builtin.
48+
// A nil receiver is a no-op.
49+
func (s *State) ClearSession(sessionID string) {
50+
if s == nil || sessionID == "" {
51+
return
52+
}
53+
s.loopDetector.clearSession(sessionID)
54+
s.maxIterations.clearSession(sessionID)
55+
}
56+
57+
// Register installs the stock builtin hooks on r and returns a [State]
58+
// handle the caller must use to clear per-session state on session_end.
59+
func Register(r *hooks.Registry) (*State, error) {
60+
state := &State{
61+
loopDetector: newLoopDetector(),
62+
maxIterations: newMaxIterations(),
63+
}
64+
if err := errors.Join(
3565
r.RegisterBuiltin(AddDate, addDate),
3666
r.RegisterBuiltin(AddEnvironmentInfo, addEnvironmentInfo),
3767
r.RegisterBuiltin(AddPromptFiles, addPromptFiles),
@@ -40,22 +70,33 @@ func Register(r *hooks.Registry) error {
4070
r.RegisterBuiltin(AddDirectoryListing, addDirectoryListing),
4171
r.RegisterBuiltin(AddUserInfo, addUserInfo),
4272
r.RegisterBuiltin(AddRecentCommits, addRecentCommits),
43-
)
73+
r.RegisterBuiltin(LoopDetector, state.loopDetector.hook),
74+
r.RegisterBuiltin(MaxIterations, state.maxIterations.hook),
75+
); err != nil {
76+
return nil, err
77+
}
78+
return state, nil
4479
}
4580

4681
// AgentDefaults captures the agent-level flags that map onto stock
47-
// builtin hook entries. Pass each AgentConfig.AddXxx flag as-is.
82+
// builtin hook entries.
83+
//
84+
// MaxConsecutiveToolCalls and ExemptLoopTools together control the
85+
// auto-injected loop_detector entry: a non-zero threshold injects a
86+
// post_tool_use hook with that threshold and the exempt tool names.
87+
// The runtime supplies the exempt list so this package stays
88+
// decoupled from any tool constants.
4889
type AgentDefaults struct {
49-
AddDate bool
50-
AddEnvironmentInfo bool
51-
AddPromptFiles []string
90+
AddDate bool
91+
AddEnvironmentInfo bool
92+
AddPromptFiles []string
93+
MaxConsecutiveToolCalls int
94+
ExemptLoopTools []string
5295
}
5396

5497
// ApplyAgentDefaults appends the stock builtin hook entries implied by
55-
// d to cfg, returning the (possibly mutated) config.
56-
//
57-
// A nil cfg is treated as empty; the returned value is non-nil iff at
58-
// least one hook (user-configured or auto-injected) is present.
98+
// d to cfg. A nil cfg is treated as empty. Returns nil iff no hook
99+
// (user-configured or auto-injected) is present.
59100
func ApplyAgentDefaults(cfg *hooks.Config, d AgentDefaults) *hooks.Config {
60101
if cfg == nil {
61102
cfg = &hooks.Config{}
@@ -69,6 +110,13 @@ func ApplyAgentDefaults(cfg *hooks.Config, d AgentDefaults) *hooks.Config {
69110
if d.AddEnvironmentInfo {
70111
cfg.SessionStart = append(cfg.SessionStart, builtinHook(AddEnvironmentInfo))
71112
}
113+
if d.MaxConsecutiveToolCalls > 0 {
114+
args := append([]string{strconv.Itoa(d.MaxConsecutiveToolCalls)}, d.ExemptLoopTools...)
115+
cfg.PostToolUse = append(cfg.PostToolUse, hooks.MatcherConfig{
116+
Matcher: "*",
117+
Hooks: []hooks.Hook{builtinHook(LoopDetector, args...)},
118+
})
119+
}
72120
if cfg.IsEmpty() {
73121
return nil
74122
}

pkg/hooks/builtins/builtins_test.go

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,8 @@ func TestRegisterInstallsAllBuiltins(t *testing.T) {
2121
t.Parallel()
2222

2323
r := hooks.NewRegistry()
24-
require.NoError(t, builtins.Register(r))
24+
_, err := builtins.Register(r)
25+
require.NoError(t, err)
2526

2627
for _, name := range []string{
2728
builtins.AddDate,
@@ -32,6 +33,8 @@ func TestRegisterInstallsAllBuiltins(t *testing.T) {
3233
builtins.AddDirectoryListing,
3334
builtins.AddUserInfo,
3435
builtins.AddRecentCommits,
36+
builtins.LoopDetector,
37+
builtins.MaxIterations,
3538
} {
3639
fn, ok := r.LookupBuiltin(name)
3740
assert.True(t, ok, "builtin %q must be registered", name)
@@ -172,7 +175,8 @@ func TestAddPromptFilesNoArgsIsNoop(t *testing.T) {
172175
func lookup(t *testing.T, name string) hooks.BuiltinFunc {
173176
t.Helper()
174177
r := hooks.NewRegistry()
175-
require.NoError(t, builtins.Register(r))
178+
_, err := builtins.Register(r)
179+
require.NoError(t, err)
176180
fn, ok := r.LookupBuiltin(name)
177181
require.True(t, ok, "builtin %q must be registered", name)
178182
require.NotNil(t, fn)

pkg/hooks/builtins/json.go

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
package builtins
2+
3+
import (
4+
"encoding/json"
5+
"maps"
6+
"slices"
7+
)
8+
9+
// sortKeys recursively sorts map keys so [json.Marshal] produces
10+
// deterministic output regardless of how the input was constructed.
11+
// Slices are walked in place; non-collection values are returned
12+
// unchanged.
13+
func sortKeys(v any) any {
14+
switch val := v.(type) {
15+
case map[string]any:
16+
sorted := make(map[string]any, len(val))
17+
for _, k := range slices.Sorted(maps.Keys(val)) {
18+
sorted[k] = sortKeys(val[k])
19+
}
20+
return sorted
21+
case []any:
22+
for i, item := range val {
23+
val[i] = sortKeys(item)
24+
}
25+
return val
26+
default:
27+
return v
28+
}
29+
}
30+
31+
// canonicalToolInput returns a stable signature for a tool's input map
32+
// suitable for equality comparison across calls. Marshalling is done
33+
// after a recursive sort so semantically identical maps with different
34+
// iteration orders produce the same bytes. An unmarshalable input or
35+
// an empty map produces an empty string — which the caller should
36+
// treat as a non-matching signature rather than a wildcard.
37+
func canonicalToolInput(in map[string]any) string {
38+
if len(in) == 0 {
39+
return ""
40+
}
41+
out, err := json.Marshal(sortKeys(in))
42+
if err != nil {
43+
return ""
44+
}
45+
return string(out)
46+
}
Lines changed: 153 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
package builtins
2+
3+
import (
4+
"context"
5+
"fmt"
6+
"log/slog"
7+
"strconv"
8+
"sync"
9+
10+
"github.com/docker/docker-agent/pkg/hooks"
11+
)
12+
13+
// LoopDetector is the registered name of the loop_detector builtin.
14+
const LoopDetector = "loop_detector"
15+
16+
// defaultLoopDetectorThreshold is the consecutive-identical-call count
17+
// at which the detector trips when no explicit threshold is configured.
18+
// Five matches the historical default of the inline tool-loop detector
19+
// previously baked into pkg/runtime.
20+
const defaultLoopDetectorThreshold = 5
21+
22+
// loopDetectorBuiltin is the post_tool_use builtin that terminates the
23+
// run when the model issues the same tool call (name + canonical
24+
// arguments) `threshold` times in a row.
25+
//
26+
// State is per-session, keyed by [hooks.Input.SessionID], so concurrent
27+
// runs on a shared runtime can't cross-contaminate each other's
28+
// counters. The state map is bounded by the number of *active* sessions
29+
// rather than all-time sessions \u2014 the runtime calls `clearSession` from
30+
// session_end so the entry is dropped when the run finishes.
31+
//
32+
// Semantics differ from the previous inline detector in one specific
33+
// way: signatures are tracked **per call**, not per batch. For the
34+
// common stuck-agent case (the model repeatedly emits a single tool
35+
// call with identical arguments) the trip point is unchanged. A model
36+
// stuck in an alternating multi-tool batch like `[A, B] [A, B] [A, B]`
37+
// is no longer flagged because each B resets A's counter; users who
38+
// hit that pattern should rely on `max_iterations` (interactive UX) or
39+
// the new max_iterations builtin (hard stop) instead.
40+
//
41+
// Polling tools listed in `args` after the threshold (e.g.
42+
// `view_background_agent`) are silently ignored: a polling call
43+
// neither increments nor resets the counter, so a looping model can't
44+
// evade detection by sneaking a single polling call between identical
45+
// stuck calls.
46+
type loopDetectorBuiltin struct {
47+
mu sync.Mutex
48+
states map[string]*loopDetectorState // keyed by SessionID
49+
}
50+
51+
type loopDetectorState struct {
52+
lastSignature string
53+
consecutive int
54+
}
55+
56+
func newLoopDetector() *loopDetectorBuiltin {
57+
return &loopDetectorBuiltin{states: map[string]*loopDetectorState{}}
58+
}
59+
60+
// hook is registered as the [hooks.BuiltinFunc] for
61+
// [hooks.EventPostToolUse]. Args layout: `[threshold, exempt1, exempt2, ...]`
62+
// where `threshold` is an optional positive integer (defaults to
63+
// [defaultLoopDetectorThreshold] when missing or invalid) and the
64+
// remaining strings are tool names to exempt from detection.
65+
func (d *loopDetectorBuiltin) hook(_ context.Context, in *hooks.Input, args []string) (*hooks.Output, error) {
66+
if in == nil || in.SessionID == "" || in.ToolName == "" {
67+
// Defensive: post_tool_use always carries SessionID and
68+
// ToolName today. Skipping unbounded, unkeyed events keeps
69+
// the state map from filling with anonymous entries.
70+
return nil, nil
71+
}
72+
73+
threshold, exempt := parseLoopDetectorArgs(args)
74+
75+
if _, ok := exempt[in.ToolName]; ok {
76+
// Polling-style tools never count toward the consecutive
77+
// total and never reset it: see the type-level comment for
78+
// why visibility-zero (rather than counter-reset) is the
79+
// right behaviour.
80+
return nil, nil
81+
}
82+
83+
sig := in.ToolName + "\x00" + canonicalToolInput(in.ToolInput)
84+
85+
d.mu.Lock()
86+
state, ok := d.states[in.SessionID]
87+
if !ok {
88+
state = &loopDetectorState{}
89+
d.states[in.SessionID] = state
90+
}
91+
if sig == state.lastSignature {
92+
state.consecutive++
93+
} else {
94+
state.lastSignature = sig
95+
state.consecutive = 1
96+
}
97+
tripped := state.consecutive >= threshold
98+
count := state.consecutive
99+
d.mu.Unlock()
100+
101+
if !tripped {
102+
return nil, nil
103+
}
104+
105+
slog.Warn("loop_detector tripped",
106+
"tool", in.ToolName, "consecutive", count,
107+
"threshold", threshold, "session_id", in.SessionID)
108+
109+
reason := fmt.Sprintf(
110+
"Agent terminated: detected %d consecutive identical calls to %s. "+
111+
"This indicates a degenerate loop where the model is not making progress.",
112+
count, in.ToolName)
113+
114+
// "block" is the post_tool_use deny verdict: aggregate() turns it
115+
// into Result.Allowed=false, which the runtime translates into
116+
// the standard Error / notification / on_error fan-out before
117+
// terminating the run.
118+
return &hooks.Output{
119+
Decision: hooks.DecisionBlockValue,
120+
Reason: reason,
121+
}, nil
122+
}
123+
124+
// clearSession drops a session's state entry, called from a
125+
// session_end hook so long-running runtimes don't accumulate
126+
// per-session entries indefinitely.
127+
func (d *loopDetectorBuiltin) clearSession(sessionID string) {
128+
d.mu.Lock()
129+
delete(d.states, sessionID)
130+
d.mu.Unlock()
131+
}
132+
133+
// parseLoopDetectorArgs splits builtin args into (threshold, exempt
134+
// tool name set). An invalid or non-positive threshold falls back to
135+
// [defaultLoopDetectorThreshold] silently \u2014 a misconfigured YAML
136+
// shouldn't disable the detector entirely.
137+
func parseLoopDetectorArgs(args []string) (int, map[string]struct{}) {
138+
threshold := defaultLoopDetectorThreshold
139+
exempt := map[string]struct{}{}
140+
141+
if len(args) == 0 {
142+
return threshold, exempt
143+
}
144+
if n, err := strconv.Atoi(args[0]); err == nil && n > 0 {
145+
threshold = n
146+
}
147+
for _, name := range args[1:] {
148+
if name != "" {
149+
exempt[name] = struct{}{}
150+
}
151+
}
152+
return threshold, exempt
153+
}

0 commit comments

Comments
 (0)