feat(config): Implement config watcher and config reloads

[Dav1dde]

Implements auto reloading of the config, for now only auto reloads health. It's a self contained config section and is already well integrated with the config snapshots.

The health config is only an example this will get more interesting once we have real use-cases we want to support and start integrating arroyo as well as potentially options automator.

Closes: INGEST-1179

[linear-code]

INGEST-1179

[sentry]

Bug: Ignoring the result of watcher.unwatch() can lead to state inconsistency between the OS watcher and currently_watched, causing spurious config reloads when the orphaned file is modified.
_{Severity: MEDIUM}

Suggested Fix

Handle the Result from watcher.unwatch(). If the call fails, do not update the internal state self.currently_watched to remove the path. This ensures the internal state remains consistent with the OS-level watcher, allowing subsequent calls to update_watch to retry unwatching the file.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.

Location: relay-server/src/services/config_reload.rs#L43-L61

Potential issue: In the `update_watch` function, the result of
`watcher.unwatch(to_remove)` is ignored. The `unwatch` operation can fail due to
OS-level issues, such as the file being deleted or permissions changing. When this
happens, the OS-level file watcher may persist, but the internal state
`self.currently_watched` is updated unconditionally, removing the file from its
tracking. This inconsistency leads to a 'ghost' watcher. If the file is later modified,
the persistent OS watcher triggers an event, causing a spurious and potentially repeated
configuration reload, making the reload service unreliable.

_{Did we get this right? 👍 / 👎 to inform future reviews.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ffd5520. Configure here.}

[cursor]

Config file watches are not re-established

Medium Severity

update_watch only calls watch for paths not already in currently_watched, then copies source_files into that set even when watch fails. After an atomic replace, inotify drops the watch on the old inode, but the path set is unchanged so the new file is never watched. Later config updates are ignored, including Kubernetes ConfigMap swaps and editor/safe-save rewrites.

 

^{Reviewed by Cursor Bugbot for commit ffd5520. Configure here.}