Skip to content

cloud-init: terminate the machine even if the script hangs or dies - #2472

Merged
alexey-milovidov merged 1 commit into
mainfrom
cloud-init-shutdown-deadline
Oct 9, 2026
Merged

alexey-milovidov merged 1 commit into
mainfrom
cloud-init-shutdown-deadline

Conversation

@alexey-milovidov

Copy link
Copy Markdown
Member

The cloud-init templates (main, versions, hardware) don't use set -e, so failing commands already fall through to the final shutdown now. But the machine kept running if the script hung or was killed before reaching it:

  • only the benchmark itself is under timeout; git clone, the dataset downloads of the versions benchmark, mkfs/mdadm, and the uploads to the sink (curl without --max-time) are not;
  • timeout ... | tee kills only its direct child: a server the benchmark started in the background keeps the pipe open, and tee waits for it forever;
  • a killed script never gets to shutdown.

Changes, in all three templates:

  • at the very start, shutdown -P +<timeout/60 + 120>: a deadline enforced by systemd regardless of the script; the final shutdown now replaces it in the normal case;
  • trap 'shutdown now' EXIT for early exits;
  • curl --max-time 300 for the uploads (no --retry: a retried POST could insert the row twice).

The legacy manual hardware/cloud-init.sh gets a 10-hour deadline as well.

🤖 Generated with Claude Code

A shutdown deadline (timeout + 2h) scheduled with systemd at the start, an
EXIT trap, and --max-time on the uploads to the sink. Before, a hang outside
of 'timeout' (git clone, dataset downloads, uploads, or a background server
keeping the pipe of 'timeout ... | tee' open) left the machine running.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@alexey-milovidov
alexey-milovidov merged commit b15266f into main Oct 9, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant