Skip to content

Stop shell_command from swallowing exceptions in bin/docker-to-squash.py - #9297

Closed
ToniAdreal wants to merge 1 commit into
apache:masterfrom
ToniAdreal:fix-9289-shell-command
Closed

ToniAdreal wants to merge 1 commit into
apache:masterfrom
ToniAdreal:fix-9289-shell-command

Conversation

@ToniAdreal

Copy link
Copy Markdown

Fixes #9289.

shell_command in bin/docker-to-squash.py returns from its finally clause, which discards every exception raised in the try/except blocks. So raise_on_error=True never raises. Reproduced on Python 3.12.3:

  • shell_command(["false"], False, False, True) returns (b'', b'', 1) instead of raising.
  • shell_command(["badcmd"], False, False, True) returns (None, None, -1); even the Popen failure is swallowed.

The fix moves the return below the try/except/finally, keeping only the timer cleanup in finally, so exceptions propagate again.

Call-site audit for raise_on_error=True: the direct callers (skopeo copy, tar, setfattr, mknod, chown, rm, mksquashfs, the hdfs_* wrappers) either ignore the return value or pass the flag through from callers that do. check_dependencies wraps each probe in try/except and returns 1 on failure; with the swallowed exceptions it could never detect a missing dependency, and it works as written after this change (verified locally: it now reports the missing hadoop binary and returns 1). Success-path behavior is unchanged: with raise_on_error=False, a failing command still returns (out, err, rc).

test_shell_command encoded the bug as its contract (the failure case only asserted rc != 0); it now asserts that a command that cannot be started and a command exiting non-zero both raise when raise_on_error is set.

Testing note: the test suite cannot be imported on Python 3.10+ until the compatibility fix in the #9291 PR lands, so locally I verified this change two ways: on top of that PR's branch, where python3 -m unittest test_docker_to_squash.py passes as-is, and against this exact tree with only the collections.abc import shimmed in a scratch copy (the one-line change from that PR), where the updated test also passes and the repro cases above raise as expected.

This is one of the four docker-to-squash.py fixes #9284 left to separate PRs.

The return statement in the finally clause of shell_command discards every exception raised in the try/except blocks, so raise_on_error=True never raises: a failing command returns its return code and a command that cannot be started returns (None, None, -1). Move the return below the try/except/finally so exceptions propagate and only the timer cleanup stays in finally. Call sites that pass raise_on_error=True either ignore the result or already expect an exception (check_dependencies wraps each call in try/except and could never detect a missing dependency before this change). Update test_shell_command to assert that failures raise.

Signed-off-by: ToniAdreal <184365500+ToniAdreal@users.noreply.github.com>
@ToniAdreal

Copy link
Copy Markdown
Author

Closing this as a duplicate. @linzch3 already opened #9293 for #9289 two days ago, with the same finally-return fix and the check_dependencies angle covered. I only spotted it after opening this one, sorry for the noise.

@ToniAdreal ToniAdreal closed this Nov 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docker-to-squash.py: shell_command swallows exceptions, so raise_on_error=True never raises

1 participant