fix: delete coglet-managed output files after upload - #3096
Conversation
Coglet creates output files (IOBase writes, oversized JSON spills) in
/tmp/coglet/predictions/{id}/outputs/ but never deletes them. This can
cause stale-output bugs where models return outputs from previous
predictions if an overwrite fails (issue #1434).
Add a managed bool field to the FileOutput IPC protocol message so
the orchestrator knows which files are safe to delete (coglet-created)
vs user-authored Path outputs (returned by reference, must not be
deleted). The orchestrator deletes managed files immediately after
reading their bytes into memory. A backstop remove_dir_all in
remove_prediction cleans up the entire prediction directory to catch
files from aborted uploads or cancelled predictions.
Closes #1434
|
LGTM |
michaeldwan
left a comment
michaeldwan
left a comment
There was a problem hiding this comment.
Thanks for taking this on. Cleaning up Coglet-created spill files is worthwhile, and making ownership explicit in the protocol is a useful direction.
I’m requesting changes because the recursive cleanup currently trusts a client-controlled prediction ID, duplicate IDs can delete a live prediction’s state, and predictor-returned paths -- the files described in #1434 -- remain intentionally undeleted. I’ve left details inline.
Please also add behavioral coverage around cleanup. The snapshots prove that managed serializes, but they don’t prove that managed files are deleted, that returned paths follow the intended contract, or that one prediction can’t remove another prediction’s directory.
Addresses review feedback on PR #3096 by validating prediction IDs, rejecting duplicate active IDs, hardening remove_dir_all, and safely handling returned cog.Path files via the managed output directory.
|
LGTM |
|
LGTM |
|
Follow-up summary for Michael’s review:
Validation completed:
|
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
There was a problem hiding this comment.
I'm Bonk, and I've done a quick review of your PR.
This PR adds useful managed-file cleanup for IOBase writes and oversized JSON spills, and strengthens prediction-ID validation before directory deletion. However, two claims in the PR description and review responses don't match the actual code.
-
Stale-file bug (#1434) not fixed for Path outputs —
send_user_file_outputcopies external paths into the managed output directory but never deletes the original file. The PR description explicitly states "removes the source only after a successful copy," and this was the core concern raised by Michael in the prior review. Without deleting the source, a predictor that returns a fixed temp path and fails to overwrite it on a subsequent prediction will still upload stale data. -
ID reservation not held through cleanup —
remove_predictionremoves the DashMap entry before callingremove_dir_all, so a reused ID could be submitted and have its new files deleted by the old cleanup task. The PR description claims the reservation is held through deletion, but the code does the opposite.
Suggested fixes are inline.
|
Posted review comments on PR #3096: #3096 (review) |
|
LGTM |
|
@anish-sahoo Bonk workflow was cancelled. View workflow run · To retry, trigger Bonk again. |
Fixes #1434 by making returned output files an explicit handoff to Coglet. Files under the per-prediction output directory or system temporary directory are moved into managed storage. Files elsewhere are copied so bundled assets and shared caches remain usable. Coglet deletes the managed copy after parsing, base64 encoding, or a successful upload. Upload failures now fail the prediction and leave the file for prediction-directory cleanup.
Prediction cleanup now validates IDs, reserves duplicate active IDs before slot acquisition, detaches a completed prediction directory before releasing its ID, and removes directory symlinks without following them. A detach failure keeps the ID reserved so a later prediction cannot reuse stale files. Client-supplied IDs are limited to 128 ASCII letters, numbers, hyphens, underscores, and periods, excluding
.and...Tests cover managed deletion, failed uploads, Python
Pathhandoff, stale-path reuse, external-file preservation, relative symlinks, duplicate IDs, permit capacity, traversal attempts, and cleanup failures. The HTTP and Python documentation now matches these rules.Validation: 255 Coglet tests, 63 coglet-python Rust tests, and 44 end-to-end Python tests pass. Rust formatting and clippy are clean.
Closes #1434.