fix(verifier): bind downloaded images end to end before caching them - #1337
Merged
Merged
Conversation
kvinwang
force-pushed
the
fix/verifier-image-cache-binding
branch
2 times, most recently
from
September 24, 2026 02:36
5557fb3 to
44966f2
Compare
kvinwang
added this pull request to stack #1342
September 24, 2026 02:37
kvinwang
force-pushed
the
fix/verifier-image-cache-binding
branch
from
September 24, 2026 02:55
44966f2 to
d945e2f
Compare
kvinwang
force-pushed
the
fix/verifier-image-cache-binding
branch
from
September 24, 2026 03:03
d945e2f to
338588e
Compare
kvinwang
force-pushed
the
fix/verifier-image-cache-binding
branch
2 times, most recently
from
September 24, 2026 03:21
6b52aa2 to
70ea3f6
Compare
- Check sha256(sha256sum.txt) against os_image_hash before parsing the manifest or hashing any file. - Require the bios, kernel and initrd that metadata.json names to be manifest entries. image_dir.join() of an absolute or `..` path, or of an unlisted name, would otherwise measure bytes os_image_hash never committed to. - Bound the downloaded archive and its extracted size at 100 MiB each; measurement images carry no rootfs.
…runtime Concurrent requests for an uncached image each downloaded it, and every finisher after the first ran remove_dir_all on the installed image while the others could be measuring from it, then raced the rename. Take a per-image lock before the cache check and hold it until the image is installed. Extraction, the manifest check and the install run in one spawn_blocking task that owns the lock guard and the temporary directory, so a caller that hits download_timeout cannot leave a still-running install to collide with the next download.
kvinwang
force-pushed
the
fix/verifier-image-cache-binding
branch
from
September 24, 2026 09:00
70ea3f6 to
e300871
Compare
kvinwang
added a commit
that referenced
this pull request
Sep 25, 2026
The verifier now retries a transient image download failure (#1388), so a two-second download timeout expires before the download error is reported. Give the one-shot rows enough time to observe the retries, and assert that a refused connection is retried, a 404 is not, and a 503 recovers. Also assert the in-process manifest digest check on a tampered archive (#1251, #1337) and that the removed debug request field is ignored and the debug response fields are gone (#1332). Signed-off-by: Kevin Wang <wy721@qq.com>
kvinwang
added a commit
that referenced
this pull request
Sep 25, 2026
…bounds #1251 renamed the manifest confinement test the image-download case ran by name. Run its replacement plus the tests that pin the sha256sum grammar, the end-to-end image binding (#1337) and the truncated-download retry (#1388), and extend the measurement cache case with shape-only keying (#1334) and the bounded cache directory (#1369). Signed-off-by: Kevin Wang <wy721@qq.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1251 (merged), which moved the image manifest check in-process.
Problem
metadata.json'sbios/kernel/initrdgo throughimage_dir.join(), so an absolute or..path, or an unlisted name, was measured from bytesos_image_hashnever committed to.sha256(sha256sum.txt)was compared withos_image_hash, and neither the download nor the extraction had a size bound.remove_dir_allon the installed image while others could be measuring from it.Fix
sha256(sha256sum.txt)withos_image_hashfirst, then check the manifest, prune unlisted files, and require the three measured files to be manifest entries.spawn_blockingtask that owns the lock and the temp dir, so a caller hittingdownload_timeoutcannot leave an install racing the next download.Verification
cargo test -p dstack-verifier(37 passed), fmt and clippy clean. New tests:extracted_image_binds_the_files_metadata_measures,concurrent_requests_download_an_image_once(8 concurrent requests, 1 download).download_imagewas run against all 31 images served atdownload.dstack.org/os-images/mr_<hash>.tar.gz(v0.5.2–v0.5.10, prod/dev/nvidia): all accepted.Supersedes the download-coalescing part of #1277.