feat: support local package artifacts in requirements - #3095
Conversation
|
LGTM |
anish-sahoo
changed the title
![ask-bonk[bot]](https://avatars.githubusercontent.com/in/2454580?s=60&v=4){kind=small}
There was a problem hiding this comment.
I'm Bonk, and I've done a quick review of your PR.
This PR adds support for staging local wheel/source-archive paths from requirements.txt through the cog_build context before pip install. Overall the approach is clean and well-tested. I found one issue where a valid pip syntax that references local paths can slip through and cause a confusing build failure.
|
I'm Bonk, and I've done a quick review of your PR. This PR adds support for staging local wheel/source-archive paths from requirements.txt through the cog_build context before pip install. Overall the approach is clean and well-tested. I found one issue where a valid pip syntax that references local paths can slip through and cause a confusing build failure. Posted a batch review with 2 suggestions on PR #3095: |
|
@anish-sahoo Bonk workflow was cancelled. View workflow run · To retry, trigger Bonk again. |
2 similar comments
|
@anish-sahoo Bonk workflow was cancelled. View workflow run · To retry, trigger Bonk again. |
|
@anish-sahoo Bonk workflow was cancelled. View workflow run · To retry, trigger Bonk again. |
|
LGTM |
There was a problem hiding this comment.
Pull request overview
Adds support for referencing local Python wheel/source-archive artifacts from build.python_requirements, staging them into the existing cog_build context so Docker builds can pip install -r /tmp/requirements.txt without copying the whole project early.
Changes:
- Parse and validate “bare” local artifact lines in requirements files, and collect/stage those files into the Docker build context.
- Update Dockerfile generation to
COPYstaged artifacts into the image before runningpip install -r ..., and rewrite requirements lines to container paths. - Add unit + integration coverage and document supported/unsupported syntax.
Reviewed changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| pkg/requirements/requirements.go | Tightens trailing-comment stripping by trimming whitespace after #. |
| pkg/requirements/requirements_test.go | Adds unit tests for local-artifact requirement parsing and updates comment-strip fixture. |
| pkg/requirements/local_artifact.go | Introduces parsing/validation for supported local artifact requirement forms. |
| pkg/dockerfile/standard_generator.go | Stages local artifacts into the build context and injects a COPY --from=cog_build ... before pip install; rejects local cog/coglet artifacts. |
| pkg/dockerfile/standard_generator_test.go | Validates artifact staging, Dockerfile ordering, and managed-package rejection behavior. |
| pkg/config/data/config_schema_v1.0.json | Updates schema description to mention local artifact support. |
| pkg/config/config.go | Collects/validates local artifact paths (exist, regular file, inside project) and exposes them to the generator. |
| pkg/config/config_test.go | Adds tests covering artifact discovery and validation failures. |
| integration-tests/tests/local_python_requirement_artifact.txtar | End-to-end test proving a local archive is installed during cog build and importable at runtime. |
| docs/yaml.md | Documents supported local artifact syntax and clarifies run limitations. |
| docs/llms.txt | Regenerated LLM doc snapshot reflecting the YAML docs change. |
| architecture/05-build-system.md | Updates the build-system architecture narrative to include staged user-provided artifacts. |
Suppressed comments (1)
pkg/requirements/local_artifact.go:70
unsupportedLocalOptioncurrently does not flag-r/--requirement/-f/--find-linkswhen the option value starts withfile:(andisRemoteRequirementtreatsfile:///...as remote due to "://"). This lets-r file:///.../--find-links file:///...through even though local file URLs and local includes are out of scope and won’t be staged into the build context.
value, ok = strings.CutPrefix(line, option+"=")
}
value = strings.TrimSpace(value)
if ok && value != "" && !isRemoteRequirement(value) && !strings.HasPrefix(value, "file:") {
return option
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if strings.HasPrefix(line, "file:") { | ||
| return "", false, fmt.Errorf("local file URL requirements are not supported: %s", line) | ||
| } |
|
@anish-sahoo Bonk workflow failed. Check the logs for details. View workflow run · To retry, trigger Bonk again. |
Make the file: URL rejection case-insensitive so FILE:/// forms are not mistaken for remote requirements, and reorder the option check ahead of the remote short-circuit so file:-scheme options are rejected too. Switch the integration test sdist from python3 -m zipfile (which flattens paths and drops the package directory) to a tar.gz that preserves the package layout, so pip install succeeds.
Summary
build.python_requirements.cog_buildbefore dependency installation.Fixes #1945
Usage
Local artifacts can be listed directly in a requirements file:
Cog supports
.whl,.zip,.tar.gz,.tgz,.tar.bz2, and.tar.xz. Paths may contain spaces, are resolved relative to the requirements file, and must point to regular files inside the project after symlink resolution.Remote URLs, remote direct references, and remote requirements options continue to pass through unchanged.
Limits
build.run: pip install ./artifact.zipdoes not stage local source files.name @ pathare rejected.--find-linksdirectories and recursive local requirements includes are rejected.cogorcogletdistribution installed by a local artifact is overridden by Cog's configured runtime. Usebuild.sdk_version,COG_SDK_WHEEL, orCOGLET_WHEELto select those packages.Testing
gotestsum -- -short -timeout 1200s -parallel 5 ./...golangci-lint run ./...go test -tags integration -run "TestIntegration/local_python_requirement_artifact" ./integration-tests/...