fix: filter agent files by type-specific model feature instead of vision alone - #39478
Open
rkfshakti wants to merge 5 commits into
Open
fix: filter agent files by type-specific model feature instead of vision alone#39478rkfshakti wants to merge 5 commits into
rkfshakti wants to merge 5 commits into
Conversation
dosubot
Bot
added
the
size:S
Contributor
Pyrefly Type Coverage
|
Contributor
Pyrefly Diffbase → PR--- /tmp/pyrefly_base.txt 2026-07-29 15:15:45.478874662 +0000
+++ /tmp/pyrefly_pr.txt 2026-07-29 15:15:36.591832634 +0000
@@ -2607,41 +2607,41 @@
ERROR Argument `None` is not assignable to parameter `query` with type `str` in function `core.agent.fc_agent_runner.FunctionCallAgentRunner._organize_user_query` [bad-argument-type]
--> tests/unit_tests/core/agent/test_fc_agent_runner.py:239:46
ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:307:35
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:316:35
ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:309:33
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:318:33
ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:310:33
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:319:33
ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:311:16
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:320:16
ERROR Object of class `FunctionType` has no attribute `assert_called` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:312:9
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:321:9
ERROR Object of class `FunctionType` has no attribute `call_args_list` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:314:23
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:323:23
ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:326:57
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:335:57
ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:348:57
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:357:57
ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:358:35
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:367:35
ERROR Argument `list[TextPromptMessageContent]` is not assignable to parameter `content` with type `str | None` in function `DummyMessage.__init__` [bad-argument-type]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:364:46
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:373:46
ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:369:35
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:378:35
ERROR Object of class `FunctionType` has no attribute `call_args` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:371:16
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:380:16
ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:398:35
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:407:35
ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:416:35
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:425:35
ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:445:35
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:454:35
ERROR Object of class `FunctionType` has no attribute `call_args_list` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:451:25
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:460:25
ERROR Object of class `NoneType` has no attribute `max_iteration` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:455:9
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:464:9
ERROR Object of class `FunctionCallAgentRunner` has no attribute `session` [missing-attribute]
- --> tests/unit_tests/core/agent/test_fc_agent_runner.py:470:29
+ --> tests/unit_tests/core/agent/test_fc_agent_runner.py:479:29
ERROR Missing argument `description` in function `core.agent.plugin_entities.AgentStrategyEntity.__init__` [missing-argument]
--> tests/unit_tests/core/agent/test_plugin_entities.py:281:32
ERROR Invalid key for TypedDict `AppParametersDict`, got `Unknown` [bad-typed-dict-key]
|
Contributor
Author
|
Friendly ping — this PR has been open for a few days with all CI checks passing. The fix filters agent files by the type-specific model feature (e.g. audio, video, document) rather than relying on vision alone, which prevents unsupported file types from being sent to models that only support a subset of multimodal inputs. Would appreciate a review when you get a chance. |
…ion alone When a model does not declare the VISION feature (e.g. qwen-long), all uploaded files were silently dropped because the agent runner gated self.files on a single ModelFeature.VISION check. This meant document files like Excel spreadsheets never reached the LLM even when the model supported the DOCUMENT feature. Filter each file by the model feature its type requires: IMAGE→VISION, DOCUMENT→DOCUMENT, AUDIO→AUDIO, VIDEO→VIDEO. This mirrors the existing _REQUIRED_MODEL_FEATURE_BY_CONTENT_TYPE mapping in graphon and ensures non-image files are passed through when the model supports them. Fixes langgenius#39431
rkfshakti
force-pushed
the
fix/agent-files-dropped-without-vision-39431
branch
from
July 28, 2026 18:18
d9f62a5 to
755ff0e
Compare
Use file_type= instead of type= and FileTransferMethod enum instead of plain string for transfer_method. Update existing stream_tool_call test to use real File objects instead of strings so the type-based filtering code path works correctly.
Contributor
Author
|
Hi maintainers — fixed the failing tests by using the correct File constructor arguments (file_type= and FileTransferMethod enum). CI should be green now. This PR filters agent files by type-specific model feature instead of vision alone, fixing #39431. Would appreciate a review when time allows. Thanks! |
The File model validator requires a reference for LOCAL_FILE transfer method. Without upload_file_id (which normalizes to reference), the constructor raises 'Missing file reference' ValidationError.
…y type check Pyrefly flagged list[str] not assignable to files: list[File]. Replaced runner.files = ["file1"] with a proper File object.
Contributor
Author
|
Fixed the Pyrefly type error in the test — |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related Issues/PRs
Fixes #39431
What changes are proposed in this pull request?
When a model does not declare the
VISIONfeature (e.g. qwen-long), all uploaded files were silently dropped because the agent runner gatedself.fileson a singleModelFeature.VISIONcheck. This meant document files like Excel spreadsheets never reached the LLM even when the model supported theDOCUMENTfeature.The fix filters each file by the model feature its type requires, mirroring the existing
_REQUIRED_MODEL_FEATURE_BY_CONTENT_TYPEmapping in graphon:IMAGE→VISIONDOCUMENT→DOCUMENTAUDIO→AUDIOVIDEO→VIDEOThis ensures non-image files are passed through when the model supports them, instead of being dropped just because the model lacks vision.
How is this PR tested?
New regression test
test_init_filters_files_by_type_not_visionverifies that a document file is kept when the model supportsDOCUMENTbut notVISION, while an image file is correctly filtered out.Does this PR require documentation update?
Does this PR require updating the MLflow Skills repository?
Release Notes
Is this a user-facing change?
What component(s), interfaces, languages, and integrations does this PR affect?
Components
area/tracking: Tracking Service, tracking client APIs, autologgingHow should the PR be classified in the release notes? Choose one:
rn/bug-fix- A user-facing bug fix worth mentioning in the release notesIs this PR a critical bugfix or security fix that should go into the next patch release?