Repository navigation
Elements from the data types to grab, complete element selectors and combining DataGrabbers - #524
Merged
Merged
Conversation
DataGrabbers: - The element keys are the replacements used by the patterns of the data types to grab, so data types that do not depend on, e.g., the task are processed once per subject. - The elements of each data type are joined on their common replacements. - The values of the replacements to grab can be given in `replacements` (as a dictionary) or by fields linked with `_REPLACEMENT_FIELDS`; elements with other values are not listed and cannot be grabbed. - The `*` and `?` wildcards in the patterns can be used to find the elements, and the paths are always absolute. - The built-in DataGrabbers (AOMIC PIOP1/PIOP2, HCP1200, DMCC13) find their elements in the files, with patterns instead of overriding `get_elements` and `get_item`. Running: - Complete elements are run without searching the dataset, and partial element selectors are no longer reported as invalid. All the selectors must have the same number of values. - A relative `datadir` in the YAML is relative to the YAML file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MultipleDataGrabber now: - Joins the elements of the DataGrabbers on their common keys, so they can have different element keys (e.g., subject and task for BOLD, and only subject for VBM), and each DataGrabber grabs an element with its keys. - Requires each data type to be grabbed by one DataGrabber (chosen with their types), except for DataGrabbers that only grab nested data types of a data type, which replace the ones of the other DataGrabber. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (93.43%) is below the target coverage (95.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #524 +/- ##
==========================================
+ Coverage 92.25% 92.33% +0.08%
==========================================
Files 148 148
Lines 6652 6696 +44
Branches 962 977 +15
==========================================
+ Hits 6137 6183 +46
+ Misses 332 328 -4
- Partials 183 185 +2
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Before Python 3.12, Path.glob skips broken symbolic links when the last component of the pattern is a literal, so files of a DataLad dataset that are not downloaded yet were not found. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DataGrabbers with many replacements (e.g., subject, session and task) listed elements that did not make sense for some data types: with only VBM (one file per subject), the elements were incomplete (
("sub-01",)) and could not be grabbed, and the built-in DataGrabbers listed one element per task, processing the same VBM data once per task. This PR makes the elements depend on the data types to grab, and reworks how elements are selected and combined.DataGrabbers
PatternDataGrabberare the replacements used by the patterns of itstypes. With only VBM, the elements are the subjects; with BOLD, the subjects and tasks._join_elements), so an element is available if it has the files of all the data types, regardless of the order of the types.replacementscan be a dictionary with the values to grab (e.g.,{"subject": null, "task": ["rest"]}), or a DataGrabber can link a field to a replacement with_REPLACEMENT_FIELDS(e.g.,{"task": "tasks"}). Elements with other values are not listed and cannot be grabbed.*and?in the patterns can be used to find the elements (the regular expression now translates them and escapes the rest), and the paths are always absolute.task-{task}_acq-*and?fMRI_{task}) instead of hard-coded subject lists andget_elements/get_itemoverrides, so they only list the elements with data, and theirtasks(and other fields) restrict the elements.MultipleDataGrabber: the DataGrabbers can have different element keys (the elements are joined on their common keys, and each DataGrabber grabs an element with its keys), each data type is grabbed by one DataGrabber (chosen with theirtypes), and a DataGrabber that only grabs nested data types (e.g., only the confounds of BOLD) replaces them.Running
select_elements): with one value for each element key, e.g., each queued job; partial selectors (e.g., only the subject) still search it, once.run, which were always reported as invalid (after computing their elements). All the selectors must have the same number of values.datadirin the YAML is relative to the YAML file, as the storage URI, so the queued jobs (run in the job directory) find the data.Docs
New sections on elements (with an example dataset and tables), combining datasets, complete and partial elements, restricting the values of the replacements and wildcards.
Tests
New tests for each of the above; some existing tests were updated, e.g., expecting partial selectors to be invalid, or a BIDS test whose pattern never matched any file (so it checked nothing).
Changelog entries to be added once the PR has a number.
🤖 Generated with Claude Code