Add WorkflowHub importer with per-tool workflow ID outputs - #52
Conversation
|
linter fails due to other scripts alread merged ! |
|
@rioualen @supernord just want to make sure you are aware of this, it would be great if you could consider reviewing |
|
Same as for tess: @mihai-sysbio imo this is ready for review, maybe @hmenager can check if the logic makes sense (ids in data folder, full metadata in dataset folder), or I can present in the next meeting, toimorrow :) @arash77 maybe you can check the code logic, worked for me locally - the question would be how to process it downstream for the atlas, so that we can see for each tool, which workflow is using it, the workflow should include at least the link, version and name I guess |
|
Thank you @mihai-sysbio for drawing my attention to this, I'll look into it! Talking about EDAM annotations and the Codex, it seems tools and workflows are annotated with EDAM labels rather than URIs, which is not ideal. I opened an issue to discuss this in the Codex repository. |
Checked the logic, works locally. On your question: link, version and name are all there. That's exactly the shape the Atlas frontend already expects for workflows, so Matches the repo conventions. CI is red from pre-existing lint/format issues on Minor: |
…WorkflowHub importer
|
@agnesbrnb can you check if there is some metadata missing that is created here https://github.com/research-software-ecosystem/micoreca/blob/main/bin/extract_workflowhub.py ? |
|
The extraction of the complete JSON file from workflowhub is what MiCoReCa needs for filtering. |
|
Thanks @paulzierep for this PR ! We have been working with FAIR-checker to extract JSON-LD semantic markup, following the Bioschemas Computational Workflow profile. I do not lanage to get references to EDAM terms, but I see that you manage to do it ? How do you proceed ? Would it be possible to document it for the workflowHub dev team so that they expose more complete Bioschemas markup on each web page ? |
Actually, the EDAM terms come directly form workflowHUB. The API response at /workflows/{id}.json includes them as topic_annotations and operation_annotations arrays with labels. |
|
@paulzierep For MiCoReCa (research-software-ecosystem/micoreca#49) I need the extraction of the workflow type (Galaxy, Nextflow, Snakemake..) under the attribute « type ». |
|
This PR looks good to me, maybe the final approval of @hmenager would be useful, provided he has some availability. As soon as it is merged, I'll work on a better RDF dump to incorporate into the SPARQL endpoint. |
@agnesbrnb |
@rioualen I'd be tempted to not wait for @hmenager's review as a blocker - if there are any issues these can be raised separately. Since this is "new" metadata in the repo, the concern of backwards compatibility isn't there. |
|
In the context of the data source for EDAM ids instead of labels, I am noting here @supernord 's confirmation that:
|
My mistake, I hadn’t noticed. I’ll update my pull request on MiCoReCA. |
|
One naïve question. Does this importer process all the revisions of an entry? Those revisions can be envisioned as "releases", some of them with their own DOI, and it might happen that some people is still using an older release for reproducibility purposes. |
hmenager
left a comment
hmenager
left a comment
There was a problem hiding this comment.
Thanks a million @paulzierep !
I'd just request to not write the tools to data if they cannot be identified formally in Galaxy with a bio.tools link.
| all_entries = [] | ||
| tool_to_wf_ids = {} | ||
| stat_mapped = 0 | ||
| stat_fallback = 0 |
There was a problem hiding this comment.
I don't think we should "fall back". If there is no bio.tools ID in the Galaxy tool, we should just not add the file to the "data" folder.
There was a problem hiding this comment.
I think this is an interesting case. What would be the reason for the absence of a bio.tools ID? If it's simply not provided at the source, indeed as soon as that would be fixed this case would disappear. However, I can also see the case of many tools outside life sciences lacking a bio.tools ID. For this latter case, I would rather favour the fall back.
Addresses the review comment: if a tool cannot be tied to a bio.tools ID,
the file should not be written at all.
A tool reference now counts as identified only when it comes from one of two
places:
* a curated WorkflowHub annotation, which states the bio.tools URL
outright (every annotation sampled from the API carries one, e.g.
{"name": "SAMtools", "id": "https://bio.tools/samtools"});
* a Galaxy tool ID that galaxy_codex maps to a bio.tool ID.
Everything else is counted and dropped. The previous `bt_id = tool_name.lower()`
fallback asserted links nobody had made, and it demonstrably invented at least
one: WorkflowHub 2239 annotates the tool named "Tiara" with
https://bio.tools/tiara-metagenomics, and the fallback used the display name to
also claim the unrelated bio.tools entry "tiara", writing
data/tiara/tiara.workflowhub.json. That file is no longer produced.
Curated display names are no longer used as identifiers either, for the same
reason -- they are labels, not IDs ("NCBI fcs" for ncbi_fcs, "Micro-Meta App"
for micro-meta_app, "Tiara" for tiara-metagenomics).
Two related fixes:
- Steps whose description is empty are skipped. A subworkflow or annotation
step has no tool reference, and the empty string previously survived into
the write loop, where os.path.join(base, "data", "") resolves to data/
itself, so os.path.isdir passed and the importer wrote data/.workflowhub.json
-- a hidden file at the root of data/ that clean() could never remove, since
its glob is data/*/*.workflowhub.json.
- The per-workflow `tools` list is sorted. It was built from a set, so its
order varied between processes with hash randomisation, and every run
rewrote all ~1500 files in imports/workflowhub/ with a reshuffled array.
Stats now report the two identification paths separately, how many identified
IDs had no data/ directory, and how many references were dropped for lack of a
bio.tools ID, so nothing is discarded silently.
Measured over the same 150 workflows, before and after: 80 per-tool files
become 70. One of the ten dropped is the data/.workflowhub.json bug and one is
the false "tiara" attribution. The remaining eight -- flexynesis, hifiasm,
kneaddata, mashmap, mitohifi, mosdepth, query_tabular -- look like genuine uses
that simply have no assertion behind them: galaxy_codex lists those Suite IDs
but leaves `bio.tool ID` null, and only 625 of its 1840 entries (34%) carry one
at all. Recovering them belongs upstream, by annotating the workflow in
WorkflowHub or filling in the galaxy_codex mapping, not by guessing here.
Also fixes the two lint failures in this file: I001 on the import block and one
over-wrapped json.dump call.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a `workflowhub` job to the weekly import, running in parallel with the eight existing importers. It follows the same shape as `biotools`: delete any stale import branch, create workflowhub-import-branch, run the composite action, then commit and push with git-auto-commit-action. Wiring the job in takes three further edits, without which the branch would be created but never consumed: - `workflowhub` added to merge-imports.needs - merge_branch_if_exists "workflowhub-import-branch" in merge-imports - delete_branch_if_exists "workflowhub-import-branch" in clean-branches Depends on research-software-ecosystem/utils#52, which adds the workflowhub-import action. Until that is merged the action path does not resolve and this job will fail; merge-imports is `if: always()`, so the other importers would still merge, but there is no reason to land this first. This makes WorkflowHub a per-tool source for the first time: it currently deposits only the bulk dumps under datasets/, and nothing under data/. Once the importer runs, scripts/stats/ecosystem.py in utils should gain ("WorkflowHub", ("*.workflowhub.json",)) in NATIVE_SOURCES, replacing the comment there that explains why WorkflowHub is absent. Runtime note: the importer walks ~1500 workflows at roughly one API request plus a 0.2 s pause each, so expect 12-15 minutes. That is longer than most of the sibling jobs, and since they run in parallel it should not extend the overall import much. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
8 participants
Add WorkflowHub importer with per-tool workflow ID outputs
Creates a new
workflowhub-import/importer that fetches all workflows from the WorkflowHub API and produces per-tool workflow ID files for the RSEc content repo.What it does:
bio.toolsannotations and Galaxyinternals.stepsdescriptionsdata/{biotool_id}/directories usinggalaxy_codexSuite ID → bio.tool ID mapping, falling back to raw tool namedata/{biotool_id}/{biotool_id}.workflowhub.jsoncontaining only a list of workflow IDsimports/workflowhub/{id}.workflowhub.json--test [N]flag to limit workflows for testingUsage:
Format of per-tool files (e.g.
data/fastp/fastp.workflowhub.json):imports/workflowhub/1104.workflowhub.json(workflow entry withmapped_tools):{ "content_blobs": [], "create_time": "2026-07-08", "creators": [ "Amirhossein Naghsh Nilchi", "Pavankumar Videm", "Bj\u00f6rn Gr\u00fcning" ], "description": "A comprehensive Galaxy workflow for the end-to-end analysis of spatial data, featuring automated QC, clustering, spatial statistics, cell type annotation, and cell-cell communication.", "discussion_links": [], "doi": "10.48546/workflowhub.workflow.2174.5", "edam_operation": [ "Clustering", "Dimensionality reduction", "Expression correlation analysis" ], "edam_topic": [ "Single-cell sequencing" ], "id": "2174", "latest_version": "5", "license": "MIT", "link": "https://workflowhub.eu/workflows/2174?version=5", "mapped_tools": [ "anndata", "celltypist", "liana", "scanpy", "spatialdata", "squidpy" ], "name": "Galaxy EISTA spatialdata + updated squidpy", "number_of_steps": 80, "projects": [], "source": "WorkflowHub", "tags": [ "bioinformatics", "galaxy", "single cell clustering", "transcriptomics", "single-cell", "spatial transcriptomics" ], "tools": [ "scanpy_inspect", "param_value_from_file", "liana", "scanpy_filter", "addvalue", "add_a_column1", "liana+", "compose_text_param", "celltypist", "tp_replace_in_line", "spatialdata_operation", "pick_value", "collection_column_join", "anndata", "scanpy", "set the value to a text input\nparam_value_from_file", "split_file_to_collection", "select the resolution\ncut1", "tp_awk_tool", "tp_cut_tool", "anndata_inspect", "scanpy_cluster_reduce_dimension", "table_compute", "tp_sort_header_tool", "scanpy_remove_confounders", "scanpy_normalize", "tp_find_and_replace", "scanpy_plot", "datamash_transpose", "anndata_manipulate", "squidpy", "spatialdata", "squidpy_graph", "liana_methods", "map_param_value" ], "update_time": "2026-07-08", "versions": "5", "workflow_class": "Galaxy" }Unfortunatly most nextflow / snakemake workflows do not have tools available via workflowHUB api, I made some tests to get it via ro-crate or the associated github, but this is messy. I would opt to fix this rather on the workflow hub side.