Add script to rollback failed import#613
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a new script, retry_failed_ingestion.py, along with its corresponding unit tests, to automate the rollback and retry process for failed Dataflow ingestion jobs. Feedback on the implementation highlights a critical correctness bug where the workflow could be retriggered even if a rollback fails, and a missing pagination implementation when fetching Dataflow job messages. Additionally, it is recommended to use posixpath instead of os.path for GCS path manipulations to avoid platform-specific path separator issues, and to clean up unused imports in the test file.
Not up to standards ⛔🔴 Issues
|
| Category | Results |
|---|---|
| Documentation | 15 minor |
| CodeStyle | 11 minor |
| Complexity | 2 medium |
🟢 Metrics 68 complexity
Metric Results Complexity 68
NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer
TIP This summary will be updated as you push new changes.
vish-cs
force-pushed
the
rollback
branch
2 times, most recently
from
a3696c4 to
5bb4e0a
Compare
| logging.error(f"Error parsing importList parameter: {e}") | ||
|
|
||
| logging.info(f"Fetching error messages for Dataflow job {job_id}...") | ||
| request = dataflow.projects().locations().jobs().messages().list( |
There was a problem hiding this comment.
This seems to only be checking dataflow for failures. What if the failures came from elsewhere in the worfklow (spanner embeddings, ingestion helper issue, bq federation).
What's the long term plan for this, should we be checking all the stages? Should the workflow be modified to propoagate errors from dataflow? (if that's even possible)
| def _revert_txn(transaction): | ||
| # 1. Get current version from ImportStatus to know what to avoid. | ||
| status_sql = """ | ||
| SELECT LatestVersion FROM ImportStatus |
There was a problem hiding this comment.
Given the changes we discussed to IngestionHistory last week, isnt that a better place to read this from? ImportStatus seems brittle given that if we have started a new workflow since, that information is gone.
| """ | ||
| status_results = transaction.execute_sql( | ||
| status_sql, | ||
| params={'importName': short_name}, |
There was a problem hiding this comment.
How would this work for a multi-provenance import?
Like if I import WHO and ILO at once?
There was a problem hiding this comment.
The script checks for the dataflow logs to see if it finds a specific failed import and reverts it. Else, it falls back to reverting all the imports in that batch.
2 participants
No description provided.