Skip to content

feat: Add concurrency limiter for upload operations - #439

Open
egalvis27 wants to merge 1 commit into
mainfrom
fix/add-concurrency-limiter-to-sync-uploads
Open

egalvis27 wants to merge 1 commit into
mainfrom
fix/add-concurrency-limiter-to-sync-uploads

Conversation

@egalvis27

@egalvis27 egalvis27 commented Aug 25, 2026

Copy link
Copy Markdown

What is Changed / Added

  • Added a concurrency limiter for temporal file uploads to cap the number of simultaneous upload/watch operations.
  • The limit is set to 5 concurrent uploads, which keeps bursts from external file managers (for example, thumbnail generation or parallel file release events) from exhausting the system inotify watcher limit.
  • Integrated the limiter into the two upload entry points that trigger TemporalFileUploader.run:
    • release flow
    • rename-based temporal upload flow
  • Added a focused unit test for the limiter to validate:
    • immediate acquisition while under the concurrency cap
    • queueing when the cap is reached
    • safe release behavior and idempotent slot cleanup
  • Verified the change with the relevant test set, which passed successfully:
    • 4 test files passed
    • 23 tests passed

Why

  • The root cause identified in the investigation was not a direct file-size issue, but a lack of concurrency control.
  • Each upload creates a file watcher and that watcher consumes a Linux inotify slot.
  • When file managers or thumbnailers trigger many release/upload operations in a burst, the application was creating watchers without any global cap.
  • This could exhaust the system watcher limit and lead to ENOSPC errors, especially under GNOME/Nautilus-like workloads.
  • The concurrency limit reduces the risk of inotify exhaustion without changing the upload logic itself, while preserving the current behavior for normal single-file operations.

Summary by CodeRabbit

  • Improvements
    • Limited simultaneous temporal-file uploads to five at a time to improve stability and resource management.
    • Additional uploads now wait in order until capacity becomes available.
    • Upload capacity is reliably restored after successful or failed uploads, including during file renaming.
    • Prevented upload slots from being held indefinitely when operations encounter errors.

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: be8c5a10-d37c-44ff-abac-e6dbeb5c9bb5

📥 Commits

Reviewing files that changed from the base of the PR and between a96ec25 and ed62412.

📒 Files selected for processing (4)
  • src/backend/features/virtual-drive/services/operations/release.service.ts
  • src/backend/features/virtual-drive/services/operations/rename/upload-temporal-file-on-rename.ts
  • src/context/storage/TemporalFiles/application/upload/upload-concurrency-limiter.test.ts
  • src/context/storage/TemporalFiles/application/upload/upload-concurrency-limiter.ts

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The change adds a five-slot upload concurrency limiter for temporal files. Release and rename uploads acquire slots before uploading and release them in finally blocks. Tests cover slot limits, queuing, and idempotent releases.

Changes

Temporal upload concurrency

Layer / File(s) Summary
Upload concurrency limiter
src/context/storage/TemporalFiles/application/upload/upload-concurrency-limiter.ts, src/context/storage/TemporalFiles/application/upload/upload-concurrency-limiter.test.ts
The limiter caps active uploads at five, queues additional callers, and provides idempotent release functions. Tests cover these behaviors.
Release upload integration
src/backend/features/virtual-drive/services/operations/release.service.ts
The release operation acquires a slot before uploading a temporal file and releases it after success or failure.
Rename upload integration
src/backend/features/virtual-drive/services/operations/rename/upload-temporal-file-on-rename.ts
The rename operation acquires a slot before uploading a temporal file and releases it after success or failure.

Estimated code review effort: 2 (Simple) | ~10 minutes

Merge Risk: ⚪ Minimal · up to ed624

The change adds a localized upload concurrency limit with focused passing tests, and no actionable merge-blocking risk remains beyond normal checks and review.

Suggested reviewers: alexismora

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 4 functions across 4 files. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: adding a concurrency limiter for upload operations.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/add-concurrency-limiter-to-sync-uploads

Warning

Some tools did not complete. Review the errors below.

🔧 ESLint

If the error stems from missing dependencies, add them to the package.json file. For unrecoverable errors (e.g., due to private dependencies), disable the tool in the CodeRabbit configuration.

src/backend/features/virtual-drive/services/operations/release.service.ts

ESLint skipped: missing config or dependency (missing-dependency). The ESLint configuration references a package that is not available in the sandbox.

src/backend/features/virtual-drive/services/operations/rename/upload-temporal-file-on-rename.ts

ESLint skipped: the matched ESLint configuration already failed (missing-dependency).

src/context/storage/TemporalFiles/application/upload/upload-concurrency-limiter.test.ts

ESLint skipped: the matched ESLint configuration already failed (missing-dependency).

  • 1 others

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant