[NAS Backup] Suppress Errors in Disk Usage Calculation that Caused Backup to Fail.

[daviftorres]

Description

This PR tried to prevent the failure of the job at the statistics section of a backup that has actually succeeded.

Apparently, it also fixes some silent failures I previously reported in #11727

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• Build/CI
• Test (unit or integration test code)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

Screenshots (if appropriate):

How Has This Been Tested?

How did you try to break this feature and the system with this change?

[daviftorres]

This is the equivalent command for applying the fix:

sed -i 's_du -sb $dest | cut -f1_du -sb $dest 2>/dev/null | cut -f1 || true_g' /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/nasbackup.sh

We haven't confirmed the exact root cause of the du failure yet. As a precaution, we applied this fix to all servers and will monitor backups over the next few days.

So, I am running tests with 2>>/var/log/cloudstack/agent/nasbackup.err so I can see what is the error message.

[daviftorres]

Proposed Changes Rationale

backup_size=$(du -sb "$dest" 2>/dev/null | cut -f1) || true

• NFS issues may cause du command to fail.
• A size retrieval failure should not invalidate a successful backup.

timeout 60 umount "$mount_point" 2>/dev/null || true
rmdir "$mount_point" 2>/dev/null || true

• Another process may keep the device busy (e.g., parallel backups).
• Network issues may cause hangs on NFS.
• Cleanup failures should not invalidate a successful backup.

echo -n "$backup_size"

• Outputs the size at the end to confirm the script completed past the potentially problematic commands.

[daviftorres]

Dear @abh1sar , do you think you can help me with this bug? Regards,

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 18.94%. Comparing base (5ed4894) to head (8dea747).
⚠️ Report is 32 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13424      +/-   ##
============================================
- Coverage     18.94%   18.94%   -0.01%     
+ Complexity    18376    18375       -1     
============================================
  Files          6192     6192              
  Lines        556550   556558       +8     
  Branches      67954    67955       +1     
============================================
- Hits         105454   105453       -1     
- Misses       439517   439526       +9     
  Partials      11579    11579              

Flag	Coverage Δ
uitests	3.51% <ø> (-0.01%)	⬇️
unittests	20.15% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

This PR adjusts the KVM NAS backup script’s “statistics/cleanup” section so that failures while computing backup disk usage (and related cleanup commands) don’t cause an otherwise successful backup job to be marked as failed.

Changes:

• Capture du output into backup_size and suppress du stderr to avoid failing the script during size calculation.
• Add timeout around umount and suppress errors from umount/rmdir.
• Emit the computed backup size at the end of backup_running_vm().

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[DaanHoogland]

clgtm

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[abh1sar]

hi @daviftorres
would it be possible to test script changes in your env where it is reproducible?
Some log outputs would be nice.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[daviftorres]

would it be possible to test script changes in your env where it i

2026-06-25 13:22:42,360 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Request:Seq 218-281474976710814:  { Cmd , MgmtId: 90520736259046, via: 218, Ver: v1, Flags: 100111, [{"org.apache.cloudstack.backup.TakeBackupCommand":{"vmName":"i-84-6493-VM","backupPath":"i-84-6493-VM/2026.06.25.13.22.42","backupRepoType":"nfs","backupRepoAddress":"10.4.2.145:/mnt/VAN3-NAS01-STOR-POOL-A1/VAN3-NAS01-DS-01","wait":"0","bypassHostMaintenance":"false"}}] }
2026-06-25 13:22:42,360 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Processing command: org.apache.cloudstack.backup.TakeBackupCommand
2026-06-25 13:25:04,120 DEBUG [cloud.agent.Agent] (AgentRequest-Handler-1:[]) (logid:e4aca261) Seq 218-281474976710814:  { Ans: , MgmtId: 90520736259046, via: 218, Ver: v1, Flags: 110, [{"org.apache.cloudstack.backup.BackupAnswer":{"size":"(47.67 GB) 51190241222","result":"true","details":"Job type:         Completed

Nothing else that I can find in the logs are related to the backup job.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[abh1sar]

@blueorangutan package

[blueorangutan]

@abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress.

[blueorangutan]

Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 18383

[DaanHoogland]

@blueorangutan test

[blueorangutan]

@DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests

[blueorangutan]

[SF] Trillian test result (tid-16435)
Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8
Total time taken: 49818 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr13424-t16435-kvm-ol8.zip
Smoke tests completed. 151 look OK, 0 have errors, 0 did not run
Only failed and skipped tests results shown below:

Test	Result	Time (s)	Test File

[DaanHoogland]

@daviftorres , can you look at https://github.com/apache/cloudstack/actions/runs/28373728590/job/84079353133?pr=13424#step:7:11412 ?

[davift]

@daviftorres , can you look at https://github.com/apache/cloudstack/actions/runs/28373728590/job/84079353133?pr=13424#step:7:11412 ?

Hey @DaanHoogland , sure!

trim trailing whitespace.................................................Failed
- hook id: trailing-whitespace
- exit code: 1
- files were modified by this hook

If I understand it right, it is caused by the echo command that produces a new line by default. I had -n in the past, but I accepted GitHub Copilot advise to remove it to prevent some error. Honestly, I am not sure if it is related or not. I am re-adding the argument now and will see if the build fails again.

Please point me in the right direction if you see that I am going the wrong way. Regards!

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.