Regarding the database schema changes, the backups table already has a type column which stores the backup type (full or incremental). Furthermore, the columns parent_backup_id, bitmap_name and chain_id are implementation specific. These should be moved to a separate table, or to the backup_details table.

Thanks @JoaoJandre, good points.

1. type column — agreed, we'll use the existing backups.type column to distinguish full vs incremental rather than adding a new one.

2. Implementation-specific columns — makes sense to keep the backups table provider-agnostic. We'll move parent_backup_id, bitmap_name, and chain_id to backup_details as key-value pairs. This also makes it easier for other backup providers to implement incremental in their own way without schema changes.

I'll update the RFC to reflect this. Thanks for the review!

cc @abh1sar

@jmsperu Thanks for working on this. This is a very useful improvement to the NAS Backup provider.

My comments below:

nas.backup.full.interval

1. I didn't understand this. Backups can be scheduled at an hourly schedule. Can be ad-hoc as well.

Full backup (Day 0 or every nas.backup.full.interval days):

1. Export the entire disk to the NAS mount
   qemu-img convert -f qcow2 -O qcow2
   /var/lib/libvirt/images/vm-disk.qcow2
   /mnt/nas/backups/vm-uuid/backup-full-20260327.qcow2

2. Create a new dirty bitmap to track changes from this point
   virsh qemu-monitor-command $DOMAIN --hmp
   'block-dirty-bitmap-add drive-virtio-disk0 backup-20260327 persistent=true'

2. Why can’t we use the libvirt’s backup-begin api for running VMs here as well?

# 1. Use libvirt backup-begin with incremental mode
# This exports only blocks dirty since bitmap "backup-20260327"
cat > /tmp/backup.xml <<'XML'
<domainbackup mode="push">
  <disks>
    <disk name="vda" backup="yes" type="file">
      <target file="/mnt/nas/backups/vm-uuid/backup-inc-20260328.qcow2"
              type="qcow2"/>
      <driver type="qcow2"/>
    </disk>
  </disks>
  <incremental>backup-20260327</incremental>
</domainbackup>
XML

virsh backup-begin $DOMAIN /tmp/backup.xml

# 2. Wait for completion
virsh domjobinfo $DOMAIN --completed

# 3. Rotate bitmaps: remove old, create new
virsh qemu-monitor-command $DOMAIN --hmp \
  'block-dirty-bitmap-remove drive-virtio-disk0 backup-20260327'
virsh qemu-monitor-command $DOMAIN --hmp \
  'block-dirty-bitmap-add drive-virtio-disk0 backup-20260328 persistent=true'

3. Checkpoints based on days won’t work as backups can be hourly and adhoc. Timestamp based bitmap name can be used instead.
4. I don't think explicit bitmap manipulation is required here. backup-begin takes care of creating the bitmap as defined in the backup xml.

Example: Restore to Day 3 (full + 3 incrementals):

# 1. Create a working copy from the full backup
cp /mnt/nas/backups/vm-uuid/backup-full-20260327.qcow2 /tmp/restored.qcow2

# 2. Apply each incremental in order using qemu-img rebase
#    Each incremental is a thin qcow2 containing only changed blocks.
#    Rebasing merges the incremental's blocks into the chain.
qemu-img rebase -u -b /tmp/restored.qcow2 \
  /mnt/nas/backups/vm-uuid/backup-inc-20260328.qcow2

qemu-img rebase -u -b /mnt/nas/backups/vm-uuid/backup-inc-20260328.qcow2 \
  /mnt/nas/backups/vm-uuid/backup-inc-20260329.qcow2

qemu-img rebase -u -b /mnt/nas/backups/vm-uuid/backup-inc-20260329.qcow2 \
  /mnt/nas/backups/vm-uuid/backup-inc-20260330.qcow2

# 3. Flatten the chain into a single image
qemu-img convert -f qcow2 -O qcow2 \
  /mnt/nas/backups/vm-uuid/backup-inc-20260330.qcow2 \
  /tmp/vm-restored-final.qcow2

# 4. Return the flattened image for CloudStack to import

5. qemu-img rebase can be done right after each backup. Then the backup files themselvess will have information about the chain. And restore task would be simple.

In addition I have the below concerns:

6. How will incremental backups work for stopped VMs.
7. Can you put some info on how Delete backups will work when the backups is an incremental backup, and a full backup containing children.
8. VMs in cloudstack are not persistent on a KVM host. When a VM stops and starts, the domain xml is recreated by Cloudstack. Cloudstack currently doesn't recreate the domain checkpoints. Checkpoints will need to be recreated when the VM starts for incremental backup with backup-begin to work.

Open Questions for Discussion

• Interest level. Is there sufficient demand for this feature to justify the implementation effort? We believe so based on mailing list threads, but would like confirmation.

Definitely.

• Dirty bitmaps vs. alternatives. Are there concerns about relying on QEMU dirty bitmaps? Alternative approaches include file-level deduplication on the NAS (less efficient, not hypervisor-aware) or qemu-img compare (slower, requires reading both images).

I don't think so. It is the way to go.

• Target release. Should this target CloudStack 4.23, or is a later release more appropriate given the scope?

Given the scope, we should definitely target 4.23. 4.23 is a non-LTS release meant for adding new functionality and features.

• Chain model. We proposed forward-incremental with periodic full backups. Would the community prefer a different model (e.g., reverse-incremental like Veeam, or forever-incremental with periodic synthetic fulls)?

I think this is the simplest model. Suitable for us.

• Scope of first PR. Should we submit the entire feature as one PR, or break it into smaller PRs (e.g., nasbackup.sh changes first, then agent, then management server, then UI)?

A single PR is better. That way reviewers have full context and can be tested end-to-end.

• Testing infrastructure. We can test against our production environment (Ubuntu 22.04, QEMU 6.2, libvirt 8.0). Are there CI environments or community test labs available for broader testing (RHEL, Rocky, older QEMU versions)?

Github CI will test build, unit tests and smokes tests. There is an integration tests for NAS plugin smokes/test_backup_recovery_nas.py. Feel free to add test case for incremental backups. It will be run with the other smokes tests.
In addition, someone will volunteer to do extensive manual testing of this feature before merging.

Looking forward to your reply. Let me know if I can help in any other way.

@abh1sar Thank you for the detailed and thoughtful review. This is exactly the kind of feedback that will make the implementation solid. Addressing each point:

1. nas.backup.full.interval

You're right — "days" is misleading since backups can be hourly or ad-hoc. The intent was to control how often a full backup is taken vs incremental. We'll change this to a count-based approach: nas.backup.full.every (default: 10) — meaning every 10th backup is a full, regardless of schedule. This works for hourly, daily, and ad-hoc backups alike.

2. Use backup-begin for full backups too

Agreed. We'll use backup-begin (libvirt's backup API) for both full and incremental backups of running VMs. For full backups, simply omit the <incremental> element from the backup XML:

<domainbackup mode="push">
  <disks>
    <disk name="vda" backup="yes" type="file">
      <target file="/mnt/nas/backups/vm-uuid/backup-full-{timestamp}.qcow2" type="qcow2"/>
      <driver type="qcow2"/>
    </disk>
  </disks>
</domainbackup>

This keeps the approach consistent and lets libvirt handle all the heavy lifting.

3. Timestamp-based bitmap names

Agreed. We'll use epoch timestamps (e.g., backup-1711584000) instead of date-based names. This avoids collisions on hourly/ad-hoc schedules and sorts naturally.

4. No explicit bitmap manipulation

Good point — backup-begin manages bitmaps through the backup XML's checkpoint definition. We'll remove the manual block-dirty-bitmap-add/remove commands and let the API handle it. The backup XML with <incremental> tells libvirt which bitmap to use and what new checkpoint to create.

5. Rebase after each backup

This is a great simplification. We'll run qemu-img rebase immediately after each incremental backup completes. This way:

• Each backup file carries its chain reference
• Restore becomes: flatten from the target point backward (no manual chain assembly)
• Simpler code, fewer edge cases

6. Stopped VMs

For stopped VMs, backup-begin is not available (no running QEMU process). We'll use a different approach:

• Full backup: qemu-img convert (same as today — copy the entire disk image)
• Incremental backup: qemu-img compare between the current disk and the last backup, then export only changed blocks using qemu-img convert with a backing file reference

This is slower than the live bitmap approach but functionally correct. The backup_details table will track whether a backup used the live (bitmap) or offline (compare) method, so restore logic can handle both.

Alternatively, we could require that incremental backups are only available for running VMs and fall back to full backup for stopped VMs. This is simpler and avoids the qemu-img compare complexity. Would the community prefer this approach?

7. Delete backup cascading

Proposed delete behavior:

• Delete a middle incremental: Merge it into its parent (rebase the child onto the grandparent), then remove the file. The chain remains valid.
• Delete a full backup with children: Refuse with an error — user must delete all children first (youngest to oldest), or we could offer a "delete chain" option that removes the full + all its incrementals.
• Delete the newest incremental: Simple file removal + update chain metadata.

We'll add a chain_position field in backup_details to track where each backup sits in the chain, making delete validation straightforward.

8. Checkpoint recreation on VM restart

This is the trickiest issue. Since CloudStack recreates the domain XML on stop/start, all checkpoints are lost. Our plan:

On VM start (in LibvirtComputingResource or the start command wrapper):

1. Query backup_details for the VM's latest bitmap name
2. If an active incremental chain exists, recreate the bitmap using:

   virsh qemu-monitor-command $DOMAIN --hmp \
     'block-dirty-bitmap-add drive-virtio-disk0 {bitmap_name} persistent=true'
   

3. Mark the bitmap as "recreated" — the next incremental will capture ALL changes since the VM restarted (slightly larger than if the bitmap had survived, but correct)

This means the first incremental after a restart will be larger (contains all writes since restart, not just since last backup), but subsequent incrementals return to normal size. This is an acceptable trade-off and matches how other backup solutions handle the same problem.

Updated implementation plan based on your feedback:

1. Use backup-begin API for both full and incremental (running VMs)
2. Use qemu-img convert/compare for stopped VMs (full always, incremental optional)
3. Timestamp-based bitmap names
4. Let libvirt manage bitmaps via backup XML (no manual bitmap commands)
5. Rebase immediately after each incremental
6. Store chain metadata in backup_details (provider-agnostic, per @JoaoJandre's feedback)
7. Recreate bitmaps on VM start
8. Add integration test cases in smokes/test_backup_recovery_nas.py
9. Single PR targeting 4.23

Will start working on this. Thank you both @abh1sar and @JoaoJandre for the guidance — it's significantly improved the design.

hey hi @jmsperu hope you are doing well. May I know if you are working on this and having a good progress ?

ping @jmsperu

@harikrishna-patnala thanks for the ping — apologies for the silence.

Yes, actively working on it. I just opened a draft PR at #13074 with the RFC committed and the implementation phases laid out. I'm targeting CloudStack 4.23 as discussed.

ETA for first reviewable commit (the bash-script + ConfigKey changes — the bulk of the new logic on the KVM side): within ~2 weeks (by 2026-05-11). Bitmap-recreation-on-start, cascade-delete, and smoke tests will land in subsequent commits on the same branch.

Will keep you posted here and on the PR.

@harikrishna-patnala @abh1sar @JoaoJandre @weizhouapache — quick update.

The full implementation has now landed on PR #13074 as 8 commits (~1.6K LoC across 11 files including the RFC). All review feedback from this thread is addressed; mapping table is in the PR description.

Highlights:

• Chain metadata lives in backup_details (per @JoaoJandre) — no schema change.
• nas.backup.full.every (count-based) replaces the day-based interval (per @abh1sar).
• Both full and incremental use backup-begin (full omits <incremental>); checkpoints created atomically via --checkpointxml.
• Bitmap loss on VM restart is handled lazily at next backup (agent recreates the missing bitmap and emits BITMAP_RECREATED= so the orchestrator can record it). Avoided touching VirtualMachineManager/StartCommand to keep the blast radius small.
• Cascade delete: middle incremental triggers a safe qemu-img rebase of its child onto the grandparent (child absorbs the to-be-deleted parent's blocks); full with children is refused unless forced=true.
• 5 new smoke tests in test_backup_recovery_nas.py cover cadence, restore-from-incremental, delete-middle, refuse-delete-full-with-children, and stopped-VM fallback.

Marking the PR ready for review now. Looking forward to your feedback.

RFC: Incremental NAS Backup Support for KVM Hypervisor

Summary

This RFC proposes adding incremental backup support to CloudStack's NAS backup provider for KVM hypervisors. By leveraging QEMU dirty bitmaps and libvirt's backup-begin API, CloudStack can track changed disk blocks between backups and export only the delta, reducing daily backup storage consumption by 80--95% and shortening backup windows from hours to minutes for large VMs. The feature is opt-in at the zone level, backward-compatible with existing full-backup behavior, and gracefully degrades on older QEMU/libvirt versions.

Motivation

CloudStack's NAS backup provider currently performs a full disk copy every time a backup is taken. For a 500 GB VM with five daily backups retained, that amounts to 2.5 TB of storage consumed. At scale -- tens or hundreds of VMs -- this becomes a serious operational and financial burden.

Problems with the current approach:

1. Storage waste. Every backup is a full copy of the entire virtual disk, regardless of how little data actually changed since the last backup.
2. Long backup windows. Copying hundreds of gigabytes over NFS or SMB takes hours, increasing the risk of I/O contention on production workloads.
3. Network bandwidth pressure. Full-disk transfers saturate the storage network during backup windows, impacting other VMs on the same host.
4. Uncompetitive feature set. VMware (Changed Block Tracking / VADP), Proxmox Backup Server, and Veeam all support incremental backups natively. CloudStack's lack of incremental backup is a common complaint on the users@ mailing list and a blocker for adoption in environments with large VMs.

What incremental backup achieves:

• Only changed blocks are transferred and stored after the initial full backup.
• A typical daily incremental for a 500 GB VM with moderate write activity is 5--15 GB, a reduction of 97--99% compared to a full copy.
• Backup completes in minutes rather than hours.
• Retention of 30+ daily restore points becomes economically feasible.

Proposed Design

Backup Chain Model

Incremental backups form a chain anchored by a periodic full backup:

Full (Day 0) -> Inc 1 (Day 1) -> Inc 2 (Day 2) -> ... -> Inc 6 (Day 6) -> Full (Day 7) -> ...

Restoring to any point in time requires the full backup plus every incremental up to the desired restore point. To bound restore complexity and protect against chain corruption, a new full backup is forced at a configurable interval.

Global settings (zone scope):

When nas.backup.incremental.enabled is false (the default), behavior is identical to today -- every backup is a full copy. Existing deployments are unaffected.

Technical Approach

1. Dirty Bitmap Tracking (QEMU Layer)

QEMU 4.0 introduced persistent dirty bitmaps: per-disk bitmaps that record which blocks have been written since the bitmap was created. These bitmaps survive QEMU restarts (they are stored in the qcow2 image header) and are the foundation for incremental backup.

Lifecycle:

1. When incremental backup is enabled for a VM, the agent creates a persistent dirty bitmap on each virtual disk via QMP:

   {
     "execute": "block-dirty-bitmap-add",
     "arguments": {
       "node": "drive-virtio-disk0",
       "name": "backup-20260327",
       "persistent": true
     }
   }

2. QEMU automatically sets bits in this bitmap whenever the guest writes to a block.
3. At backup time, the bitmap tells the backup process exactly which blocks to read.
4. After a successful backup, a new bitmap is created for the next cycle and the old bitmap is optionally removed.

2. Backup Flow

Full backup (Day 0 or every nas.backup.full.interval days):

# 1. Export the entire disk to the NAS mount
qemu-img convert -f qcow2 -O qcow2 \
  /var/lib/libvirt/images/vm-disk.qcow2 \
  /mnt/nas/backups/vm-uuid/backup-full-20260327.qcow2

# 2. Create a new dirty bitmap to track changes from this point
virsh qemu-monitor-command $DOMAIN --hmp \
  'block-dirty-bitmap-add drive-virtio-disk0 backup-20260327 persistent=true'

Incremental backup (Day 1 through Day N):

# 1. Use libvirt backup-begin with incremental mode
# This exports only blocks dirty since bitmap "backup-20260327"
cat > /tmp/backup.xml <<'XML'
<domainbackup mode="push">
  <disks>
    <disk name="vda" backup="yes" type="file">
      <target file="/mnt/nas/backups/vm-uuid/backup-inc-20260328.qcow2"
              type="qcow2"/>
      <driver type="qcow2"/>
    </disk>
  </disks>
  <incremental>backup-20260327</incremental>
</domainbackup>
XML

virsh backup-begin $DOMAIN /tmp/backup.xml

# 2. Wait for completion
virsh domjobinfo $DOMAIN --completed

# 3. Rotate bitmaps: remove old, create new
virsh qemu-monitor-command $DOMAIN --hmp \
  'block-dirty-bitmap-remove drive-virtio-disk0 backup-20260327'
virsh qemu-monitor-command $DOMAIN --hmp \
  'block-dirty-bitmap-add drive-virtio-disk0 backup-20260328 persistent=true'

New full backup cycle (Day 7):

# Remove all existing bitmaps
virsh qemu-monitor-command $DOMAIN --hmp \
  'block-dirty-bitmap-remove drive-virtio-disk0 backup-20260327'

# Take a full backup (same as Day 0)
# Optionally prune expired chains from NAS

3. Restore Flow

Restoring from an incremental chain requires replaying the full backup plus all incrementals up to the target restore point. This is handled entirely within nasbackup.sh and is transparent to the management server and the end user.

Example: Restore to Day 3 (full + 3 incrementals):

# 1. Create a working copy from the full backup
cp /mnt/nas/backups/vm-uuid/backup-full-20260327.qcow2 /tmp/restored.qcow2

# 2. Apply each incremental in order using qemu-img rebase
#    Each incremental is a thin qcow2 containing only changed blocks.
#    Rebasing merges the incremental's blocks into the chain.
qemu-img rebase -u -b /tmp/restored.qcow2 \
  /mnt/nas/backups/vm-uuid/backup-inc-20260328.qcow2

qemu-img rebase -u -b /mnt/nas/backups/vm-uuid/backup-inc-20260328.qcow2 \
  /mnt/nas/backups/vm-uuid/backup-inc-20260329.qcow2

qemu-img rebase -u -b /mnt/nas/backups/vm-uuid/backup-inc-20260329.qcow2 \
  /mnt/nas/backups/vm-uuid/backup-inc-20260330.qcow2

# 3. Flatten the chain into a single image
qemu-img convert -f qcow2 -O qcow2 \
  /mnt/nas/backups/vm-uuid/backup-inc-20260330.qcow2 \
  /tmp/vm-restored-final.qcow2

# 4. Return the flattened image for CloudStack to import

An alternative approach uses qemu-img commit to merge each layer down. The implementation will benchmark both methods and choose the faster one for large images.

4. Database Schema Changes

Modified table: backups

New table: backup_chains

Schema migration will be provided as a Liquibase changeset, consistent with CloudStack's existing migration framework. The new columns are nullable to maintain backward compatibility with existing backup records.

5. Management Server Changes

BackupManagerImpl (orchestration):

• Before taking a backup, query the active chain for the VM.
• If no active chain exists, or the chain has reached nas.backup.incremental.max.chain incrementals, or nas.backup.full.interval days have elapsed since the last full backup: schedule a full backup and start a new chain.
• Otherwise: schedule an incremental backup linked to the previous backup in the chain.
• On backup failure: if the bitmap is suspected corrupt, mark the chain as SEALED and force a full backup on the next run.

NASBackupProvider.takeBackup():

• Accept a new parameter BackupType (FULL or INCREMENTAL).
• For incremental: pass the parent backup's bitmap name and NAS path to the agent command.

TakeBackupCommand / TakeBackupAnswer:

• Add fields: backupType (FULL/INCREMENTAL), parentBackupId, bitmapName, parentBackupPath.
• The answer includes the actual size of the backup (important for incrementals, which are much smaller than the disk size).

RestoreBackupCommand:

• Add field: backupChain (ordered list of backup paths from full through the target incremental).
• The agent reconstructs the full image from the chain before importing.

6. KVM Agent Changes

LibvirtTakeBackupCommandWrapper:

• For FULL backups: existing behavior (qemu-img convert), plus create the initial dirty bitmap.
• For INCREMENTAL backups: use virsh backup-begin with <incremental> XML, then rotate bitmaps.
• Pre-flight check: verify QEMU version >= 4.0 and libvirt version >= 6.0. If not met, fall back to full backup and log a warning.

nasbackup.sh enhancements:

• New flag -i for incremental mode.
• New flag -p <parent_path> to specify the parent backup on the NAS.
• New flag -b <bitmap_name> to specify which dirty bitmap to use.
• New subcommand restore-chain that accepts an ordered list of backup paths and produces a flattened image.

LibvirtRestoreBackupCommandWrapper:

• If the restore target is an incremental backup, request the full chain from the management server and pass it to nasbackup.sh restore-chain.

7. API Changes

Existing API: createBackup

No change to the API signature. The management server automatically decides full vs. incremental based on the zone configuration and the current chain state. Callers do not need to specify the backup type.

Existing API: listBackups

Response gains two new fields:

• backuptype (string): Full or Incremental
• parentbackupid (string): UUID of the parent backup (null for full backups)

Existing API: restoreBackup

No change. The management server resolves the full chain internally.

8. UI Changes

• Backup list view: Add a "Type" column showing Full or Incremental, with a visual indicator (e.g., a small chain icon for incrementals).
• Backup detail view: Show the backup chain as a vertical timeline: full backup at the top, incrementals branching down, with sizes and timestamps.
• Restore dialog: When the user selects an incremental restore point, display a note: "This restore will replay N backups (total chain size: X GB)."
• Backup schedule settings (zone-level): Toggle for incremental backup, full backup interval slider, max chain length input.

Storage Savings Projections

The following estimates assume a moderate write workload (2--5% of disk blocks changed per day), which is typical for application servers, databases with WAL, and file servers.

For environments with higher change rates (e.g., heavy database writes), incremental sizes will be larger, but savings still typically exceed 60%.

Requirements

Graceful degradation: If a KVM host runs QEMU < 4.0 or libvirt < 6.0, the agent will detect this at startup and report incrementalBackupCapable=false to the management server. Backups for VMs on that host will remain full-only, with a warning logged. No manual intervention is required.

Risks and Mitigations

Implementation Plan

Total estimated effort: 7--8 weeks.

We (Xcobean Systems) intend to implement this and submit PRs against the main branch. We would appreciate early design feedback before starting implementation to avoid rework.

Prior Art

• VMware VADP / Changed Block Tracking (CBT): VMware's CBT is the industry-standard approach. A change tracking driver inside the hypervisor records changed blocks, and backup vendors query the CBT via the vSphere API. This RFC's approach is analogous, using QEMU dirty bitmaps as the CBT equivalent.

• Proxmox Backup Server (PBS): PBS uses QEMU dirty bitmaps to implement incremental backups natively. Their implementation validates that the dirty bitmap approach is production-ready for KVM/QEMU environments. PBS has been stable since Proxmox VE 6.4 (2020).

• Veeam Backup & Replication: Veeam uses a "reverse incremental" model where the most recent backup is always a synthetic full, and older backups are stored as reverse deltas. This simplifies restore (always restore from the latest full) at the cost of more I/O during backup. We chose the forward-incremental model for simplicity and because it aligns with how QEMU dirty bitmaps work natively.

• libvirt backup API: The virsh backup-begin command and its underlying virDomainBackupBegin() API were specifically designed for this use case. The libvirt documentation includes examples of incremental backup using dirty bitmaps. See: https://libvirt.org/kbase/incremental-backup.html

About the Author

Xcobean Systems Limited operates a production Apache CloudStack deployment providing IaaS to 50+ client VMs. We use the NAS backup provider daily and have contributed several improvements to it:

• PR #12805 -- NAS backup NPE fix
• PR #12822 -- Backup restore improvements
• PR #12826 -- NAS backup script hardening
• PRs #12843--#12848 -- Various NAS backup fixes
• PR #12872 -- Additional backup provider fixes

We experience the storage and bandwidth cost of full-only backups firsthand and are motivated to solve this problem upstream rather than maintaining a fork.

Open Questions for Discussion

We welcome feedback from the community on the following:

1. Interest level. Is there sufficient demand for this feature to justify the implementation effort? We believe so based on mailing list threads, but would like confirmation.

2. Dirty bitmaps vs. alternatives. Are there concerns about relying on QEMU dirty bitmaps? Alternative approaches include file-level deduplication on the NAS (less efficient, not hypervisor-aware) or qemu-img compare (slower, requires reading both images).

3. Target release. Should this target CloudStack 4.23, or is a later release more appropriate given the scope?

4. Chain model. We proposed forward-incremental with periodic full backups. Would the community prefer a different model (e.g., reverse-incremental like Veeam, or forever-incremental with periodic synthetic fulls)?

5. Scope of first PR. Should we submit the entire feature as one PR, or break it into smaller PRs (e.g., nasbackup.sh changes first, then agent, then management server, then UI)?

6. Testing infrastructure. We can test against our production environment (Ubuntu 22.04, QEMU 6.2, libvirt 8.0). Are there CI environments or community test labs available for broader testing (RHEL, Rocky, older QEMU versions)?

This RFC is posted as a GitHub Discussion to gather community feedback before implementation begins. Please share your thoughts, concerns, and suggestions.

Appendix: Related Proposal — CloudStack Infrastructure Backup to NAS

Problem

CloudStack's NAS backup provider only backs up VM disks. The management server database, agent configurations, SSL certificates, and global settings are not backed up. If the management server fails, all metadata is lost unless someone manually configured mysqldump.

Proposed Solution

Add a new scheduled task that automatically backs up CloudStack infrastructure to the same NAS backup storage.

What gets backed up:

Configuration:

• nas.infra.backup.enabled (global, default: false)
• nas.infra.backup.schedule (cron expression, default: 0 2 * * * — daily at 2am)
• nas.infra.backup.retention (number of backups to keep, default: 7)

Implementation:

• New class: InfrastructureBackupTask extending ManagedContextRunnable
• Runs on management server (not KVM agent)
• Uses existing NAS mount point from backup storage pool
• Creates timestamped directory: infra-backup/2026-03-27/
• Runs mysqldump --single-transaction for hot backup
• Tars config directories
• Manages retention (delete backups older than N days)
• Logs to CloudStack events for audit trail

Restore:

• Manual via CLI: mysql cloud < backup.sql + extract config tars
• Future: one-click restore from UI

This is a much simpler change (~200 lines of Java) that addresses a real operational gap. Could target 4.22.1 or 4.23.