Unable to delete failed Instance Snapshot on stopped KVM VM with RAW/RBD storage

[akoskuczi-bw]

problem

When using KVM with RBD primary storage and kvm.vmstoragesnapshot.enabled=true, an Instance Snapshot that fails during creation before the underlying storage snapshot is created cannot be deleted while the VM is stopped.

The failed Instance Snapshot remains in CloudStack in an error state. When CloudStack later attempts to delete the snapshot, the snapshot is handled by DefaultVMSnapshotStrategy instead of StorageVMSnapshotStrategy.

For a stopped VM backed by RAW/RBD storage, DeleteVMSnapshotCommand then fails because the libvirt domain does not exist:

Delete Instance Snapshot failed due to org.libvirt.LibvirtException:
Domain not found: no domain with matching name 'i-2-43-VM'

Starting the VM allows the same failed Instance Snapshot to be deleted successfully.

This appears to be a RAW/RBD variant of #11673. PR #11687 fixes a similar case for QCOW2/stopped VMs, but the fallback implemented there only handles QCOW2 volumes.

versions

Apache CloudStack: 4.22.1
Hypervisor: KVM
Primary storage: Ceph RBD
Volume format: RAW
Instance Snapshot type: Disk
kvm.vmstoragesnapshot.enabled=true

The steps to reproduce the bug

Deploy a KVM VM whose ROOT volume is stored on Ceph RBD.
Enable: kvm.vmstoragesnapshot.enabled=true
Run the VM without a working QEMU guest agent.
Create a disk-only Instance Snapshot with VM quiescing enabled.
Snapshot creation fails during the freeze operation: StorageVMSnapshotStrategy

Failed to frozen VM - i-2-43-VM due to Guest agent is not responding:
QEMU guest agent is not connected

Failed to create Instance Snapshot
The Instance Snapshot remains in CloudStack in a failed/error state.
Stop the VM.
Try to delete the failed Instance Snapshot, or destroy the VM so that CloudStack runs VmWorkDeleteAllVMSnapshots.

Actual result

CloudStack does not use StorageVMSnapshotStrategy to clean up the failed snapshot.

The deletion is handled by DefaultVMSnapshotStrategy:

KvmFileBasedStorageVmSnapshotStrategy ... cannot handle [...]
...
DeleteVMSnapshotCommand
...
DefaultVMSnapshotStrategy

The command sent to the KVM agent contains an RBD/RAW ROOT volume:

"poolType":"RBD"
"format":"RAW"
"volumeType":"ROOT"
"vmName":"i-2-43-VM"

The agent then fails because the stopped VM has no libvirt domain:

DeleteVMSnapshotAnswer:
result=false

Delete Instance Snapshot failed due to org.libvirt.LibvirtException:
Domain not found: no domain with matching name 'i-2-43-VM'

The VM destroy operation also fails because orchestrateDeleteAllVMSnapshots() cannot remove the failed Instance Snapshot:

CloudRuntimeException:
Failed to destroy vm with specified vmId

Caused by:
Delete Instance Snapshot ... failed due to
org.libvirt.LibvirtException:
Domain not found: no domain with matching name 'i-2-43-VM'

Expected result

A failed storage-based Instance Snapshot whose creation stopped before any underlying storage snapshot was created should be removable while the VM is stopped.

The cleanup should not require a libvirt domain to exist.

In particular, if:

the Instance Snapshot is in Error state,
it was created using the storage-based KVM snapshot strategy,
there are no kvmStorageSnapshot details / underlying volume snapshots to remove,

then deleting the Instance Snapshot should clean up the CloudStack metadata and return success.

VM destruction should also not be blocked by such a failed Instance Snapshot.

Additional observations

The Instance Snapshot creation originally uses StorageVMSnapshotStrategy:

StorageVMSnapshotStrategy
Freeze of virtual machine took 0 milliseconds.

Failed to create Instance Snapshot ...
Could not freeze Instance.
QEMU guest agent is not connected

The failure happens during the VM freeze operation, before the actual RBD snapshot creation.
As a result, there is no underlying RBD snapshot associated with the failed Instance Snapshot.

During a later delete, however, CloudStack falls back to DefaultVMSnapshotStrategy.
For stopped RAW/RBD VMs, the KVM LibvirtDeleteVMSnapshotCommandWrapper cannot use the QCOW2 offline fallback, because the ROOT volume format is RAW.

Workaround

Starting the VM creates the libvirt domain again.
After starting the VM, deleting the exact same failed Instance Snapshot succeeds:

DeleteVMSnapshotAnswer: result=true

No RBD snapshot is actually being deleted in this case. The presence of the libvirt domain merely allows the wrapper to determine that the requested libvirt snapshot does not exist and return success.

The current workaround is therefore:

Start VM

• delete failed Instance Snapshot
• stop/destroy VM

This should not be necessary for a failed storage-based Instance Snapshot.

What to do about it?

No response

[github-actions]

🎯 Triage report

Failed disk-only Instance Snapshots on stopped KVM VMs using RAW/RBD (Ceph) primary storage cannot be deleted, because deletion falls back to DefaultVMSnapshotStrategy and the KVM agent's LibvirtDeleteVMSnapshotCommandWrapper requires a live libvirt domain — which doesn't exist for a stopped VM. This also blocks VM destruction (orchestrateDeleteAllVMSnapshots() fails). A well-documented workaround (start VM, delete snapshot, stop/destroy VM) exists but shouldn't be necessary.

📊 Assessment

Dimension	Value	Reasoning
Type	type:bug	Clear defect: failed snapshot cleanup is blocked under specific conditions.
Component	component:kvm, component:ceph	Explicitly KVM hypervisor + Ceph RBD primary storage; RAW volume format.
Severity	Severity:Major	Blocks VM snapshot cleanup and can block VM destruction, but a workaround exists (start VM first), so not a full blocker.
Labels	type:bug, component:kvm, component:ceph, Severity:Major	See above.
Coding agent	Suitable	Well-defined scope: handle the case where a storage-based Instance Snapshot is in Error state with no underlying storage snapshot/kvmStorageSnapshot details, allowing metadata cleanup without requiring a libvirt domain. Clear reproduction steps and expected behavior provided.

🔗 Similar issues

• Reporter explicitly references Unable to delete Instance snapshot in "Error' state on stopped Instances #11673 and PR KVM: fix delete vm snapshot if it does not exist with a Stopped vm #11687 as a related/precedent case (same class of bug for QCOW2/stopped VMs; PR KVM: fix delete vm snapshot if it does not exist with a Stopped vm #11687's fallback only covers QCOW2, not RAW/RBD). No other duplicate or related open issues were found via search.

💡 Notes and suggestions

• Root cause: KvmFileBasedStorageVmSnapshotStrategy cannot handle RBD/RAW volumes, so deletion falls through to DefaultVMSnapshotStrategy, which dispatches DeleteVMSnapshotCommand to the KVM agent and requires a live libvirt domain.
• Suggested fix direction (per reporter): extend the check in the delete path so that when an Instance Snapshot is in Error state, was created via the storage-based (RBD) strategy, and has no associated storage snapshot / kvmStorageSnapshot details, the deletion short-circuits to a metadata-only cleanup (similar to the existing QCOW2 offline fallback added in KVM: fix delete vm snapshot if it does not exist with a Stopped vm #11687), without invoking DeleteVMSnapshotCommand against a non-existent libvirt domain.
• Also verify orchestrateDeleteAllVMSnapshots() (used during VM destroy) benefits from the same fix so VM destruction isn't blocked by such failed snapshots.
• Reproduction requires: KVM + Ceph RBD primary storage, kvm.vmstoragesnapshot.enabled=true, a VM without a functioning QEMU guest agent, and a disk-only Instance Snapshot with quiescing enabled to force a freeze failure.

Generated by Daily Issue Triage · sonnet50 98.8K · ◷

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9