Unable to remove vcenter/esxi host from cloudstack with force/clean if the vcenter is deleted externally

[kiranchavala]

problem

Unable to remove vcenter/esxi host from cloudstack with force/cleanly if the vcenter is deleted externally

versions

ACS 4.22

The steps to reproduce the bug

1. Set up Cloudstack env with a vcenter

2. Vcenter gets deleted externally

3. Admin user goes to the Ui > Zones > Remove vmware details from zone

Exception :

Logs

2026-05-18 06:08:03,827 DEBUG [c.c.a.ApiServlet] (qtp2038105753-25:[ctx-04dda2ab]) (logid:f6e52355) ===START===  10.0.3.251 -- POST
command=removeVmwareDc
response=json
zoneid=7696337c-f532-41a9-a37a-a974b6fff240
sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8

2026-05-18 06:08:03,827 DEBUG [c.c.a.ApiServlet] (qtp2038105753-25:[ctx-04dda2ab]) (logid:f6e52355) Two factor authentication is already verified for the user 2, so skipping
2026-05-18 06:08:03,832 DEBUG [c.c.a.ApiServer] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2026-05-18 06:08:03,834 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) Account for user id 97067b1a-4f5e-11f1-9840-1e00e50001f3 is Root Admin or Domain Admin, all APIs are allowed.
2026-05-18 06:08:03,834 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-05-18 06:08:03,834 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-05-18 06:08:03,840 WARN  [o.a.c.a.c.a.z.RemoveVmwareDcCmd] (qtp2038105753-25:[ctx-04dda2ab, ctx-4d64e8b9]) (logid:f6e52355) The zone has one or more resources (like cluster), hence not able to remove VMware datacenter from zone. Please remove all resource from zone, and retry. Exception: com.cloud.exception.ResourceInUseException: Zone has one or more clusters. Can't remove VMware datacenter to zone which already has clusters.
	at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.validateZoneWithResources(VmwareManagerImpl.java:1433)
	at com.cloud.hypervisor.vmware.manager.VmwareManagerImpl.removeVmwareDatacenter(VmwareManagerImpl.java:1350)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

4. Next admin tries to delete the host in the vmware cluster from cloudstack by keeping the host in maintenance mode. There is no option to force delete the host

logs

2026-05-18 06:14:03,175 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-c7171a5b]) (logid:3f7a543d) ===START===  10.0.3.251 -- POST
command=prepareHostForMaintenance
response=json
id=4a0c6597-86e8-4409-adfe-63970f27ca09
sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8

2026-05-18 06:14:03,175 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-c7171a5b]) (logid:3f7a543d) Two factor authentication is already verified for the user 2, so skipping
2026-05-18 06:14:03,179 DEBUG [c.c.a.ApiServer] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2026-05-18 06:14:03,180 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) Account for user id 97067b1a-4f5e-11f1-9840-1e00e50001f3 is Root Admin or Domain Admin, all APIs are allowed.
2026-05-18 06:14:03,180 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-05-18 06:14:03,180 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-05-18 06:14:03,193 DEBUG [c.c.a.ApiServer] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) Retrieved cmdEventType from job info: MAINT.PREPARE
2026-05-18 06:14:03,195 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-33:[ctx-1f2955e7, job-41]) (logid:1e20138f) Add job-41 into job monitoring
2026-05-18 06:14:03,197 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) submit async job-41, details: AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.host.PrepareForHostMaintenanceCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"QAZorHbuhisqM_pl6NxnR1wC3b8\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"154\",\"id\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"ctxDetails\":\"{\\\"interface com.cloud.host.Host\\\":\\\"4a0c6597-86e8-4409-adfe-63970f27ca09\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"cmdEventType\":\"MAINT.PREPARE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":41,"initMsid":32989190816243,"instanceId":1,"instanceType":"Host","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"8aa5c307-73fc-4cad-bd3b-5caaeafb483a"}
2026-05-18 06:14:03,198 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-c7171a5b, ctx-66699209]) (logid:3f7a543d) ===END===  10.0.3.251 -- POST
command=prepareHostForMaintenance
response=json
id=4a0c6597-86e8-4409-adfe-63970f27ca09
sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8

2026-05-18 06:14:03,198 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl$5] (API-Job-Executor-33:[ctx-1f2955e7, job-41]) (logid:8aa5c307) Executing AsyncJob {"accountId":2,"cmd":"org.apache.cloudstack.api.command.admin.host.PrepareForHostMaintenanceCmd","cmdInfo":"{\"response\":\"json\",\"ctxUserId\":\"2\",\"sessionkey\":\"QAZorHbuhisqM_pl6NxnR1wC3b8\",\"httpmethod\":\"POST\",\"ctxStartEventId\":\"154\",\"id\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"ctxDetails\":\"{\\\"interface com.cloud.host.Host\\\":\\\"4a0c6597-86e8-4409-adfe-63970f27ca09\\\"}\",\"ctxAccountId\":\"2\",\"uuid\":\"4a0c6597-86e8-4409-adfe-63970f27ca09\",\"cmdEventType\":\"MAINT.PREPARE\"}","cmdVersion":0,"completeMsid":null,"created":null,"id":41,"initMsid":32989190816243,"instanceId":1,"instanceType":"Host","lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"8aa5c307-73fc-4cad-bd3b-5caaeafb483a"}
2026-05-18 06:14:03,213 INFO  [c.c.r.ResourceManagerImpl] (API-Job-Executor-33:[ctx-1f2955e7, job-41, ctx-c137d8d5]) (logid:8aa5c307) Maintenance: attempting maintenance of host Host {"id":1,"name":"10.0.32.191","type":"Routing","uuid":"4a0c6597-86e8-4409-adfe-63970f27ca09"}
2026-05-18 06:14:03,216 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-33:[ctx-1f2955e7, job-41, ctx-c137d8d5]) (logid:8aa5c307) Can not send command com.cloud.agent.api.MaintainCommand due to Host Host {"id":1,"name":"10.0.32.191","type":"Routing","uuid":"4a0c6597-86e8-4409-adfe-63970f27ca09"} not being up
2026-05-18 06:14:03,216 WARN  [c.c.r.ResourceManagerImpl] (API-Job-Executor-33:[ctx-1f2955e7, job-41, ctx-c137d8d5]) (logid:8aa5c307) Unable to send MaintainCommand to host: Host {"id":1,"name":"10.0.32.191","type":"Routing","uuid":"4a0c6597-86e8-4409-adfe-63970f27ca09"}

5. Next admin tries to delete the primary storage associated with the VMware cluster

logs

2026-05-18 06:18:34,608 DEBUG [c.c.s.StorageManagerImpl] (qtp2038105753-23:[ctx-eef67450, ctx-7f6db4e9]) (logid:2da339a0) Cannot delete storage pool StoragePool {"id":1,"name":"ref-trl-11757-v-Mol9-harikrishna-patnala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"4ac4c575-ace8-391c-892a-b207bc1248c6"} as the following non-destroyed volumes are on it: [Volume [36d6c20e-33f6-4403-a4fb-64efa7e2cdde] (attached to VM [4195861e-b559-40fc-8829-61217d75d9da])].
2026-05-18 06:18:34,608 ERROR [c.c.a.ApiServer] (qtp2038105753-23:[ctx-eef67450, ctx-7f6db4e9]) (logid:2da339a0) unhandled exception executing api command: [Ljava.lang.String;@7c5be73a com.cloud.utils.exception.CloudRuntimeException: Cannot delete pool StoragePool {"id":1,"name":"ref-trl-11757-v-Mol9-harikrishna-patnala-esxi-pri1","poolType":"NetworkFilesystem","uuid":"4ac4c575-ace8-391c-892a-b207bc1248c6"} as there are non-destroyed volumes associated to this pool.
	at com.cloud.storage.StorageManagerImpl.deleteDataStoreInternal(StorageManagerImpl.java:1765)

6. Admin user tries to expunge the vm which is having volumes in the primary storage

logs

2026-05-18 06:20:16,910 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Cleaning storage for VM [VM instance {"id":3,"instanceName":"i-2-3-VM","state":"Expunging","type":"User","uuid":"4195861e-b559-40fc-8829-61217d75d9da"}].
2026-05-18 06:20:16,911 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Skipping destroy for the volume [{"name":"ROOT-3","uuid":"36d6c20e-33f6-4403-a4fb-64efa7e2cdde"}] as it is in [Destroy] state.
2026-05-18 06:20:16,931 DEBUG [c.c.h.XenServerGuru] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) We are returning the default host to execute commands because the command is not of Copy type.
2026-05-18 06:20:16,931 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Wait time setting on org.apache.cloudstack.storage.command.DeleteCommand is 1800 seconds
2026-05-18 06:20:16,932 DEBUG [o.a.c.s.RemoteHostEndPoint] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Failed to send command, due to Agent:2, com.cloud.exception.AgentUnavailableException: Resource [Host:2] is unreachable: Host 2: Host with specified id is not in the right state: Alert
2026-05-18 06:20:16,932 DEBUG [o.a.c.s.d.d.CloudStackPrimaryDataStoreDriverImpl] (API-Job-Executor-37:[ctx-2be7236d, job-49, ctx-c4244590]) (logid:d918c6d7) Unable to destroy volume [id: 3, uuid: 36d6c20e-33f6-4403-a4fb-64efa7e2cdde] com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:2, com.cloud.exception.AgentUnavailableException: Resource [Host:2] is unreachable: Host 2: Host with specified id is not in the right state: Alert
	at org.apache.cloudstack.storage.RemoteHostEndPoint.sendMessage(RemoteHostEndPoint.java:141)
	at org.apache.cloudstack.storage.datastore.driver.CloudStackPrimaryDataStoreDriverImpl.deleteAsync(CloudStackPrimaryDataStoreDriverImpl.java:260)
	at org.apache.cloudstack.storage.volume.VolumeServiceImpl.expungeVolumeAsync(VolumeServiceImpl.java:459)
	at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.cleanupVolumes(VolumeOrchestrator.java:1357)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)

7. User Tries to destroy the volume

Logs

2026-05-18 06:24:41,199 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-7f941028]) (logid:ffa4161f) ===START===  10.0.3.251 -- POST
command=deleteVolume
response=json
id=36d6c20e-33f6-4403-a4fb-64efa7e2cdde
sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8

2026-05-18 06:24:41,199 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-7f941028]) (logid:ffa4161f) Two factor authentication is already verified for the user 2, so skipping
2026-05-18 06:24:41,204 DEBUG [c.c.a.ApiServer] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2026-05-18 06:24:41,206 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) Account for user id 97067b1a-4f5e-11f1-9840-1e00e50001f3 is Root Admin or Domain Admin, all APIs are allowed.
2026-05-18 06:24:41,206 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2026-05-18 06:24:41,206 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) API rate limiting is disabled. We will not use ApiRateLimitService.
2026-05-18 06:24:41,213 DEBUG [c.c.u.AccountManagerImpl] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) Account [Account [{"accountName":"admin","id":2,"uuid":"970585ac-4f5e-11f1-9840-1e00e50001f3"}]] has access to resource.
2026-05-18 06:24:41,229 INFO  [c.c.a.ApiServer] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) Please specify a volume that is not attached to any VM.
2026-05-18 06:24:41,230 DEBUG [c.c.a.ApiServlet] (qtp2038105753-23:[ctx-7f941028, ctx-4d144df9]) (logid:ffa4161f) ===END===  10.0.3.251 -- POST
command=deleteVolume
response=json
id=36d6c20e-33f6-4403-a4fb-64efa7e2cdde
sessionkey=QAZorHbuhisqM_pl6NxnR1wC3b8

Admin user has to manually execute the DB queries to remove the vcenter details from the cloudstack zone

What to do about it?

Expected behaviour

Provide an option of force removing the vcenter details from cloudstack if the vcenter is destroyed externally and is unrecoverable