Repository navigation
Degraded cloudstack agent #11141
Description
Activity
Thanks for opening your first issue here! Be sure to follow the issue template!
I have a similar problem, with other symptoms as well. For example, alerts in Cloudstack like "Health checks failed" for virtual routers on affected hosts. The agent is constantly consuming 100% CPU, even when there are no jobs or any current actions on that host.
@poli1025 is degradation specific only for the agent or is there overall slowness in handling start-stop API calls? (So we can rule out any MS issue)
You mention bulk operations. Can you please give some insight on that?
In the agent logs, do you see any related logs, like some stuck task of a background job giving an error?
Would ut be possible for you provide logs and may be heap dump to analyze this?We’ve identified that the issue is specifically related to the agent — there is no noticeable delay in the API calls.
We haven’t found any errors in the logs, but we are observing that the agent is progressively taking longer to perform the required actions.The bulk operations are being executed via Terraform, such as the creation of 50 machines. The next day, when we destroy and recreate them, the process takes significantly more time. However, if we restart the agent, the performance returns to normal. Over time, the degradation happens again, and the creation time can eventually triple.
2025-07-28 22:21:59,724 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-51:[]) (logid:) Trying to fetch storage pool 0f79bc6b-d365-3005-a4b7-8a41d336c6c6 from libvirt 2025-07-28 22:21:59,726 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-51:[]) (logid:) Asking libvirt to refresh storage pool 0f79bc6b-d365-3005-a4b7-8a41d336c6c6 2025-07-28 22:23:00,459 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-62:[]) (logid:) Trying to fetch storage pool 0f79bc6b-d365-3005-a4b7-8a41d336c6c6 from libvirt 2025-07-28 22:23:00,460 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-62:[]) (logid:) Asking libvirt to refresh storage pool 0f79bc6b-d365-3005-a4b7-8a41d336c6c6 2025-07-28 22:23:00,863 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-60:[]) (logid:) Trying to fetch storage pool 00b05d76-e83c-377e-9005-32f9a1d18679 from libvirt 2025-07-28 22:23:00,864 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-60:[]) (logid:) Asking libvirt to refresh storage pool 00b05d76-e83c-377e-9005-32f9a1d18679 2025-07-28 22:23:00,918 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-65:[]) (logid:) Trying to fetch storage pool e65cc751-30c7-3404-8932-d9d0c1577411 from libvirt 2025-07-28 22:23:00,919 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-65:[]) (logid:) Asking libvirt to refresh storage pool e65cc751-30c7-3404-8932-d9d0c1577411 2025-07-28 22:23:01,420 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-64:[]) (logid:) Trying to fetch storage pool 6aa94ea7-c6a7-344c-a11f-9a00654e1c23 from libvirt 2025-07-28 22:23:01,421 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-64:[]) (logid:) Asking libvirt to refresh storage pool 6aa94ea7-c6a7-344c-a11f-9a00654e1c23 2025-07-28 22:23:01,824 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-67:[]) (logid:) Trying to fetch storage pool 149ab1fd-3757-3bde-b359-60f522b8a17c from libvirt 2025-07-28 22:23:01,825 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-67:[]) (logid:) Asking libvirt to refresh storage pool 149ab1fd-3757-3bde-b359-60f522b8a17c 2025-07-28 22:24:01,665 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-55:[]) (logid:) Trying to fetch storage pool 00b05d76-e83c-377e-9005-32f9a1d18679 from libvirt 2025-07-28 22:24:01,667 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-55:[]) (logid:) Asking libvirt to refresh storage pool 00b05d76-e83c-377e-9005-32f9a1d18679 2025-07-28 22:24:02,026 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-68:[]) (logid:) Trying to fetch storage pool 0f79bc6b-d365-3005-a4b7-8a41d336c6c6 from libvirt 2025-07-28 22:24:02,028 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-68:[]) (logid:) Asking libvirt to refresh storage pool 0f79bc6b-d365-3005-a4b7-8a41d336c6c6 2025-07-28 22:24:02,092 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-75:[]) (logid:) Trying to fetch storage pool 5d8d05bd-451b-33be-ab65-b9d8e3f02528 from libvirt 2025-07-28 22:24:02,094 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-75:[]) (logid:) Asking libvirt to refresh storage pool 5d8d05bd-451b-33be-ab65-b9d8e3f02528 2025-07-28 22:24:02,205 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-61:[]) (logid:) Trying to fetch storage pool 6aa94ea7-c6a7-344c-a11f-9a00654e1c23 from libvirt 2025-07-28 22:24:02,207 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-61:[]) (logid:) Asking libvirt to refresh storage pool 6aa94ea7-c6a7-344c-a11f-9a00654e1c23 2025-07-28 22:24:02,312 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-70:[]) (logid:) Trying to fetch storage pool cacc288a-c318-3b0e-858d-1293480acec6 from libvirt 2025-07-28 22:24:02,313 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-70:[]) (logid:) Asking libvirt to refresh storage pool cacc288a-c318-3b0e-858d-1293480acec6 2025-07-28 22:24:02,492 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-72:[]) (logid:) Trying to fetch storage pool 00b05d76-e83c-377e-9005-32f9a1d18679 from libvirt 2025-07-28 22:24:02,493 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-72:[]) (logid:) Asking libvirt to refresh storage pool 00b05d76-e83c-377e-9005-32f9a1d18679 2025-07-28 22:24:02,543 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-73:[]) (logid:) Trying to fetch storage pool e65cc751-30c7-3404-8932-d9d0c1577411 from libvirt 2025-07-28 22:24:02,544 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-73:[]) (logid:) Asking libvirt to refresh storage pool e65cc751-30c7-3404-8932-d9d0c1577411 2025-07-28 22:25:02,455 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-83:[]) (logid:) Trying to fetch storage pool dec8139e-7163-3ea4-b488-cfce1f138e58 from libvirt 2025-07-28 22:25:02,456 INFO [kvm.storage.LibvirtStorageAdaptor] (AgentRequest-Handler-83:[]) (logid:) Asking libvirt to refresh storage pool dec8139e-7163-3ea4-b488-cfce1f138e58I created 50 vm instances in parallel, the deployments and agents worked well
need more testing and investigation
@weizhouapache The initial deployments and agents work well. However, after a few days we observe degradation: VM creation, power cycling, and console access start to slow down.
The problem seems to be a leak in the handler threads while checking storage usage. The more agent threads you configure in agent.properties and the less time you configure to retrieve volume usage metrics, the worse it gets, and the faster it happens.
On a fresh start of the agent, you get the 'Trying to fetch storage pool xxxx from libvirt' message whenever the usage service is getting updated metrics. Those requests are either leaking or not getting garbage collected or something like that in time. Those requests start to overlap with time, and you end up seeing the same request to the same primary storage tens or hundreds of times. The only way to recover from that is to restart the agent, limit the number of threads of the agent and try to read the usage metrics in longer time spans (I think it defaults to 10 minutes or something like that, setting it to once every two hours mitigates it a bit, just enough so you don't have to restart the agent every few hours so it doesn't hog the kvm node cpu).
Here's a log with redacted storage uuids so it's easier to see (take a look at the timestamps)
storage-log-no-uuids.logThis happens at least since Cloudstack 4.19
thanks for the information @vgarcia-linube
68 remaining items
Cloudstack doesn't really advertise Debian as a first class citizen of a distro as it doesn't list community packages: https://cloudstack.apache.org/downloads/#community-packages
Interestingly, I am at least seeing bookworm (debian 12) packages here:
https://download.cloudstack.org/debian/dists/bookworm/It would be good to get trixie in there and make them more official by referencing them in the download page. I probably would have gone Debian on a deployment if I thought it was supported at the same level as Ubuntu.
I dont know how widespread debian is as platform for acs, this might need testing.
The Debian 12 life cycle encompasses five years: the initial three years of full Debian support, until June 10th, 2026, and two years of Long Term Support (LTS), until June 30th, 2028
Bookworm has libvirt 9.0.0, Trixie is using 11.3.0. Thats a huge version step, none of the EL is on 11.x yet afaik. I will try to find time adding a Debian 13 host in our dev environment.
@bhouse-nexthop Shapeblue has generic marked Debian / Ubuntu packages https://www.shapeblue.com/cloudstack-packages/
@DaanHoogland why tempted?
shear prejudice
Reacted by JuergenI am testing Debian 13 since yesterday with automated high concurrent load tests around the instance and volume lifecycle, so far no signs of problems. We will probably move on to Debian.
Reacted by dahnhey folks, I bring good news. I managed to identify where the big is and is not on
libvirtdversions we are bisecting:- v10.0 bug confirmed (unhealthy)
- v10.2 bug confirmed (unhealthy)
- v10.4 bug confirmed (unhealthy)
- v10.6 bug not present (healthy)
FYI, @cpaelzer
Reacted by Juergen and Tadios Abebehmm, the changeset between 10.4 and 10.6 isn't exactly small, don't know how easy it would be to identify the actual commit that fixes this: libvirt/libvirt@v10.4.0...v10.6.0
I can confirm that 11.3 also works perfectly fine
hmm, the changeset between 10.4 and 10.6 isn't exactly small, don't know how easy it would be to identify the actual commit that fixes this: libvirt/libvirt@v10.4.0...v10.6.0
So far, the plan is to dissect even more. I have increased my testing environment from 3 to 5 KVM hosts and I have 2 more on the way. This might speed up the process to identify when the change that fixed the bug.
After two weeks of operation in the production environment, I can confirm that the agent with libvirt-10.6.0-1ubuntu3.3 behaves correctly without any apparent problems. The CPU load for the agent process still ranges from 0.x% to 1.8%.
For now, we will have libvirt-10.6.0-1ubuntu3.3 in production as a temporary solution. If the bug fix is backported, we will return (after testing) to the original libvirt from the Ubuntu distribution. If the bug is not fixed, we will remain on libvirt-10.6.0-1ubuntu3.3 until the major upgrade to Ubuntu 26.04 (depending on compatibility with CloudStack).
- locked and limited conversation to collaborators
on Jan 16, 2026 - moved this from Discuss to Done in Apache CloudStack BugFest - Issues
on Jan 16, 2026
problem
Over time, the CloudStack agent becomes increasingly slow when starting or stopping virtual machines. This performance degradation is especially noticeable during bulk VM operations, where the delays can become significant. The only effective workaround I've found so far is to restart the CloudStack agent service, which temporarily restores normal performance.
versions
the version is 4.20.1
Ubuntu 24
kvm
The steps to reproduce the bug
What to do about it?