Management server hangs when js.interpretation.enabled=true

problem

Enabling the global setting js.interpretation.enabled=true causes the CloudStack management server to hang during startup at the module loading phase. This prevents using the secondary storage selector feature (createSecondaryStorageSelector, updateSecondaryStorageSelector APIs).

versions

4.20 branch (4.20.3.0-SNAPSHOT) - AFFECTED - needs fix before 4.20.3.0 release
4.20.2.0 (released) - Not affected (commit not present, setting does not exist, APIs work unconditionally)

Introduced by commit 03a4b9f ("server,utils: improve js interpretation functionality") merged on Oct 16, 2025 into the 4.20 branch.

The commit:

Added js.interpretation.enabled as a hidden ConfigKey (default: false)
Made CreateSecondaryStorageSelectorCmd and UpdateSecondaryStorageSelectorCmd conditional on this setting
Made significant changes to JsInterpreter.java (security hardening, daemon workers, etc.)

When the setting is enabled, something in the initialization causes the MS server to hang.

The steps to reproduce the bug

Deploy CloudStack 4.20.3.0
Verify server is running:

   curl -s "http://localhost:8080/client/api?command=listZones" | head -5

Enable the js.interpretation.enabled setting:

   mysql -u root -p cloud -e "UPDATE configuration SET value='true' WHERE name='js.interpretation.enabled';"

Restart management server:

   systemctl restart cloudstack-management

Wait 3+ minutes and check server status:

   curl -s "http://localhost:8080/client/api?command=listZones" | head -5

Expected Result

Server starts successfully and createSecondaryStorageSelector API is available.

Actual Result

Server returns 503 Service Unavailable. Log shows server stuck at module loading phase:

2026-01-26 19:53:01,937 DEBUG [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Trying to obtain module [redfish] context.
2026-01-26 19:53:01,937 WARN  [o.a.c.s.m.m.i.DefaultModuleDefinitionSet] (main:[]) (logid:) Application context not found for module definition [redfish]

Log timestamp remains static while current time progresses - server is hung.

Evidence

4.20 branch build (shapeblue18197) - Bug present:

[root@ref-trl-10730-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# rpm -qa | grep cloudstack
cloudstack-common-4.20.3.0-shapeblue18197.noarch
cloudstack-management-4.20.3.0-shapeblue18197.noarch

[root@ref-trl-10730-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# mysql -u root -p cloud -e "UPDATE configuration SET value='true' WHERE name='js.interpretation.enabled';"
[root@ref-trl-10730-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# systemctl restart cloudstack-management
[root@ref-trl-10730-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# sleep 180
[root@ref-trl-10730-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# curl -s "http://localhost:8080/client/api?command=listZones" | head -5
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 503 Service Unavailable</title>
</head>

4.20.2.0 released version (shapeblue17631) - Works correctly:

[root@ref-trl-10714-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# rpm -qa | grep cloudstack
cloudstack-common-4.20.2.0-shapeblue17631.noarch
cloudstack-management-4.20.2.0-shapeblue17631.noarch

mysql> SELECT name, value FROM configuration WHERE name LIKE '%interpret%';
Empty set (0.00 sec)

[root@ref-trl-10714-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# cmk list apis 2>/dev/null | grep -i "name.*selector"
      "name": "createSecondaryStorageSelector",
      "name": "updateSecondaryStorageSelector",
      "name": "listSecondaryStorageSelectors",
      "name": "removeSecondaryStorageSelector",

Impact

Secondary storage selector feature is unusable on current 4.20 branch
Quota tariff activation rules may also be affected (uses same JsInterpreter)
Blocks testing of PR Consider secondary storage selectors during cold volume migration #10957 and any other functionality that depends on this feature

What to do about it?

No response

Update

The root cause has been identified. The js.interpretation.enabled setting requires an encrypted value in the database, not plain text.

Workaround:

Get encryption key: cat /etc/cloudstack/management/key
Encrypt the value: java -classpath /usr/share/cloudstack-common/lib/cloudstack-utils.jar com.cloud.utils.crypt.EncryptionCLI -p <key> -i true
Update with encrypted value: UPDATE configuration SET value='<encrypted>' WHERE name='js.interpretation.enabled';

Remaining issues:

Server hangs silently instead of showing a clear error when decryption fails
No documentation that this hidden setting requires encryption
Consider adding validation/error handling for encrypted config values

Could be closed as user error, but suggesting improvement to error handling.

@winterhazel

@winterhazel , any thoughts?

@DaanHoogland

@winterhazel , any thoughts?

@DaanHoogland regarding the startup failure, it is the behavior I find correct for this situation. There should also be a clear message in the logs informing that there was an issue while decrypting this configuration's value, so that operators know what to look into.

I have some thoughts about js.interpretation.enabled though. I do not have access to the discussion regarding the CVE that prompted the introduction of this setting to know why it was handled that way. However, I think that it should not have been made a hidden setting with an encrypted value, and should be enabled by default. The vulnerability was fixed as far as I am aware; also, the APIs that allow configuring scripts (host, Quota tariff, and secondary storage selector configuration) should only be accessible to people with access to the infrastructure. Hence, if a new vulnerability with the interpreter gets discovered and exploited, that is not an issue with the platform, but internal permission granting issues. Other features that come enabled by default may have vulnerabilities that we are not aware of yet, but that's not a reason for we to disable them by default. Having it as a hidden encrypted setting just makes it unnecessarily difficult for operators to use the features.

@winterhazel

3. Consider adding validation/error handling for encrypted config values

in short @winterhazel , you would agree with ^ this? Should be a simple PR. I’ll put it on my list

@DaanHoogland

@DaanHoogland yes, I agree with @RosiKyu's suggestions in 1 and 3.

2 not so much as I think that it should not be hidden at all.

@winterhazel

@winterhazel @RosiKyu cc @DaanHoogland The idea behind keeping some settings hidden is to not allow root admin accounts to change them. Only a privileged user having access to DB would be able to change them.
Incidentally all hidden settings are encrypted currently which can be documented.

@shwstppr

@winterhazel @RosiKyu cc @DaanHoogland The idea behind keeping some settings hidden is to not allow root admin accounts to change them. Only a privileged user having access to DB would be able to change them. Incidentally all hidden settings are encrypted currently which can be documented.

@shwstppr ok, but my point is that js.interpretation.enabled should not be kept hidden. Is there a reason for restricting root admin accounts, which can already do anything they want with all resources in the environment as long as they receive API permission, from enabling a feature that only they will be able to use?

@winterhazel

@winterhazel while in most cases ROOT admin will have access to the system as well (underlying server), but there can be cases when ROOT admin is just the CloudStack admin. In those cases, a non-hidden config can be changed by this CloudStack admin and the system can be under security risk as highlighted by the CVE for which this setting was introduced. There could be better means to alter such configs, but for no,w ACS provides only the hidden configs

@shwstppr

@shwstppr as I commented in #12523 (comment): the known vulnerabilities pertaining the JS were patched, weren't they? Other features that come enabled by default almost certainly have vulnerabilities that we are not aware of yet, but we cannot put them behind a hidden setting due to a hypothetical security issue. If we were consistent with how the JS interpretation was handled, then we would have to also disable by default and place behind a hidden setting features like direct download, volume/template upload and register, which had CVEs in the past, and all other functionalities that uses administrator/user input in a shell/python script.

The extraconfig feature could also be used to abuse the environment. Following your logic, root admins should not be able to see/change this configuration, yet the configurations related to this feature are not hidden.

@winterhazel

@winterhazel it was not my logic alone. It was a PMC decision. I was just explaining the case. If you're not happy with it please raise with it.
Also, you may create a change PR to make it a regular config and get it merged ✌️

@shwstppr

@shwstppr @winterhazel

Reviewing the private discussion, I see that the fix started with the config as hidden, and it was mentioned that we could discuss it, but we never had the actual discussion about whether it should or should not be hidden (perhaps because of the urgency to release the fixes or lack of time/attention; water under the bridge). Now that we no longer have such urgency, it is a good opportunity to discuss whether it makes sense to change it or keep it as it is.

@winterhazel, if you will, create a PR with your proposal so we can move forward with the discussion.

@RosiKyu

@RosiKyu I read your comments and then the discussions following. What in you opinion is to be done (in order of priority)?

@winterhazel same question to you..

@DaanHoogland

@DaanHoogland I think that the startup failure when an encrypted setting has a decrypted value in the database is the correct behavior. This indicates to operators that there is something wrong that they need to check. We just need to improve the logging when this failure happens. Maybe something like:

We expect the value of setting '<name>' to be encrypted in the database with the Management Server's key, but we were unable to decrypt it using this key. This issue may happen when the value was manually changed in the database to a plain decrypted value. For reference on how to change the value of encrypted settings through the database, see <URL to the docuementation>.'

🎯 Triage report

When js.interpretation.enabled is set to a plain-text true value in the database (rather than the required encrypted form), the management server hangs silently during startup with no actionable error message. The root cause is user configuration error — the setting is a hidden encrypted key that must be set via EncryptionCLI — but the actual bug is that the server hangs indefinitely instead of failing fast with a clear diagnostic. Community consensus is to improve error handling for misconfigured encrypted settings.

📊 Assessment

Dimension	Value	Reasoning
Type	`type:bug`	Server hangs silently rather than emitting a clear error for misconfigured encrypted settings
Component	`component:management-server`	Startup/config loading phase
Severity	`Severity:Minor`	Setting is not commonly used; workaround is documented in issue comments
Labels	`type:bug`, `component:management-server`, `Severity:Minor`	See above
Coding agent	Suitable	Narrow, well-scoped fix: emit a clear error message when config decryption fails at startup instead of hanging

💡 Notes and suggestions

Workaround (documented in comments):

Get encryption key: cat /etc/cloudstack/management/key
Encrypt the value: java -classpath /usr/share/cloudstack-common/lib/cloudstack-utils.jar com.cloud.utils.crypt.EncryptionCLI -p <key> -i true
Update DB with the encrypted value.

Fix suggestion: In the configuration loading/decryption code, catch EncryptionException / decryption failure and emit a log message such as:

"Failed to decrypt value for setting 'js.interpretation.enabled'. This setting requires an encrypted value set via EncryptionCLI. See documentation at (URL)."

Then fail fast with a clear startup error rather than hanging.

Related: PR #12605 proposes making js.interpretation.enabled a regular (non-hidden) configuration.

Generated by Daily Issue Triage · ◷

Add this agentic workflows to your repo

To install this agentic workflow, run

gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9

This makes it really painful to enable js.interpretation.enabled. There are legitimate reasons to use this flag.

Navigation Menu

Repository navigation

Management server hangs when js.interpretation.enabled=true #12523

problem

versions

The steps to reproduce the bug

Expected Result

Actual Result

Evidence

Impact

Related

What to do about it?

Update

3 remaining items

🎯 Triage report

📊 Assessment

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Management server hangs when js.interpretation.enabled=true #12523

Description

problem

versions

The steps to reproduce the bug

Expected Result

Actual Result

Evidence

Impact

Related

What to do about it?

Activity

RosiKyu commented on Jan 26, 2026

Update

DaanHoogland commented on Jan 27, 2026

winterhazel commented on Jan 28, 2026

DaanHoogland commented on Jan 28, 2026

winterhazel commented on Feb 2, 2026

shwstppr commented on Feb 2, 2026

winterhazel commented on Feb 4, 2026

shwstppr commented on Feb 5, 2026

winterhazel commented on Feb 5, 2026

shwstppr commented on Feb 5, 2026

GutoVeronezi commented on Feb 5, 2026

3 remaining items

DaanHoogland commented on Mar 3, 2026

winterhazel commented on Apr 10, 2026

github-actions commented on Jul 2, 2026

🎯 Triage report

📊 Assessment

scottsignal commented on Sep 9, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions