feat(roles/bootloader): add role

CHANGELOG.md

@@ -8,7 +8,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-**Highlights:** On RHEL 8, a MariaDB package upgrade no longer cuts applications on the same host off from their database. Apache no longer loads `mod_info`, which served the complete configuration including other modules' credentials. A broken PHP-FPM configuration aborts the run instead of taking the service down on the restart. Sudo rules deployed by `freeipa_server` can carry their commands again. The Bitwarden lookup can be told to abort instead of silently generating a new password, for runs against hosts whose credentials must already exist. The Grafana graph configuration for the Monitoring Plugins is no longer deployed on every ordinary run and has to be requested explicitly by its tag.
+**Highlights:** Kernel parameters that only take effect at boot time, `psi=1` among them, can be deployed from the inventory with the new `bootloader` role. On RHEL 8, a MariaDB package upgrade no longer cuts applications on the same host off from their database. Apache no longer loads `mod_info`, which served the complete configuration including other modules' credentials. A broken PHP-FPM configuration aborts the run instead of taking the service down on the restart. Sudo rules deployed by `freeipa_server` can carry their commands again. The Bitwarden lookup can be told to abort instead of silently generating a new password, for runs against hosts whose credentials must already exist. The Grafana graph configuration for the Monitoring Plugins is no longer deployed on every ordinary run and has to be requested explicitly by its tag.
 
 ### Breaking Changes
 
@@ -20,6 +20,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+* **role:bootloader**: New role that manages the kernel command line, for parameters that only take effect at boot time such as `psi=1`. Options are applied to every boot entry of the host, on the Red Hat family with `grubby` and on Debian and Ubuntu through a GRUB drop-in of its own. A changed command line requests a reboot at the maintenance window instead of rebooting right away, and a `--check` run reports what it would change without touching the host.
 * **role:files**: A file can opt out of the backup copy that is written before it is overwritten, via the `backup` subkey of `files__files__*_var`.
 * **role:collabora**: The `collabora:configure` tag deploys `coolwsd.xml` and the logrotate configuration without touching the packages.
 * **role:docker**: The address pools docker assigns container network subnets from (`default-address-pools`) can be configured.

COMPATIBILITY.md

@@ -15,6 +15,7 @@ Which Ansible role is proven to run on which OS?
 | audit                                 |        |        |   x    |   x    |   (x)   |           |           |           |                                              |
 | bind                                  |        |        |   x    |   x    |    x    |           |           |           |                                              |
 | blocky                                |        |        |   x    |   x    |   (x)   |           |           |           |                                              |
+| bootloader                            |  (x)   |   x    |   x    |   x    |    x    |    (x)    |    (x)    |    (x)    | Fedora 44                                    |
 | borg_local                            |        |        |   x    |  (x)   |   (x)   |           |           |           |                                              |
 | chromium_headless                     |   x    |  (x)   |   x    |   x    |    x    |           |           |           |                                              |
 | chrony                                |        |        |   x    |   x    |    x    |           |           |           |                                              |

extensions/molecule/bootloader/converge.yml

@@ -0,0 +1,2 @@
+- name: 'Converge bootloader playbook'
+  ansible.builtin.import_playbook: 'linuxfabrik.lfops.bootloader'

extensions/molecule/bootloader/inventory/group_vars/systems_under_test.yml

@@ -0,0 +1,3 @@
+bootloader__cmdline_options__group_var:
+  - name: 'psi'
+    value: 1

extensions/molecule/bootloader/inventory/hosts.yml

@@ -0,0 +1,15 @@
+# yamllint disable rule:empty-values
+lfops_bootloader:
+  children:
+    systems_under_test:
+
+systems_under_test:
+  hosts:
+    debian12-vm:
+    debian13-vm:
+    rocky8-vm:
+    rocky9-vm:
+    rocky10-vm:
+    ubuntu2204-vm:
+    ubuntu2404-vm:
+    ubuntu2604-vm:

extensions/molecule/bootloader/molecule.yml

@@ -0,0 +1 @@
+# Molecule scenario marker

extensions/molecule/bootloader/verify.yml

@@ -0,0 +1,88 @@
+# The kernel command line can only be observed on the running kernel, and it only changes on a
+# reboot. The scenario therefore reboots the host once, on the run where the option is still
+# missing from /proc/cmdline, and asserts the running kernel came up with it. verify runs twice
+# (after converge and after idempotence, see the test_sequence in config.yml); on the second run
+# the host already carries the option, so it neither expects a reboot request nor reboots again.
+- name: 'Verify the kernel command line carries the configured option'
+  hosts: 'systems_under_test'
+  tasks:
+
+    # Persistent state, visible before the reboot: on the Red Hat family the role writes the
+    # boot entries with grubby, on the Debian family update-grub regenerates grub.cfg.
+    - name: 'grubby --info=ALL'
+      ansible.builtin.command: 'grubby --info=ALL'
+      register: '__molecule__grubby_info_result'
+      changed_when: false
+      when:
+        - 'ansible_facts["os_family"] == "RedHat"'
+
+    - name: 'Assert that every boot entry has psi=1 on its command line'
+      ansible.builtin.assert:
+        that: '__molecule__grubby_info_result["stdout_lines"]
+            | select("match", "^args=")
+            | reject("search", "psi=1")
+            | list | length == 0'
+      when:
+        - 'ansible_facts["os_family"] == "RedHat"'
+
+    - name: 'grep "^[[:space:]]*linux" /boot/grub/grub.cfg'
+      ansible.builtin.command: 'grep --extended-regexp "^[[:space:]]*linux" /boot/grub/grub.cfg'
+      register: '__molecule__grub_cfg_linux_result'
+      changed_when: false
+      when:
+        - 'ansible_facts["os_family"] == "Debian"'
+
+    - name: 'Assert that every menu entry has psi=1 on its command line'
+      ansible.builtin.assert:
+        that: '__molecule__grub_cfg_linux_result["stdout_lines"]
+            | reject("search", "psi=1")
+            | list | length == 0'
+      when:
+        - 'ansible_facts["os_family"] == "Debian"'
+
+    - name: 'slurp /proc/cmdline'
+      ansible.builtin.slurp:
+        src: '/proc/cmdline'
+      register: '__molecule__proc_cmdline_result'
+
+    # Only true right after converge changed the boot loader configuration. The role requests the
+    # reboot instead of performing it, so the request has to be waiting in the schedule_reboot spool.
+    - name: 'Verify the pending reboot and apply it'
+      when: '"psi=1" not in (__molecule__proc_cmdline_result["content"] | ansible.builtin.b64decode)'
+      block:
+
+        - name: 'stat /run/schedule-reboot/bootloader'
+          ansible.builtin.stat:
+            path: '/run/schedule-reboot/bootloader'
+          register: '__molecule__reboot_request_stat_result'
+
+        - name: 'Assert that the role requested a reboot'
+          ansible.builtin.assert:
+            that: '__molecule__reboot_request_stat_result["stat"]["exists"] | bool'
+
+        - name: 'systemctl reboot'
+          ansible.builtin.reboot:  # yamllint disable-line rule:empty-values
+
+        - name: 'slurp /proc/cmdline'
+          ansible.builtin.slurp:
+            src: '/proc/cmdline'
+          register: '__molecule__proc_cmdline_result'
+
+    - name: 'Assert that the running kernel booted with psi=1'
+      ansible.builtin.assert:
+        that: '"psi=1" in (__molecule__proc_cmdline_result["content"] | ansible.builtin.b64decode)'
+
+    # The option is only worth setting if the kernel acts on it. RHEL 8 ships kernels without
+    # pressure stall information, so this is checked everywhere else.
+    - name: 'slurp /proc/pressure/cpu'
+      ansible.builtin.slurp:
+        src: '/proc/pressure/cpu'
+      register: '__molecule__proc_pressure_cpu_result'
+      when:
+        - 'not (ansible_facts["os_family"] == "RedHat" and ansible_facts["distribution_major_version"] | int < 9)'
+
+    - name: 'Assert that the kernel reports pressure stall information'
+      ansible.builtin.assert:
+        that: '(__molecule__proc_pressure_cpu_result["content"] | ansible.builtin.b64decode) is search("^some ")'
+      when:
+        - 'not (ansible_facts["os_family"] == "RedHat" and ansible_facts["distribution_major_version"] | int < 9)'

playbooks/README.md

@@ -99,6 +99,16 @@ Calls the following roles (in order):
 * [blocky](https://github.com/Linuxfabrik/lfops/tree/main/roles/blocky)
 
 
+## bootloader.yml
+
+Calls the following roles (in order):
+
+* [postfix](https://github.com/Linuxfabrik/lfops/tree/main/roles/postfix): `bootloader__skip_postfix`
+* [mailto_root](https://github.com/Linuxfabrik/lfops/tree/main/roles/mailto_root): `bootloader__skip_mailto_root`
+* [schedule_reboot](https://github.com/Linuxfabrik/lfops/tree/main/roles/schedule_reboot): `bootloader__skip_schedule_reboot`
+* [bootloader](https://github.com/Linuxfabrik/lfops/tree/main/roles/bootloader)
+
+
 ## borg_local.yml
 
 Calls the following roles (in order):

playbooks/all.yml

@@ -9,6 +9,7 @@
 - import_playbook: 'audit.yml'
 - import_playbook: 'bind.yml'
 - import_playbook: 'blocky.yml'
+- import_playbook: 'bootloader.yml'
 - import_playbook: 'borg_local.yml'
 - import_playbook: 'chromium_headless.yml'
 - import_playbook: 'chrony.yml'

playbooks/bootloader.yml

@@ -0,0 +1,49 @@
+- name: 'Playbook linuxfabrik.lfops.bootloader'
+  hosts:
+    - 'lfops_bootloader'
+
+  pre_tasks:
+    - ansible.builtin.import_role:
+        name: 'shared'
+        tasks_from: 'log-start.yml'
+      tags:
+        - 'always'
+
+    - ansible.builtin.import_role:
+        name: 'shared'
+        tasks_from: 'global-variables.yml'
+      tags:
+        - 'always'
+
+
+  roles:
+
+    - role: 'linuxfabrik.lfops.postfix'
+      postfix__aliases__dependent_var: '{{
+          mailto_root__postfix__aliases__dependent_var
+        }}'
+      postfix__sender_canonicals__dependent_var: '{{
+          mailto_root__postfix__sender_canonicals__dependent_var
+        }}'
+      when:
+        - 'not bootloader__skip_postfix | d(false)'
+
+    - role: 'linuxfabrik.lfops.mailto_root'
+      when:
+        - 'not bootloader__skip_mailto_root | d(false)'
+
+    # deployed before the bootloader role, so the schedule-reboot command is in place when a
+    # changed kernel command line requests a reboot
+    - role: 'linuxfabrik.lfops.schedule_reboot'
+      when:
+        - 'not bootloader__skip_schedule_reboot | d(false)'
+
+    - role: 'linuxfabrik.lfops.bootloader'
+
+
+  post_tasks:
+    - ansible.builtin.import_role:
+        name: 'shared'
+        tasks_from: 'log-end.yml'
+      tags:
+        - 'always'

roles/bootloader/README.md

@@ -0,0 +1,113 @@
+# Ansible Role linuxfabrik.lfops.bootloader
+
+This role manages the kernel command line of a host, for parameters that only take effect at boot time.
+
+On the Red Hat family the boot entries are written with `grubby`. Debian and Ubuntu do not package `grubby`, so there the role deploys a GRUB drop-in of its own and regenerates the boot loader configuration.
+
+
+*Available in the next LFOps release.*
+
+
+## How the Role Behaves
+
+* Red Hat family: options are applied to every boot entry of the host (`grubby --update-kernel=ALL`), so the running kernel and every kernel still installed alongside it carry the same command line. `grubby` keeps `GRUB_CMDLINE_LINUX` in `/etc/default/grub` in sync while doing so, appending only the managed options and leaving the rest of the file alone. Nothing else in `/etc/default/grub` and nothing in `grub.cfg` is touched.
+* Debian family: the options are written to `/etc/default/grub.d/z00-lfops.cfg` and `update-grub` regenerates `/boot/grub/grub.cfg` from it. `grub-mkconfig` sources `/etc/default/grub` first and every `/etc/default/grub.d/*.cfg` after it, so the drop-in wins without the packaged configuration file ever being edited, and it appends to whatever `GRUB_CMDLINE_LINUX` already holds instead of replacing it (the sourcing order was read from the `grub-mkconfig` of grub-common 2.12-9+deb13u2 on Debian 13, 2.12-1ubuntu7.3 on Ubuntu 24.04 and 2.14-2ubuntu2.1 on Ubuntu 26.04). Once no option is left to set, the drop-in is removed instead of being left behind empty.
+* A configured option counts as present only when **every** boot entry carries it, and it is compared as a whole word with the option escaped, so an option containing a dot matches a dot.
+* A run against a host that already carries the configured command line changes nothing and reports no change, and it neither requests a reboot nor touches any file. Changing the value of an option that is already set replaces it rather than adding a second one.
+* `--check` changes nothing. The dry run reads the current boot entries and reports what it would add or remove.
+* The change only takes effect on the next boot. When the [schedule_reboot](https://github.com/Linuxfabrik/lfops/tree/main/roles/schedule_reboot) mechanism is deployed, a changed command line requests a reboot at the next maintenance window (spool entry `bootloader`). Without it, the role only prints a message and leaves the reboot to the operator.
+* On the Red Hat family a kernel installed later inherits the command line from the running kernel. `kernel-install` builds the boot entry of a new kernel from `/etc/kernel/cmdline`, from `/usr/lib/kernel/cmdline`, or, when neither exists, from `/proc/cmdline` of the running kernel (verified against `/usr/lib/kernel/install.d/20-grub.install` on Rocky 9). A kernel installed between the change and the reboot therefore still comes up without the new options; run the role again afterwards. On the Debian family this cannot happen, because installing a kernel regenerates `/boot/grub/grub.cfg` from the drop-in.
+* The role manages the kernel command line only. It does not add, remove or reorder boot entries, does not change the boot loader timeout, and does not manage the GRUB password.
+
+
+## Known Limitations
+
+* GRUB 2 only. Hosts booted by zipl or systemd-boot are not supported.
+* Debian family: `state: 'absent'` only drops an option from the command line this role writes. An option that comes from `/etc/default/grub` or from another drop-in stays, because the role never edits files it does not own. On the Red Hat family the same option is removed with `grubby --remove-args`.
+
+
+## Dependent Roles
+
+Any [LFOps playbook](https://github.com/Linuxfabrik/lfops/blob/main/playbooks/README.md) that installs this role runs these for you. Optional ones can be disabled via the playbook's skip variables.
+
+* Optional: the reboot mechanism should be in place (role: [linuxfabrik.lfops.schedule_reboot](https://github.com/Linuxfabrik/lfops/tree/main/roles/schedule_reboot)), so a changed kernel command line reboots the host at the maintenance window instead of waiting for a manual reboot.
+
+
+## Requirements
+
+* The host is booted by GRUB 2.
+* Red Hat family: `grubby` is installed. It is part of every GRUB installation there, since `kernel-install` relies on it.
+* Debian family: `grub2-common` is installed. It provides `update-grub`, which the role calls.
+
+
+## Tags
+
+`bootloader`
+
+* Configures the kernel command line.
+* Requests a reboot when the kernel command line changed.
+* Triggers: none.
+
+
+## Optional Role Variables
+
+These variables are intended to be used in a host / group variable file in the Ansible inventory. Note that the group variable can only be used in one group at a time.
+
+`bootloader__cmdline_options__host_var` / `bootloader__cmdline_options__group_var`
+
+* Kernel command line options. An option that is already present with a different value is overwritten. On the Debian family the options end up in `GRUB_CMDLINE_LINUX`, so they apply to the recovery entries as well.
+* Type: List of dictionaries.
+* Default: `[]`
+* Subkeys:
+
+    * `name`:
+
+        * Mandatory. Name of the option, for example `psi`.
+        * Type: String.
+
+    * `value`:
+
+        * Optional. Value of the option. Omit it for options that stand on their own, for example `quiet`. Quote a value YAML reads as a boolean, `'on'` and `'off'` among them, otherwise it reaches the command line as `True` or `False`.
+        * Type: String or Number.
+
+    * `state`:
+
+        * Optional. Whether the option is added to or removed from the kernel command line. One of `present` or `absent`. On the Debian family see "Known Limitations".
+        * Type: String.
+        * Default: `'present'`
+
+Example:
+```yaml
+# optional
+bootloader__cmdline_options__group_var:
+  - name: 'psi'
+    value: 1
+  - name: 'quiet'
+  - name: 'nosmt'
+    state: 'absent'
+```
+
+
+## Troubleshooting
+
+**The option is configured, but `/proc/cmdline` does not contain it**
+
+* The host has not been rebooted since the change. Check the boot entries with `grubby --info=ALL` respectively `grep linux /boot/grub/grub.cfg`; they carry the new command line right away, `/proc/cmdline` only after the reboot.
+
+**On a Red Hat-family host a newly installed kernel boots without the configured options**
+
+* The kernel was installed while the change was still pending a reboot, so it inherited the command line of the running kernel. Run the role again to update the entry of the new kernel.
+
+**On a Debian-family host an option is still on the command line although it is set to `state: 'absent'`**
+
+* The option comes from `/etc/default/grub` or from another drop-in in `/etc/default/grub.d/`, which this role does not touch. Remove it there.
+
+
+## License
+
+[The Unlicense](https://unlicense.org/)
+
+
+## Author Information
+
+[Linuxfabrik GmbH, Zurich](https://www.linuxfabrik.ch)

roles/bootloader/defaults/main.yml

@@ -0,0 +1,12 @@
+bootloader__cmdline_options__combined_var: '{{ (
+      bootloader__cmdline_options__role_var +
+      bootloader__cmdline_options__dependent_var +
+      bootloader__cmdline_options__group_var +
+      bootloader__cmdline_options__host_var
+    ) | linuxfabrik.lfops.combine_lod
+  }}'
+
+bootloader__cmdline_options__dependent_var: []
+bootloader__cmdline_options__group_var: []
+bootloader__cmdline_options__host_var: []
+bootloader__cmdline_options__role_var: []

roles/bootloader/meta/argument_specs.yml

@@ -0,0 +1,30 @@
+argument_specs:
+  main:
+    options:
+
+      bootloader__cmdline_options__dependent_var:
+        type: 'list'
+        elements: 'dict'
+        required: false
+        default: []
+        description: >-
+          Kernel command line options to apply (`name`/`value`/`state`).
+          Dependent-role injection.
+
+      bootloader__cmdline_options__group_var:
+        type: 'list'
+        elements: 'dict'
+        required: false
+        default: []
+        description: >-
+          Kernel command line options to apply (`name`/`value`/`state`).
+          Group-level override.
+
+      bootloader__cmdline_options__host_var:
+        type: 'list'
+        elements: 'dict'
+        required: false
+        default: []
+        description: >-
+          Kernel command line options to apply (`name`/`value`/`state`).
+          Host-level override.