composefs deployment path discovery

[jmpolom]

An installer wrapping bootc install to-filesystem --composefs-backend may need to write machine-specific configuration into the installed deployment's /etc before the first boot. This is an ordinary and extremely common installer function that many would consider bare minimum functionality. For composefs, the /etc path is generally of the form <target>/state/deploy/<deployment-id>/etc. Bootc computes the deployment ID and creates this directory, but does not provide a method to obtain the ID or path after installation. I believe deployment-id is synonymous with "composefs digest" here.

My installer currently finds the directory by assuming that <target>/state/deploy contains exactly one deployment. That works for a fresh installation in a clean directory tree, but that's a naive assumption. Determining the digest via a boot entry is not a viable substitute: pre-existing entries and bootloader defaults can affect which entry is primary, and the running system cannot identify a new deployment that has not booted yet. The OCI image digest is also not the composefs deployment ID.

Bootc needs a simple feature (read: not something that requires the overhead of an IPC API) to provide the deployment ID or absolute deployment path created by that specific install invocation in a stable, parseable format that a wrapping installer can capture. This would let the installer configure the right /etc without relying on assumptions and inferences (directory counts, log parsing, or bootloader ordering). The analogous ostree workflow can query the deployment path with ostree admin --sysroot=<target> --print-current-dir.

[jmpolom]

Adding some more detail about workarounds that still fall short of what is needed:

1. There is a new bootc install mount that purports to mount the deployed system into a target directory. This is certainly a welcome feature for interactive system debugging/rescue but to me it is a bit too heavy for an installer (that will certainly have mounting/partition handling functionality). Additionally at present time bootc install mount uses the same naive logic I'm already employing (single deployment directory, assume that is what was installed). At present time this is a heavyweight solution that does not fully address the need for deterministic, calculated discovery.
2. The hidden compute-composefs-digest commands (there is also compute-composefs-digest-from-storage) which are both seemingly experimental and unstable commands. A feature like these honestly is about ideal in terms of usage mechanics however they're not able to calculate the composefs digest from an installed system, they need a directory of image contents to do so. Providing bootc access to the image contents has proven cumbersome to handle in an installer and can result in race-y/TOCTOU concerns depending how images are referenced, and/or concerns about pulling images multiple times if container caches are not shared precisely the correct way.

For these 2 reasons I am led back to thinking a simple feature where bootc outputs somewhere the composefs digest which is already computed at installation time (but currently discard). This could be a uniquely formatted output line that could be grepped or dropped into a text file somewhere easily discoverable in the directory structure.

[cgwalters]

A lot of prior discussion on this in #542 (and thanks for filing that)

There is a new bootc install mount that purports to mount the deployed system into a target directory. This is certainly a welcome feature for interactive system debugging/rescue but to me it is a bit too heavy for an installer (that will certainly have mounting/partition handling functionality).

Can you explain more of why you feel this way? For sure an installer will support mounting, but...while there is mount.composefs that's a prior implementation (we're working to unify them!) - but even then there's the initramfs handling logic which we really do want to re-use.

I totally agree that we should have an API, and that will likely happen via varlink, but I don't see us in the short term getting out of needing to offer a higher level API that handles mounting the root in the same way bootc would.

Is there more specific technical implementation issues you hit with bootc install mount? (Just guessing: mount namespacing?)

[jmpolom]

Can you explain more of why you feel this way? For sure an installer will support mounting, but...while there is mount.composefs that's a prior implementation (we're working to unify them!) - but even then there's the initramfs handling logic which we really do want to re-use.

I totally agree that we should have an API, and that will likely happen via varlink, but I don't see us in the short term getting out of needing to offer a higher level API that handles mounting the root in the same way bootc would.

Is there more specific technical implementation issues you hit with bootc install mount? (Just guessing: mount namespacing?)

I don't need to mount the whole root filesystem to drop files into /etc (<target>/state/deploy/<deployment-id>/etc), and strongly prefer not to delegate more mounting functions to bootc during installation to-filesystem. This is important since I also use a separate partition for /var that must be mounted at install time so it's just cleaner to keep all the major mounting functionality in 1 place.

The other issue is the logic bootc install mount employs which today is rather naive. It does not actually compute the composefs digest and locate the deployment, it only handles a case where there is a singular deployment on disk and assumes that is the one which was just installed. I am already using this naive/presumptive method today but it could be prone to error if there happens to be >1 deployment for some reason, or perhaps even an extraneous directory in the path. Need a more deterministic and precise way to locate the deployment.

Ultimately I want an analog to ostree admin --print-current-dir that just spits out the path to the deployment in /state/deploy or just the composefs digest and I can handle assembling the full path. This could already be happening at install time since the digest is obviously available but right now it appears that digest is not logged anywhere to the user.

The closest candidates are the compute-composefs-digest commands which are not stable/hidden and have other edge cases that are cumbersome to work around. It would flatly be easier to just have the install command produce the composefs digest at install time somewhere when the composefs backend is selected.

I do agree that having a high level "mount" feature is essentially for system rescuing/debugging -- essential functionality honestly but maybe adjacent to the needs an installer will have.

[cgwalters]

I don't need to mount the whole root filesystem to drop files into /etc

I agree, but the rationale behind the current design is Anaconda for example has %post which is defined to run code in the context of the target environment pre-reboot, which needs the root mounted properly.

(And Anaconda is pretty strategic for my employer, it predates virtualization for us as the way to install bare metal systems)

---

Ultimately I want an analog to ostree admin --print-current-dir

How about bootc install mount --etc that only mounts /etc at the target path?

That'd be pretty easy to do I think.

But...yes, we can probably do some kind of bootc install status that output JSON e.g. if you really prefer it, I am not excited about basically exposing the implementation paths, but OTOH we can't easily change them anyways.

[jmpolom]

I don't need to mount the whole root filesystem to drop files into /etc

I agree, but the rationale behind the current design is Anaconda for example has %post which is defined to run code in the context of the target environment pre-reboot, which needs the root mounted properly.

(And Anaconda is pretty strategic for my employer, it predates virtualization for us as the way to install bare metal systems)

You don't say? Yeah I can see a "mount all" function being rather essential for a legacy heavyweight installer. In my case though I am embedding the installer into the image itself so the image is "self installing".

Ultimately I want an analog to ostree admin --print-current-dir

How about bootc install mount --etc that only mounts /etc at the target path?

That'd be pretty easy to do I think.

But...yes, we can probably do some kind of bootc install status that output JSON e.g. if you really prefer it, I am not excited about basically exposing the implementation paths, but OTOH we can't easily change them anyways.

Even simpler. I would prefer it not to perform any mounting. A status/text output would be ideal. My initial thought was writing out an installation summary file (specifically a file so it is a point in time static document) into the sysroot somewhere that contains (at a minimum) the digest for what was just installed. The key is the digest needs to be calculated (and bootc install already does this), not just inferred because there is only one dir under /state/deploy/.

Calculating the digest from an installation doesn't appear to be feasible? It appears the digest calculation requires the original source image filesystem tree. So making this an on-demand output like bootc status or even ostree admin --print-current-dir seems like it might be fighting the way things want to work.

Perhaps I'm not understanding what the "source of truth" is here but if there are reliable ways to derive the specific digest that was installed in an authoritative and deterministic manner I'm all ears. Another candidate could be to inspect boot configuration but I think that could also be potentially cumbersome/ambiguous if there are pre-existing artifacts on disk (ie: a converted installation, re-used filesystem, etc).

[cgwalters]

In my case though I am embedding the installer into the image itself so the image is "self installing".

Yes! I'm pretty happy with the podman run <image> bootc install style flow, it's quite powerful. There's a larger picture question here of whether what you want could be put underneath bootc install, I think #2314 is going to really help with this, although there's quite a lot in this space.

Perhaps I'm not understanding what the "source of truth" is

The kernel commandline is what we use for booting, and the ideal is to use that stuff when offline too. That's being worked on in #2515

Even simpler. I would prefer it not to perform any mounting. A status/text output would be ideal.

I think we're kind of just re-hashing #542 (comment) and below at this point, can we continue there?

[jmpolom]

In my case though I am embedding the installer into the image itself so the image is "self installing".

Yes! I'm pretty happy with the podman run <image> bootc install style flow, it's quite powerful. There's a larger picture question here of whether what you want could be put underneath bootc install, I think #2314 is going to really help with this, although there's quite a lot in this space.

In my case it is more like podman run <...> <image> install-{composefs,ostree}.sh which then handles partitioning, filesystems, and of course the actual bootc install deployment bit. But yes having everything self contained is nice.

The apparent lack of LUKS support in the linked PR would be a deal breaker for me. I believe repart can on paper handle filesystems inside a LUKS container though, so I'm not sure why bootc falls back to the internal partitioning logic?

I am not using bootc install to-disk and likely won't due to bespoke partitioning needs. Is there any known incompatibility with bootc and systemd-repart run externally as part of an install routine? I would consider trying it if it is expected to work.

Perhaps I'm not understanding what the "source of truth" is

The kernel commandline is what we use for booting, and the ideal is to use that stuff when offline too. That's being worked on in #2515

Kernel command line would vary based on how many deployments exist, and is opaque if a UKI is used? How can the kernel command line be inspected and how can the active/primary deployment determined? This did not seem straightforward especially in the case where /boot or ESP might be contaminated with other artifacts.