Skip to content

[DRAFT - held on OSv fork/COW landing] postgres18-musl: stock PostgreSQL 18 on OSv demo - #73

Draft
gburd wants to merge 6 commits into
cloudius-systems:masterfrom
gburd:wip/postgres18-musl-demo
Draft

[DRAFT - held on OSv fork/COW landing] postgres18-musl: stock PostgreSQL 18 on OSv demo#73
gburd wants to merge 6 commits into
cloudius-systems:masterfrom
gburd:wip/postgres18-musl-demo

Conversation

@gburd

@gburd gburd commented Jul 28, 2026

Copy link
Copy Markdown

Draft — held on the OSv fork/COW work landing on OSv master

Adds a postgres18-musl/ demo app that runs stock, unmodified PostgreSQL 18
(musl build) on OSv.

This will NOT run on stock OSv master. It requires the OSv-side
thread-backed fork() + per-child COW address space + per-process signals + the
fork-completeness stack (AS-aware mmap, fd inheritance, timer parking, cross-AS
coherence) to be present on OSv master first. Until then this is filed as a
draft so the app is visible alongside the kernel work it depends on.

Requires (on OSv, not this repo)

All of the above are opt-in / CONF_fork on OSv; PG-per-backend fork is the
whole reason this app needs them.

Status

Ready to draft; held on fork-merge. Do not merge until the OSv fork/COW
stack (#1455/#1456/#1457 + #1458) is on OSv master; add ZFS (#1423) for the
ZFS-on-NVMe /data variant.

gburd added 5 commits July 27, 2026 08:11
Builds PostgreSQL 18 (REL_18_STABLE) from source with musl-gcc as an OSv PIE,
boots a small baked cluster, and serves SQL over TCP. Single-backend demo;
requires the OSv fork/COW coherence work (conf_fork=1 + the fork-arena kernel
fixes) which is not yet on OSv master, so this stays a WIP branch until that
lands.
…ady-serving)

Add a documented full-VM snapshot/restore capability to the demo: boot
OSv+PostgreSQL under Firecracker to "ready to accept connections", take a Full
snapshot (guest memory + device/vCPU state), and on later launches restore that
snapshot so PostgreSQL is already serving. No reboot, no re-initdb, no musl-PIE
re-mmap.

Measured on m5d.metal (Firecracker v1.7.0, OSv v0.57 PG18.4-musl + OpenZFS,
single-file ZFS pool recordsize=8k):
  cold launch    -> first external select 1 = 11.75 s
  restore        -> first external select 1 =  0.24 s  (median of 5, ~48x)
  FC resume itself                          =  0.05 s
The ~6.7 s PostgreSQL musl-PIE mmap+init is eliminated (baked into the snapshot).

OSv resumes cleanly with NO OSv kernel changes required (already
snapshot-ready). Proof it is a resume not a reboot: pg_postmaster_start_time()
after restore is the original pre-snapshot boot time. Disk consistency via a
fresh COW copy (cp --reflink=auto) of the pool at snapshot time so the guest
ARC/shared_buffers agree with disk. Honest gotchas documented: clock skew
(kvmclock re-syncs, ~50-85 ms stable offset, tolerated), virtio re-attach (host
tap+DNAT must pre-exist), RNG (no observed duplication, weak probe).

Files: fc-snapshot.sh (driver), fc-snap.py (Firecracker API helper), README
section with build/snapshot/restore steps, measured numbers, and gotchas.

Author: Greg Burd
Add a 'lifecycle' subcommand that proves a PERSISTENT WARM checkpoint: a first
warm restore (~0.2s to first query), insert rows in the restored session, then
FREEZE (a fresh Full snapshot plus a preserved ZFS backing file) followed by a
CLEAN Firecracker shutdown (the VM process exits entirely), and finally a
SECOND warm restore from the frozen state (~0.2s again). The table, its rows,
and the original pg_postmaster_start_time() all survive the
freeze/clean-shutdown/warm-restart cycle, so the instance identity and its data
are a durable resumable checkpoint.

Also add a scripted 'seed' subcommand (build the single-file ZFS pool and
cpiod-push an initdb'd cluster into it) with its cpio_push.py helper, and fix
the boot argument passing so the leading --rootfs is not mis-parsed by the
Firecracker driver argument parser.

Measured on m5d.metal with Firecracker v1.7 (4 GiB guest, single-file ZFS
pool): both warm launches were about 0.17s to first query and
pg_postmaster_start_time() stayed equal to the very first launch across every
freeze/thaw cycle.

Signed-off-by: Greg Burd <greg@burd.me>
GCC 13 libgcc lse-init.o references glibc __getauxval (absent in musl); every
link fails and PG configure cannot detect the GCC atomic builtins.  Disable the
LSE runtime-detection path on aarch64 so links succeed and atomics are found.
… note

Two unrelated one-line problems in the same module.

The Makefile declared '.PHONY: module' but not 'build', while 'module: build'
depends on a target named 'build' and the module also creates a real 'build/'
directory.  Make then considers 'build' up to date because the directory
exists, skips the recipe, and 'module' succeeds without building anything.
The result is an image that contains no PostgreSQL while the build reports
success, which has silently produced a 10 MB image where a working one is
about 65 MB.  Declaring 'build' phony as well makes the recipe always run.

Separately, the comment claimed OSv's worker AIO path is incomplete and
lumped it together with io_uring.  io_method=worker is verified working on
OSv: PostgreSQL 18.6 starts, forks its three IO worker processes and serves
queries, with SHOW io_method reporting 'worker' to an external client.  It
needs no kernel AIO, being shared memory plus a latch plus forked workers
calling preadv/pwritev.  io_uring is the incomplete one, and it also needs
--with-liburing at configure time.  Since io_method is PGC_POSTMASTER, one
build can select any method at run time with -c io_method=..., so the demo
default is left at sync and the comment now says why.

Signed-off-by: Greg Burd <greg@burd.me>
@gburd

gburd commented Sep 9, 2026

Copy link
Copy Markdown
Author

Two one-line fixes pushed, both found while verifying PostgreSQL 18's asynchronous I/O on OSv.

The build target was shadowed by the build/ directory. The Makefile declared .PHONY: module but not build, while module: build depends on a target named build and the module also creates a real build/ directory. Make sees the directory, considers the target up to date, skips the recipe, and module succeeds having built nothing. The observable result is an image with no PostgreSQL in it while the build reports success: about 10 MB where a working image is roughly 65 MB. Two separate people lost time to this before it was diagnosed. Declaring build phony as well makes the recipe always run.

Worth knowing as a check regardless of this fix: image size is a reliable smoke test here. A PostgreSQL-bearing usr.img is around 65 MB with seven manifest entries for the module; a shadowed build yields roughly 10 MB and still exits zero.

The io_method comment was wrong, and in a way that mattered. It said OSv's "io_uring/worker AIO path is incomplete", conflating two unrelated mechanisms. io_method=worker needs no kernel AIO at all: it is shared memory plus an LWLock plus a latch, with forked IO worker processes calling pg_preadv/pg_pwritev, all of which OSv already supports. Verified on OSv with PostgreSQL 18.6, checked from a client outside the guest rather than from inside it:

version:    PostgreSQL 18.6 on x86_64-pc-linux-musl, compiled by gcc (GCC) 13.3.1
io_method:  worker
io_workers: 3

PostgreSQL started cleanly, forked its three IO worker processes, reported "database system is ready to accept connections", and served queries with no faults, with PGDATA on a ZFS dataset rather than a memory filesystem.

io_uring is the genuinely incomplete one for this module, and it additionally needs --with-liburing at configure time plus liburing in the image.

Since io_method is PGC_POSTMASTER, one build can select any method at run time with -c io_method=..., so the demo default stays at sync and the comment now explains the choice instead of misstating the platform's capabilities.

seed_copy faithfully preserved the source mode, which is correct on a
filesystem that has modes.  rofs does not: scripts/gen-rofs-img.py stores
only the S_IFMT type bits (DIR_MODE 0x4000, REG_MODE 0x8000), discarding the
permission field entirely, and fs/rofs/rofs_vnops.cc's rofs_getattr() then
reports a hardcoded 0555 for every inode.

So copying the observed mode created the destination tree without an
owner-write bit, and the next level down failed: O_CREAT hit
vn_access(VWRITE) and returned EACCES.  Seeding a PostgreSQL cluster from a
baked read-only image therefore stopped at the second directory level.  A
0555 PGDATA would have been rejected by PostgreSQL regardless.

Use the modes initdb itself produces, 0600 for files and 0700 for
directories, rather than propagating modes the source filesystem cannot
represent.  With this, seeding a 19 GB 200-warehouse cluster onto a ZFS
dataset completes with no errors.

Signed-off-by: Greg Burd <greg@burd.me>
@gburd

gburd commented Sep 11, 2026

Copy link
Copy Markdown
Author

One more fix, found while seeding a PostgreSQL cluster onto ZFS inside an OSv guest.

seed_copy preserved the source file mode, which is the right thing to do on a filesystem that has modes. rofs does not have them. scripts/gen-rofs-img.py lines 53-55 define

DIR_MODE  = int('0x4000', 16)
REG_MODE  = int('0x8000', 16)
LINK_MODE = int('0xA000', 16)

and assigns those unconditionally, so only the S_IFMT type bits are stored and the permission field is discarded entirely. rofs_getattr() then reports a hardcoded 0555 for every inode.

The consequence is that copying the observed mode created the destination tree without an owner-write bit, and the next level down failed: O_CREAT reached vn_access(VWRITE) and returned EACCES. Seeding stopped at the second directory level. A 0555 PGDATA would have been rejected by PostgreSQL anyway, so preserving that mode could never have worked.

This uses the modes initdb itself produces, 0600 for files and 0700 for directories, rather than propagating modes the source filesystem cannot represent. With it, seeding a 19 GB 200-warehouse cluster onto a ZFS dataset completes with no errors.

Worth noting separately, since it is a defect in the image tooling rather than in this module: rofs discarding permissions is invisible until something tries to write into a tree copied out of it. Anything that reads a mode from a rofs image is reading an invented value.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant