libs/libc/elf: Load FDPIC modules through the ELF loader - #19673
libs/libc/elf: Load FDPIC modules through the ELF loader#19673casaroli wants to merge 12 commits into
Conversation
Thank you, please all related changes / PRs / bundles as breaking, just to mark important change (like "breaking news"), hopefully there will be no impact, but we had situations like this before that broke stuff somewhere else, and we just got warning about hitting CI quotas :-P |
|
Please take a look at https://nuttx.apache.org/docs/latest/contributing/guide.html#breaking-changes-handling-process :-)
|
A function pointer under FDPIC is not a code address. Because each PT_LOAD segment is placed independently, a pointer has to carry the data base its callee will need, so it is a two-word descriptor: the entry point, and the base to install in the PIC register before branching. R_ARM_FUNCDESC_VALUE says "the thing you are patching is such a descriptor", and R_ARM_FUNCDESC says "manufacture one and give me its address". Both need state a relocation cannot carry. A descriptor's second word is the *object's* data base, from DT_PLTGOT, and R_ARM_FUNCDESC carves descriptors from a pool whose cursor has to survive from one relocation to the next. up_relocate() is handed only a relocation, a resolved symbol and an address to patch. arch_data is the existing channel for exactly this -- RISC-V already uses it to remember a HI20 relocation while its LO12 partner is processed -- but nothing has ever put loader state into it: it is declared zeroed and written only by up_relocate() itself. So ARCH_ELFDATA_INIT and ARCH_ELFDATA_FINI are added, seeding the block from the loadinfo before the relocation loop and reading the cursor back after. Both default to nothing, so an architecture that does not define them is unaffected, and RISC-V's use of arch_data is untouched. libelf_relocatedyn() walks both dynamic tables under one arch_data, so the cursor spans the whole object. The addend handling is the part that is easy to get wrong. REL format keeps the addend in place, in the word about to become the entry point, and a pointer to a static function is referenced through its *section* symbol -- the value is the section base and the offset, including the Thumb bit, is entirely in the addend. Dropping it yields an even address and the core faults trying to execute it as ARM code. The GOT written into a descriptor is the loading object's own, even for an imported function, which is what makes a callback work: when the base firmware's qsort() calls back into a module's comparison function, the module needs its own data base in the PIC register. libelf_relocatedyn()'s imported-symbol path needed a change to suit. It stores the resolved address directly and never calls up_relocate(), which cannot produce a two-word descriptor, so under FDPIC the resolved value now goes through up_relocate() and the relocation type decides what to write. Implemented for armv7-m and armv8-m, the profiles FDPIC targets; the other ARM variants gain the arch_data block but no new relocations. Built and booted mps3-an547:picostest and lm3s6965-ek:qemu-nxflat, the ELF PIC and NXFLAT users of this code, both unchanged. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
🔗 Cross-repo PR dependenciesThe read-only Build run reported the following dependent PR(s) and fetched head SHA(s):
CI run: https://github.com/apache/nuttx/actions/runs/31046857417 |
🔗 Cross-repo PR dependenciesThe read-only Build run reported the following dependent PR(s) and fetched head SHA(s):
CI run: https://github.com/apache/nuttx/actions/runs/31076754676 |
With placement, the dynamic tags and the relocations in hand, the last thing an FDPIC module needs is for the ELF loader to recognise it and hand the scheduler its data base. The data base arrives by a different route than for everything else. A PIC ELF object has it as the address of its .got section, which the loader finds by name; an FDPIC object names it in DT_PLTGOT, which is read while the dynamic tags are parsed. Both end up in the dspace_s that up_initial_state() installs in the PIC base register when the task starts, so both kinds of module run the same way from there on. A module carrying DT_NEEDED is refused rather than loaded. Shared libraries belong to dlopen() rather than to a loader that walks dependencies itself, and nothing in the tree resolves DT_NEEDED today -- the tag appears exactly once, as a constant in include/elf.h. Loading such a module anyway would leave it to fault on its first call into a library that was never brought in, so it fails at load with a message naming the cause. This also makes CONFIG_BINFMT_CONSTRUCTORS do what it says for a module loaded through exec(). It has never had any effect there: elf_loadbinary() recorded .init_array and .fini_array and nothing ever called them, so a C++ module's global objects were left as .bss and its constructors were dropped without a word. They now run, at the end of the load, which is where libelf_insert() has always run them for a module arriving through dlopen(). That is a fix rather than a break -- a global that should have been constructed now is -- but it is worth knowing about, because a module that worked around the gap by initializing from its entry point will find the constructor has already run. Such a module should drop the workaround; left in place it initializes twice. A module with no constructors is unaffected, as is any configuration with CONFIG_BINFMT_CONSTRUCTORS disabled. Built and booted mps3-an547:picostest, which shares this path. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
The base firmware and an FDPIC module disagree about what a function pointer is. Firmware is not built FDPIC, so to it a pointer is a code address and it branches there. A module passes the address of a two word descriptor instead, because its code and data are placed independently and a bare code address would leave the callee unable to find its own data. A firmware routine that takes a callback therefore branches into the module's data segment and faults. So the ten entry points that can be handed a callback by a module resolve the descriptor before storing or branching to it: qsort, bsearch, pthread_create, signal, sigaction, task_create and task_create_with_stack, task_spawn, pthread_once, scandir, and mq_notify and timer_create with SIGEV_THREAD. Which one resolves matters as much as that one does. Resolving twice would take an already resolved code address for a descriptor and read two words from the instruction stream, so each pointer is resolved exactly once, at the outermost point that sees it. signal() passes its argument through untouched because sigaction() and then nxsig_action() will resolve it, which covers a module calling sigaction() directly as well. qsort() is split so that the public entry resolves and the recursive implementation does not. scandir() resolves its filter but not its comparison function, which it hands to qsort(). Whether a caller is a module at all is asked of the PIC base register, which up_initial_state() sets only for a task that has a D-Space. A plain kernel task therefore reads zero and is left alone. SIGEV_THREAD is the case the register cannot answer, because the callback runs later on a work queue worker that carries no module's base at all. The base is captured instead when the notification is registered, in the module's own context, and installed around the call. fdpic_invoke() keeps hand written assembly rather than using up_setpicbase(). The register has to hold the module's base for exactly one call and then go back, and nothing in C tells the compiler the register is live across that call, so saving, installing, branching and restoring have to be a single sequence. All of it is behind CONFIG_ELF_FDPIC, which is new here and defaults off. Built mps3-an547:picostest both ways; with it off the entry points compile to what they were, and with it on qsort() calls fdpic_callback(), which reads the PIC base register through up_getpicbase(). Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
Running one for the first time turned up two holes in the ET_DYN path. Neither shows up in a build. An undefined symbol is resolved with libelf_findglobal(), which searches only the table of globally registered symbols. The export table that exec() hands its caller went no further than the ET_REL path, so an ET_DYN module could not import anything the caller supplied. Invisible while such modules resolved everything internally; an FDPIC module imports its libc, and every import failed with "Unable to resolve addr of ext ref printf" although the caller had passed a table containing printf. The export table is now threaded into libelf_relocatedyn() and consulted when the global table has no answer, leaving the existing lookup order intact. A relocation naming a symbol defined inside the object was dropped silently. The code handles a relocation with no symbol, and one against an undefined symbol, but a defined symbol fell through both. That was harmless while every dynamic relocation arriving here had symbol index zero, which is the case for R_ARM_RELATIVE. FDPIC brings the first ones that do not: a pointer to a static function is emitted against the *section* symbol, so the value is the section base and the offset within it -- including the Thumb bit -- is carried as the addend. Deriving a value from the word being patched, as the no-symbol case does, would translate that addend as though it were an address. Confirmed against a real module: .text at 0x23c plus an addend of 0x95 gives 0x2d1, which is the function with its Thumb bit. Also stop libelf_symname() reporting a nameless symbol as an error. A section symbol has no name, and libelf_findsymbol() walks the whole table looking for optional entries such as nx_stacksize, so it meets these routinely and checks for -ESRCH itself. At error level it printed ten or more lines per module load and buried the diagnostics that matter. Built and run on lm3s6965-ek with the examples/elf ROMFS. The ET_REL test modules load as before, and an FDPIC module now loads, relocates, resolves printf and puts from the table exec() supplied, and calls through a function descriptor of its own. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
A module that dlopen()s a library gets back function addresses from dlsym() and calls them. Under FDPIC a bare code address is not enough: the callee needs its own data base as well, so what dlsym() returns has to be a function descriptor. The exported symbol table carries no type information -- symtab_s is a name and a value, and its own comment says typing would have to be added to support anything but function pointers -- so by the time dlsym() is asked there is no way to tell a function from an object. libelf_insertsymtab() is the last point that can: st_info is still in hand there. So an FDPIC object's exported functions are published as the address of a descriptor carved from the module's pool, and dlopen(), dlsym() and the module registry need no knowledge of FDPIC at all. The pool is sized for the dynamic symbol table as well as the relocations, since both can draw from it. That leaves the symbol values themselves, which were wrong for any ET_DYN object. libelf_loadsymtab() adds the symbol's section address to its value, which is right for ET_REL, where the section address is where the section was actually placed and the value is relative to it. In a shared object both are already full link-time addresses, so adding them counts the section twice. It needs translating onto wherever the object was placed instead. Library data is shared between everything that dlopen()s it, because the registry holds one instance per name. Giving each user its own copy would mean teaching the registry about instances, which is a much larger change to shared code; an executable loaded through exec() already gets its own data, since that path loads a fresh copy each time. Built and run on lm3s6965-ek with the examples/elf ROMFS; the FDPIC module continues to load, relocate and call through its own descriptors. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
A module that names a shared library in DT_NEEDED now gets it loaded and its imports bound against it, rather than being left with undefined symbols. dlopen() does the loading. It is already the loader for a shared library, so the work goes there rather than into a dependency walker of the loader's own: the library lands in the module registry like anything else, its exports come back through libelf_getsymbol() -- the same call dlsym() uses -- and a library named by two modules is opened once and reference counted. Undefined symbols are resolved against the globally registered symbols first, then the opened libraries, then the table exec() supplied. The handles are closed when the module is removed. Four things had to be fixed to make it work, none of which a build shows. reldata was a file-scope global. Opening a library from inside libelf_relocatedyn() makes that function reentrant, so the nested load overwrote the outer one's relocation offsets and the module resumed binding with the library's DT_REL. It is now per call. A cross-object call needs the callee's data base, not the caller's. A symbol resolved from an FDPIC library comes back as a descriptor, and R_ARM_FUNCDESC_VALUE was treating it as a code address and pairing it with the importing module's GOT. It now copies both words, so the library runs with its own. An object with no imports has no PLT and so no DT_PLTGOT, but it still has a GOT and still has to be entered with it. Without the fallback its descriptors carried a data base of zero and the library read its globals through a null pointer. libelf_symname() was static, and reading a DT_NEEDED name needs it. Nothing happens without CONFIG_LIBC_DLFCN; a module with DT_NEEDED is refused there, since there is no way to load what it asks for. Built and run on lm3s6965-ek: a module naming a library in DT_NEEDED calls into it and gets the right answer, and the library keeps its own data. mps3-an547:picostest and lm3s6965-ek:qemu-nxflat, which have CONFIG_LIBC_DLFCN off, build and run unchanged. CONFIG_FDPIC depends on the flat build. A module's read-only segment is held by a filesystem pin that has to be given back when the module is unloaded, which happens on a task other than the one that loaded it, so it is held through a reference to the file rather than a descriptor -- and the file interface is not reachable from the loader in the protected and kernel builds. Selecting it there would leak the pin and leave the filesystem unable to compact. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
🔗 Cross-repo PR dependenciesThe read-only Build run reported the following dependent PR(s) and fetched head SHA(s):
CI run: https://github.com/apache/nuttx/actions/runs/31093453964 |
hello, this is not a breaking change anymore. it is a bug fix (now c++ constructors will work). it will break just if someone found a way around C++ constructors not working, then in this case, they would run twice. so i consider this not to be a breaking change anymore, and a bugfix instead. can you please remove the breaking-change tag? |
apps/examples/fdpicxip and apps/testing/fs/xipfs carry their modules as committed byte arrays, and regenerate them with make -C apps/examples/fdpicxip/modules regen NUTTX_DIR=/path/to/nuttx which reads nuttx-fdpic.mk and fdpic-embed.py from here. Both apps are already upstream and cannot rebuild their own blobs from source without this. nuttx-fdpic.mk also builds a module out of tree, which is what anyone writing one starts from; the README describes the four link flags that matter and why. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
FDPIC has no page, and the parts of it a reader has to get right are spread across binfmt/Kconfig, the ELF loader and the ARM toolchain definitions. The page covers what an FDPIC module is and what it buys over the position independent ELF support already in the tree, how the loader places one, where shared libraries come from and how they are found, which entry points resolve a function descriptor and the rules for adding another, and how to build a module and a library with a toolchain that can emit FDPIC. A comparison table places it against NXFLAT and PIC ELF, and the reference section records the object layout and the relocations. The known limitation is stated: global constructors are not run for a module loaded through binfmt. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
xipfs-fdpic is the xipfs configuration plus the FDPIC loader and the fdpicxip demo, so that the fdpic and reject sections of the XIPFS test suite have something to run. It is the configuration the loader was tested with, and until now none was in the tree: FDPIC needed a dozen options set by hand, three of which are not obvious. CONFIG_LIBC_ENVPATH and CONFIG_LDPATH_INITIAL, because a library named in DT_NEEDED is opened by name and dlopen() searches LD_LIBRARY_PATH. And CONFIG_LIBC_ELF_HAVE_SYMTAB, because a library resolves its own imports against the module registry's symbol table rather than the one exec() supplies. CONFIG_DEFAULT_TASK_STACKSIZE goes to 4096, from the board's 2048, and both sides of the loader need it. A module that calls into the firmware's printf family overflows 2048, and with no MPU that is a lockup with no diagnostic at all; CONFIG_ELF_STACKSIZE is not set here because it defaults to CONFIG_DEFAULT_TASK_STACKSIZE and so follows it to 4096. The XIPFS test task is sized from it directly, and at 2048 that task overflows in printf partway through the module tests, so the suite stops mid-run with no failure reported and no dump -- which reads as a loader bug rather than as the test running out of stack. Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
the content is now Documentation/components/tools/fdpic.rst. tools/fdpic/README.md is dropped |
|
If this gets merged, next is to make signals and C++ exceptions work (for FDPIC and ELF PIC - just not sure about NXFLAT, but we can try) |
❌ Cross-repo dependency could not be appliedThe Build report says the declared dependency PR(s) could not be applied, so CI did not run against the combined code: Reason: cherry-pick failed (if your PR has merge commits, rebase instead) CI run: https://github.com/apache/nuttx/actions/runs/31159022159 |
|
I will review in this week end. |
| * on the flat build. | ||
| */ | ||
|
|
||
| #ifdef CONFIG_FDPIC |
|
|
||
| bool fdpic; | ||
| bool textpin; | ||
|
|
There was a problem hiding this comment.
change to pic? the difference between pic and fdpic is that the normal pic doesn't support so.
| #ifdef HAVE_LIBC_ELF_PIN | ||
| /* The file the pin is held through, handed to the module once it loads. */ | ||
|
|
||
| FAR struct file *pinfile; |
There was a problem hiding this comment.
why not let mmap to hold the pin
| * | ||
| ****************************************************************************/ | ||
|
|
||
| void libelf_pinrelease(FAR struct file **pinfile); |
There was a problem hiding this comment.
let mmap pin the storage and ummap to unpin it.
There was a problem hiding this comment.
If the loader uses mmap(), which group owns the mapping? the caller's, which may exit while the module is still running, or the module's, which does not exist yet at load time?
| { | ||
| if (phdr->p_flags & PF_X) | ||
| { | ||
| if (loadinfo->fdpic) |
There was a problem hiding this comment.
should we fallback to copy if the fs or mtd doesn't support xip.
| * segments were placed separately and | ||
| * the text is media, not an allocation | ||
| */ | ||
| uintptr_t gotaddr; /* An FDPIC object's data base, to |
There was a problem hiding this comment.
could we check gotaddr |= 0 and remove fdpic
There was a problem hiding this comment.
I don't think so, because we can have a FDPIC module with no imports has no DT_PLTGOT, and it will have gotaddr = 0
| bool "FDPIC modules" | ||
| default n | ||
| select PIC | ||
| depends on ARCH_ARMV7M || ARCH_ARMV8M |
There was a problem hiding this comment.
remove the dependence
| static inline void fdpic_invoke(uintptr_t entry, uintptr_t arg, | ||
| uintptr_t got) | ||
| { | ||
| register uintptr_t r0v __asm__ ("r0") = arg; |
There was a problem hiding this comment.
not work for other arch, should keep jn arch.h or arch/elf.h
| This is the default stack size that will be used when starting ELF binaries. | ||
|
|
||
| config ELF_FDPIC | ||
| config FDPIC |
There was a problem hiding this comment.
move to the firdt patch which add ELF_FDPIC
| # | ||
| ############################################################################ | ||
|
|
||
| # nuttx-fdpic.mk -- build out-of-tree FDPIC modules for NuttX |
There was a problem hiding this comment.
why not modify the original elf generation scirpt in place
Summary
Loads ELF shared objects built for the ARM FDPIC ABI, as a mode of the existing ELF loader rather than as a separate binary format.
An FDPIC object places its two
PT_LOADsegments independently, so its read-only segment is executed straight out of flash where the filesystem already holds it, and only the writable segment is copied to RAM, once per running instance. Several instances of one module therefore share one copy of the text. What FDPIC adds over the position independent ELF support already present is a function pointer that carries its own data base, as a two word descriptor rather than a bare code address; that is what lets a module be called back on a thread it did not create, such as the work queue worker that runs aSIGEV_THREADnotification.This supersedes #19584, which added a separate
binfmt/fdpic.c. Everything is folded intolibs/libc/elfandbinfmt/elf.cinstead, andDT_NEEDEDis implemented by callingdlopen(), as requested in that review.#19600 puts ARM PIC on r9, which an FDPIC module requires and which the base firmware has to reserve. This branch is based on it, so its four commits are already here and there is nothing for CI to fetch — which is why it is not in the list above. It does still have to merge first: it is a breaking change carrying its own §1.13 vote, and merging this PR ahead of it would land the r9 change without one.
#19639 made
dlopen()return an already loaded library and count the opens, and has now merged, so it is no longer listed above.DT_NEEDEDis implemented throughdlopen(), so without it a library named by a second module would be refused and the tests below would fail.apache/nuttx-apps#3693 updates the two test apps, which are already upstream and assert the previous library semantics.
Semantics worth reviewing
A library named in
DT_NEEDEDis opened withdlopen(), which returns the object already in the module registry rather than loading a second copy. There is therefore one instance of a library, its data included, shared by every module that names it. A module started withexec()is different: that path loads the module afresh each time, so two running instances have separate data while sharing one copy of the text.apps/testing/fs/xipfsandapps/examples/fdpicxipasserted the opposite, because the loader that walkedDT_NEEDEDitself gave each instance a private copy. The paired apps PR updates them.Impact
CONFIG_BINFMT_CONSTRUCTORSstarts doing what it says for a module loaded throughexec(). It has never had any effect there:elf_loadbinary()recorded.init_arrayand.fini_arrayand nothing ever called them, so a C++ module's global objects were left as.bssand its constructors were dropped without a word. They now run, at the end of the load, which is wherelibelf_insert()has always run them for a module arriving throughdlopen().This is a fix, not a break: a global that should have been constructed now is, and C++ modules under
exec()stop silently losing their constructors. It is called out here only because it is worth knowing — a module that worked around the gap by initializing from its entry point will find the constructor has already run, and should drop the workaround, since left in place it initializes twice. A module with no constructors is unaffected, as is any configuration withCONFIG_BINFMT_CONSTRUCTORSdisabled.This PR is no longer marked a breaking change. It carried that marking for two reasons: the constructors above, and prebuilt NXFLAT modules being refused once ARM PIC moved to r9. The second is #19600, which has merged, so it belongs to master rather than to this PR and this branch no longer carries those commits. The first is a bug fix, so the
!and theBREAKING CHANGE:block are gone.A third change is visible but breaks nothing upstream: a library named in
DT_NEEDEDis now one shared instance rather than one per module instance. That is described below, and it differs only from the superseded #19584, never from a released NuttX.CONFIG_DEFAULT_TASK_STACKSIZEwants 4096 rather than the rp23xx default of 2048, and thexipfs-fdpicboard configuration added here sets it. Both sides of the loader need it. A module that calls into the firmware's printf family overflows 2048, and with no MPU that is a lockup rather than a diagnostic;CONFIG_ELF_STACKSIZEis not set separately because it defaults toCONFIG_DEFAULT_TASK_STACKSIZEand follows it to 4096. Andapps/testing/fs/xipfssizes its own task from it, so at 2048 that task overflows inside printf partway through the module tests. The suite then stops mid-run with no failure reported and no dump, which reads as a loader bug rather than as the test running out of stack — it cost me most of a day, so it is worth knowing before you point this at a board whose default is 2048.How to verify this on your board
Help welcome — §1.14 asks for runtime logs from more than one real architecture, and I have only an RP2350.
The r9 half is #19600, which this branch includes; its verification instructions are in that PR and apply unchanged.
The constructors half needs only a board that can load an ELF module. Enable
CONFIG_BINFMT_CONSTRUCTORSand runapps/examples/sotestorapps/examples/module. If the module has a global constructor — a C++ static object, or__attribute__((constructor))— it now runs at load time, where before it was silently skipped and the global was left as.bss. No FDPIC, no xipfs and no special toolchain are needed for this part.The FDPIC half needs a filesystem that can expose its media, so in practice xipfs or ROMFS on a target with XIP flash.
pimoroni-pico-2-plus:xipfs-fdpicis added by this PR and is the configuration I tested;xipfs_test fdpic,xipfs_test rejectandfdpicxipare the runs to make. The module blobs are committed, so noarm-uclinuxfdpiceabitoolchain is needed to run them.Testing
apps/testing/fs/xipfsandapps/examples/fdpicxipare already upstream and carry the assertions; both were run in full.QEMU,
mps2-an500(Cortex-M7, xipfs on a rammtd that answersXIPFSIOC_PIN):Hardware, Pimoroni Pico Plus 2 (RP2350, Cortex-M33, real QSPI flash), same numbers: 131/131, 34/34, 7/7 and all four demos. That covers the armv8-m relocation path and a filesystem that really pins extents.
The 131/131 was re-run end to end on the
xipfs-fdpicconfiguration exactly as it ships here, from an erased flash, against the reference counting of #19639 in its current form.Between them these reach
R_ARM_FUNCDESCand the descriptor pool, all ten libc and sched callback entry points,SIGEV_THREADdelivery throughmq_notify()andtimer_create(),DT_JMPRELbinding,XIPFSIOC_PIN, two modules sharing one library, C++ modules with constructors in dependency order, and the six rejection cases.