[DNM][WiP][PoC] userspace LL scheduling: LLEXT & multicore#10945
Draft
lyakh wants to merge 91 commits into
Draft
[DNM][WiP][PoC] userspace LL scheduling: LLEXT & multicore#10945lyakh wants to merge 91 commits into
lyakh wants to merge 91 commits into
Conversation
Merged
Replace the k_thread_cpu_mask_clear() + k_thread_cpu_mask_enable()
sequence with a single k_thread_cpu_pin() call.
The clear-then-enable pattern momentarily leaves the thread with a
zero CPU mask, which is invalid under CONFIG_SCHED_CPU_MASK_PIN_ONLY
and triggers an assertion after Zephyr commit 7798570a031d ("kernel:
sched: enforce non-zero CPU mask invariant in PIN_ONLY mode").
k_thread_cpu_pin() atomically sets exactly one bit in the mask,
avoiding the transient zero-mask state.
Signed-off-by: Jyri Sarha <jyri.sarha@linux.intel.com>
The real fix is to remove the locking around dai_get_properties() altogether, but this depends on fixes in Zephyr DAI drivers. To unblock user-space work, remove the calls to spinlocks for now. This opens up possibility to hit issues with concurrent playback and capture cases on multiple cores, so this commit remains a WIP until fixes in Zephyr drivers land. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Modify code to allocate DAI properties object on stack and use dai_get_properties_copy(). This is required when DAI code is run in user-space and a syscall is needed to talk to the DAI driver. It's not possible to return a pointer to kernel memory, so instead data needs to be copied to caller stack. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Turn the pdata->sem into a dynamic object in userspace LL builds, implemented with Zephyr k_sem. Add POSIX no-op stubs for sys_sem to maintain testbench build compatibility. Keep statically allocated semaphore for kernel LL builds. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add function scheduler_get_data_for_core() to look up scheduler data for a particular type of scheduler. This variant allows to pass the core number as an argument, so it can be called from unprivileged code. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add function user_ll_grant_access() to allow other threads to access the scheduler mutex. This is needed if work is submitted from other threads to the scheduler. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
When LL scheduler is run in user-space, use a different Zephyr thread name. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
The COHERENT_CHECK_NONSHARED_CORES debug macros call cpu_get_id() which invokes arch_proc_id() - a privileged hardware register read that faults in user-space context. Disable the entire debug block at compile time when CONFIG_SOF_USERSPACE_LL is enabled. This also fixes the same latent issue in CORE_CHECK_STRUCT and CORE_CHECK_STRUCT_INIT. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Modify the checks in zephyr_ll_assert_core() to make them safe to call from user-space LL threads. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Needs more review, but makes the tests pass again. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Build fails when building with CONFIG_THREAD_NAME disabled. Fix the issue by conditional compilation of code using CONFIG_THREAD_MAX_NAME_LEN. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add support to run pipeline_schedule_triggered() in user-space. Use the user_ll_lock/unlock_sched() interface if building with CONFIG_SOF_USERSPACE_LL. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
In user-space LL builds the low-latency scheduler runs its work in a dedicated privileged domain thread, created together with its timer and access grants by scheduler_init_context() (zephyr_ll_init_context() -> domain_thread_init()). This context is per-core and must exist on every core that runs LL tasks. So far it was only established for the primary core, so LL tasks could not be scheduled on secondary cores when CONFIG_SOF_USERSPACE_LL is enabled. Allocate an LL task in secondary_core_init() and run scheduler_init_context() on it, giving each secondary core its own LL domain thread. A dedicated sec_core_init UUID is registered for the task. The whole block is compiled in only for CONFIG_SOF_USERSPACE_LL. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Make comp_drivers_get() usable from user-space when SOF is built with CONFIG_SOF_USERSPACE_LL. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Add CONFIG_SOF_USERSPACE_LL support to chain DMA component: - Allocate comp_dev, private data, and DMA buffer from user heap - Use k_work_delayable for periodic task instead of LL timer scheduler to keep DMA operations in kernel context - Guard all changes with #ifdef CONFIG_SOF_USERSPACE_LL Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Use a module context instead of heap to allocate objects in chain-dma module. This is required to support vregion use. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Commit f515f3b kept chain DMA in kernel context for CONFIG_SOF_USERSPACE_LL by driving chain_task_run() from a k_work_delayable on the system workqueue, because chain DMA used the raw Zephyr DMA API (dma_get_status/dma_reload/dma_start/dma_stop) which is not callable from an unprivileged user thread. Running a ~1ms (500us on AMD) audio DMA cadence on the low-priority system workqueue, rescheduled completion-relative via k_work_reschedule(), is prone to preemption and drift and risks xruns. Port chain DMA to SOF's sof_dma_* syscall wrappers (as host and dai already do) so it can run unprivileged, and schedule it as a normal SOF_SCHEDULE_LL_TIMER task on the user LL thread. This removes the kernel-workqueue path entirely; both userspace and non-userspace builds now share the LL scheduler path with scheduler-managed task lifecycle. Channel handles are stored as integer indices instead of kernel-only struct dma_chan_data pointers. The user-heap allocations for the component, private data and DMA buffer are retained so the user thread can access them. Signed-off-by: Kai Vehmanen <kai.vehmanen@linux.intel.com>
Currently building with userspace LL and the debug overlay generates non-functional images. This is due to two incompatibilities: (1) cpu_get_id() called in userspace in zephyr_ll_task_init() and (2) the heap_alloc boot-test currently causing an exception. Fix the former and disable the latter to re-enabld debug builds with the userspace enabled. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Eliminate multiple instances of IPC4 data copying, use simple type- casts instead. This removes stack objects and replaces run-time copying with compile-time pointer substitution. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Prepare for multi-core support: allocate the IPC thread dynamically and extract thread initialisation into a separate function. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Userspace IPC context is global, allocate it uncached. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Use current core when calling scheduler_get_data_for_core(). Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Sometimes 10ms aren't enough for userspace IPC processing, increase it to 20ms. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
The SOF_IPC4_GLB_SET_PIPELINE_STATE IPC can apply to one or to multiple pipelines. Extract pipeline ID detection into a function to be re-used with userspace IPC processing. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Userspace LL allocation context can be used by all cores, allocate it as uncached. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Make scheduling LL thread and synchronisation objects per-core and forward IPCs and scheduling events accordingly. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
The global driver list should never be accessed from the audio context. Add an assertion for that. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
scheduler_get_data_for_core() returns void pointer, no need to type-cast it to other pointer types. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
syscall-related code can be used when CONFIG_USERSPACE is undefined too. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Define the "core" variable only when it is used. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Define the "ipc" variable only for cases, when it's used. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
ipc4_init_module_instance() should be called when CONFIG_SOF_USERSPACE_LL isn't selected but also when initializing a DP module. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
If LL runs in userspace, it needs access to loaded LLEXT modules, running in DP more too. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
In kernel context cpu_is_me() should return a correct result for correct multicore support. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Add two syscall functions to allocate and map, and to unmap vregion for userspace modules. For now only used for DP modules. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
vregion_get(), vregion_put() and vregion_set_interim() should also be callable from the userspace. Make them syscalls. Also remove redundant symbol exporting since the vregion API shouldn't be used directly by LLEXT modules. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
scheduler_dp_ll_tick() is currently registered as a notifier callback, but it's always triggered deterministically, always with the same-core-only flag, which leads to it being called immediately. So the notifier only adds a layer of indirection and reduces clarity. Replace it with a direct function call. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
scheduler_dp_ll_tick(() has to recalculate DP deadlines and reschedule DP threads. Make it a syscall to be able to call it from the userspace LL scheduler. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Both arch_schedulers_get() and arch_user_schedulers_get() should only be called from the privileged context. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Make scheduler_dp_internal_free() a syscall in the "application" DP implementation. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
In case of userspace LL scheduling the (also userspace) IPC thread needs access rights to DP assets like the thread itself and its stack and synchronisation primitives. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
zephyr_ll_domain() and scheduler_get_task_info_ll() are needed in non-userspace builds too, make them universally available. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
The LL userspace thread has to interact with the DP one. Grant required rights. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
DP scheduler operations, instance data and DP module memory have to be accessible to the userspace LL scheduler. Allocate dynamic data on the userspace heap and place static data in the userspace accessible ELF section. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Remove duplicate ll_schedule_domain.h inclusion in ipc-common.c. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
Switching to the userspace mode in DP and LL cases differs. Add a comment to explain that. Signed-off-by: Guennadi Liakhovetski <guennadi.liakhovetski@linux.intel.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This includes #10558 and my patches on top to enable LLEXT and multicore. Current status: passes simple tests with nocodec with both core 0 and core 1 streaming. 2 streams simultaneously run into a problem when the first of them terminates. WiP.