feat: wake io_uring workers through eventfd

[thweetkomputer]

What problem does this PR solve?

Issue Number: N/A

Problem Summary:

With use_io_uring=true, each brpc worker currently owns a separate polling pthread. That pthread waits for CQEs and wakes the worker through a condition variable, adding one thread and a cross-thread handoff per worker and preventing deferred task execution from being driven solely by the ring's issuer.

What is changed and the side effects?

Changed:

• Make use_io_uring the single switch for the optimized scheduler path; there is no separate eventfd-wakeup gflag.
• Create one nonblocking eventfd per io_uring RingListener and arm a persistent multishot poll request for scheduler wakeups.
• Let the owning worker block in io_uring_submit_and_wait() instead of starting poll_thd_.
• Initialize worker rings with IORING_SETUP_SINGLE_ISSUER, IORING_SETUP_DEFER_TASKRUN, and IORING_SETUP_TASKRUN_FLAG.
• Route TaskGroup::Notify() and NotifyIfWaiting() through eventfd while retaining notification coalescing and lost-wakeup checks.
• Wake workers through eventfd during shutdown so they can observe the stopped parking-lot state.
• Preserve the original polling-thread and condition-variable behavior when use_io_uring=false.

Side effects:

• Performance effects(性能影响): io_uring mode removes one polling pthread per worker and its condition-variable handoff. Waking an idle worker instead performs a nonblocking eventfd write.
• Breaking backward compatibility(向后兼容性): non-io_uring users are unchanged. Enabling use_io_uring now also requires kernel/liburing support for deferred task execution and multishot poll; initialization fails explicitly when those capabilities are unavailable.

Material design decisions:

• Eventfd wakeup is an implementation detail of io_uring mode, so a second configuration flag and invalid mixed configurations are avoided.
• The multishot poll is expected to remain armed. A CQE without IORING_CQE_F_MORE fails fast because silently continuing could permit a future permanent worker sleep.
• EAGAIN from eventfd write is success: a saturated eventfd is already readable and therefore already provides the required wakeup.

Verification:

cmake -S /tmp/brpc-eventfd-src -B /tmp/brpc-eventfd-build2 \
  -DIO_URING_ENABLED=ON -DWITH_GLOG=ON \
  -DCMAKE_PREFIX_PATH=/opt/eloq/third_party
PASS

cmake --build /tmp/brpc-eventfd-build2 --parallel 8
PASS (full build; existing compiler warnings only)

git diff --check master...HEAD
PASS

The current GitHub Actions compile, format/license, and Bazel jobs have passed; the unit-test job is still running at the time of this update. End-to-end EloqKV and EloqDoc matrices run in the linked dependency-validation PRs.

Performance benchmark (600.004 seconds, read-only GET workload):

Metric	Before	After	Change
Ops/sec	399,941.56	399,978.18	+0.01%
Average latency	0.23624 ms	0.21039 ms	-10.94%
p99.9 latency	0.57500 ms	0.48700 ms	-15.30%
p99.99 latency	3.08700 ms	2.60700 ms	-15.55%
Throughput	995,164.74 KB/sec	995,255.90 KB/sec	+0.01%

Post-change CPU utilization was 4,248.587 seconds total: 637.575 seconds user and 3,611.012 seconds system. No pre-change CPU measurement was supplied, so no CPU improvement is claimed.

Additional rate-limited memtier samples:

Metric	Before	After	Change
Observed duration	12.6 sec	6.1 sec	—
Ops/sec	665,708.44	750,805.62	+12.78%
Average latency	0.48018 ms	0.42535 ms	-11.42%
p99.9 latency	1.26300 ms	1.03900 ms	-17.74%
p99.99 latency	3.23100 ms	2.87900 ms	-10.89%
Throughput	1,656,714.13 KB/sec	1,869,030.02 KB/sec	+12.82%

Both rate-limited commands requested 600 seconds but were interrupted with Ctrl+C after unequal short durations. They are supporting observations, not a stable comparison.

Risk / rollback / reviewer focus:

• Primary risks are lost wakeups, shutdown ordering, and preserving the single-issuer invariant.
• Disable use_io_uring to restore the legacy runtime path, or revert this PR to retain the previous io_uring polling-thread implementation.
• Please focus on TaskGroup::Wait()/Notify(), RingListener::WaitForCqe()/HandleCqe(), and TaskControl::stop_and_join().

---

Check List:

• I have performed a self-review of my own code
• I have compiled the final diff with io_uring enabled
• End-to-end dependency-validation CI is complete

[liangjchen]

What's the motivation of this PR? If I understand correctly, it is "Let the owning brpc worker block in io_uring_submit_and_wait() instead of starting poll_thd_". But this is problematic. The reason is as follows: we use brpc worker thread as the worker thread to drive execution of all modules, including networking, runtime, tx service and storage. This worker cannot park in any module, but in a place that can be woken by any module if it has incoming tasks/requests/responses. Parking in the networking means that only the network event may wake it. What if there are cc requests combing from other cores for processing?

as for the perf improvement, what I don't get is this: when the workload is high, the brpc worker thread should be busy pooling all modules. The backgroud thread of the networking module should never get the chance to wake up. This change won't make a difference, at least not better. If this change is better, it must be something else.

btw, for the io uring, one lession I learned from Eloqstore is that because we use DEFER_TASKRUN, cqe may not arrive proactively in time. we need to frequently call io_uring_enter to get the result back. Is this relevant here?

[thweetkomputer]

What's the motivation of this PR? If I understand correctly, it is "Let the owning brpc worker block in io_uring_submit_and_wait() instead of starting poll_thd_". But this is problematic. The reason is as follows: we use brpc worker thread as the worker thread to drive execution of all modules, including networking, runtime, tx service and storage. This worker cannot park in any module, but in a place that can be woken by any module if it has incoming tasks/requests/responses. Parking in the networking means that only the network event may wake it. What if there are cc requests combing from other cores for processing?

as for the perf improvement, what I don't get is this: when the workload is high, the brpc worker thread should be busy pooling all modules. The backgroud thread of the networking module should never get the chance to wake up. This change won't make a difference, at least not better. If this change is better, it must be something else.

btw, for the io uring, one lession I learned from Eloqstore is that because we use DEFER_TASKRUN, cqe may not arrive proactively in time. we need to frequently call io_uring_enter to get the result back. Is this relevant here?

The motivation of this PR is to enable IORING_SETUP_DEFER_TASKRUN for the worker-owned io_uring. The performance improvement is not from replacing the condition variable with eventfd. I verified this with an ablation test: I kept SINGLE_ISSUER, eventfd, and the scheduler wakeup logic unchanged, and removed only DEFER_TASKRUN and TASKRUN_FLAG. The performance improvement disappeared.

In the previous design, the brpc worker submitted requests while poll_thd_ waited for CQEs. That design does not require eventfd, but DEFER_TASKRUN requires io_uring_enter(GETEVENTS) to be called by the same thread that submitted the requests (io_uring_setup(2) (https://man7.org/linux/man-pages/man2/io_uring_setup.2.html)). Therefore, the brpc worker must own both submission and CQE waiting. Once the worker waits in io_uring_submit_and_wait(), a condition-variable notification cannot wake it. Eventfd is therefore registered on the same ring so that scheduler notifications from other modules generate a CQE and wake the worker.

Before waiting, TaskGroup checks HasTask() for every registered module, and checks them again immediately before WaitForCqe(). If work arrives after the final check, NotifyWorker() eventually writes to the eventfd. Therefore, the worker is not waiting only for networking events.

While the worker is active, io_uring_peek_cqe() uses IORING_SQ_TASKRUN to let liburing enter with GETEVENTS when deferred task work is pending. When idle, io_uring_submit_and_wait(..., 1) also enters with GETEVENTS.
DEFER_TASKRUN improves performance by allowing completion task work to be processed and batched when the owning worker enters io_uring, instead of proactively interrupting its user-space execution. @liangjchen