[Bug] BE SIGSEGV crash in bthread::TaskGroup::sched_to and Page Cache on 4.0.5-rc01

[iversonleo587-cpu]

Search before asking

• I had searched in the issues and found no similar issues.

Version

doris-4.0.5-rc01-59de8c4c524
Git commit: 59de8c4
JDK: OpenJDK 17.0.18+8 (/opt/jdk-17.0.18+8/bin/java)
OS: Linux x86_64

What's Wrong?

Our BE node (BackendId: 1780885838226, Host: 10.66.7.1) crashes repeatedly with SIGSEGV on version 4.0.5-rc01. This is NOT caused by Linux OOM killer (no matching records in dmesg).
Two distinct crash patterns observed in be.out:
Pattern 1 – Query execution (most frequent)
SIGSEGV at address @0x38 in bthread worker thread:
bthread::TaskGroup::sched_to → run_main_task → worker_thread
Pattern 2 – Background cache management (2026-06-25 00:20)
SIGSEGV at address @0x8 during Page Cache capacity adjustment:
Daemon::cache_adjust_capacity_thread
→ CacheManager::for_each_cache_refresh_capacity
→ LRUCache::set_capacity
→ SegmentFooterPB::~SegmentFooterPB
Most recent production incident (2026-06-25):

• 08:11:34 BE started (PID 419300)
• ~13:40:22 BE crashed (Aborted at unix time 1782366022)
• 13:41:10 Client ETL job failed: (1105, 'errCode = 2, detailMessage = backend 1780885838226 is down')
• 13:41:55 BE restarted automatically
  Crash frequency on this single BE (same version):
  | Date | Query id | Stack |
  |------------|---------------------------------------|--------------------------------|
  | 2026-06-22 | 6c0e5ba4c3ab45f2-a3734229ec2d08ba | bthread SIGSEGV @0x38 |
  | 2026-06-23 | 0-0 | bthread SIGSEGV @0x38 |
  | 2026-06-24 | 7133b64a2c24c1d-942bd6304aa27668 | bthread SIGSEGV @0x38 |
  | 2026-06-24 | 594b7141026d19bf-7a05c24fda60e79f | bthread SIGSEGV @0x38 |
  | 2026-06-25 | 0-0 | Page Cache/SegmentFooterPB @0x8 |
  | 2026-06-25 | 0-0 | bthread SIGSEGV @0x38 |
  BE environment:
• CPU: 32 cores, Memory: ~125 GB
• TabletNum: 17860, DataUsedCapacity: ~585 GB, Disk UsedPct: ~35%
• Workload mode: mix
  Stack trace – most recent crash (2026-06-25 ~13:40, before restart at 13:41:55):

What You Expected?

BE should remain stable under heavy query/load pressure. If resource limits are exceeded, Doris should fail the query gracefully (e.g., memory limit exceeded, query cancelled), not crash the entire BE process with SIGSEGV.
After a failure, clients should receive a query-level error, not "backend is down" due to process crash.

How to Reproduce?

We have not isolated a minimal reproducible case yet. Crashes appear under sustained production load on a single BE node running 4.0.5-rc01.
Observed trigger pattern:

1. BE runs for several hours under mixed query/load workload
2. A heavy INSERT INTO ... SELECT ETL job starts
3. BE crashes with SIGSEGV in bthread::TaskGroup::sched_to
4. FE reports "backend is down" to clients
5. BE is restarted (~1 minute later)
   Recent failing workload (high level, no full SQL):

• Job: ETL via DolphinScheduler (pre environment)
• SQL pattern:
  • TRUNCATE TABLE dws.dws_finance_sku_order_time_expense_stats_h
  • Two large INSERT INTO ... SELECT statements (is_combination=0 and is_combination=1)
  • Source: large DWD fact tables (dwd_finance_order_expense_*)
  • Operations: GROUP BY on 20+ columns, UNION ALL, multiple LEFT JOINs (dim tables + ods table)
• Task runtime before failure: ~3 minutes (13:37:58 → 13:41:10 CST)
  Cluster info:
• Single affected BE: 10.66.7.1 (BackendId 1780885838226)
• 17860 tablets on this BE (relatively heavy)
• No disk pressure (UsedPct ~35%)
• No Linux OOM kill records in dmesg
  What we ruled out:
• Linux OOM killer: dmesg shows no OOM/kill records around crash time
• Disk full: UsedPct ~35%
• Manual shutdown: isShutdown=false in SHOW BACKENDS
• One-off incident: same BE crashed 6+ times in 3 days with identical stack patterns
  To help reproduce:
  We can provide full be.out crash logs and SQL privately if needed. The crash also occurs without a specific query id (Query id: 0-0), suggesting it may be triggered by general load rather than one specific SQL shape.

Anything Else?

Impact:

• Production ETL jobs fail with "backend is down"
• BE requires repeated manual/automatic restarts
• Data risk: jobs use TRUNCATE before INSERT, leaving target table empty/incomplete on failure
  Questions for maintainers:

1. Are there known fixes for bthread::TaskGroup::sched_to SIGSEGV (@0x38) in 4.0.x after commit 59de8c4?
2. Is the SegmentFooterPB destructor crash during cache_adjust_capacity_thread() a known issue in 4.0.5-rc01?
3. Is 4.0.5-rc01 recommended for production? Which stable version should we upgrade to?
4. Any workaround besides version upgrade (cache/memory config, query-side mitigation)?
   Attachments we can provide:

• Full be.out crash sections
• be.WARNING / be.INFO around 13:40-13:42 CST on 2026-06-25
• SHOW BACKENDS output
• Full SQL text (privately if needed)

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[zclllyybb]

Breakwater-GitHub-Analysis-Slot: slot_2bb1eec8775e
This content is generated by AI for reference only.

Initial triage from the issue text and the branch-4.0 code at 59de8c4c524:

1. This should be treated as a real BE crash bug, not as normal query memory-limit behavior. The absence of Linux OOM-killer records only rules out a kernel kill; it does not explain repeated SIGSEGV inside the BE process.

2. The bthread::TaskGroup::sched_to frame alone is not enough to identify the Doris operator that corrupted state. Doris 4.0.x bundles brpc 1.4.0, where sched_to(pg, tid) immediately resolves and dereferences TaskMeta. A SIGSEGV at a very small address such as 0x38 in that frame usually means invalid scheduler/task metadata or prior memory corruption. The useful frames are the Doris frames below the bthread scheduler, which are not included in the issue body.

3. For the page-cache stack, the reported path is plausible as an eviction/destruction site: cache_adjust_capacity_thread() calls CacheManager::for_each_cache_refresh_capacity(), which calls LRUCachePolicy::adjust_capacity_weighted(), which calls LRUCache::set_capacity() and may evict cached entries. Segment footers are cached in the storage page cache as a MemoryTrackedPageWithPagePtr<std::shared_ptr<SegmentFooterPB>>, so SegmentFooterPB::~SegmentFooterPB can legitimately appear while evicting index-page-cache entries. However, the normal lifetime path appears intended to be safe: PageCacheHandle holds a cache handle while copying out the shared_ptr, and Segment stores only a weak_ptr. So a crash in the destructor is more suggestive of memory corruption or a cache-entry lifetime bug than of ordinary page-cache capacity adjustment by itself.

4. I checked the local 4.0 tags: 4.0.5 and 4.0.5-rc01 point to the same commit 59de8c4c524, while 4.0.6 points to 4.0.6-rc02. The same-line 4.0.6 tag contains several crash fixes after 59de8c4c524, including:

   • #63257: fixes a branch-4.0 BE SIGSEGV when shared-hash-table hash joins publish runtime filters during close. This is potentially relevant because the workload includes multiple joins and heavy INSERT INTO ... SELECT.
   • #62427: removes SchemaCache to fix a concurrent crash in OlapScanner::prepare.
   • #61444: fixes a file-cache background LRU SIGSEGV when clearing cache.

None of those is proven to be the same root cause as the SegmentFooterPB destructor crash from the shortened stack, but they are enough evidence that staying on 4.0.5/4.0.5-rc01 is risky for a production workload that is already repeatedly crashing.

Suggested next steps:

1. Upgrade the affected BE from 4.0.5/4.0.5-rc01 to the latest available 4.0.x release, at least the local same-line 4.0.6 tag, before spending too much time tuning around the crash. This is the strongest immediate mitigation because there are known branch-4.0 crash fixes after the reported SHA.

2. As a temporary query-side mitigation for the ETL job, try disabling shared hash table for broadcast joins at the session/job level:

   SET enable_share_hash_table_for_broadcast_join = false;

   This targets the #63257 class of crash. It may reduce performance, and it only helps if the full stack or profile confirms the failing plan uses broadcast hash joins/shared runtime filters.

3. Avoid TRUNCATE followed by large INSERT INTO ... SELECT as the production recovery pattern while this BE is unstable. Use a staging table plus atomic publish/rename/swap pattern where possible, so a BE crash does not leave the target table empty.

4. If the page-cache crash recurs before upgrade and it always occurs in cache_adjust_capacity_thread, disable_memory_gc=true can be considered only as emergency containment with close memory monitoring. It skips that background adjustment path but can worsen memory pressure, so it should not be the primary fix.

Information still needed to root-cause this precisely:

1. Full be.out fatal sections for all crash patterns, including the signal header and complete symbolized stack, not only the top frames.
2. be.INFO and be.WARNING for at least 5-10 minutes around each crash, especially [MemoryGC] cache-capacity logs, fragment cancellation logs, runtime-filter/hash-join logs, and any file/page-cache warnings.
3. The full SQL or at least EXPLAIN/profile for the failing ETL statements, with session variables used by DolphinScheduler.
4. BE config/session values for disable_memory_gc, enable_memory_orphan_check, storage_page_cache_limit, index_page_cache_percentage, enable_share_hash_table_for_broadcast_join, query/load memory limits, and whether page/file cache was enabled.
5. A core dump or addr2line output produced with the exact doris_be binary/build id from 59de8c4c524.
6. Whether the repeated bthread::TaskGroup::sched_to @0x38 crashes include Doris frames such as RuntimeFilterWrapper, HashJoinBuildSink, OlapScanner, exchange/data-stream code, or cache code below the bthread scheduler.