Sandbox ID or Build ID
No response
Environment
Nomad v1.10.5 (LTS)
resource memory=100G
resouce memory_max=-1
Orchestrator: Go process managing Firecracker sandbox lifecycle
~120 concurrent sandboxes per node
What we've ruled out so far:
No Go heap leak detected via pprof (retained heap is normal)
NBD dispatch buffers (~492MB) and Firecracker metrics buffers (~121MB) are per-sandbox fixed costs, totaling ~600MB — not the 75GB culprit
The 75GB anon-rss is likely guest memory backing / UFFD mappings outside Go's heap profile
Timestamp of the issue
Tue Sep 15 19:17:16
Frequency
Happens occasionally
Expected behavior
no oom,Our questions:
Troubleshooting: What's the recommended approach to pinpoint which anonymous mmap regions (guest memory, UFFD, or something else) are consuming the majority of anon-rss? Are there specific smaps patterns or Nomad/cgroup metrics we should focus on?
Per-sandbox memory accounting: Is there a known relationship between the number of concurrent Firecracker sandboxes and orchestrator anon-rss growth? Any best practices for bounding per-sandbox memory overhead on the orchestrator side?
Nomad version: We're currently on Nomad v1.10.5 (LTS, supported until April 2027). Is this the recommended version for production E2B deployments, or should we consider upgrading to 2.0.x? Are there any known Nomad-related memory/cgroup issues in 1.10.x that could contribute to this?
Actual behavior
oom
Any guidance or pointers to relevant docs/issues would be greatly appreciated. Thanks!
Issue reproduction
no steps to reproduce
Additional context
OOM details (from dmesg):
mesg -T | grep oom
[Wed Sep 9 20:22:01 2026] orchestrator invoked oom-killer: gfp_mask=0x400cc0(GFP_KERNEL_ACCOUNT), order=1, oom_score_adj=0
[Wed Sep 9 20:22:01 2026] oom_kill_process+0x118/0x280
[Wed Sep 9 20:22:01 2026] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[Wed Sep 9 20:22:01 2026] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=699af8c1-34d3-f80e-88a0-b82bc06b3fdd.start.scope,mems_allowed=0-1,oom_memcg=/nomad.slice/share.slice/699af8c1-34d3-f80e-88a0-b82bc06b3fdd.start.scope,task_memcg=/nomad.slice/share.slice/699af8c1-34d3-f80e-88a0-b82bc06b3fdd.start.scope,task=orchestrator,pid=1146907,uid=0
[Wed Sep 9 20:22:01 2026] Memory cgroup out of memory: Killed process 1146907 (orchestrator) total-vm:18482419796kB, anon-rss:78803048kB, file-rss:15898016kB, shmem-rss:0kB, UID:0 pgtables:5599708kB oom_score_adj:0
[Wed Sep 9 20:22:20 2026] oom_reaper: reaped process 1146907 (orchestrator), now anon-rss:0kB, file-rss:14483156kB, shmem-rss:0kB
Any guidance or pointers to relevant docs/issues would be greatly appreciated. Thanks!
Sandbox ID or Build ID
No response
Environment
Nomad v1.10.5 (LTS)
resource memory=100G
resouce memory_max=-1
Orchestrator: Go process managing Firecracker sandbox lifecycle
~120 concurrent sandboxes per node
What we've ruled out so far:
No Go heap leak detected via pprof (retained heap is normal)
NBD dispatch buffers (~492MB) and Firecracker metrics buffers (~121MB) are per-sandbox fixed costs, totaling ~600MB — not the 75GB culprit
The 75GB anon-rss is likely guest memory backing / UFFD mappings outside Go's heap profile
Timestamp of the issue
Tue Sep 15 19:17:16
Frequency
Happens occasionally
Expected behavior
no oom,Our questions:
Troubleshooting: What's the recommended approach to pinpoint which anonymous mmap regions (guest memory, UFFD, or something else) are consuming the majority of anon-rss? Are there specific smaps patterns or Nomad/cgroup metrics we should focus on?
Per-sandbox memory accounting: Is there a known relationship between the number of concurrent Firecracker sandboxes and orchestrator anon-rss growth? Any best practices for bounding per-sandbox memory overhead on the orchestrator side?
Nomad version: We're currently on Nomad v1.10.5 (LTS, supported until April 2027). Is this the recommended version for production E2B deployments, or should we consider upgrading to 2.0.x? Are there any known Nomad-related memory/cgroup issues in 1.10.x that could contribute to this?
Actual behavior
oom
Any guidance or pointers to relevant docs/issues would be greatly appreciated. Thanks!
Issue reproduction
no steps to reproduce
Additional context
OOM details (from dmesg):
mesg -T | grep oom
[Wed Sep 9 20:22:01 2026] orchestrator invoked oom-killer: gfp_mask=0x400cc0(GFP_KERNEL_ACCOUNT), order=1, oom_score_adj=0
[Wed Sep 9 20:22:01 2026] oom_kill_process+0x118/0x280
[Wed Sep 9 20:22:01 2026] [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[Wed Sep 9 20:22:01 2026] oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=699af8c1-34d3-f80e-88a0-b82bc06b3fdd.start.scope,mems_allowed=0-1,oom_memcg=/nomad.slice/share.slice/699af8c1-34d3-f80e-88a0-b82bc06b3fdd.start.scope,task_memcg=/nomad.slice/share.slice/699af8c1-34d3-f80e-88a0-b82bc06b3fdd.start.scope,task=orchestrator,pid=1146907,uid=0
[Wed Sep 9 20:22:01 2026] Memory cgroup out of memory: Killed process 1146907 (orchestrator) total-vm:18482419796kB, anon-rss:78803048kB, file-rss:15898016kB, shmem-rss:0kB, UID:0 pgtables:5599708kB oom_score_adj:0
[Wed Sep 9 20:22:20 2026] oom_reaper: reaped process 1146907 (orchestrator), now anon-rss:0kB, file-rss:14483156kB, shmem-rss:0kB
Any guidance or pointers to relevant docs/issues would be greatly appreciated. Thanks!