Pathological slowdown when the host denies LuaJIT executable trace memory (jit.status() true, but every trace aborts) #55

Symptom

On hosts that embed LuaJIT but cannot grant it executable memory, shen-lua boots and runs pathologically slowly — while jit.status() still reports the JIT as on:

• kernel boot: 40–66 s (vs ~1 s cold / ~0.2 s warm under a healthy local luajit)
• a 20M-iteration probe loop: 19.9 s inside the affected host vs 0.036 s under local luajit (~550×) — slower even than plain interpretation

Where it happens

Observed concretely: Envoy's Lua HTTP filter on macOS (Homebrew Envoy 1.39.0, arm64, macOS 25.x), while building examples/envoy. macOS's hardened runtime denies a binary without the JIT entitlement (com.apple.security.cs.allow-jit) the MAP_JIT executable mappings LuaJIT needs for trace mcode.

The same mechanism applies to any embedder whose host process is denied executable memory (hardened/sandboxed binaries, some SELinux deny_execmem configurations), so this is not Envoy-specific.

Root cause

jit.status() only reports whether the JIT engine is enabled, not whether mcode allocation works. With the engine on but allocation denied, every hot path attempts a trace, recording succeeds, the final mcode allocation fails (failed to allocate mcode memory), the trace aborts with a penalty — and is re-attempted on the next hot path, forever. The kernel generates ~1500 traces during boot, so boot degenerates into a continuous record/abort loop. This is the unbounded version of the trace-cache-flush cost already noted at the top of boot.lua.

Diagnosis recipe

Run a hot loop and compare against expectations, or watch trace events directly:

local compiled = false
jit.attach(function(what) if what == "stop" then compiled = true end end, "trace")
local s = 0; for i = 1, 400 do s = s + i end
-- compiled == false with jit.status() == true  =>  this issue

Fix

PR #54 (commit 2ef93b7) adds exactly that probe to boot.lua: at load, compile one throwaway hot loop under a jit.attach trace watcher; if no trace materializes, call jit.off() and note it on stderr. Measured on the affected host: 2 s cold / 1 s warm to a serving Envoy (from 40–66 s), request latency unchanged (1.5–3 ms through the proxy). Healthy hosts compile the single probe trace and are unaffected (verified silent under local luajit and OpenResty's nginx).

Overrides: SHEN_JIT=on skips the probe (opt-in for verified hosts); SHEN_JIT=off skips straight to the interpreter, as before.

Related: #43 (arm64 boot SIGSEGV on old LuaJIT betas — a different failure with the same SHEN_JIT escape hatch).