fix(cf-workers): log the request-body pipe's rejection instead of dropping it - #135
fix(cf-workers): log the request-body pipe's rejection instead of dropping it#135alukach wants to merge 1 commit into
Conversation
…pping it `forward` piped the inbound body into a FixedLengthStream and discarded the `pipe_to` promise, on the reasoning that a pipe failure also errors `readable` so the awaited outbound fetch would surface it. Production says otherwise. When the pipe fails the origin sees a truncated request and closes without replying; the runtime turns that into a Cloudflare-minted `error code: 520` -- a 16-byte text/plain body with no origin request id and no error document. The proxy relays that status faithfully, so the caller gets a 520 that no AWS SDK can deserialize, and the reason the pipe failed is gone. Observe the promise on a spawned task and log the rejection. The distinct causes -- inbound body short of or past the declared Content-Length (either errors the FixedLengthStream), the inbound stream erroring, the outbound fetch cancelling `readable` -- are indistinguishable downstream but named here. Diagnostic only: no behaviour change, and `forward` still returns as soon as the outbound response headers arrive. Cargo.lock is included because it was stale on main at the workspace's previous version; the pre-commit clippy hook regenerates it on every run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Claude finished @alukach's task in 2m 39s —— View job Review complete
✅ No blocking issues — safe to merge. The change is diagnostic-only, matches the stated scope, and doesn't touch the One non-blocking robustness note:
Simplify (ponytail) Nothing to cut — single-purpose helper, no new dependency, no speculative abstraction. (Minor, not worth a line item: the rationale for why the pipe rejection matters is written out twice — once in 💰 Estimated review cost: $0.80 · 2m39s · 23 turns |
|
🚀 Latest commit deployed to https://multistore-proxy-pr-135.development-seed.workers.dev
|
What I'm changing
WorkerBackend::forwardpipes a streamed PUT body into aFixedLengthStreamand discards thepipe_topromise. The comment justified it: a pipe failure also errorsreadable, so the awaited outbound fetch would surface the error.It does not. Chasing a persistent ~0.1% failure rate on
PutObject/UploadPartin a downstream proxy, the observed sequence is:error code: 520—server: cloudflare, a subrequestcf-ray,text/plain, 16 bytes, no origin request id and no error document;GatewayResponse::Forwardrelays that status faithfully, so the caller gets a 520 whose body no AWS SDK can deserialize.Every trace of why the pipe failed is gone, because the only object that carried it was dropped on the floor. That is the gap this closes. It does not fix the failure — it makes the failure legible.
Confirmed from production logs across two independent invocations (4.9 MB
PutObjectand 16 MiBUploadPart), both showing the identical Cloudflare-origin signature with zerox-amz-*headers.How I did it
crates/cf-workers/src/backend.rs— newlog_pipe_rejection(pipe, declared_len). Observes thepipe_topromise onwasm_bindgen_futures::spawn_localand logs the rejection at WARN with the declaredContent-Length. Spawned rather than awaited:forwardmust still return as soon as the outbound response headers arrive, and the pipe is driven by that fetch's backpressure. The task only observes a promise — it issues no I/O and does not extend the request's lifetime.let _ = stream.pipe_to(...)with the call, and rewrote the comment: the old one asserted behaviour that production contradicts, which is worse than no comment.Content-Length(either errors theFixedLengthStream), the inbound stream erroring, or the outbound fetch cancellingreadable.Cargo.lock— was stale onmainat the workspace's previous version; the pre-commit clippy hook regenerates it on every run, so it is included rather than fought.Diagnostic only. No behaviour change, and the
None(aws-chunked) branch is untouched — it has no pipe to observe.Test plan
cargo fmt --checkcargo clippy --fix -p multistore-cf-workers --target wasm32-unknown-unknowncargo checkcargo check -p multistore-cf-workers --target wasm32-unknown-unknownNo test accompanies this. The crate has no wasm test harness — the only
cfg(test)module is a host-side one incors.rs— and the behaviour needs a liveReadableStreamunder a real Workers runtime to exercise. Flagging that rather than papering over it: the repo convention is a failing test first, and I could not write one that would actually fail.Follow-up
Once the rejection reason is visible, the real fix follows. If the pipe is failing on a length mismatch, the outbound request should fail with a proper
ProxyErrorrather than letting a truncated body reach the origin and turn into an opaque 520.