Skip to content

fix(ios): keep one upload's body stream from holding up every other upload - #814

Merged
jkmassel merged 3 commits into
fix/own-media-delegate-stronglyfrom
jkmassel/upload-writer-never-waits
Oct 21, 2026
Merged

jkmassel merged 3 commits into
fix/own-media-delegate-stronglyfrom
jkmassel/upload-writer-never-waits

Conversation

@jkmassel

@jkmassel jkmassel commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Fixes a hang where one upload body could stall every other upload that GutenbergKit's own media client sends. On CI, it hung two EditorViewControllerMediaTeardownTests upload tests until the job was cancelled, in 4 of 21 iOS Simulator Tests runs on 2026-10-19:

  • "with no handler, the page's upload reaches the site through GutenbergKit's own client"
  • "the page can read the attachment ID off a failed upload"

It cost #804 a run (build 3286) and three of #813's soak runs (build 3287).

This targets #625's branch, where MultipartBodyWriter arrived with #747.

Who is affected

The CI hang came from a stream type only the tests use, so only test runs have hung.

Apps can hit a related problem. It applies to hosts that don't supply their own media upload handler, so uploads go through InternalMediaClient. If a "room available" event reaches a writer whose buffer is already full, the old writer waits for that upload's reader, and every other upload waits with it. We have not seen this in an app, and haven't reproduced it end to end.

Root Cause

Every MultipartBodyWriter shares one serial queue. That is safe only if no write blocks. Two things let a write block it.

1. The stream was opened off the queue

start() handed the stream to the queue, then called open() on the caller's thread. The stream could deliver its first event to the queue while open() was still running. The two threads' calls into the stream could leave it stuck "opening", and a write then waited in waitForOpen forever.

A sample of a hung CI run (build 3290, from a throwaway branch) showed:

  • the shared queue's thread in CFWriteStreamWrite → waitForOpen
  • five other threads waiting to read bodies that never came

In that job's log, "the writer sends the full body and closing boundary when the file reads cleanly" never finished. That test writes to OutputStream.toMemory(); the other memory-stream test passed in 1.6 s. The upload tests hung because they share the queue it blocked. A stress run on an iPhone 15 Pro reproduced the hang: with the old writer, the shared queue was held for 25 s in the first round.

2. The writer wrote without checking for room

The writer wrote on every "room available" event, but an event doesn't promise room: one can be left over from before the last write filled the buffer. Writing to a full bound pair, which upload bodies use, blocks until the reader reads.

Why the stuck writer wasn't an upload's
  • In the iOS 26.5 Simulator CoreFoundation that CI ran, a bound pair reports its open as finished as soon as anything asks for its status. So CFWriteStreamWrite never reaches waitForOpen for one. A memory stream has no such check, and signals room from inside its open.
  • On the iPhone, the memory streams that got stuck stayed at status "opening" with nothing written, 20 s later. No bound pair got stuck in any run.

Fix

ios/Sources/GutenbergKit/Sources/Media/MultipartBodyWriter.swift:

  • Open the stream on the queue, so once start() has run, only the queue calls into the stream.
  • Write only when output.hasSpaceAvailable says the stream can take bytes; otherwise do nothing. With any room at all, a bound pair's write takes what fits and returns at once.
  • Also write on .openCompleted, so an event skipped because the stream hadn't finished opening is retried.
  • queue is now internal (was private), so the tests can use it.

Test plan

  • New test "a writer whose stream can't take bytes doesn't hold up other writers". It sends a "room available" event to a writer whose buffer is full and nobody is reading, then checks that another block on the shared queue still runs. Without the room check it fails every time (3 of 3, timing out after 60 s), and the rest of the suite still finishes. With it, it passes in about 20 ms.
  • New test "a writer opens its stream on the shared queue, not on the caller's thread". It holds the queue, starts a writer, and checks that the stream isn't open yet. With the old start() it fails every time (3 of 3); with the fix it passed 10 of 10.
  • New test "a writer writes when its stream finishes opening". Without the .openCompleted change it fails; with it, it passes.
  • Whole Swift package on the Mac: 799 + 396 pass.
  • Whole iOS Simulator suite on CI: build 3311.
  • SwiftLint is clean.
  • A throwaway stress run, not committed, on the Mac, iOS 26.5 and 27.0 simulators, and an iPhone 15 Pro. Each run sends thousands of bodies to readers that stall, leave, never open, or are cancelled, plus 160 real URLSession uploads to a local server that trickles, stalls or hangs up. Every body is checked byte for byte.
    • It catches the bug: the old writer failed on every platform, and on the iPhone it reproduced the CI hang.
    • It found cause 1: with only the room check, a memory-stream writer still got stuck opening in 3 of 5 iPhone runs. With the open moved onto the queue, none did in 10 runs (about 38,400 writers).
    • Not changed by this PR: on the iOS 27.0 simulator, with about 100 streams at once, a write occasionally waited 100–150 ms on CoreFoundation's own locks. It didn't happen on the iPhone.
  • 20 runs of the iOS Simulator Tests job on CI at low priority (build 3298): all passed. That was before the open moved onto the queue.

@jkmassel jkmassel added [Type] Bug An existing feature does not function as intended iOS labels Oct 19, 2026
@jkmassel jkmassel self-assigned this Oct 19, 2026
@wpmobilebot

wpmobilebot commented Oct 19, 2026 •

Copy link
Copy Markdown

XCFramework Build

This PR's XCFramework is available for testing. Add the following to your Package.swift:

.package(url: "https://github.com/wordpress-mobile/GutenbergKit", branch: "pr-build/814")

Built from 312a1c7

@jkmassel
jkmassel requested a review from dcalhoun October 19, 2026 18:47
@jkmassel
jkmassel force-pushed the jkmassel/upload-writer-never-waits branch 2 times, most recently from 56b3ab5 to bb83201 Compare October 19, 2026 18:54
…pload

Every `MultipartBodyWriter` shares one serial queue, so no write may
block. Two things let a write block it.

`start()` opened the stream on the caller's thread after handing it to
the queue, so the stream could deliver an event to the queue while
`open` was still running. The two threads' calls into the stream could
leave it stuck opening, and a write then waited in `waitForOpen`
forever. That hung two `EditorViewControllerMediaTeardownTests` upload
tests on CI, in 4 of 21 runs on 2026-10-19: the stuck writer was a
test's, writing to `OutputStream.toMemory()`, and every writer behind
it on the queue stopped. A stress run on an iPhone 15 Pro reproduced
it.

The writer also wrote on every "room available" event without
checking, but an event doesn't promise room. Writing to a full bound
pair, which upload bodies use, blocks until its reader reads. That
hasn't been seen in an app.

The writer now opens the stream on the queue, writes only when the
stream says it can take bytes, and also writes on `.openCompleted`.
New tests check that a writer whose stream is full leaves the queue
free, that `.openCompleted` writes, and that `start()` opens the
stream on the queue. The queue is internal so the tests can use it.
@jkmassel
jkmassel force-pushed the jkmassel/upload-writer-never-waits branch from bb83201 to 2623b72 Compare October 19, 2026 21:21

@dcalhoun dcalhoun left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a draft, but my review is requested, so I presumed the draft state was erroneous. This looks good.

I captured one finding by Claude worth considering. I could be misguided.

Comment thread ios/Tests/GutenbergKitTests/Media/InternalMediaClientTests.swift Outdated
`writerNeverHoldsUpTheSharedQueue` waited for the stream's buffer to
fill by calling `queue.sync` inside `waitUntil`. `waitUntil` checks its
deadline only between calls. So if the queue was stuck, `queue.sync`
never returned, and the run hung without naming this test.

The check now runs on the queue with `async` and reports back through a
lock, so a stuck queue fails this wait after 60 s. If the stuck write
is this test's own, the `defer` then frees the queue, and other tests
waiting on it carry on. A queue stuck by anything else stays stuck, and
tests that wait on a writer with no timeout still hang the run.

The check also requires the stream to be open. A closed stream also
reports no room, so a writer that closed its stream early would have
passed the old wait, and the test would have checked nothing.

Co-authored-by: David Calhoun <github@davidcalhoun.me>
jkmassel added a commit that referenced this pull request Oct 20, 2026
`writerOpensOnTheQueue` held the shared writer queue on a semaphore
until the test's own task ran again. Other tests in the suite read a
body with a blocking read on a Swift concurrency thread, and those reads
wait on that queue. When they took every thread, the test's task could
never run, so the hold was never released. The whole run then hung
without naming a test. With the thread pool narrowed to 4 threads, 4 of
5 runs hung this way. With this test skipped, none did.

The test no longer holds the queue. It calls `start` from a block on the
queue and reads the stream's status in that same block. The writer
queues its own `open` behind the block, so the status still shows
whether `start` opened the stream itself, and nothing waits on anything.
@jkmassel
jkmassel marked this pull request as ready for review October 21, 2026 01:13
@jkmassel
jkmassel merged commit 7bfaecb into fix/own-media-delegate-strongly Oct 21, 2026
1 check passed
@jkmassel
jkmassel deleted the jkmassel/upload-writer-never-waits branch October 21, 2026 04:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

iOS [Type] Bug An existing feature does not function as intended

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants