From 7750238080d0bc731a86bd8404e0d3a18d2b967e Mon Sep 17 00:00:00 2001 From: Joseph <162703152+josephnef@users.noreply.github.com> Date: Tue, 10 Nov 2026 16:38:06 +0300 Subject: [PATCH 1/3] =?UTF-8?q?cell=20M6:=20the=20switch's=20own=20interru?= =?UTF-8?q?ption=20=E2=80=94=20first=20pass,=20TSF=20placed=20at=20the=20r?= =?UTF-8?q?etune,=20fresh=20DL=20first,=20A3=20frozen=20in=20flight?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The UE loop's pass after a switch is scheduled for the target's first superframe; the TSF is placed for the target's uplink slot right after the switch's retune (Kestrel's first beacon after a TSF step fires at the old phase: guarded against the target's active span); the CU seeds the target identity's uplink slot from the handover command; a DL frame after a switch leads with not-yet-offered PDUs (an offered one only just before the fresh PDU that would push it out of the UE's reorder window); a UE's A3 is frozen while its handover is in flight. Rig (m6_daps_xch, 4 runs per case): DL gap across the switch 122-123 ms with visits (225 before), no uplink lost before the switch. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/cell-m6.md | 89 ++++++++++++++++++++++++++++++++--- examples/cell/cell.h | 36 ++++++++++---- examples/cell/main.cpp | 65 ++++++++++++++++++++++++- tests/cell_node_selftest.cpp | 14 +++++- tools/ran-controller/main.cpp | 12 +++++ 5 files changed, 196 insertions(+), 20 deletions(-) diff --git a/docs/cell-m6.md b/docs/cell-m6.md index 12691afd..1c2e46f3 100644 --- a/docs/cell-m6.md +++ b/docs/cell-m6.md @@ -505,10 +505,10 @@ measurements. Each UE is strongest to the DU on its own box: - **The misses behind 99.9 %:** every switch between cells of different phase (1↔3: 51.2 ms apart) had exactly one target uplink out of slot, of 724–786. It came in the second superframe after the switch, 49–51 ms off, - about the phase difference, as if that one uplink still ran on the source - cell's timing. The switch 2→3, between cells of the same phase, had none - (0 of 530 and 596). Its cause is not established, and no fallback - followed. + about the phase difference. The switch 2→3, between cells of the same + phase, had none (0 of 530 and 596), and no fallback followed. The cause + is Kestrel's beacon timer: the first beacon after the switch's TSF step + fires at the old phase (see "The switch's own interruption"). - **Every switch:** TSF continuity (18–134 ms after the network instant, under 2 P), `ran.ho_done` 5–17 superframes after it, and no fallback. - **No DL in a deaf window:** no DU aired a UE a DL frame inside a suspend @@ -1308,9 +1308,12 @@ to the first after it. - An answer that is not queued, expires (3 ms after the copy that prompted it) or fails gives its PDUs back: the offered mark rolls back. - After the switch, `pick_dl_pdus` sends not-yet-offered PDUs first, but - an offered PDU before it would fall 24 sns behind them. A lost answer's + an offered PDU goes just before the fresh one that would push it out of + the UE's reorder window while it is still recoverable. A lost answer's PDU then reaches the UE inside its reorder window (selftest: a lost - answer behind a 60-PDU backlog). + answer behind a 60-PDU backlog). (This rule is from the switch- + interruption change below; the first version, "before it falls 24 sns + behind", let a large offered backlog lead the queue.) - **What is left** is the switch's own pause: ~2 superframes (225 ms). The UE leaves at the instant, retunes, and the target's grant takes effect from its next superframe. @@ -1328,6 +1331,80 @@ to the first after it. switch with visits). Same-channel DL DAPS, the three-cell crossing with a source failure, and the plain and fault handovers all pass. +### The switch's own interruption (inter-frequency) + +With the visits, the only losses left were at the switch itself. A plain +inter-frequency handover cost the UE ~2 superframes of DL (225 ms) and +1–2 uplinks. The timeline (rig, cell 2's phase 60 ms after cell 1's): +- **The switch:** the UE switches at the instant, on time (0.07 ms into the + source superframe, 42 ms into the target's), and retunes in 24 ms. +- **The target's grant** takes effect from its next superframe, 60 ms + after the instant; its first DL frame there is received. +- **The uplink beacon** was restarted only by the UE loop's NEXT pass, 100 ms + later, past that superframe's uplink slot. That uplink was lost. +- **The next uplink** fired 40.7 ms off the slot: at the source's phase. + +**Kestrel's beacon timer.** The first beacon after a large TSF step fires +at the OLD phase, whether the step comes before the beacon's restart or +right after it. The engine evidently schedules each TBTT from the previous +one and takes a TSF write only from the TBTT after next (a hypothesis that +fits every observation). The UE now predicts where that beacon lands, at +the target slot + the step: 57.2 ms predicted, 57.3 observed. This is +also the "one uplink out of slot per switch, about the phase difference +off" of the three-cell handover. + +**What changed** (`examples/cell`, `tools/ran-controller`): +- **The pass after a switch** is scheduled for the new cell's first + superframe (+1 ms), not a full loop period later. +- **The TSF is placed for the target's uplink slot** right after the + switch's retune, while the beacon is stopped (`cell.ue.switch_place`), so + every beacon from the second one is in the target's slot. +- **The stale-phase beacon** is harmless in the target's idle time, and + there it is heard and forwarded. If it would land in the target's active + span, the UE holds the restart until that TBTT has passed: one uplink + given up rather than a collision. On the rig the guard held 5 of 18 + switches, all in three-cell layouts. +- **The CU** records the target identity's uplink slot from the handover + command. The first uplinks after the switch come before the target's + report names the UE; before this, they were mapped to the wrong network + superframe, and the switch superframe counted as lost. +- **The A3 of a UE in flight is frozen** until its handover concludes. The + UE leaves at the instant and the source's reports lose its own + measurements; with a slow conclusion (the target's report naming the UE + up to ~15 superframes later, as before), the DUs' stale uplink evidence + cleared the candidate and raised a second one for the same move (no + second command, but a wrong record). + +**Rig result** (testrig `scripts/m6_daps_xch.sh`): 4 runs per case, source cell +on 36, target on 157, phase difference 60 ms, a CU DL stream of one PDU per +100 ms. "Before the switch" counts uplinks lost in the window before it; "DL +gap" is the last DL PDU received from the source to the first from the +target. + +| case | UL lost before the switch | DL silence (ms) | DL gap across the switch (ms) | +|---|---|---|---| +| source dies, visits | 0 / 0 / 0 / 0 | 205 (all 4) | 122–123 | +| source dies, no visits | 10–16 (median 13.5) | 1498–2010 (median 1805) | the same | +| source alive, visits | 0 / 0 / 0 / 0 | 205 (all 4) | 123 (all 4) | + +Before this change the gap across the switch was 225 ms with one run at +327: the target's queue after the switch led with its large offered +backlog, so the first fresh PDU waited. Now a DL frame takes the +not-yet-offered PDUs first (`pick_dl_pdus`). An offered PDU goes only just +before the fresh one that would push it out of the UE's reorder window +while it is still recoverable, so a lost answer's PDU is still a late copy, +not a confirmed gap. +Uplinks lost in the whole window, switch superframe included: 0 in 7 of the +8 visit runs, 1 in one. + +Re-run on the final build (with the frozen A3 above), 2 runs per case: +gap across the switch 122–123 ms in 3 of the 4 visit runs, 225 in one. In +that run the last visit's answer had already carried the newest PDUs, and +the next one (the CU sends one per 100 ms) was not yet at the target when +its first frame after the switch was built: that frame carried only +already-received PDUs, and the next superframe the new one. Cadence, not +queue order; no UL lost before the switch in any visit run. + ## Replay of a signed neighbour list A signature proves who wrote a beacon, not when it was aired, so the diff --git a/examples/cell/cell.h b/examples/cell/cell.h index 545fa5fe..ee2d6fe8 100644 --- a/examples/cell/cell.h +++ b/examples/cell/cell.h @@ -396,22 +396,38 @@ inline std::optional parse_dl_frame(const uint8_t *p, size_t len, uint1 } // The PDUs one DL frame carries from a UE's queue (taken out of it, in -// order): those no DAPS-visit answer has offered yet first (the UE most -// likely has the offered ones), but an offered PDU goes first once it would -// fall more than `lag` sns behind the fresh one next in line, so that a lost -// answer's PDU still reaches the UE inside its reorder window (a late copy, -// never a confirmed gap). Without an offered mark: plain queue order. +// order): those no DAPS-visit answer has offered yet first — the UE most +// likely took the offered ones, so its reception top is about the offered +// mark. Sending a fresh sn f moves that top to f, which would push an offered +// sn p out of the UE's reorder window (`window`) while p could still be +// recovered (offered - p <= window) and f - p > window: such a p goes just +// before f, so a lost answer's PDU arrives as a late copy, never a confirmed +// gap. Normally (f = offered + 1) that is at most one PDU, in the same frame. +// Without an offered mark: plain queue order. inline std::vector pick_dl_pdus(std::deque &q, const std::optional &offered, - size_t max, uint32_t lag = 24) { + size_t max, uint32_t window = 32) { std::deque fresh, old; for (uint32_t sn : q) (offered && !devourer::cell::seq_lt(*offered, sn) ? old : fresh).push_back(sn); std::vector out; while (out.size() < max && (!fresh.empty() || !old.empty())) { - const bool take_old = fresh.empty() || (!old.empty() && fresh.front() - old.front() > lag); - auto &from = take_old ? old : fresh; - out.push_back(from.front()); - from.pop_front(); + auto urgent = old.end(); + if (!fresh.empty()) + for (auto it = old.begin(); it != old.end(); ++it) + if (*offered - *it <= window && fresh.front() - *it > window) { + urgent = it; + break; + } + if (urgent != old.end()) { + out.push_back(*urgent); + old.erase(urgent); + } else if (!fresh.empty()) { + out.push_back(fresh.front()); + fresh.pop_front(); + } else { + out.push_back(old.front()); + old.pop_front(); + } } for (auto it = q.begin(); it != q.end();) it = std::find(out.begin(), out.end(), *it) != out.end() ? q.erase(it) : std::next(it); diff --git a/examples/cell/main.cpp b/examples/cell/main.cpp index a4eb0b71..6c5cf90d 100644 --- a/examples/cell/main.cpp +++ b/examples/cell/main.cpp @@ -3299,6 +3299,9 @@ int run_ue(IRadio *dev, const Cfg &c) { int64_t last_susp_sf = -1; bool susp_infeasible = false; bool had_susp_prev = false; /* last pass had neighbours only a suspend gap reaches */ + std::optional next_pass_at; /* (a switch's) */ + std::optional place_ul_slot; /* an inter-frequency switch: place the TSF after its retune */ + std::optional beacon_hold_until; /* no beacon restart before */ std::vector last_bcn; /* the uplink beacon frame on air (for DAPS visits) */ int64_t last_visit_sf = -1; uint64_t visits = 0, visits_ok = 0, steps_held = 0; @@ -3307,7 +3310,16 @@ int run_ue(IRadio *dev, const Cfg &c) { /* A DAPS-visit window: passes 30 ms apart, so the next superframe's * visit is in reach after one ends ~75 ms into its superframe. */ const bool visiting = daps_visit && ho_pending && ho_pending->channel != 0 && ho_pending->channel != cur_ch; - std::this_thread::sleep_for(std::chrono::milliseconds(visiting ? 30 : 100)); + { + /* Right after a switch: the next pass at the new cell's first superframe + * (+1 ms), so the uplink beacon restarts — and the step places it — + * before that superframe's uplink slot, not a whole pass later. */ + auto until = std::chrono::steady_clock::now() + std::chrono::milliseconds(visiting ? 30 : 100); + if (next_pass_at && *next_pass_at < until) + until = *next_pass_at; + next_pass_at.reset(); + std::this_thread::sleep_until(until); + } uint64_t t = 0, f = 0; if (!dev->ReadTsfAndFreeRun(t, f)) continue; @@ -3445,8 +3457,12 @@ int run_ue(IRadio *dev, const Cfg &c) { emit(l); rearm_ack = true; ho_back = HoBack{source_key, cur_ch, h.cell, h.ue, h.tsf_delta_us, sfn + ho_fallback_sf, h.id}; - if (h.channel != 0 && h.channel != cur_ch) + if (du_now >= 0) + next_pass_at = std::chrono::steady_clock::now() + std::chrono::microseconds((sfn + 1) * P + 1000 - du_now); + if (h.channel != 0 && h.channel != cur_ch) { retune_to = h.channel; /* inter-frequency: done right after this lock */ + place_ul_slot = h.ul_slot; /* and the TSF placed for the target's slot */ + } } /* Committed once the target's map in force grants us; no grant by the * deadline = back to the source (conditional: the UE never ends up @@ -3524,6 +3540,46 @@ int run_ue(IRadio *dev, const Cfg &c) { emit(l); } const auto r1 = std::chrono::steady_clock::now(); + if (tuned && place_ul_slot && grid && *place_ul_slot < grid->slots().size()) { + /* The switch's TSF placement, while the beacon is stopped: its + * restart then fires in the target's uplink slot from the first + * one (a step after the restart came too late for it: the first + * uplink landed at the source's phase, 40 ms off). */ + uint64_t t2 = 0, f2 = 0; + if (dev->ReadTsfAndFreeRun(t2, f2)) { + int64_t du2 = -1; + { + std::lock_guard lk(mu); + du2 = sync.RemoteForLocal(static_cast(wrap.extend(static_cast(f2)))); + } + const int64_t st = ue_tsf_step(static_cast(t2), du2, + grid->slots()[*place_ul_slot].start_us, 0, P, 2); + const bool ok = st == 0 || dev->WriteTsf(static_cast( + static_cast(dev->ReadTsf()) + st + + static_cast(std::llround(comp_us)))); + /* Kestrel's beacon timer takes a TSF write from the TBTT after next: + * the first beacon after this step fires at the OLD phase, i.e. at + * (slot + step) on the target's grid. Harmless in its idle time; + * inside its active span it would collide, so the restart then + * waits that TBTT out (one uplink, not a collision). */ + int64_t span = 0; + for (const auto &x : grid->slots()) + span = std::max(span, x.start_us + x.len_us); + const int64_t stale = ((grid->slots()[*place_ul_slot].start_us + st) % P + P) % P; + const bool collide = st != 0 && (stale < span + 1000 || stale > P - 1000); + if (collide && du2 >= 0) { + const int64_t s_at = (du2 / P + 1) * P + stale; /* the stale TBTT, next superframe */ + beacon_hold_until = std::chrono::steady_clock::now() + std::chrono::microseconds(s_at - du2 + 2000); + } + char l[200]; + snprintf(l, sizeof l, + "{\"ev\":\"cell.ue.switch_place\",\"ue\":%d,\"step_us\":%lld,\"ok\":%d,\"stale_us\":%lld," + "\"hold\":%d}", + me_a.load(), (long long)st, ok ? 1 : 0, (long long)stale, collide ? 1 : 0); + emit(l); + } + } + place_ul_slot.reset(); if (tuned) { cur_ch = to; retune_done_us.store(std::chrono::duration_cast( @@ -3760,6 +3816,11 @@ int run_ue(IRadio *dev, const Cfg &c) { const bool content_moved = force_beacon || !beaconing || running_epoch != mcopy.epoch || fb.base != last_fb.base || fb.bitmap != last_fb.bitmap || snr_gen != last_snr_gen; + if (content_moved && !beaconing && beacon_hold_until) { + if (std::chrono::steady_clock::now() < *beacon_hold_until) + continue; /* a switch's stale TBTT is still ahead (see switch_place) */ + beacon_hold_until.reset(); + } if (content_moved) { auto b = build_ue_beacon(cell_a.load(), me, c.tu, mcopy.epoch, fb, snr_half, snr_gen, auth, ue_area); { diff --git a/tests/cell_node_selftest.cpp b/tests/cell_node_selftest.cpp index c48e6dd9..a48bd430 100644 --- a/tests/cell_node_selftest.cpp +++ b/tests/cell_node_selftest.cpp @@ -162,8 +162,9 @@ int main() { dlt[26 + 13] ^= 1; /* seq */ CHECK(!parse_dl_frame(dlt.data(), dlt.size(), 5, k1)); /* What a DL frame takes from the queue after a switch: not-yet-offered - * first; an offered one before it falls 24 sns behind (a lost answer's - * PDU stays inside the UE's reorder window); plain order without a mark. */ + * first; an offered one just before the fresh one that would push it out + * of the UE's reorder window while still recoverable; plain order without + * a mark. */ { std::deque q; for (uint32_t s = 100; s < 110; ++s) @@ -192,6 +193,15 @@ int main() { } } CHECK(ok); + /* After a DAPS handover: a 40-PDU offered backlog (100..139, the UE has + * them up to 139) and the next fresh PDU (140): the fresh one rides the + * FIRST frame (only 107, which 140 would push out of the window while + * still recoverable, goes just before it). */ + std::deque h; + for (uint32_t s = 100; s < 141; ++s) + h.push_back(s); + auto f = pick_dl_pdus(h, 139u, 8); + CHECK(f.size() == 8 && f[0] == 107 && f[1] == 140); } /* A DAPS-visit DL answer: broadcast RA, the target identity in the body, * tagged with the target cell's key; not a unicast DL frame. */ diff --git a/tools/ran-controller/main.cpp b/tools/ran-controller/main.cpp index a80bf703..99373b80 100644 --- a/tools/ran-controller/main.cpp +++ b/tools/ran-controller/main.cpp @@ -981,6 +981,12 @@ int main() { retired.erase(r); } seen.insert(key); + /* A UE whose handover is in flight: its A3 is frozen until the flight + * ends. The UE leaves at the instant, and the source's reports then + * lose its own measurements; the DUs' stale ones must not clear the + * candidate and raise another for the same move. */ + if (flights.count(key)) + continue; auto it = ho.find(key); if (it == ho.end()) it = ho.emplace(key, HoState{devourer::cell::A3Trigger(cell, a3c), std::nullopt}).first; @@ -1079,6 +1085,12 @@ int main() { std::to_string(cell) + ",\"to\":" + std::to_string(force->tgt) + "}"); } ul.alias({cmd.target, cmd.target_ue}, {cmd.source, cmd.ue}); /* DAPS: one stream */ + /* The target identity's uplink slot from the command itself: + * its first uplinks come before the target's report names it, + * and must map to the slot they belong to. */ + if (cmd.target_pos >= 1 && cmd.target_pos <= tr.positions.size()) + ul_slot[{cmd.target, cmd.target_ue}] = {tr.phase_us, tr.period_us, + tr.positions[cmd.target_pos - 1].ul.start_us, ms_now()}; if (auto d = dls.find({cmd.source, cmd.ue}); d != dls.end()) { char db[200]; snprintf(db, sizeof db, From 07eb812930a55fa6380b96e08472d2952e1c3bbc Mon Sep 17 00:00:00 2001 From: Joseph <162703152+josephnef@users.noreply.github.com> Date: Wed, 11 Nov 2026 04:17:47 +0300 Subject: [PATCH 2/3] docs: the switch rig run's source channel is 149 Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/cell-m6.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/cell-m6.md b/docs/cell-m6.md index 1c2e46f3..a5542453 100644 --- a/docs/cell-m6.md +++ b/docs/cell-m6.md @@ -1376,7 +1376,7 @@ off" of the three-cell handover. second command, but a wrong record). **Rig result** (testrig `scripts/m6_daps_xch.sh`): 4 runs per case, source cell -on 36, target on 157, phase difference 60 ms, a CU DL stream of one PDU per +on 149, target on 157, phase difference 60 ms, a CU DL stream of one PDU per 100 ms. "Before the switch" counts uplinks lost in the window before it; "DL gap" is the last DL PDU received from the source to the first from the target. From b5a3b792046a6e77e15b3340b50327c828acce85 Mon Sep 17 00:00:00 2001 From: Joseph <162703152+josephnef@users.noreply.github.com> Date: Wed, 11 Nov 2026 08:43:31 +0300 Subject: [PATCH 3/3] =?UTF-8?q?cell=20M6=20switch=20interruption:=20review?= =?UTF-8?q?=20fixes=20=E2=80=94=20out-of-window=20offered=20PDUs=20before?= =?UTF-8?q?=20the=20fresh=20one,=20placement=20retried,=20expedited=20pass?= =?UTF-8?q?=20kept,=20A3=20restarted=20after=20a=20failed=20flight?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - pick_dl_pdus: an answer may have been lost, so every queued offered PDU the fresh one would push out of the UE's reorder window goes first (selftest: a 40-PDU backlog, every answer lost, through DlStreamRx: no gap). - switch_place: the stale TBTT counted from the expedited restart; a failed clock read/write keeps the placement pending, retried before the restart. - The expedited pass survives a failed clock read (2 ms retry, bounded). - The CU restarts a frozen A3 when its flight ends without the move. - cell.dl logs sn0/fresh/offered after visit answers. Co-Authored-By: Claude Opus 5.5 (1M context) --- docs/cell-m6.md | 55 +++++++----- examples/cell/cell.h | 15 ++-- examples/cell/main.cpp | 154 +++++++++++++++++++++++----------- tests/cell_node_selftest.cpp | 29 +++++-- tools/ran-controller/main.cpp | 11 +++ 5 files changed, 183 insertions(+), 81 deletions(-) diff --git a/docs/cell-m6.md b/docs/cell-m6.md index a5542453..7e68f515 100644 --- a/docs/cell-m6.md +++ b/docs/cell-m6.md @@ -1308,12 +1308,13 @@ to the first after it. - An answer that is not queued, expires (3 ms after the copy that prompted it) or fails gives its PDUs back: the offered mark rolls back. - After the switch, `pick_dl_pdus` sends not-yet-offered PDUs first, but - an offered PDU goes just before the fresh one that would push it out of - the UE's reorder window while it is still recoverable. A lost answer's - PDU then reaches the UE inside its reorder window (selftest: a lost - answer behind a 60-PDU backlog). (This rule is from the switch- - interruption change below; the first version, "before it falls 24 sns - behind", let a large offered backlog lead the queue.) + every queued offered PDU that the fresh one would push out of the UE's + reorder window goes before it. A lost answer's PDU then reaches the UE + inside its reorder window (selftests: a lost answer behind a 60-PDU + backlog; a 40-PDU backlog with every answer lost, through DlStreamRx). + (This rule is from the switch-interruption change below; the first + version, "before it falls 24 sns behind", let a large offered backlog + lead the queue.) - **What is left** is the switch's own pause: ~2 superframes (225 ms). The UE leaves at the instant, retunes, and the target's grant takes effect from its next superframe. @@ -1362,8 +1363,13 @@ off" of the three-cell handover. - **The stale-phase beacon** is harmless in the target's idle time, and there it is heard and forwarded. If it would land in the target's active span, the UE holds the restart until that TBTT has passed: one uplink - given up rather than a collision. On the rig the guard held 5 of 18 - switches, all in three-cell layouts. + given up rather than a collision. The beacon restarts no earlier than + the expedited pass, so the stale TBTT is the first at that phase from + then. A failed clock read or write keeps the placement pending: it is + retried before the restart (3 tries, then the ordinary step corrects it a + pass later), and the expedited pass itself survives a failed clock read + (retried 2 ms later). On the rig the guard held 5 of 18 switches, all in + three-cell layouts. - **The CU** records the target identity's uplink slot from the handover command. The first uplinks after the switch come before the target's report names the UE; before this, they were mapped to the wrong network @@ -1373,7 +1379,9 @@ off" of the three-cell handover. measurements; with a slow conclusion (the target's report naming the UE up to ~15 superframes later, as before), the DUs' stale uplink evidence cleared the candidate and raised a second one for the same move (no - second command, but a wrong record). + second command, but a wrong record). A flight that ends without the move + (lost DU, timeout) restarts that A3, so a new candidate needs its own + time-to-trigger. **Rig result** (testrig `scripts/m6_daps_xch.sh`): 4 runs per case, source cell on 149, target on 157, phase difference 60 ms, a CU DL stream of one PDU per @@ -1390,20 +1398,27 @@ target. Before this change the gap across the switch was 225 ms with one run at 327: the target's queue after the switch led with its large offered backlog, so the first fresh PDU waited. Now a DL frame takes the -not-yet-offered PDUs first (`pick_dl_pdus`). An offered PDU goes only just -before the fresh one that would push it out of the UE's reorder window -while it is still recoverable, so a lost answer's PDU is still a late copy, -not a confirmed gap. +not-yet-offered PDUs first (`pick_dl_pdus`). Only the offered PDUs that the +fresh one would push out of the UE's reorder window go before it (the +answers may have been lost: the offered mark is not the UE's top), so a +lost answer's PDU is still a late copy, not a confirmed gap. The queue +after a switch normally spans no more than that window, and the fresh PDU +leads. Uplinks lost in the whole window, switch superframe included: 0 in 7 of the 8 visit runs, 1 in one. -Re-run on the final build (with the frozen A3 above), 2 runs per case: -gap across the switch 122–123 ms in 3 of the 4 visit runs, 225 in one. In -that run the last visit's answer had already carried the newest PDUs, and -the next one (the CU sends one per 100 ms) was not yet at the target when -its first frame after the switch was built: that frame carried only -already-received PDUs, and the next superframe the new one. Cadence, not -queue order; no UL lost before the switch in any visit run. +Re-run on the final build (review fixes; `cell.dl` now logs, after visit +answers, a frame's first sn, how many of its PDUs were not yet offered, and +the offered mark), 3 runs per case: gap across the switch 123 ms in 4 of +the 6 visit runs, 225 in 2. In all four 123 ms runs the first frame after +the switch carried the next fresh PDU; in both 225 ms runs the target's +queue held no fresh PDU yet when that frame was built (31 PDUs, all +offered — the CU sends one per 100 ms), and the next frame led with it. +The rest of the 225 ms is PDU cadence, not queue order. Over all the runs +on this branch's builds, 18 of 24 visit runs were at 122–123 ms. The +target's queue spans ~39 sns behind the fresh PDU, so ~7 offered PDUs that +the fresh one would push out of the UE's window go before it, in the same +8-PDU frame. No UL was lost before the switch in any visit run. ## Replay of a signed neighbour list diff --git a/examples/cell/cell.h b/examples/cell/cell.h index ee2d6fe8..d68a4964 100644 --- a/examples/cell/cell.h +++ b/examples/cell/cell.h @@ -397,12 +397,13 @@ inline std::optional parse_dl_frame(const uint8_t *p, size_t len, uint1 // The PDUs one DL frame carries from a UE's queue (taken out of it, in // order): those no DAPS-visit answer has offered yet first — the UE most -// likely took the offered ones, so its reception top is about the offered -// mark. Sending a fresh sn f moves that top to f, which would push an offered -// sn p out of the UE's reorder window (`window`) while p could still be -// recovered (offered - p <= window) and f - p > window: such a p goes just -// before f, so a lost answer's PDU arrives as a late copy, never a confirmed -// gap. Normally (f = offered + 1) that is at most one PDU, in the same frame. +// likely took the offered ones. But an answer may have been lost, so the +// offered mark says nothing certain about the UE's reception top: sending a +// fresh sn f moves that top to f and confirms as missing every sn more than +// `window` behind it (DlStreamRx's reorder window). Every queued offered sn p +// with f - p > window therefore goes before f, oldest first: a lost answer's +// PDU arrives as a late copy, never a confirmed gap. Normally (the queue +// spans at most `window` sns behind f) that is none, and f leads. // Without an offered mark: plain queue order. inline std::vector pick_dl_pdus(std::deque &q, const std::optional &offered, size_t max, uint32_t window = 32) { @@ -414,7 +415,7 @@ inline std::vector pick_dl_pdus(std::deque &q, const std::op auto urgent = old.end(); if (!fresh.empty()) for (auto it = old.begin(); it != old.end(); ++it) - if (*offered - *it <= window && fresh.front() - *it > window) { + if (fresh.front() - *it > window) { urgent = it; break; } diff --git a/examples/cell/main.cpp b/examples/cell/main.cpp index 6c5cf90d..a28430c1 100644 --- a/examples/cell/main.cpp +++ b/examples/cell/main.cpp @@ -2064,6 +2064,7 @@ int run_du(IRadio *dev, const Cfg &c) { std::vector pdus; /* the CU DL PDUs this frame carries */ uint64_t pdu_epoch = 0; size_t q_left = 0; + std::optional offered_mark; /* the queue's DAPS-visit offered mark (logged) */ { std::lock_guard lk(mu); DlUe &d = dlue.at(ue); @@ -2081,6 +2082,7 @@ int run_du(IRadio *dev, const Cfg &c) { } } else if (auto qi = dlq.find(ue); qi != dlq.end()) { pdu_epoch = qi->second.epoch; + offered_mark = qi->second.offered; /* PDUs a DAPS-visit answer has not offered go first; the offered * ones (the UE most likely took them before the switch) fill the * rest, still sent in case an answer was lost — the UE takes a @@ -2133,12 +2135,21 @@ int run_du(IRadio *dev, const Cfg &c) { qi->second.sns.insert(qi->second.sns.begin(), pdus.begin(), pdus.end()); } } - char l[260]; + /* After DAPS-visit answers: what the frame carried against the mark. */ + std::string pick; + if (offered_mark && !pdus.empty()) { + size_t fresh = 0; + for (uint32_t sn : pdus) + fresh += devourer::cell::seq_lt(*offered_mark, sn) ? 1 : 0; + pick = ",\"sn0\":" + std::to_string(pdus.front()) + ",\"fresh\":" + std::to_string(fresh) + + ",\"offered\":" + std::to_string(*offered_mark); + } + char l[320]; snprintf(l, sizeof l, "{\"ev\":\"cell.dl\",\"ue\":%u,\"sf\":%lld,\"seq\":%u,\"attempt\":%d," - "\"rate\":%d,\"late_us\":%lld,\"ok\":%d,\"pdus\":%zu,\"q\":%zu}", + "\"rate\":%d,\"late_us\":%lld,\"ok\":%d,\"pdus\":%zu,\"q\":%zu%s}", ue, (long long)sf, tx->seq, tx->attempt, rate, (long long)(sub - start), ok ? 1 : 0, - pdus.size(), q_left); + pdus.size(), q_left, pick.c_str()); emit(l); } /* DL queues of UEs this DU does not serve and the CU stopped feeding @@ -3302,10 +3313,75 @@ int run_ue(IRadio *dev, const Cfg &c) { std::optional next_pass_at; /* (a switch's) */ std::optional place_ul_slot; /* an inter-frequency switch: place the TSF after its retune */ std::optional beacon_hold_until; /* no beacon restart before */ + int place_tries = 0; /* ... retried before the beacon restart if it failed */ + int64_t restart_du = -1; /* the switch's expedited pass (DU time): the beacon restarts there */ + /* The switch's TSF placement, while the beacon is stopped: its restart + * then fires in the target's uplink slot from the first one (a step after + * the restart came too late for it: the first uplink landed at the + * source's phase, 40 ms off). False (still pending) if the clock could not + * be read or written: retried before the restart, a few times. */ + auto switch_place = [&]() -> bool { + if (!place_ul_slot || *place_ul_slot >= grid->slots().size()) { + place_ul_slot.reset(); + return true; + } + uint64_t t2 = 0, f2 = 0; + int64_t du2 = -1; + if (dev->ReadTsfAndFreeRun(t2, f2)) { + std::lock_guard lk(mu); + if (sync.Ready() && wrap.ready()) + du2 = sync.RemoteForLocal(static_cast(wrap.extend(static_cast(f2)))); + } + int64_t st = 0; + bool ok = du2 >= 0; + if (ok) { + st = ue_tsf_step(static_cast(t2), du2, grid->slots()[*place_ul_slot].start_us, 0, P, 2); + ok = st == 0 || dev->WriteTsf(static_cast(static_cast(dev->ReadTsf()) + st + + static_cast(std::llround(comp_us)))); + } + if (!ok) { + char l[160]; + snprintf(l, sizeof l, "{\"ev\":\"cell.ue.switch_place_fail\",\"ue\":%d,\"try\":%d,\"what\":\"%s\"}", + me_a.load(), place_tries + 1, du2 < 0 ? "read" : "write"); + emit(l); + if (++place_tries >= 3) + place_ul_slot.reset(); /* the ordinary step corrects it, a pass later */ + return false; + } + /* Kestrel's beacon timer takes a TSF write from the TBTT after next: the + * first beacon after this step fires at the OLD phase, i.e. at + * (slot + step) on the target's grid. Harmless in its idle time; inside + * its active span it would collide, so the restart then waits that TBTT + * out (one uplink, not a collision). The beacon restarts no earlier than + * the expedited pass (restart_du), so the stale TBTT is the first one at + * that phase from then on. */ + int64_t span = 0; + for (const auto &x : grid->slots()) + span = std::max(span, x.start_us + x.len_us); + const int64_t stale = ((grid->slots()[*place_ul_slot].start_us + st) % P + P) % P; + const bool collide = st != 0 && (stale < span + 1000 || stale > P - 1000); + if (collide) { + const int64_t from = std::max(du2, restart_du); + int64_t s_at = from / P * P + stale; + if (s_at < from) + s_at += P; + beacon_hold_until = std::chrono::steady_clock::now() + std::chrono::microseconds(s_at - du2 + 2000); + } + char l[200]; + snprintf(l, sizeof l, + "{\"ev\":\"cell.ue.switch_place\",\"ue\":%d,\"step_us\":%lld,\"ok\":1,\"stale_us\":%lld," + "\"hold\":%d}", + me_a.load(), (long long)st, (long long)stale, collide ? 1 : 0); + emit(l); + place_ul_slot.reset(); + return true; + }; std::vector last_bcn; /* the uplink beacon frame on air (for DAPS visits) */ int64_t last_visit_sf = -1; uint64_t visits = 0, visits_ok = 0, steps_held = 0; uint64_t susp_gaps = 0, susp_heard = 0, susp_fail = 0; + bool expedited = false; + int expedite_tries = 0; while (std::chrono::steady_clock::now() < deadline) { /* A DAPS-visit window: passes 30 ms apart, so the next superframe's * visit is in reach after one ends ~75 ms into its superframe. */ @@ -3317,19 +3393,31 @@ int run_ue(IRadio *dev, const Cfg &c) { auto until = std::chrono::steady_clock::now() + std::chrono::milliseconds(visiting ? 30 : 100); if (next_pass_at && *next_pass_at < until) until = *next_pass_at; + expedited = next_pass_at.has_value(); next_pass_at.reset(); std::this_thread::sleep_until(until); } + /* The expedited pass is not given up on one failed clock read: retried + * 2 ms later (a few times), not at the normal pass interval. */ + auto retry_expedited = [&] { + if (expedited && ++expedite_tries <= 5) + next_pass_at = std::chrono::steady_clock::now() + std::chrono::milliseconds(2); + }; uint64_t t = 0, f = 0; - if (!dev->ReadTsfAndFreeRun(t, f)) + if (!dev->ReadTsfAndFreeRun(t, f)) { + retry_expedited(); continue; + } int64_t du_now = 0; const GrantMap *m = nullptr; GrantMap mcopy; { std::lock_guard lk(mu); - if (!sync.Ready() || !wrap.ready()) + if (!sync.Ready() || !wrap.ready()) { + retry_expedited(); continue; + } + expedite_tries = 0; du_now = sync.RemoteForLocal(static_cast(wrap.extend(static_cast(f)))); if (sync.Steps() != fit_steps_seen || sync.Rejected() != fit_rejected_seen) { char l[200]; @@ -3457,11 +3545,14 @@ int run_ue(IRadio *dev, const Cfg &c) { emit(l); rearm_ack = true; ho_back = HoBack{source_key, cur_ch, h.cell, h.ue, h.tsf_delta_us, sfn + ho_fallback_sf, h.id}; - if (du_now >= 0) - next_pass_at = std::chrono::steady_clock::now() + std::chrono::microseconds((sfn + 1) * P + 1000 - du_now); + if (du_now >= 0) { + restart_du = (sfn + 1) * P + 1000; + next_pass_at = std::chrono::steady_clock::now() + std::chrono::microseconds(restart_du - du_now); + } if (h.channel != 0 && h.channel != cur_ch) { retune_to = h.channel; /* inter-frequency: done right after this lock */ place_ul_slot = h.ul_slot; /* and the TSF placed for the target's slot */ + place_tries = 0; } } /* Committed once the target's map in force grants us; no grant by the @@ -3540,46 +3631,10 @@ int run_ue(IRadio *dev, const Cfg &c) { emit(l); } const auto r1 = std::chrono::steady_clock::now(); - if (tuned && place_ul_slot && grid && *place_ul_slot < grid->slots().size()) { - /* The switch's TSF placement, while the beacon is stopped: its - * restart then fires in the target's uplink slot from the first - * one (a step after the restart came too late for it: the first - * uplink landed at the source's phase, 40 ms off). */ - uint64_t t2 = 0, f2 = 0; - if (dev->ReadTsfAndFreeRun(t2, f2)) { - int64_t du2 = -1; - { - std::lock_guard lk(mu); - du2 = sync.RemoteForLocal(static_cast(wrap.extend(static_cast(f2)))); - } - const int64_t st = ue_tsf_step(static_cast(t2), du2, - grid->slots()[*place_ul_slot].start_us, 0, P, 2); - const bool ok = st == 0 || dev->WriteTsf(static_cast( - static_cast(dev->ReadTsf()) + st + - static_cast(std::llround(comp_us)))); - /* Kestrel's beacon timer takes a TSF write from the TBTT after next: - * the first beacon after this step fires at the OLD phase, i.e. at - * (slot + step) on the target's grid. Harmless in its idle time; - * inside its active span it would collide, so the restart then - * waits that TBTT out (one uplink, not a collision). */ - int64_t span = 0; - for (const auto &x : grid->slots()) - span = std::max(span, x.start_us + x.len_us); - const int64_t stale = ((grid->slots()[*place_ul_slot].start_us + st) % P + P) % P; - const bool collide = st != 0 && (stale < span + 1000 || stale > P - 1000); - if (collide && du2 >= 0) { - const int64_t s_at = (du2 / P + 1) * P + stale; /* the stale TBTT, next superframe */ - beacon_hold_until = std::chrono::steady_clock::now() + std::chrono::microseconds(s_at - du2 + 2000); - } - char l[200]; - snprintf(l, sizeof l, - "{\"ev\":\"cell.ue.switch_place\",\"ue\":%d,\"step_us\":%lld,\"ok\":%d,\"stale_us\":%lld," - "\"hold\":%d}", - me_a.load(), (long long)st, ok ? 1 : 0, (long long)stale, collide ? 1 : 0); - emit(l); - } - } - place_ul_slot.reset(); + if (!tuned) + place_ul_slot.reset(); /* stays on the source: nothing to place */ + else + switch_place(); if (tuned) { cur_ch = to; retune_done_us.store(std::chrono::duration_cast( @@ -3816,6 +3871,11 @@ int run_ue(IRadio *dev, const Cfg &c) { const bool content_moved = force_beacon || !beaconing || running_epoch != mcopy.epoch || fb.base != last_fb.base || fb.bitmap != last_fb.bitmap || snr_gen != last_snr_gen; + if (content_moved && !beaconing && place_ul_slot && !switch_place()) { + /* the switch's placement failed: no restart at the old phase; again soon */ + next_pass_at = std::chrono::steady_clock::now() + std::chrono::milliseconds(2); + continue; + } if (content_moved && !beaconing && beacon_hold_until) { if (std::chrono::steady_clock::now() < *beacon_hold_until) continue; /* a switch's stale TBTT is still ahead (see switch_place) */ diff --git a/tests/cell_node_selftest.cpp b/tests/cell_node_selftest.cpp index a48bd430..205344e1 100644 --- a/tests/cell_node_selftest.cpp +++ b/tests/cell_node_selftest.cpp @@ -8,6 +8,7 @@ #include #include "cell.h" +#include "cell/Handover.h" using namespace cellnode; @@ -178,7 +179,7 @@ int main() { CHECK(b == std::vector({106, 107, 108, 109, 100, 101, 102, 103})); CHECK(r == std::deque({104, 105})); /* A lost answer with a large backlog: 100..109 offered, 110..169 fresh — - * the offered ones go before the fresh run gets 24 ahead of them. */ + * each offered one goes before the fresh run gets a window ahead of it. */ std::deque w; for (uint32_t s = 100; s < 170; ++s) w.push_back(s); @@ -193,15 +194,29 @@ int main() { } } CHECK(ok); - /* After a DAPS handover: a 40-PDU offered backlog (100..139, the UE has - * them up to 139) and the next fresh PDU (140): the fresh one rides the - * FIRST frame (only 107, which 140 would push out of the window while - * still recoverable, goes just before it). */ + /* After a DAPS handover, the usual queue (offered 108..139, at most a + * window behind the next fresh PDU 140): the fresh one leads. */ std::deque h; - for (uint32_t s = 100; s < 141; ++s) + for (uint32_t s = 108; s < 141; ++s) h.push_back(s); auto f = pick_dl_pdus(h, 139u, 8); - CHECK(f.size() == 8 && f[0] == 107 && f[1] == 140); + CHECK(f.size() == 8 && f[0] == 140 && f[1] == 108); + /* A 40-PDU offered backlog (100..139) whose answers were ALL lost (the + * UE's top is 99) and fresh 140..149: the UE's DlStreamRx confirms no + * gap — 100..107 go before 140 would push them out of its window. */ + std::deque g; + for (uint32_t s = 100; s < 150; ++s) + g.push_back(s); + devourer::cell::DlStreamRx rx; + rx.accept(1, 99); + auto g1 = pick_dl_pdus(g, 139u, 8); + CHECK(g1 == std::vector({100, 101, 102, 103, 104, 105, 106, 107})); + for (uint32_t sn : g1) + rx.accept(1, sn); + while (!g.empty()) + for (uint32_t sn : pick_dl_pdus(g, 139u, 8)) + rx.accept(1, sn); + CHECK(rx.missing_confirmed() == 0 && rx.missing_total() == 0 && rx.delivered() == 51); } /* A DAPS-visit DL answer: broadcast RA, the target identity in the body, * tagged with the target cell's key; not a unicast DL frame. */ diff --git a/tools/ran-controller/main.cpp b/tools/ran-controller/main.cpp index 99373b80..3dc8e947 100644 --- a/tools/ran-controller/main.cpp +++ b/tools/ran-controller/main.cpp @@ -1126,12 +1126,22 @@ int main() { } /* In flight: (re)send to both DUs until staged; conclude when the * target reports the UE; give up long after the instant. */ + /* A flight that ends without the move: the source key's A3 (frozen in + * flight) starts over, a new candidate needs its own time-to-trigger. */ + auto a3_restart = [&](const devourer::cell::HoCommand &c, const char *why) { + if (auto h = ho.find({c.source, c.ue}); h != ho.end()) { + if (h->second.firing) + log_clear(c.ue, c.source, *h->second.firing, why); + ho.erase(h); + } + }; for (auto it = flights.begin(); it != flights.end();) { HoFlight &fl = it->second; auto sit = dus.find(fl.cmd.source), tit = dus.find(fl.cmd.target); if (sit == dus.end() || tit == dus.end()) { ul.forget({fl.cmd.target, fl.cmd.target_ue}); /* the provisional alias */ dl_undup(fl.cmd); + a3_restart(fl.cmd, "flight_lost"); it = flights.erase(it); continue; } @@ -1177,6 +1187,7 @@ int main() { cancels[fl.cmd.id] = Cancel{fl.cmd}; /* never committed: free the target */ ul.forget({fl.cmd.target, fl.cmd.target_ue}); /* and its provisional alias */ dl_undup(fl.cmd); + a3_restart(fl.cmd, "timeout"); it = flights.erase(it); continue; }