From c38e141926685a6e316369f6d9af4c15ff4ce5b2 Mon Sep 17 00:00:00 2001 From: Jason Evans Date: Thu, 8 Oct 2026 17:07:06 -0400 Subject: [PATCH 1/7] Add proxy-side identity capture design spec for lockr-issued IDs Lockr-issued IDs reach client-side bidders through ortb2.user.ext.eids but never the server-side auction, the ts-eids cookie, or the EC identity graph (#1246). This spec captures the IDs in the first-party proxy response from lockr's API, sets a server-owned HttpOnly cookie on that response, and defers the KV upsert to the post-send slot so no KV write sits on the hot path. --- ...026-10-08-lockr-identity-capture-design.md | 461 ++++++++++++++++++ 1 file changed, 461 insertions(+) create mode 100644 docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md diff --git a/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md b/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md new file mode 100644 index 000000000..a51b336ce --- /dev/null +++ b/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md @@ -0,0 +1,461 @@ +# Proxy-side identity capture for lockr-issued IDs + +Status: Design proposed, awaiting engineering review. + +Issue: [#1246](https://github.com/IABTechLab/trusted-server/issues/1246). + +Related: [2026-07-10 KV EID request snapshot and EC recovery](2026-07-10-kv-eid-request-snapshot-ec-recovery-design.md) +defines the snapshot and post-send primitives this design reuses. + +## 1. Purpose + +The lockr SDK obtains partner IDs (ID5 today; RampID, UID2, EUID, FirstID, +Yahoo ConnectID, PanoramaID, Criteo and Epsilon by configuration) and hands +them to Prebid through `pbjs.setConfig({ortb2: {user: {ext: {eids}}}})`. +Trusted Server collects auction EIDs from `pbjs.getUserIdsAsEids()`, which only +reports the Prebid User ID module. The lockr IDs therefore reach client-side +bidders and never reach the server-side auction, the `ts-eids` cookie, or the +EC identity graph. This was measured on the production publisher route on +2026-10-06 (Safari) and reconfirmed on 2026-10-08 (Chrome, direct versus TS +matched arm): six captured `/auction` payloads contained no `id5-sync.com` +entry while the same page carried the ID in `ortb2.user.ext.eids`. + +The fix proposed in the issue thread (merge configured `ortb2` EIDs into the +JavaScript collector) corrects the symptom but keeps three dependencies this +design removes: + +1. The ID still travels through Prebid configuration. +2. The ID still depends on the tsjs prebid module running in the page. +3. The `ts-eids` cookie is written in `bidsBackHandler`, after the first + auction request has left, and KV ingestion happens on a later eligible + navigation. + +This design captures the IDs where Trusted Server already sees them, in the +first-party proxy response from lockr's API, and writes them to a server-owned +first-party cookie immediately and to KV after the response has been sent. + +## 2. Observed lockr flow + +All lockr traffic on a TS page goes through `/integrations/lockr/api/*`, proxied +by `handle_api_proxy` in `crates/trusted-server-core/src/integrations/lockr.rs` +to `https://identity.loc.kr`. The trust-server SDK build hard-codes +`host: "/integrations/lockr/api"`, so no client shim is involved in routing. + +Measured call sequence after the publisher's consent gate opens: + +| Order | Request | Carries | +| ----- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| 1 | `GET /integrations/lockr/sdk` | SDK body, 118,542 bytes | +| 2 | `POST .../publisher/app/v2/identityLockr/settings` | Request body `{appID}`. Response: publisher flags, `rampJSClientId`, client IP and geo, `hashedUserAgent`, `vvFlag` | +| 3 | `POST .../publisher/app/v2/identityLockr/page-view` | Request body: `LTID`, existing `tokens` keyed by provider, `noGenerate`, consent strings, URL, referrer. Response on first issue: `aimTokens[]` and `ids..eid` | +| 4 | `POST .../refresh-tokens`, `.../generate-tokens`, `.../sync-no-hem-ids` | Later token updates, same shapes | +| 5 | `POST .../revoke-consent` | `LTID` and `optOutReason` | + +Measured `page-view` response on first issue (identifier values elided): + +```json +{ + "success": true, + "data": true, + "pvStatus": true, + "aimTokens": [ + { + "advertising_token": "%7B%22universal_uid%22%3A%22ID5*...%22%7D", + "identity_expires": 1791315739717, + "key_name": "id5id", + "settings": { "dropCookie": false, "dropLocalStorage": false } + } + ], + "ids": { + "id5id": { + "eid": { + "source": "id5-sync.com", + "uids": [ + { + "id": "ID5*...", + "atype": 1, + "ext": { "linkType": 1, "pba": "..." } + } + ] + } + } + } +} +``` + +The response is 603 bytes. The SDK then writes cookie and localStorage `id5id`, +and exposes the EID to Prebid with `inserter: "Lockr-For-Publishers"`, +`matcher: ""`, `mm: 2`. + +The request arrives at the proxy as a same-origin XHR, so it carries the +`ts-ec`, `__gpp`, `__gpp_sid` and `us_privacy` cookies, and its body carries +lockr's own `gppString`, `consentString` and `ccpaString`. The integration route +already builds an `EcContext` from the request and attaches `EcFinalizeState` +to the response (`dispatch_fallback` in +`crates/trusted-server-adapter-fastly/src/app.rs`). The Fastly entry point +already runs post-send work after `send_edgezero_response` (pull sync in +`crates/trusted-server-adapter-fastly/src/main.rs`). + +Nothing reads the proxied response body today. `handle_api_proxy` returns the +upstream response as-is. + +## 3. Requirements + +1. Lockr-issued IDs reach the server-side auction without the Prebid User ID + module and without any client-side JavaScript module. +2. The IDs are written to a first-party cookie by the server on the same + response that delivered them, and to the EC identity graph without a KV + write before response bytes are sent. +3. The IDs are available to the next auction on the same page load through the + cookie, and to later navigations through KV, with the cookie as fallback. +4. Token refresh and consent revocation from lockr are reflected without a new + mechanism. +5. OpenRTB 2.6 EID provenance (`inserter`, `matcher`, `mm`) survives into the + auction payload. +6. Existing consent gating, size caps and fail-closed KV behavior are + unchanged. + +Out of scope: changing the JavaScript EID collector, server-to-server calls to +lockr, direct proxies for ID5 or LiveRamp, and storing provenance metadata in +KV. + +## 4. Design + +### 4.1 Capture hook on the integration proxy + +Add an optional trait to the integration registry: + +```rust +/// Identity tokens an integration proxy observed in an upstream response. +pub struct CapturedEid { + /// OpenRTB `source` domain, for example `id5-sync.com`. + pub source: String, + pub uid: String, + pub atype: Option, + /// Unix seconds; absent when the provider gave no expiry. + pub expires_at: Option, + pub inserter: Option, + pub matcher: Option, + pub mm: Option, +} + +/// Outcome of inspecting one proxied response. +pub enum IdentityCapture { + None, + Tokens(Vec), + ConsentRevoked, +} + +pub trait IntegrationIdentityCapture { + /// Paths (relative to the integration's `/api` prefix) whose responses + /// should be buffered and inspected. + fn capture_paths(&self) -> &'static [&'static str]; + + /// Parses a buffered upstream response body. + fn capture(&self, path: &str, status: StatusCode, body: &[u8]) -> IdentityCapture; +} +``` + +`IntegrationRegistration::builder` gains `.with_identity_capture()`. The +registry's `handle_proxy` checks whether the matched proxy implements the trait +and whether the request path is in `capture_paths()`. If so it buffers the +upstream response with `collect_response_bounded` at a new +`IDENTITY_CAPTURE_MAX_RESPONSE_BYTES` of 64 KiB, calls `capture`, then rebuilds +the response from the buffered bytes. Every other path streams through +unchanged. Buffering is limited to the listed JSON endpoints; the SDK body and +unlisted paths are never buffered by this hook. + +### 4.2 Lockr implementation + +`LockrIntegration` implements the trait for `page-view`, `generate-tokens`, +`refresh-tokens`, `sync-no-hem-ids` and `revoke-consent`. + +For token endpoints the parser reads `aimTokens[]` and, when present, +`ids..eid`. The `eid` object is preferred because it already carries +`source`, `atype` and provenance. When only `aimTokens[]` is present the +`key_name` is mapped to a source domain with a static table taken from the +SDK's own `tokenMappings` and `tokenSourceMappings`: + +| `key_name` | `source` | `atype` | +| -------------------------- | ---------------------------------- | ------- | +| `id5id` | `id5-sync.com` | 1 | +| `_lr_env` | `liveramp.com` | 3 | +| `__uid2_advertising_token` | `uidapi.com` | 3 | +| `__euid_advertising_token` | `euid.eu` | 3 | +| `firstid` | `first-id.fr` | 1 | +| `connectId` | `yahooinc.com` | 3 | +| `panoramaId` | `panorama.com` (verify with lockr) | 1 | +| `cto_bidid` | `criteo.com` | 3 | +| `_publink` | `epsilon.com` | 3 | + +Unknown `key_name` values are logged at debug level and dropped. The ID5 +`advertising_token` is URL-encoded JSON `{"universal_uid": "..."}`; the parser +extracts `universal_uid`. Tokens are validated with the existing +`is_valid_eid_uid` rule (non-empty, at most `MAX_UID_LENGTH` of 512 bytes). +`identity_expires` is milliseconds and is converted to seconds. + +`revoke-consent` with a 2xx upstream status yields `ConsentRevoked`. + +Non-2xx upstream responses and unparseable bodies yield `None` and are passed +through to the browser unchanged. + +### 4.3 Consent gate + +Capture runs only when all of the following hold: + +1. `ec_context.consent()` passes `ec_consent_granted`, evaluated from the + request's consent cookies exactly as on publisher navigations. +2. The request carries a valid `ts-ec` cookie, so there is an EC ID to attach + the IDs to. Integration XHRs never generate a new EC ID; that rule is + unchanged. +3. The lockr request body's `gppString` or `consentString`, when present, does + not indicate an opt-out. The body is already buffered for POST forwarding by + `collect_body_bounded`; this adds a parse, not a read. + +When the gate fails the response is passed through and nothing is written. + +### 4.4 Server-owned first-party cookie + +A new cookie, `ts-eids-s`, is set on the proxy response that delivered the +tokens: + +- Value: base64 of the structured JSON array already accepted by + `parse_prebid_eids_cookie`, holding `source` and one `id` per source and + nothing else. No `atype`, no `ext`, no provenance. Those are re-attached on + the server from a static per-source table when the auction payload is built + (section 4.6). Example before encoding: + + ```json + [{ "source": "id5-sync.com", "uids": [{ "id": "ID5*..." }] }] + ``` + + The cookie rides on every same-origin request, not only auctions, so it is + kept as small as the ID itself. One ID5 token encodes to 184 bytes + (measured). All nine lockr-issued sources with representative token lengths + encode to 1,716 bytes (modeled with the measured ID5, RampID, UID2 and + Criteo values); UID2 and Criteo tokens account for most of that. + +- Attributes: `Path=/; Secure; HttpOnly; SameSite=Lax`, `Domain` as configured + for `ts-ec`, `Max-Age` = the smallest `expires_at` minus now, capped at 30 + days, defaulting to 30 days when no expiry was given. +- Merge policy: the new value is the union of the existing `ts-eids-s` cookie + on the request and the captured EIDs, keyed by `source`; captured values + replace existing values for the same source. Sources whose stored expiry has + passed are dropped at merge time. +- Size: encoded value capped at 2,048 bytes. When the merged value exceeds + the cap, sources are dropped oldest-expiry first until it fits; dropping is + logged. The cap is below the 3,072-byte `ts-eids` cap because this cookie + carries one ID per source and nothing else, and above the modeled full set + so no lockr-issued source is dropped in practice. +- `ConsentRevoked` sets `ts-eids-s` with `Max-Age=0`. + +The cookie is separate from `ts-eids` for two reasons. The tsjs prebid module +rewrites `ts-eids` on every auction from `getUserIdsAsEids()`, so a server +value stored there would be overwritten within seconds. And a server-set +`HttpOnly` cookie is not subject to the script-written cookie lifetime cap in +WebKit, and cannot be edited by page scripts. + +`enforce_set_cookie_cache_privacy` in the Fastly middleware already marks any +response carrying `Set-Cookie` as private and uncacheable, so no cache change +is needed. + +### 4.5 Readers of the new cookie + +Two readers change: + +- `resolve_client_auction_eids` in `crates/trusted-server-core/src/auction/endpoints.rs` + currently prefers the request body and falls back to `ts-eids`. It will + parse `ts-eids-s` and merge it with whichever of body or `ts-eids` applied, + through `merge_auction_eids`. Before merging, each `ts-eids-s` entry is + expanded with `atype` and provenance from the static table in section 4.6. + For the same `source`, the `ts-eids-s` entry is ordered first so its values + win under the existing "first value wins, fill missing fields" merge rule. +- `collect_eid_cookie_updates` in `crates/trusted-server-core/src/ec/prebid_eids.rs` + takes a third optional cookie value and feeds it through + `collect_prebid_eid_updates_from_eids`. `EcFinalizeState` and + `EcRequestState` carry the extra cookie value. Existing partner-registry + gating applies: a source without an `[[ec.partners]]` entry still reaches the + auction body but is not written to KV. + +The JavaScript collector is not changed by this design. Its `ts-eids` output +continues to cover User ID module providers. + +The `x-ts-eids` response header on `/auction` is unchanged. It is emitted +after the merge and consent gating from the final `user.eids`, is on the +`INTERNAL_HEADERS` strip list for inbound requests, and is read by nothing on +the server. It remains the diagnostic view of what was sent to bidders. + +### 4.6 EID provenance + +`Eid` in `crates/trusted-server-core/src/openrtb.rs` gains three optional +fields with `skip_serializing_if = "Option::is_none"`: `inserter: Option`, +`matcher: Option`, `mm: Option`. `deny_unknown_fields` stays. +`StructuredCookieEid` and `structured_cookie_eids_to_openrtb` carry the same +three fields. `parse_client_auction_eids` and `merge_auction_eids` preserve +them, with the conflict rule "first non-empty value wins", matching how `atype` +and `ext` are merged today. + +Provenance is not stored in the cookie or in KV. A static table in the lockr +integration, keyed by `source`, supplies `atype`, `inserter` +(`Lockr-For-Publishers`), `matcher` (empty) and `mm` (2) when a `ts-eids-s` +entry is expanded for the auction, matching what the SDK sets on +`ortb2.user.ext.eids`. The `atype` column is the one from the mapping table in +section 4.2. Provider `ext` payloads such as ID5's `pba` are not carried; they +are not stored today either and bidders that need them read the provider +cookie directly. + +Legacy flattened cookies decode as before. KV entries are unchanged: the +identity graph still stores one UID per partner source and no provenance. + +### 4.7 KV write after send + +Capture never writes KV before the response is sent. Instead the proxy handler +attaches a response extension: + +```rust +pub struct DeferredIdentityUpdates { + pub updates: Vec, + pub revoke: bool, +} +``` + +`PartnerIdUpdate` values are built with the existing partner-registry lookup, +so only configured sources are included. In the Fastly entry point, after +`send_edgezero_response` returns and before pull sync, a new +`run_identity_updates_after_send` applies them: + +- `updates` go through `KvIdentityGraph::upsert_partner_ids_from_snapshot` + with the request's `EcKvSnapshot`, which already skips unchanged UIDs, fails + closed on a missing or unreadable root row, and refuses to write when the + stored `consent.ok` is false. +- `revoke` calls `write_withdrawal_tombstone`, the same path the CMP + withdrawal flow uses. + +Both are best-effort and log on failure. The Wasm instance stays alive until +the handler returns, so this work costs nothing on the response path. On a +`page-view` that carries no new token (the common returning-visitor case) the +extension is absent and no KV operation runs. + +The Axum, Cloudflare and Spin adapters apply the same extension in their own +post-send positions where one exists; where an adapter has no post-send hook +the updates are applied before send, matching how those adapters already +handle pull sync. The hot-path constraint is a Fastly production constraint. + +### 4.8 Data flow summary + +``` +browser -> POST /integrations/lockr/api/.../page-view (ts-ec, consent cookies, body) + TS proxy -> identity.loc.kr -> 200 {aimTokens, ids} + TS: consent gate -> capture -> Set-Cookie ts-eids-s (HttpOnly) + TS: attach DeferredIdentityUpdates -> send response to browser + TS: after send -> KV upsert (CAS, skip-unchanged) or tombstone + +next /auction on this page -> reads ts-eids-s -> user.eids includes id5-sync.com +next navigation -> KV snapshot (hot-path read already present) + -> ts-eids-s fallback when the row is missing +``` + +## 5. Freshness + +| Event | Cookie | KV | Auction sees it | +| ------------------------------------------ | -------------------------------------- | ------------------------------------- | --------------------------- | +| First issue (`page-view` with `aimTokens`) | Same response | After that response is sent | Next `/auction` on the page | +| Lockr refresh (`refresh-tokens`) | Same response, new value and `Max-Age` | After send, only when the UID changed | Next `/auction` | +| Lockr revoke (`revoke-consent`) | Cleared on that response | Tombstone after send | Next request | +| Returning visitor, no new token | Untouched | No operation | From KV snapshot | + +Modeled cost per captured response: one JSON parse of at most 64 KiB and one +`Set-Cookie` of at most 2,048 bytes. Modeled cost on every later same-origin +request: the cookie bytes, 184 for ID5 alone and up to 1,716 for all nine +lockr-issued sources. No additional KV read is added; the deferred upsert reuses the +request snapshot and performs its own refresh read only on a CAS retry, as it +does today for `ts-eids` ingestion. + +## 6. Configuration + +No new lockr options. KV persistence for a lockr-issued source requires the +existing partner registry entry, for example: + +```toml +[[ec.partners]] +name = "ID5" +source_domain = "id5-sync.com" +openrtb_atype = 1 +bidstream_enabled = true +``` + +Without this entry the ID still reaches the auction body through the cookie +and is never written to KV, matching current `ts-eids` behavior. The deployed +registry for the affected publisher should be checked for this entry as part of +rollout. + +## 7. Error handling + +- Upstream non-2xx or malformed JSON: pass-through, no cookie, no KV, debug log. +- Response larger than 64 KiB on a capture path: pass-through with a warning; + the limit is an order of magnitude above the measured 603-byte response. +- Consent gate failure: pass-through, nothing written. +- Cookie over 2,048 bytes after merge: drop oldest-expiry sources, log. +- KV upsert failure after send: logged, cookie already delivered, next + navigation retries through normal cookie ingestion. +- Missing `ts-ec`: pass-through, nothing written; the SDK call still succeeds. + +## 8. Testing + +Rust, alongside the code: + +- `integrations/lockr.rs`: parser tests for `aimTokens` only, `ids.*.eid` + present, URL-encoded ID5 JSON, unknown `key_name`, oversized UID, non-2xx, + malformed body, `revoke-consent`. +- `integrations/registry.rs`: capture path buffering versus pass-through + streaming for unlisted paths; response bytes identical after buffering. +- `auction/endpoints.rs`: `ts-eids-s` merged with body, with `ts-eids`, alone; + expansion from the static table; provenance precedence; caps. +- `ec/prebid_eids.rs`: structured cookie with and without provenance fields; + legacy cookie unchanged; third cookie input. +- `ec/finalize.rs` and adapter entry point: deferred updates applied after send; + absent when no tokens; tombstone on revoke; skip-unchanged. +- Consent-denied regression coverage retained for every new path. + +JavaScript: no changes, existing suites must still pass. + +Live acceptance on the TS matched arm: + +1. Fresh profile, accept consent, wait for the `page-view` response. +2. The `page-view` response carries `Set-Cookie: ts-eids-s=...; HttpOnly`, + and the decoded value holds only `source` and `id`. +3. The next `/auction` request body decodes with `id5-sync.com` in + `user.eids`, with `inserter` and `mm` present. +4. `GET /_ts/admin/ec` for the session's EC ID shows the ID5 UID under + `ids["id5-sync.com"]` after one further eligible request. +5. Reload: same UID, no new KV write (admin generation unchanged). +6. Trigger `revoke-consent` from the lockr SDK: cookie cleared, tombstone + present. + +## 9. Rollout + +1. Land the `Eid` provenance fields and cookie parser changes first; they are + additive and safe on their own. +2. Land the capture hook with lockr behind a `capture_identity` boolean on + `[integrations.lockr]`, default `true`, so a publisher can switch it off + without a deploy. +3. Add the `id5-sync.com` partner entry on the affected publisher. +4. Verify with the live acceptance steps above. + +## 10. Open questions + +1. Should `ts-eids-s` also be exposed to client-side auctions, for example + through `/identify` or a response header, or is the server-side auction the + only consumer? This design assumes server-side only. +2. The `panoramaId` source domain and `atype` should be confirmed with lockr + before the mapping table ships. +3. Whether a captured `expires_at` should also bound the KV entry. The identity + graph has no per-partner expiry today, and this design leaves that as is. + +## 11. Future-state options + +- Register lockr as a pull-sync partner and resolve tokens server-to-server + from the `ltid` value, removing the SDK from the identity path entirely. + Requires an S2S contract with lockr that does not exist today. +- Apply the same capture hook to direct ID5 and LiveRamp proxies once those + endpoints are proxied first-party. From 287468929ea7b280e8e2b46fa7928edf55c7f7ff Mon Sep 17 00:00:00 2001 From: Christian Date: Fri, 9 Oct 2026 16:23:01 -0500 Subject: [PATCH 2/7] Implement server-side identity foundation and Lockr response capture --- crates/trusted-server-adapter-axum/src/app.rs | 11 +- .../tests/routes.rs | 40 +- .../src/app.rs | 13 +- .../tests/routes.rs | 11 +- .../trusted-server-adapter-fastly/src/app.rs | 20 +- .../trusted-server-adapter-fastly/src/main.rs | 35 +- crates/trusted-server-adapter-spin/src/app.rs | 20 +- .../tests/routes.rs | 34 +- .../src/auction/endpoints.rs | 636 +++++++++--- .../src/auction/formats.rs | 30 +- .../src/auction/test_support.rs | 3 + .../trusted-server-core/src/auction/types.rs | 2 +- crates/trusted-server-core/src/config.rs | 221 +++- crates/trusted-server-core/src/consent/mod.rs | 84 ++ crates/trusted-server-core/src/constants.rs | 6 +- .../src/cookies/template_cache_policy.rs | 6 +- crates/trusted-server-core/src/ec/admin.rs | 391 +------- crates/trusted-server-core/src/ec/auth.rs | 1 + .../trusted-server-core/src/ec/batch_sync.rs | 109 ++ crates/trusted-server-core/src/ec/eids.rs | 195 ++-- crates/trusted-server-core/src/ec/finalize.rs | 259 +++-- crates/trusted-server-core/src/ec/identify.rs | 70 +- crates/trusted-server-core/src/ec/identity.rs | 941 ++++++++++++++++++ crates/trusted-server-core/src/ec/kv.rs | 434 +++++++- crates/trusted-server-core/src/ec/kv_types.rs | 190 +++- crates/trusted-server-core/src/ec/mod.rs | 44 +- .../trusted-server-core/src/ec/prebid_eids.rs | 755 +------------- .../trusted-server-core/src/ec/pull_sync.rs | 32 +- .../src/ec/pull_sync_marker.rs | 63 ++ crates/trusted-server-core/src/ec/registry.rs | 6 +- .../src/integrations/identity.rs | 447 +++++++++ .../src/integrations/lockr.rs | 530 +++++++++- .../src/integrations/mod.rs | 1 + .../src/integrations/prebid.rs | 9 + .../src/integrations/registry.rs | 121 ++- crates/trusted-server-core/src/openrtb.rs | 17 +- crates/trusted-server-core/src/publisher.rs | 126 ++- crates/trusted-server-core/src/settings.rs | 8 + .../trusted-server-js/lib/src/core/auction.ts | 3 + .../lib/src/integrations/prebid/index.ts | 134 +-- .../test/integrations/prebid/index.test.ts | 193 +--- docs/guide/configuration.md | 19 + docs/guide/integrations/lockr.md | 185 ++-- docs/guide/integrations/prebid.md | 89 +- ...rver-side-identity-foundation-and-lockr.md | 115 +++ ...026-10-08-lockr-identity-capture-design.md | 729 +++++++------- ...-server-side-identity-foundation-design.md | 473 +++++++++ trusted-server.example.toml | 2 + 48 files changed, 5427 insertions(+), 2436 deletions(-) create mode 100644 crates/trusted-server-core/src/ec/identity.rs create mode 100644 crates/trusted-server-core/src/integrations/identity.rs create mode 100644 docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md create mode 100644 docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index caba714d6..fea635bcc 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -16,9 +16,8 @@ use trusted_server_core::auction::{ use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::ec::EcContext; use trusted_server_core::ec::admin::{ - admin_ec_lookup_not_supported, deny_admin_diagnostic_fallback, handle_admin_eids_lookup, + admin_ec_lookup_not_supported, admin_eids_lookup_retired, deny_admin_diagnostic_fallback, }; -use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; use trusted_server_core::proxy::{ @@ -495,11 +494,9 @@ fn named_route_handler( // dev server has no store to read. Ok(admin_ec_lookup_not_supported()) } - NamedRouteHandler::AdminEidsLookup => { - let partner_registry = - PartnerRegistry::from_config(&state.settings.ec.partners)?; - handle_admin_eids_lookup(&partner_registry, &req) - } + // Keep the authenticated route local. Removing it would + // risk forwarding admin credentials to publisher fallback. + NamedRouteHandler::AdminEidsLookup => Ok(admin_eids_lookup_retired()), NamedRouteHandler::LegacyAdminDenied => Ok(legacy_admin_alias_denied()), NamedRouteHandler::Auction => { // Build the geo-aware EC context so the auction consent diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index 7126b2a71..0711574e0 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -412,14 +412,32 @@ async fn admin_ec_route_without_credentials_returns_401() { } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn authenticated_admin_eids_route_returns_200() { - // The EIDs echo is pure request inspection (no KV), so the dev server - // serves the real handler. +async fn retired_admin_eids_route_still_requires_authentication() { + let req = Request::builder() + .method("GET") + .uri("/_ts/admin/eids") + .body(AxumBody::empty()) + .expect("should build request"); + let resp = make_service() + .ready() + .await + .expect("should be ready") + .call(req) + .await + .expect("should respond"); + assert_eq!(resp.status().as_u16(), 401); + assert!(resp.headers().contains_key("www-authenticate")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_eids_route_returns_local_retirement_denial() { + // Retired routes must not forward admin credentials to publisher fallback. let mut svc = make_service(); let req = Request::builder() .method("GET") .uri("/_ts/admin/eids") .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .header("cookie", "ts-eids=private-cookie-value") .body(AxumBody::empty()) .expect("should build request"); let resp = svc @@ -431,8 +449,20 @@ async fn authenticated_admin_eids_route_returns_200() { .expect("should respond"); assert_eq!( resp.status().as_u16(), - 200, - "/_ts/admin/eids should serve the real EIDs echo handler" + 410, + "/_ts/admin/eids should deny locally after authentication" + ); + assert_eq!( + resp.headers() + .get("cache-control") + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + assert_eq!( + resp.headers() + .get("x-content-type-options") + .and_then(|v| v.to_str().ok()), + Some("nosniff") ); } diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index 87b9567e7..a0907b24b 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -19,10 +19,9 @@ use trusted_server_core::config_payload::CONFIG_BLOB_KEY; use trusted_server_core::config_payload::{DEFAULT_SECRET_STORE_ID, settings_from_config_blob}; use trusted_server_core::ec::EcContext; use trusted_server_core::ec::admin::{ - admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, - deny_admin_diagnostic_fallback, handle_admin_eids_lookup, + admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, admin_eids_lookup_retired, + deny_admin_diagnostic_fallback, }; -use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; use trusted_server_core::platform::RuntimeServices; @@ -643,13 +642,11 @@ fn build_router(state: &Arc) -> RouterService { .get("/_ts/admin/ec/{id}", |_ctx: RequestContext| async { Ok::(admin_ec_lookup_not_supported()) }) - // Admin EIDs echo: pure request inspection (no KV), so this - // adapter serves the real handler. + // Keep the retired diagnostic local and behind existing authentication. .get( "/_ts/admin/eids", - make_handler(Arc::clone(&state), |s, _services, req| async move { - let partner_registry = PartnerRegistry::from_config(&s.settings.ec.partners)?; - handle_admin_eids_lookup(&partner_registry, &req) + make_handler(Arc::clone(&state), |_s, _services, _req| async move { + Ok(admin_eids_lookup_retired()) }), ) .post( diff --git a/crates/trusted-server-adapter-cloudflare/tests/routes.rs b/crates/trusted-server-adapter-cloudflare/tests/routes.rs index f385bf529..8ae9e6c4c 100644 --- a/crates/trusted-server-adapter-cloudflare/tests/routes.rs +++ b/crates/trusted-server-adapter-cloudflare/tests/routes.rs @@ -370,9 +370,8 @@ async fn admin_ec_route_without_credentials_returns_401() { } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn authenticated_admin_eids_route_returns_200() { - // The EIDs echo is pure request inspection (no KV), so this adapter - // serves the real handler. +async fn authenticated_admin_eids_route_returns_local_retirement() { + // Retired diagnostics must not forward authenticated requests to origin. let req = request_builder() .method("GET") .uri("/_ts/admin/eids") @@ -383,9 +382,11 @@ async fn authenticated_admin_eids_route_returns_200() { assert_eq!( resp.status().as_u16(), - 200, - "/_ts/admin/eids should serve the real EIDs echo handler" + 410, + "should deny the retired EID diagnostic locally" ); + assert_eq!(resp.headers()["cache-control"], "no-store"); + assert_eq!(resp.headers()["x-content-type-options"], "nosniff"); } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index d1da9b61f..5e5fd7728 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -24,7 +24,7 @@ //! | POST | `/_ts/admin/keys/deactivate` | [`handle_deactivate_key`] | //! | GET | `/_ts/admin/ec` | [`handle_admin_ec_lookup`] | //! | GET | `/_ts/admin/ec/{id}` | [`handle_admin_ec_lookup`] | -//! | GET | `/_ts/admin/eids` | [`handle_admin_eids_lookup`] | +//! | GET | `/_ts/admin/eids` | [`admin_eids_lookup_retired`] | //! | POST | `/_ts/api/v1/batch-sync` | [`handle_batch_sync`] | //! | GET | `/_ts/api/v1/identify` | [`handle_identify`] | //! | GET | `/_ts/set-tester` | [`handle_set_tester`] | @@ -107,9 +107,9 @@ use trusted_server_core::auction::{ }; use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::config_payload::DEFAULT_SECRET_STORE_ID; -use trusted_server_core::constants::{COOKIE_SHAREDID, COOKIE_TS_EIDS}; +use trusted_server_core::constants::COOKIE_SHAREDID; use trusted_server_core::ec::admin::{ - deny_admin_diagnostic_fallback, handle_admin_ec_lookup, handle_admin_eids_lookup, + admin_eids_lookup_retired, deny_admin_diagnostic_fallback, handle_admin_ec_lookup, }; use trusted_server_core::ec::batch_sync::handle_batch_sync; use trusted_server_core::ec::consent::ec_consent_withdrawn; @@ -317,7 +317,6 @@ pub(crate) struct EcFinalizeState { /// in response extensions, so `edgezero_main` rebuilds the graph from /// settings when this is set. pub(crate) use_finalize_kv: bool, - pub(crate) eids_cookie: Option, pub(crate) sharedid_cookie: Option, pub(crate) is_real_browser: bool, /// Per-request services carried to the entry point so the pull-sync @@ -333,7 +332,6 @@ struct EcRequestState { ec_context: EcContext, kv_graph: Option, finalize_kv_graph: Option, - eids_cookie: Option, sharedid_cookie: Option, is_real_browser: bool, services: RuntimeServices, @@ -353,7 +351,6 @@ impl EcRequestState { EcFinalizeState { ec_context: self.ec_context, use_finalize_kv: self.finalize_kv_graph.is_some(), - eids_cookie: self.eids_cookie, sharedid_cookie: self.sharedid_cookie, is_real_browser: self.is_real_browser, services: self.services, @@ -411,7 +408,6 @@ fn build_ec_request_state( ); } - let eids_cookie = crate::extract_cookie_value(req, COOKIE_TS_EIDS); let sharedid_cookie = crate::extract_cookie_value(req, COOKIE_SHAREDID); let geo_info = services @@ -455,7 +451,6 @@ fn build_ec_request_state( ec_context, kv_graph, finalize_kv_graph, - eids_cookie, sharedid_cookie, is_real_browser, services: services.clone(), @@ -579,7 +574,7 @@ async fn execute_named( let kv = crate::maybe_identity_graph(&state.settings); handle_admin_ec_lookup(kv.as_ref(), ®istry, &req) } - NamedRouteHandler::AdminEidsLookup => handle_admin_eids_lookup(®istry, &req), + NamedRouteHandler::AdminEidsLookup => Ok(admin_eids_lookup_retired()), _ => unreachable!("admin diagnostics should use early dispatch"), }) .unwrap_or_else(|error| http_error(&error)); @@ -731,7 +726,6 @@ async fn run_named_route( fn run_batch_sync(state: &AppState, services: &RuntimeServices, req: Request) -> Response { let device_signals = device_signals_for(&req); let is_real_browser = device_signals.looks_like_browser(); - let eids_cookie = crate::extract_cookie_value(&req, COOKIE_TS_EIDS); let sharedid_cookie = crate::extract_cookie_value(&req, COOKIE_SHAREDID); let result = crate::require_identity_graph(&state.settings).and_then(|kv| { @@ -746,7 +740,6 @@ fn run_batch_sync(state: &AppState, services: &RuntimeServices, req: Request) -> response.extensions_mut().insert(EcFinalizeState { ec_context: EcContext::default(), use_finalize_kv: false, - eids_cookie, sharedid_cookie, is_real_browser, services: services.clone(), @@ -2553,13 +2546,14 @@ mod tests { let response = route(&router, request); - assert_eq!(response.status(), StatusCode::OK); + assert_eq!(response.status(), StatusCode::GONE); + assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); assert!( response .extensions() .get::() .is_none(), - "admin EIDs diagnostics should not attach EC finalization state" + "retired admin diagnostics should not attach EC finalization state" ); } diff --git a/crates/trusted-server-adapter-fastly/src/main.rs b/crates/trusted-server-adapter-fastly/src/main.rs index b9f3b86be..ec91935f1 100644 --- a/crates/trusted-server-adapter-fastly/src/main.rs +++ b/crates/trusted-server-adapter-fastly/src/main.rs @@ -16,6 +16,7 @@ use fastly::{Request as FastlyRequest, Response as FastlyResponse}; use trusted_server_core::cache_policy::{EdgeCacheHeader, cache_control_headers_have_directive}; use trusted_server_core::ec::device::DeviceSignals; use trusted_server_core::ec::finalize::ec_finalize_response; +use trusted_server_core::ec::identity::{IdentityEffects, send_then_apply_identity}; use trusted_server_core::ec::kv::KvIdentityGraph; use trusted_server_core::ec::pull_sync::{ PullSyncContext, build_pull_sync_context, dispatch_pull_sync, @@ -502,12 +503,14 @@ fn edgezero_main(mut req: FastlyRequest, sandbox: &mut Sandbox, ordinal: u64, re if let Some(settings) = settings_snapshot.as_deref() { match apply_edgezero_ec_finalize(settings, &mut ec_state, &mut response) { Ok(partner_registry) => { - send_edgezero_response( + send_identity_response( response, request_filter_effects.as_ref(), counters.as_ref(), + settings, + &partner_registry, + &mut ec_state, ); - run_edgezero_pull_sync_after_send(settings, &partner_registry, &ec_state); return; } Err(e) => { @@ -521,15 +524,13 @@ fn edgezero_main(mut req: FastlyRequest, sandbox: &mut Sandbox, ordinal: u64, re Ok(settings) => { match apply_edgezero_ec_finalize(&settings, &mut ec_state, &mut response) { Ok(partner_registry) => { - send_edgezero_response( + send_identity_response( response, request_filter_effects.as_ref(), counters.as_ref(), - ); - run_edgezero_pull_sync_after_send( &settings, &partner_registry, - &ec_state, + &mut ec_state, ); return; } @@ -601,13 +602,33 @@ fn apply_edgezero_ec_finalize( &mut ec_state.ec_context, finalize_kv_graph.as_ref(), &partner_registry, - ec_state.eids_cookie.as_deref(), ec_state.sharedid_cookie.as_deref(), response, ); Ok(partner_registry) } +/// Keeps capture effects through cookie finalization, then consumes them after send. +fn send_identity_response( + mut response: HttpResponse, + filters: Option<&RequestFilterEffects>, + counters: Option<&SandboxCounters>, + settings: &Settings, + partners: &PartnerRegistry, + state: &mut EcFinalizeState, +) { + let effects = response.extensions_mut().remove::(); + if send_then_apply_identity( + effects, + &mut state.ec_context, + partners, + || send_edgezero_response(response, filters, counters), + || require_identity_graph(settings).ok(), + ) { + run_edgezero_pull_sync_after_send(settings, partners, state); + } +} + fn run_edgezero_pull_sync_after_send( settings: &Settings, partner_registry: &PartnerRegistry, diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index b8e8a2492..c8400bb16 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -21,10 +21,9 @@ use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::config_payload::settings_from_config_blob; use trusted_server_core::ec::EcContext; use trusted_server_core::ec::admin::{ - admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, - deny_admin_diagnostic_fallback, handle_admin_eids_lookup, + admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, admin_eids_lookup_retired, + deny_admin_diagnostic_fallback, }; -use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::http_util::sanitize_forwarded_headers; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; @@ -645,18 +644,9 @@ fn build_router(state: &Arc) -> RouterService { Ok::(admin_ec_lookup_not_supported()) }; - // Admin EIDs echo: pure request inspection (no KV), so this adapter - // serves the real handler. - let s = Arc::clone(&state); - let admin_eids_handler = move |ctx: RequestContext| { - let s = Arc::clone(&s); - async move { - let req = ctx.into_request(); - let result = PartnerRegistry::from_config(&s.settings.ec.partners) - .and_then(|registry| handle_admin_eids_lookup(®istry, &req)); - Ok::(result.unwrap_or_else(|e| http_error(&e))) - } - }; + // Retain authenticated local denial, never publisher fallback. + let admin_eids_handler = + |_ctx: RequestContext| async { Ok::(admin_eids_lookup_retired()) }; // /auction let s = Arc::clone(&state); diff --git a/crates/trusted-server-adapter-spin/tests/routes.rs b/crates/trusted-server-adapter-spin/tests/routes.rs index 613832b05..be9add93c 100644 --- a/crates/trusted-server-adapter-spin/tests/routes.rs +++ b/crates/trusted-server-adapter-spin/tests/routes.rs @@ -168,21 +168,45 @@ async fn admin_ec_route_without_credentials_returns_401() { } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn authenticated_admin_eids_route_returns_200() { - // The EIDs echo is pure request inspection (no KV), so this adapter - // serves the real handler. +async fn retired_admin_eids_route_still_requires_authentication() { + let req = request_builder() + .method("GET") + .uri("/_ts/admin/eids") + .body(edgezero_core::body::Body::empty()) + .expect("should build request"); + let resp = route(test_router(), req).await; + assert_eq!(resp.status().as_u16(), 401); + assert!(resp.headers().contains_key("www-authenticate")); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn authenticated_admin_eids_route_returns_local_retirement_denial() { + // Retired routes must not forward admin credentials to publisher fallback. let req = request_builder() .method("GET") .uri("/_ts/admin/eids") .header("authorization", "Basic YWRtaW46YWRtaW4tcGFzcw==") + .header("cookie", "ts-eids=private-cookie-value") .body(edgezero_core::body::Body::empty()) .expect("should build request"); let resp = route(test_router(), req).await; assert_eq!( resp.status().as_u16(), - 200, - "/_ts/admin/eids should serve the real EIDs echo handler" + 410, + "/_ts/admin/eids should deny locally after authentication" + ); + assert_eq!( + resp.headers() + .get("cache-control") + .and_then(|v| v.to_str().ok()), + Some("no-store") + ); + assert_eq!( + resp.headers() + .get("x-content-type-options") + .and_then(|v| v.to_str().ok()), + Some("nosniff") ); } diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index f36446456..26ccce8eb 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -10,14 +10,13 @@ use serde_json::Value as JsonValue; use crate::auction::formats::AdRequest; use crate::auction::orchestrator::OrchestrationResult; use crate::consent::{consent_allows_server_side_auction, gate_eids_by_consent}; -use crate::constants::COOKIE_TS_EIDS; -use crate::cookies::extract_cookie_value; use crate::ec::EcContext; use crate::ec::EcKvSnapshot; use crate::ec::eids::{resolve_partner_ids, to_eids}; use crate::ec::kv::KvIdentityGraph; use crate::ec::kv_types::MAX_UID_LENGTH; -use crate::ec::prebid_eids::parse_prebid_eids_cookie; +use crate::ec::partner::normalize_partner_source_domain; +use crate::ec::prebid_eids::collect_prebid_eid_updates_from_eids; use crate::ec::registry::PartnerRegistry; use crate::error::TrustedServerError; use crate::openrtb::{Eid, Uid}; @@ -181,6 +180,21 @@ pub async fn handle_auction( }; let consent_context = ec_context.consent().clone(); + // Keep normalized body updates on the adapter-owned context, not the + // response, so handled provider errors cannot lose persistence work. + let client_eids = if ec_id.is_some() { + parse_client_auction_eids(body.eids.as_ref()) + } else { + None + }; + if let Some(registry) = registry + && let Some(eids) = &client_eids + && consent_allows_server_side_auction(&consent_context) + { + ec_context.stage_browser_eid_updates(collect_prebid_eid_updates_from_eids(eids, registry)); + ec_context.set_eid_sync_source(crate::ec::EidSyncSource::Auction); + } + if !orchestrator.is_enabled() { log::info!("/auction: auction is disabled; returning no-bid response"); let auction_request = convert_tsjs_to_auction_request( @@ -277,34 +291,6 @@ pub async fn handle_auction( ); } - // Parse client-provided EIDs from the current request body. When the - // current request does not include them, fall back to the persisted - // `ts-eids` cookie so later requests can still forward the browser's - // full OpenRTB-style EID structure. - // - // Gate this on the same identity-consent condition as the EC ID - // (`ec_id.is_some()`, which is already filtered by `ec_context.ec_allowed()`). - // Otherwise a US/GPC or US-Privacy opt-out context — where EC identity use is - // denied but a non-personalized auction may still run — could forward - // persistent client EIDs from the body/cookie, since `gate_eids_by_consent` - // only strips on TCF/GDPR signals. This matches the publisher and - // `/_ts/page-bids` paths, which also resolve client EIDs only when - // `ec_id.is_some()`. - let client_eids = if ec_id.is_some() { - resolve_client_auction_eids( - body.eids.as_ref(), - extract_cookie_value(&http_req, COOKIE_TS_EIDS).as_deref(), - ) - } else { - None - }; - // Carry the full request-local EID set to response finalization so KV - // ingestion uses what this request actually sent, not just whatever fits - // in the size-capped `ts-eids` cookie (see `ec::finalize::ec_finalize_response`). - if let Some(eids) = &client_eids { - ec_context.set_client_eids(eids.clone()); - } - // Resolve partner EIDs from the KV identity graph when the user has a valid // EC and both KV and partner stores are available. Gate the read on a // present registry: without one, `resolve_auction_eids` yields no @@ -315,12 +301,11 @@ pub async fn handle_auction( }; // Hand the loaded row to the request context so response finalization — // which runs on an EC context the adapter owns, after this handler returns - // — ingests `ts-eids`/`sharedId` updates from this read instead of paying + // — ingests body/sharedId updates from this read instead of paying // for a second lookup. if !matches!(auction_kv_snapshot, EcKvSnapshot::NotRead) { ec_context.set_kv_snapshot(auction_kv_snapshot.clone()); } - let eids = resolve_auction_eids(&auction_kv_snapshot, registry, ec_context); // Look up geo for device info. let geo = services @@ -344,7 +329,8 @@ pub async fn handle_auction( // Merge current-request client EIDs with KV-resolved EIDs, then apply // consent gating before attaching them to the auction request. - let merged_eids = merge_auction_eids(client_eids, eids); + let merged_eids = + resolve_and_merge_auction_eids(client_eids, &auction_kv_snapshot, registry, ec_context); let had_eids = merged_eids.as_ref().is_some_and(|v| !v.is_empty()); auction_request.user.eids = gate_eids_by_consent(merged_eids, auction_request.user.consent.as_ref()); @@ -448,6 +434,7 @@ pub(crate) fn resolve_auction_eids( snapshot: &EcKvSnapshot, registry: Option<&PartnerRegistry>, ec_context: &EcContext, + now: Option, ) -> Option> { let registry = registry?; @@ -465,27 +452,38 @@ pub(crate) fn resolve_auction_eids( return Some(Vec::new()); } - let resolved = resolve_partner_ids(registry, entry); + let resolved = resolve_partner_ids(registry, entry, now); Some(to_eids(&resolved)) } -pub(crate) fn resolve_client_auction_eids( - raw: Option<&JsonValue>, - cookie_value: Option<&str>, +/// Merges browser and KV EIDs without letting browser input revive an unusable UID. +/// +/// Uses one checked time for both filters and only the active full EC ID's row. +/// A different browser UID remains usable for this request, not a KV replacement. +pub(crate) fn resolve_and_merge_auction_eids( + mut client_eids: Option>, + snapshot: &EcKvSnapshot, + registry: Option<&PartnerRegistry>, + ec_context: &EcContext, ) -> Option> { - parse_client_auction_eids(raw).or_else(|| parse_cookie_auction_eids(cookie_value)) -} - -fn parse_cookie_auction_eids(cookie_value: Option<&str>) -> Option> { - let cookie_value = cookie_value?; - match parse_prebid_eids_cookie(cookie_value) { - Ok(eids) if eids.is_empty() => None, - Ok(eids) => Some(eids), - Err(_) => { - log::trace!("Auction EIDs: failed to parse ts-eids cookie; dropping"); - None + if !ec_context.ec_allowed() { + return None; + } + let ec_id = ec_context.ec_value()?; + let now = crate::ec::checked_current_timestamp(); + if let (Some(eids), Some(entry)) = (&mut client_eids, snapshot.entry_for(ec_id)) { + for eid in eids.iter_mut() { + if let Ok(source) = normalize_partner_source_domain(eid.source.trim()) + && let Some(record) = entry.ids.get(&source) + && !record.is_usable(now) + { + eid.uids.retain(|uid| uid.id != record.uid); + } } + eids.retain(|eid| !eid.uids.is_empty()); } + let resolved = resolve_auction_eids(snapshot, registry, ec_context, now); + merge_auction_eids(client_eids, resolved) } fn parse_client_auction_eids(raw: Option<&JsonValue>) -> Option> { @@ -530,7 +528,22 @@ fn parse_client_auction_eids(raw: Option<&JsonValue>) -> Option> { continue; } - eids.push(Eid { source, uids }); + eids.push(Eid { + source, + uids, + inserter: entry + .get("inserter") + .and_then(JsonValue::as_str) + .map(str::to_owned), + matcher: entry + .get("matcher") + .and_then(JsonValue::as_str) + .map(str::to_owned), + mm: entry + .get("mm") + .and_then(JsonValue::as_i64) + .and_then(|mm| i32::try_from(mm).ok()), + }); } if eids.is_empty() { None } else { Some(eids) } @@ -585,11 +598,24 @@ pub(crate) fn merge_auction_eids( merged.push(Eid { source: eid.source.clone(), uids: Vec::new(), + ..Eid::default() }); merged.len() - 1 } }; + // Browser provenance wins over missing KV provenance. Empty matcher + // and zero-valued mm are both meaningful supplied values. + if eid.inserter.is_some() { + merged[source_index].inserter = eid.inserter; + } + if eid.matcher.is_some() { + merged[source_index].matcher = eid.matcher; + } + if eid.mm.is_some() { + merged[source_index].mm = eid.mm; + } + for uid in eid.uids { if uid.id.trim().is_empty() || uid.id.len() > MAX_UID_LENGTH { continue; @@ -630,6 +656,11 @@ mod tests { use crate::auction::types::{AuctionRequest, AuctionResponse}; use crate::consent::jurisdiction::Jurisdiction; use crate::consent::types::ConsentContext; + use crate::constants::COOKIE_TS_EIDS; + use crate::ec::kv_backend::test_support::InMemoryEcKv; + use crate::ec::kv_backend::{ + EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteMode, EcKvWriteOutcome, + }; use crate::error::IntoHttpResponse as _; use crate::openrtb::Uid; use crate::platform::test_support::{ @@ -689,6 +720,7 @@ mod tests { fn counting_test_partner(source_domain: &str) -> crate::settings::EcPartner { crate::settings::EcPartner { + identity_owner: None, name: format!("Partner {source_domain}"), source_domain: source_domain.to_owned(), openrtb_atype: crate::settings::EcPartner::default_openrtb_atype(), @@ -706,13 +738,51 @@ mod tests { } } + struct BrowserRecordingKv { + inner: InMemoryEcKv, + lookups: Arc, + writes: Arc>>, + } + + impl EcKvStore for BrowserRecordingKv { + fn store_name(&self) -> &str { + self.inner.store_name() + } + fn lookup(&self, key: &str) -> Result, Report> { + self.lookups + .fetch_add(1, std::sync::atomic::Ordering::Relaxed); + self.inner.lookup(key) + } + fn key_exists(&self, key: &str) -> Result> { + self.inner.key_exists(key) + } + fn insert( + &self, + key: &str, + write: EcKvWrite<'_>, + ) -> Result> { + self.writes + .lock() + .expect("should lock write modes") + .push(write.mode); + self.inner.insert(key, write) + } + fn list_keys_with_prefix( + &self, + prefix: &str, + limit: u32, + ) -> Result, Report> { + self.inner.list_keys_with_prefix(prefix, limit) + } + fn delete(&self, key: &str) -> Result<(), Report> { + self.inner.delete(key) + } + } + #[tokio::test] async fn auction_endpoint_snapshot_is_reused_by_response_finalization() { - // `/auction` loads the identity-graph row to resolve server-side EIDs. - // Finalization runs afterwards on the same EC context and ingests - // `ts-eids`/`sharedId` updates. Both must be served by a single billable - // read: before the snapshot was shared, finalization saw `NotRead` and - // paid for a second lookup. + // Body updates survive the provider error on the adapter-owned context. + // Finalization combines sharedId and reuses the one auction lookup. let settings = create_test_settings(); let mut orchestrator = AuctionOrchestrator::new(AuctionConfig { enabled: true, @@ -722,18 +792,27 @@ mod tests { ..Default::default() }); orchestrator.register_provider(Arc::new(EidCapturingProvider { - had_eids: Arc::new(std::sync::Mutex::new(None)), + requests: Arc::new(Mutex::new(Vec::new())), })); - let registry = PartnerRegistry::from_config(&[counting_test_partner("sharedid.org")]) - .expect("should build partner registry"); + let registry = PartnerRegistry::from_config(&[ + counting_test_partner("sharedid.org"), + counting_test_partner("id5-sync.com"), + ]) + .expect("should build partner registry"); let lookups = Arc::new(std::sync::atomic::AtomicUsize::new(0)); - let graph = KvIdentityGraph::counting("counting-store", Arc::clone(&lookups)); + let writes = Arc::new(Mutex::new(Vec::new())); + let graph = KvIdentityGraph::new(BrowserRecordingKv { + inner: InMemoryEcKv::new("counting-store"), + lookups: Arc::clone(&lookups), + writes: Arc::clone(&writes), + }); let ec_id = format!("{}.ABC123", "a".repeat(64)); let mut live = crate::ec::kv_types::KvEntry::tombstone(1000); live.consent.ok = true; graph.create(&ec_id, &live).expect("should seed live row"); lookups.store(0, std::sync::atomic::Ordering::Relaxed); + writes.lock().expect("should lock write modes").clear(); let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); let req = Request::builder() @@ -746,6 +825,11 @@ mod tests { "code": "div-gpt-ad-1", "mediaTypes": { "banner": { "sizes": [[300, 250]] } } } + ], + "eids": [ + {"source":"id5-sync.com", "inserter":"forged", "matcher":"", "mm":0, + "uids":[{"id":"body-id5", "ext":{"writer":"forged"}}]}, + {"source":"unknown.example", "uids":[{"id":"body-only"}]} ] })) .expect("should serialize body"), @@ -754,7 +838,7 @@ mod tests { // The capturing provider deliberately fails its launch; identity // resolution — the subject of this test — completes before dispatch. - let _ = handle_auction( + let result = handle_auction( &settings, &orchestrator, Some(&graph), @@ -764,6 +848,10 @@ mod tests { req, ) .await; + assert!( + result.is_err(), + "provider launch failure should return a handled error" + ); assert_eq!( lookups.load(std::sync::atomic::Ordering::Relaxed), @@ -775,14 +863,12 @@ mod tests { "the endpoint must hand its snapshot to the request context" ); - ec_context.set_eid_sync_source(crate::ec::EidSyncSource::Auction); let mut response = http::Response::new(EdgeBody::empty()); crate::ec::finalize::ec_finalize_response( &settings, &mut ec_context, Some(&graph), ®istry, - None, Some("shared-cookie-id"), &mut response, ); @@ -792,6 +878,14 @@ mod tests { 1, "finalization must reuse the endpoint snapshot instead of reading again" ); + let recorded = writes.lock().expect("should lock write modes"); + assert_eq!( + recorded.len(), + 1, + "body and sharedId must make only one write" + ); + assert!(matches!(recorded[0], EcKvWriteMode::IfGenerationMatch(_))); + drop(recorded); let (stored, _) = graph .get(&ec_id) .expect("should read store") @@ -801,6 +895,131 @@ mod tests { Some("shared-cookie-id"), "the sharedId update must still be ingested from the shared snapshot" ); + assert_eq!( + stored.ids.get("id5-sync.com").map(|id| id.uid.as_str()), + Some("body-id5") + ); + assert!(!stored.ids.contains_key("unknown.example")); + let serialized = serde_json::to_value(&stored).expect("should serialize persisted row"); + assert!(serialized["ids"]["id5-sync.com"].get("inserter").is_none()); + assert!(!serialized.to_string().contains("forged")); + } + + #[tokio::test] + async fn body_persistence_on_no_bid_never_creates_or_recovers_a_root() { + for case in [ + "live", + "no-token", + "denied", + "missing", + "failed", + "tombstone", + "cookie-only", + ] { + let settings = create_test_settings(); + let orchestrator = AuctionOrchestrator::new(AuctionConfig { + enabled: false, + ..Default::default() + }); + let registry = PartnerRegistry::from_config(&[counting_test_partner("id5-sync.com")]) + .expect("should build registry"); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let lookups = Arc::new(std::sync::atomic::AtomicUsize::new(0)); + let writes = Arc::new(Mutex::new(Vec::new())); + let graph = KvIdentityGraph::new(BrowserRecordingKv { + inner: InMemoryEcKv::new("browser-test"), + lookups: Arc::clone(&lookups), + writes: Arc::clone(&writes), + }); + if !matches!(case, "missing" | "failed") { + let mut entry = crate::ec::kv_types::KvEntry::tombstone(1_000); + entry.consent.ok = case != "tombstone"; + graph.create(&ec_id, &entry).expect("should seed row"); + } + let mut context = make_ec_context( + Jurisdiction::NonRegulated, + (case != "no-token").then_some(ec_id.as_str()), + ); + if case == "denied" { + context.consent_mut().jurisdiction = Jurisdiction::Unknown; + } + if case == "failed" { + context.set_kv_snapshot(EcKvSnapshot::Failed { + ec_id: ec_id.clone(), + }); + } else { + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + } + lookups.store(0, std::sync::atomic::Ordering::Relaxed); + writes.lock().expect("should lock writes").clear(); + let payload = json!({"adUnits":[{"code":"slot", "mediaTypes":{"banner":{"sizes":[[300,250]]}}}], + "eids": if case == "cookie-only" { JsonValue::Null } else { + json!([{"source":"id5-sync.com", "uids":[{"id":"body-id"}]}]) + }}); + let cookie = BASE64.encode( + serde_json::to_vec(&json!([ + {"source":"id5-sync.com", "uids":[{"id":"retired-cookie-id"}]} + ])) + .expect("should encode legacy cookie"), + ); + let request = Request::builder() + .method("POST") + .uri("https://publisher.example/auction") + .header("cookie", format!("ts-eids={cookie}")) + .body(EdgeBody::from( + serde_json::to_vec(&payload).expect("should serialize body"), + )) + .expect("should build request"); + let mut response = handle_auction( + &settings, + &orchestrator, + Some(&graph), + Some(®istry), + &mut context, + &noop_services(), + request, + ) + .await + .expect("should return no bid"); + crate::ec::finalize::ec_finalize_response( + &settings, + &mut context, + Some(&graph), + ®istry, + None, + &mut response, + ); + assert_eq!( + writes.lock().expect("should lock writes").len(), + usize::from(case == "live"), + "{case}" + ); + // A Missing point-read snapshot is revalidated once by the + // existing browser CAS path. Present generations and Failed + // snapshots are reused; a miss never authorizes root creation. + assert_eq!( + lookups.load(std::sync::atomic::Ordering::Relaxed), + usize::from(case == "missing"), + "{case}: reuse usable snapshot, with one bounded miss revalidation" + ); + assert!( + !context.ec_generated(), + "{case}: auction must not create or recover roots" + ); + let stored = graph.get(&ec_id).expect("should inspect row"); + if case == "live" { + assert_eq!( + stored.expect("live root remains").0.ids["id5-sync.com"].uid, + "body-id" + ); + } else if let Some((entry, _)) = stored { + assert!(entry.ids.is_empty(), "{case}: no browser enrichment"); + } else { + assert!(matches!(case, "missing" | "failed")); + } + assert!(response.headers().get("x-ts-eids").is_none()); + assert!(response.headers().get("x-ts-eids-truncated").is_none()); + } } #[tokio::test] @@ -820,7 +1039,7 @@ mod tests { ..Default::default() }); orchestrator.register_provider(Arc::new(EidCapturingProvider { - had_eids: Arc::new(std::sync::Mutex::new(None)), + requests: Arc::new(Mutex::new(Vec::new())), })); let registry = PartnerRegistry::from_config(&[counting_test_partner("id5-sync.com")]) .expect("should build partner registry"); @@ -865,22 +1084,12 @@ mod tests { ) .await; - assert_eq!( - ec_context.client_eids().map(|eids| eids - .iter() - .map(|eid| eid.source.as_str()) - .collect::>()), - Some(vec!["id5-sync.com"]), - "the endpoint must hand its parsed body EIDs to the request context" - ); - let mut response = http::Response::new(EdgeBody::empty()); crate::ec::finalize::ec_finalize_response( &settings, &mut ec_context, Some(&graph), ®istry, - None, // No `ts-eids` cookie on this request at all. None, &mut response, ); @@ -1254,10 +1463,9 @@ mod tests { ); } - /// Provider that records whether the auction request it received carried - /// EIDs, then fails its launch so no real transport handle is needed. + /// Records bidder-facing requests, then fails without a transport handle. struct EidCapturingProvider { - had_eids: Arc>>, + requests: Arc>>, } #[async_trait::async_trait(?Send)] @@ -1271,8 +1479,10 @@ mod tests { request: &AuctionRequest, _context: &AuctionContext<'_>, ) -> Result> { - *self.had_eids.lock().expect("should lock captured eids") = - Some(request.user.eids.is_some()); + self.requests + .lock() + .expect("should lock captured requests") + .push(request.clone()); Err(Report::new(TrustedServerError::Auction { message: "capture only".to_string(), })) @@ -1312,9 +1522,9 @@ mod tests { ..Default::default() }; let mut orchestrator = AuctionOrchestrator::new(config); - let had_eids = Arc::new(std::sync::Mutex::new(None)); + let requests = Arc::new(Mutex::new(Vec::new())); orchestrator.register_provider(Arc::new(EidCapturingProvider { - had_eids: Arc::clone(&had_eids), + requests: Arc::clone(&requests), })); let services = noop_services(); @@ -1374,20 +1584,134 @@ mod tests { ) .await; - assert_eq!( - *had_eids.lock().expect("should lock captured eids"), - Some(false), - "outgoing auction request must carry no EIDs when EC identity is denied" + let captured = requests.lock().expect("should lock captured requests"); + let request = captured + .last() + .expect("should capture the bidder-facing request"); + assert!( + request.user.eids.is_none(), + "should carry no EIDs when EC identity is denied" ); } + #[tokio::test] + async fn auction_body_and_cookie_cannot_resubmit_expired_identity() { + for (client_uids, from_cookie, expected_uids) in [ + (vec!["expired-uid"], false, vec![]), + (vec!["expired-uid"], true, vec![]), + (vec!["different-uid"], false, vec!["different-uid"]), + ( + vec!["expired-uid", "different-uid"], + false, + vec!["different-uid"], + ), + ] { + let settings = create_test_settings(); + let mut orchestrator = AuctionOrchestrator::new(AuctionConfig { + enabled: true, + providers: AuctionConfig::legacy_provider_map(&["eid_capturing_provider"]), + ..Default::default() + }); + let requests = Arc::new(Mutex::new(Vec::new())); + orchestrator.register_provider(Arc::new(EidCapturingProvider { + requests: Arc::clone(&requests), + })); + let registry = + PartnerRegistry::from_config(&[counting_test_partner("ids.example.com")]) + .expect("should build registry"); + let graph = KvIdentityGraph::in_memory("auction-store"); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let mut entry = + crate::ec::kv_types::KvEntry::minimal("ids.example.com", "expired-uid", 1_000); + entry + .ids + .get_mut("ids.example.com") + .expect("should contain record") + .expires_at = Some(1); + graph.create(&ec_id, &entry).expect("should seed identity"); + let (_, generation_before) = graph + .get(&ec_id) + .expect("should read identity") + .expect("should find identity"); + let mut ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + let eids = json!([{ + "source": "IDS.EXAMPLE.COM.", + "uids": client_uids.iter().map(|uid| json!({"id": uid})).collect::>() + }]); + let mut body = json!({ "adUnits": [{ + "code": "example-slot", + "mediaTypes": { "banner": { "sizes": [[300, 250]] } } + }] }); + let mut builder = Request::builder() + .method("POST") + .uri("https://publisher.example.com/auction"); + if from_cookie { + let cookie = + BASE64.encode(serde_json::to_vec(&eids).expect("should encode cookie EIDs")); + builder = builder.header("cookie", format!("{COOKIE_TS_EIDS}={cookie}")); + } else { + body["eids"] = eids; + } + let request = builder + .body(EdgeBody::from( + serde_json::to_vec(&body).expect("should serialize body"), + )) + .expect("should build auction request"); + + let _ = handle_auction( + &settings, + &orchestrator, + Some(&graph), + Some(®istry), + &mut ec_context, + &noop_services(), + request, + ) + .await; + + let captured = requests.lock().expect("should lock captured requests"); + let sent = captured + .last() + .expect("should capture the bidder-facing request"); + let actual_uids: Vec<&str> = sent + .user + .eids + .iter() + .flatten() + .flat_map(|eid| &eid.uids) + .map(|uid| uid.id.as_str()) + .collect(); + assert_eq!( + actual_uids, expected_uids, + "should remove the matching expired UID but preserve different current-request IDs" + ); + let (retained, generation_after) = graph + .get(&ec_id) + .expect("should read retained identity") + .expect("should retain identity"); + assert_eq!( + retained, entry, + "should not replace or delete retained expired records" + ); + assert_eq!( + generation_before, generation_after, + "should not write on auction resolution" + ); + } + } + #[test] fn resolve_auction_eids_returns_empty_without_snapshot() { let registry = PartnerRegistry::empty(); let ec_id = format!("{}.ABC123", "a".repeat(64)); let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, Some(®istry), &ec_context); + let result = resolve_auction_eids( + &EcKvSnapshot::NotRead, + Some(®istry), + &ec_context, + Some(1_000), + ); assert!( result.is_some_and(|eids| eids.is_empty()), "should degrade to empty EIDs without a snapshot" @@ -1399,7 +1723,7 @@ mod tests { let ec_id = format!("{}.ABC123", "a".repeat(64)); let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, None, &ec_context); + let result = resolve_auction_eids(&EcKvSnapshot::NotRead, None, &ec_context, Some(1_000)); assert!( result.is_none(), "should return None when registry is missing" @@ -1412,7 +1736,12 @@ mod tests { let ec_id = format!("{}.ABC123", "a".repeat(64)); let ec_context = make_ec_context(Jurisdiction::Unknown, Some(&ec_id)); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, Some(®istry), &ec_context); + let result = resolve_auction_eids( + &EcKvSnapshot::NotRead, + Some(®istry), + &ec_context, + Some(1_000), + ); assert!( result.is_none(), "should return None when consent is denied" @@ -1424,7 +1753,12 @@ mod tests { let registry = PartnerRegistry::empty(); let ec_context = make_ec_context(Jurisdiction::NonRegulated, None); - let result = resolve_auction_eids(&EcKvSnapshot::NotRead, Some(®istry), &ec_context); + let result = resolve_auction_eids( + &EcKvSnapshot::NotRead, + Some(®istry), + &ec_context, + Some(1_000), + ); assert!( result.is_none(), "should return None when no EC value is present" @@ -1440,7 +1774,7 @@ mod tests { let snapshot = EcKvSnapshot::Failed { ec_id: ec_id.clone(), }; - let result = resolve_auction_eids(&snapshot, Some(®istry), &ec_context); + let result = resolve_auction_eids(&snapshot, Some(®istry), &ec_context, Some(1_000)); let eids = result.expect("should return Some on KV error (degraded mode)"); assert!( eids.is_empty(), @@ -1449,60 +1783,65 @@ mod tests { } #[test] - fn resolve_client_auction_eids_falls_back_to_ts_eids_cookie() { - let cookie_payload = json!([ - { - "source": "sharedid.org", - "uids": [ - { "id": "shared_cookie", "atype": 3 }, - { "id": "shared_cookie_2", "ext": { "provider": "example" } } - ] - } - ]); - let encoded = BASE64 - .encode(serde_json::to_vec(&cookie_payload).expect("should serialize cookie payload")); - - let resolved = resolve_client_auction_eids(None, Some(&encoded)) - .expect("should fall back to structured ts-eids cookie"); - - assert_eq!(resolved.len(), 1, "should preserve cookie source entry"); - assert_eq!(resolved[0].source, "sharedid.org"); - assert_eq!( - resolved[0].uids.len(), - 2, - "should preserve multiple cookie UIDs" - ); - assert_eq!(resolved[0].uids[0].id, "shared_cookie"); + fn expiry_matching_uses_normalized_sources_and_full_ec_identity() { + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let mut entry = + crate::ec::kv_types::KvEntry::minimal("ids.example.com", "expired-uid", 1_000); + entry + .ids + .get_mut("ids.example.com") + .expect("should contain record") + .expires_at = Some(1); + let snapshot = EcKvSnapshot::Present { + ec_id: ec_id.clone(), + entry: Box::new(entry), + generation: Some(1), + }; + let context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + for source in ["ids.example.com", "IDS.EXAMPLE.COM.", " ids.example.com "] { + let client = parse_client_auction_eids(Some( + &json!([{"source": source, "uids": [{"id": "expired-uid"}]}]), + )); + assert!( + resolve_and_merge_auction_eids(client, &snapshot, None, &context).is_none(), + "should not bypass expiry with a source alias" + ); + } + let other_ec_id = format!("{}.XYZ789", "a".repeat(64)); + let other_context = make_ec_context(Jurisdiction::NonRegulated, Some(&other_ec_id)); + let client = parse_client_auction_eids(Some( + &json!([{"source": "ids.example.com", "uids": [{"id": "expired-uid"}]}]), + )); + let resolved = resolve_and_merge_auction_eids(client, &snapshot, None, &other_context) + .expect("should preserve unrelated current-request input"); assert_eq!( - resolved[0].uids[1].ext, - Some(json!({ "provider": "example" })), - "should preserve UID ext from cookie fallback" + resolved[0].uids[0].id, "expired-uid", + "should not apply another full EC ID's lifecycle metadata" ); } #[test] - fn resolve_client_auction_eids_prefers_request_body_over_cookie() { - let raw = json!([ - { - "source": "id5-sync.com", - "uids": [{ "id": "body_uid", "atype": 1 }] - } - ]); - let cookie_payload = json!([ - { - "source": "sharedid.org", - "uids": [{ "id": "cookie_uid", "atype": 3 }] - } - ]); - let encoded = BASE64 - .encode(serde_json::to_vec(&cookie_payload).expect("should serialize cookie payload")); - - let resolved = resolve_client_auction_eids(Some(&raw), Some(&encoded)) - .expect("should prefer request body EIDs"); - - assert_eq!(resolved.len(), 1, "should use request body when present"); - assert_eq!(resolved[0].source, "id5-sync.com"); - assert_eq!(resolved[0].uids[0].id, "body_uid"); + fn body_provenance_survives_parse_merge_and_serialization() { + let raw = json!([{"source":"id5-sync.com", "inserter":"Lockr-For-Publishers", + "matcher":"", "mm":0, "uids":[{"id":"uid", "ext":{"linkType":1}}]}]); + let client = parse_client_auction_eids(Some(&raw)); + let resolved = Some(vec![Eid { + source: "id5-sync.com".to_owned(), + uids: vec![Uid { + id: "uid".to_owned(), + atype: Some(1), + ext: None, + }], + ..Eid::default() + }]); + let merged = merge_auction_eids(client, resolved).expect("should merge body and KV"); + let output = serde_json::to_value(&merged).expect("should serialize"); + assert_eq!(output[0]["inserter"], "Lockr-For-Publishers"); + assert_eq!(output[0]["matcher"], ""); + assert_eq!(output[0]["mm"], 0); + assert_eq!(output[0]["uids"][0]["atype"], 1); + assert_eq!(output[0]["uids"][0]["ext"]["linkType"], 1); + assert!(parse_client_auction_eids(None).is_none()); } #[test] @@ -1655,6 +1994,9 @@ mod tests { #[test] fn merge_auction_eids_deduplicates_client_and_resolved_ids() { let client_eids = Some(vec![Eid { + inserter: None, + matcher: None, + mm: None, source: "id5-sync.com".to_string(), uids: vec![Uid { id: "ID5_abc".to_string(), @@ -1664,6 +2006,9 @@ mod tests { }]); let resolved_eids = Some(vec![ Eid { + inserter: None, + matcher: None, + mm: None, source: "id5-sync.com".to_string(), uids: vec![Uid { id: "ID5_abc".to_string(), @@ -1672,6 +2017,9 @@ mod tests { }], }, Eid { + inserter: None, + matcher: None, + mm: None, source: "liveramp.com".to_string(), uids: vec![Uid { id: "LR_xyz".to_string(), @@ -1693,6 +2041,9 @@ mod tests { #[test] fn merge_auction_eids_preserves_multiple_uids_per_source() { let client_eids = Some(vec![Eid { + inserter: None, + matcher: None, + mm: None, source: "sharedid.org".to_string(), uids: vec![Uid { id: "shared_client".to_string(), @@ -1701,6 +2052,9 @@ mod tests { }], }]); let resolved_eids = Some(vec![Eid { + inserter: None, + matcher: None, + mm: None, source: "sharedid.org".to_string(), uids: vec![Uid { id: "shared_server".to_string(), @@ -1720,6 +2074,9 @@ mod tests { #[test] fn merge_auction_eids_prefers_server_resolved_metadata_on_conflict() { let client_eids = Some(vec![Eid { + inserter: None, + matcher: None, + mm: None, source: "adserver.org".to_string(), uids: vec![Uid { id: "shared_uid".to_string(), @@ -1728,6 +2085,9 @@ mod tests { }], }]); let resolved_eids = Some(vec![Eid { + inserter: None, + matcher: None, + mm: None, source: "adserver.org".to_string(), uids: vec![Uid { id: "shared_uid".to_string(), diff --git a/crates/trusted-server-core/src/auction/formats.rs b/crates/trusted-server-core/src/auction/formats.rs index a0df63d94..c9bb9e8f2 100644 --- a/crates/trusted-server-core/src/auction/formats.rs +++ b/crates/trusted-server-core/src/auction/formats.rs @@ -15,9 +15,8 @@ use uuid::Uuid; use crate::auction::context::ContextValue; use crate::consent::ConsentContext; -use crate::constants::{HEADER_X_TS_EC_CONSENT, HEADER_X_TS_EIDS, HEADER_X_TS_EIDS_TRUNCATED}; +use crate::constants::HEADER_X_TS_EC_CONSENT; use crate::creative; -use crate::ec::eids::encode_eids_header; use crate::error::TrustedServerError; use crate::geo::GeoInfo; use crate::openrtb::{ @@ -533,21 +532,6 @@ pub(crate) fn convert_to_openrtb_response_with_report( .insert(HEADER_X_TS_EC_CONSENT, HeaderValue::from_static("ok")); } - // Attach EID response headers when consent-gated EIDs are available. - if let Some(ref eids) = auction_request.user.eids { - let (encoded, truncated) = encode_eids_header(eids)?; - let header_val = - HeaderValue::from_str(&encoded).change_context(TrustedServerError::Auction { - message: "Failed to encode EIDs header value".to_string(), - })?; - response.headers_mut().insert(HEADER_X_TS_EIDS, header_val); - if truncated { - response - .headers_mut() - .insert(HEADER_X_TS_EIDS_TRUNCATED, HeaderValue::from_static("true")); - } - } - Ok(OpenRtbResponseConversion { response, delivery }) } @@ -561,6 +545,7 @@ mod tests { use crate::auction::types::{ ApsRendererV1, ApsTagType, AuctionResponse, Bid, BidRenderer, BidStatus, }; + use crate::constants::{HEADER_X_TS_EIDS, HEADER_X_TS_EIDS_TRUNCATED}; use crate::openrtb::{Eid, Uid}; use crate::platform::test_support::noop_services; use crate::test_support::tests::create_test_settings; @@ -811,9 +796,12 @@ mod tests { } #[test] - fn response_includes_eid_headers_when_eids_present() { + fn response_keeps_eid_diagnostics_retired_when_eids_present() { let mut request = make_auction_request(); request.user.eids = Some(vec![Eid { + inserter: None, + matcher: None, + mm: None, source: "ssp.com".to_owned(), uids: vec![Uid { id: "uid-1".to_owned(), @@ -829,8 +817,8 @@ mod tests { .expect("should build response"); assert!( - response.headers().get(&HEADER_X_TS_EIDS).is_some(), - "should include x-ts-eids header when EIDs are present" + response.headers().get(&HEADER_X_TS_EIDS).is_none(), + "EID diagnostic headers stay retired even when EIDs are present" ); assert_eq!( response @@ -845,7 +833,7 @@ mod tests { .headers() .get(&HEADER_X_TS_EIDS_TRUNCATED) .is_none(), - "should not include truncated header for small payload" + "should never emit the retired truncation header" ); } diff --git a/crates/trusted-server-core/src/auction/test_support.rs b/crates/trusted-server-core/src/auction/test_support.rs index a83f8899a..c0b9d31f7 100644 --- a/crates/trusted-server-core/src/auction/test_support.rs +++ b/crates/trusted-server-core/src/auction/test_support.rs @@ -93,6 +93,9 @@ pub(crate) fn canonical_parity_auction_request() -> AuctionRequest { ..Default::default() }), eids: Some(vec![Eid { + inserter: None, + matcher: None, + mm: None, source: "identity.example".to_string(), uids: vec![Uid { id: "fictional-uid".to_string(), diff --git a/crates/trusted-server-core/src/auction/types.rs b/crates/trusted-server-core/src/auction/types.rs index 406915706..f852f389d 100644 --- a/crates/trusted-server-core/src/auction/types.rs +++ b/crates/trusted-server-core/src/auction/types.rs @@ -93,7 +93,7 @@ pub struct UserInfo { /// cookies/headers, not from stored data. #[serde(skip)] pub consent: Option, - /// Extended User IDs parsed from the [`crate::constants::COOKIE_TS_EIDS`] cookie. + /// Extended User IDs from the current auction body and usable registered KV records. /// /// Raw (un-gated) values from the browser; consent gating via /// [`crate::consent::gate_eids_by_consent`] is applied centrally in the diff --git a/crates/trusted-server-core/src/config.rs b/crates/trusted-server-core/src/config.rs index 79f100e2f..9b72f4e7d 100644 --- a/crates/trusted-server-core/src/config.rs +++ b/crates/trusted-server-core/src/config.rs @@ -256,6 +256,7 @@ pub fn validate_settings_for_deploy(settings: &Settings) -> Result<(), Report (Settings, String) { + let mut settings = valid_settings(); + settings + .integrations + .insert_config( + "lockr", + &serde_json::json!({ + "enabled": true, + "capture_identity": false, + "app_id": "synthetic-lockr-app", + }), + ) + .expect("should insert disabled-capture Lockr config"); + let registration = crate::integrations::lockr::register(&settings) + .expect("should build Lockr registration") + .expect("Lockr config should register"); + let capture = registration + .identity_capture + .expect("Lockr registration should retain its capture descriptor"); + assert!( + !capture.enabled(), + "fixture should exercise the capture kill switch" + ); + let source = capture + .sources() + .first() + .expect("Lockr should declare at least one source") + .to_string(); + + settings.ec.partners.push(crate::settings::EcPartner { + name: "Owned identity source".to_owned(), + source_domain: source.clone(), + openrtb_atype: crate::settings::EcPartner::default_openrtb_atype(), + bidstream_enabled: true, + api_token: Some(Redacted::new( + "owned-api-token-key-reference-32-bytes".to_owned(), + )), + batch_rate_limit: crate::settings::EcPartner::default_batch_rate_limit(), + pull_sync_enabled: false, + pull_sync_url: None, + pull_sync_allowed_domains: vec![], + pull_sync_ttl_sec: crate::settings::EcPartner::default_pull_sync_ttl_sec(), + pull_sync_rate_limit: crate::settings::EcPartner::default_pull_sync_rate_limit(), + identity_owner: Some("lockr".to_owned()), + ts_pull_token: None, + }); + (settings, source) + } + + #[test] + fn owner_validation_accepts_disabled_capture_and_unresolved_deploy_secret_refs() { + let (settings, source) = lockr_identity_owner_settings(); + + validate_settings_for_deploy(&settings) + .expect("deploy validation should accept a disabled capture descriptor"); + let partner = settings + .ec + .partners + .last() + .expect("should retain the configured owner partner"); + assert_eq!(partner.source_domain, source); + assert_eq!( + partner + .api_token + .as_ref() + .expect("should retain the API token reference") + .expose(), + "owned-api-token-key-reference-32-bytes", + "deploy validation should keep the unresolved secret reference" + ); + } + + #[test] + fn owner_validation_rejects_unknown_and_nonclaiming_owners_in_deploy_and_runtime() { + for (owner, source_override) in [ + ("missing_identity_module", None), + ("lockr", Some("unclaimed.identity.test")), + ] { + let (mut settings, _) = lockr_identity_owner_settings(); + { + let partner = settings + .ec + .partners + .last_mut() + .expect("should retain the configured owner partner"); + partner.identity_owner = Some(owner.to_owned()); + if let Some(source) = source_override { + partner.source_domain = source.to_owned(); + } + } + let deploy_error = validate_settings_for_deploy(&settings) + .expect_err("deploy validation should reject an invalid owner claim"); + assert!( + deploy_error.to_string().contains("identity owner"), + "deploy error should identify the owner claim: {deploy_error:?}" + ); + let runtime_error = validate_settings_for_runtime(&settings) + .expect_err("runtime validation should reject an invalid owner claim"); + assert!( + runtime_error.to_string().contains("identity owner"), + "runtime error should identify the owner claim: {runtime_error:?}" + ); + } + } + + #[test] + fn owner_validation_rejects_enabled_legacy_pull_in_deploy_and_runtime() { + let (mut settings, _) = lockr_identity_owner_settings(); + let partner = settings + .ec + .partners + .last_mut() + .expect("should retain the configured owner partner"); + partner.pull_sync_enabled = true; + partner.pull_sync_url = Some("https://pull.identity.test/v1".to_owned()); + partner.pull_sync_allowed_domains = vec!["pull.identity.test".to_owned()]; + partner.ts_pull_token = Some(Redacted::new( + "owned-pull-token-key-reference-32-bytes".to_owned(), + )); + + let deploy_error = validate_settings_for_deploy(&settings) + .expect_err("deploy validation should reject legacy pull on an owned source"); + assert!( + deploy_error.to_string().contains("legacy pull disabled"), + "deploy error should identify the owned-source pull conflict: {deploy_error:?}" + ); + let runtime_error = validate_settings_for_runtime(&settings) + .expect_err("runtime validation should reject legacy pull on an owned source"); + assert!( + runtime_error.to_string().contains("legacy pull disabled"), + "runtime error should identify the owned-source pull conflict: {runtime_error:?}" + ); + } + + #[test] + fn startup_blob_validation_keeps_disabled_owner_descriptor_and_resolves_partner_token() { + let (mut settings, source) = lockr_identity_owner_settings(); + settings.publisher.proxy_secret = Redacted::new("owner-proxy-secret".to_owned()); + settings.ec.passphrase = Redacted::new("owner-ec-passphrase".to_owned()); + for (index, handler) in settings.handlers.iter_mut().enumerate() { + handler.password = Redacted::new(format!("owner-handler-password-{index}")); + } + settings + .ec + .partners + .last_mut() + .expect("should retain the configured owner partner") + .api_token = Some(Redacted::new("owner-partner-api-token".to_owned())); + + validate_settings_for_deploy(&settings) + .expect("deploy should accept unresolved references before startup resolves them"); + + let mut secrets = HashMap::new(); + for (key, value) in [ + ( + "owner-proxy-secret", + "resolved-proxy-secret-not-placeholder", + ), + ( + "owner-ec-passphrase", + "resolved-ec-passphrase-32-bytes-valid", + ), + ( + "owner-handler-password-0", + "resolved-handler-password-0-32-bytes", + ), + ( + "owner-handler-password-1", + "resolved-handler-password-1-32-bytes", + ), + ( + "owner-partner-api-token", + "resolved-owned-api-token-32-bytes-minimum", + ), + ] { + secrets.insert(key.to_owned(), value.as_bytes().to_vec()); + } + let store = HashMapSecretStore::new(secrets); + let envelope = BlobEnvelope::new( + serde_json::to_value(&settings).expect("should serialize settings"), + "2026-10-09T00:00:00Z".to_owned(), + ); + let envelope_json = serde_json::to_string(&envelope).expect("should serialize envelope"); + + let started = crate::config_payload::settings_from_config_blob( + &envelope_json, + &store, + &StoreName::from("identity-owner-test-secrets"), + ) + .expect("startup should resolve secrets and validate the disabled owner descriptor"); + + let started_partner = started + .ec + .partners + .last() + .expect("startup should retain the configured owner partner"); + assert_eq!(started_partner.identity_owner.as_deref(), Some("lockr")); + assert_eq!(started_partner.source_domain, source); + assert_eq!( + started_partner + .api_token + .as_ref() + .expect("startup should retain the identify token") + .expose(), + "resolved-owned-api-token-32-bytes-minimum", + "startup should resolve the identify token for the owned source" + ); + let registration = crate::integrations::lockr::register(&started) + .expect("should build Lockr registration after startup") + .expect("Lockr should remain registered while capture is disabled"); + assert!( + registration.identity_capture.is_some(), + "the disabled capture descriptor should remain discoverable" + ); + } } diff --git a/crates/trusted-server-core/src/consent/mod.rs b/crates/trusted-server-core/src/consent/mod.rs index 76aa6ad85..72c694a32 100644 --- a/crates/trusted-server-core/src/consent/mod.rs +++ b/crates/trusted-server-core/src/consent/mod.rs @@ -404,6 +404,90 @@ pub fn is_consent_expired(tcf: &types::TcfConsent, max_age_days: u32) -> bool { now_deciseconds().saturating_sub(tcf.last_updated_ds) > max_age_ds } +/// Checks restrictive capture signals with the shared decoders and expiry policy. +/// +/// Populated malformed signals never become permission through a permissive +/// jurisdiction or proxy mode. Body signals may only restrict request consent. +pub(crate) fn identity_capture_consent_allowed( + request: &ConsentContext, + body: &RawConsentSignals, + config: &ConsentConfig, +) -> bool { + if request.gpc || request.expired || !allows_ec_creation(request) { + return false; + } + let request_signals = RawConsentSignals { + raw_tc_string: request.raw_tc_string.clone(), + raw_gpp_string: request.raw_gpp_string.clone(), + raw_us_privacy: request.raw_us_privacy.clone(), + ..Default::default() + }; + restrictive_identity_signals_allow(&request_signals, config) + && restrictive_identity_signals_allow(body, config) + && effective_tcf(request).is_none_or(allows_eid_transmission) + && request.gpp.as_ref().and_then(|gpp| gpp.us_sale_opt_out) != Some(true) + && !request + .us_privacy + .as_ref() + .is_some_and(|usp| usp.opt_out_sale == PrivacyFlag::Yes) +} + +fn restrictive_identity_signals_allow(signals: &RawConsentSignals, config: &ConsentConfig) -> bool { + let tcf_allowed = |tcf: &TcfConsent| { + if !allows_eid_transmission(tcf) { + return false; + } + if !config.check_expiration { + return true; + } + let Some(now) = crate::ec::checked_current_timestamp() else { + return false; + }; + now.saturating_mul(10).saturating_sub(tcf.last_updated_ds) + <= u64::from(config.max_consent_age_days) * DECISECONDS_PER_DAY + }; + if let Some(raw) = signals + .raw_tc_string + .as_deref() + .filter(|raw| !raw.is_empty()) + { + let Ok(tcf) = tcf::decode_tc_string(raw) else { + return false; + }; + if !tcf_allowed(&tcf) { + return false; + } + } + if let Some(raw) = signals + .raw_gpp_string + .as_deref() + .filter(|raw| !raw.is_empty()) + { + let Ok(gpp) = gpp::decode_gpp_string(raw) else { + return false; + }; + if gpp.us_sale_opt_out == Some(true) + || gpp.eu_tcf.as_ref().is_some_and(|tcf| !tcf_allowed(tcf)) + || (gpp.us_sale_opt_out.is_none() && gpp.eu_tcf.is_none()) + { + return false; + } + } + if let Some(raw) = signals + .raw_us_privacy + .as_deref() + .filter(|raw| !raw.is_empty()) + { + let Ok(usp) = us_privacy::decode_us_privacy(raw) else { + return false; + }; + if usp.opt_out_sale == PrivacyFlag::Yes { + return false; + } + } + !signals.gpc +} + /// Constructs a US Privacy string from `Sec-GPC` and publisher config defaults. /// /// Called when `gpc = true` but no explicit `us_privacy` cookie exists and the diff --git a/crates/trusted-server-core/src/constants.rs b/crates/trusted-server-core/src/constants.rs index 1b55679e2..68ccff808 100644 --- a/crates/trusted-server-core/src/constants.rs +++ b/crates/trusted-server-core/src/constants.rs @@ -3,16 +3,18 @@ use http::header::HeaderName; pub const COOKIE_TS_EC: &str = "ts-ec"; /// Short-lived signed proof that the current EC row has every pull-partner UID. pub const COOKIE_TS_EC_PULL_COMPLETE: &str = "ts-ec-pull-complete"; -/// Cookie written by the Trusted Server JS SDK containing a standard-base64-encoded -/// JSON array of Extended User IDs (`[{ source, uids }]`) from identity providers. +/// Retired EID cookie name, retained only for transition regression fixtures. +/// New servers neither read nor write it. pub const COOKIE_TS_EIDS: &str = "ts-eids"; pub const COOKIE_TS_TESTER: &str = "ts-tester"; pub const COOKIE_SHAREDID: &str = "sharedId"; pub const HEADER_X_PUB_USER_ID: HeaderName = HeaderName::from_static("x-pub-user-id"); pub const HEADER_X_TS_EC: HeaderName = HeaderName::from_static("x-ts-ec"); +/// Retired diagnostic header. Keep inbound stripping and withdrawal clearing. pub const HEADER_X_TS_EIDS: HeaderName = HeaderName::from_static("x-ts-eids"); pub const HEADER_X_TS_EC_CONSENT: HeaderName = HeaderName::from_static("x-ts-ec-consent"); +/// Retired diagnostic header. Never emitted by current servers. pub const HEADER_X_TS_EIDS_TRUNCATED: HeaderName = HeaderName::from_static("x-ts-eids-truncated"); pub const HEADER_X_TS_ERROR: HeaderName = HeaderName::from_static("x-ts-error"); pub const HEADER_X_TS_JS_ASSET_PROXY: HeaderName = HeaderName::from_static("x-ts-js-asset-proxy"); diff --git a/crates/trusted-server-core/src/cookies/template_cache_policy.rs b/crates/trusted-server-core/src/cookies/template_cache_policy.rs index 61c3b9139..71ec389ee 100644 --- a/crates/trusted-server-core/src/cookies/template_cache_policy.rs +++ b/crates/trusted-server-core/src/cookies/template_cache_policy.rs @@ -2,7 +2,7 @@ use std::collections::{HashMap, HashSet}; -use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EC, COOKIE_TS_EIDS}; +use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EC}; use crate::platform::TemplateCookieValue; /// Whether the request can share a template and its explicit cookie dimensions. @@ -32,7 +32,7 @@ pub(crate) fn validate_cookie_names( return Err(format!("{field} contains invalid cookie name `{name}`")); } if field == "template_cache_key_cookies" - && [COOKIE_TS_EC, COOKIE_TS_EIDS, COOKIE_SHAREDID].contains(&name.as_str()) + && [COOKIE_TS_EC, COOKIE_SHAREDID].contains(&name.as_str()) { return Err(format!( "{field} must not key on Trusted Server identity cookie `{name}`" @@ -232,7 +232,7 @@ mod tests { #[test] fn template_cookie_policy_rejects_identity_keys_but_allows_bypass() { - for name in [COOKIE_TS_EC, COOKIE_TS_EIDS, COOKIE_SHAREDID] { + for name in [COOKIE_TS_EC, COOKIE_SHAREDID] { let error = validate_cookie_names(&names(&[name]), &[]) .expect_err("should reject identity cookie keys"); assert!( diff --git a/crates/trusted-server-core/src/ec/admin.rs b/crates/trusted-server-core/src/ec/admin.rs index e21fab702..aad4d4ed6 100644 --- a/crates/trusted-server-core/src/ec/admin.rs +++ b/crates/trusted-server-core/src/ec/admin.rs @@ -6,9 +6,7 @@ //! attach, so operators can debug KV-to-auction propagation without KV //! console access. //! -//! Also serves `GET /_ts/admin/eids`, which echoes the request's `ts-eids` -//! and `sharedId` cookies with an ingestion preview — the client-side half -//! of EID propagation that is never stored server-side. +//! The retired `GET /_ts/admin/eids` diagnostic stays locally denied after authentication. //! //! Authentication is enforced by the `^/_ts/admin` basic-auth handler //! configuration; startup validation rejects configs that leave these paths @@ -17,7 +15,6 @@ //! internal detail (raw consent strings, partner UIDs, parse errors). use std::borrow::Cow; -use std::collections::BTreeMap; use http::{HeaderValue, Method, Request, Response, StatusCode, header}; use serde::Serialize; @@ -26,8 +23,6 @@ use serde_json::Value as JsonValue; use edgezero_core::body::Body as EdgeBody; use error_stack::{Report, ResultExt as _}; -use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EIDS}; -use crate::cookies::extract_cookie_value; use crate::error::TrustedServerError; use crate::openrtb::Eid; @@ -37,9 +32,6 @@ use super::kv::KvIdentityGraph; use super::kv_backend::EcKvLookup; use super::kv_types::{KvEntry, KvMetadata}; use super::log_id; -use super::prebid_eids::{ - analyze_prebid_eids_cookie, collect_sharedid_update, dedupe_partner_updates, is_valid_eid_uid, -}; use super::registry::PartnerRegistry; /// Route prefix shared by the cookie-based and explicit-ID lookup routes. @@ -482,13 +474,22 @@ fn iso_timestamp(unix_seconds: u64) -> Option { /// Derives the auction EID view for a valid entry, mirroring the filters in /// [`resolve_partner_ids`] and reporting why each stored ID was skipped. fn build_auction_view(registry: &PartnerRegistry, entry: &KvEntry) -> AuctionEidsView { - let resolved = resolve_partner_ids(registry, entry); + let now = super::checked_current_timestamp(); + let resolved = resolve_partner_ids(registry, entry, now); let eids = to_eids(&resolved); let mut skipped = Vec::new(); for (source_domain, partner_uid) in &entry.ids { - let reason = if partner_uid.uid.is_empty() { + let reason = if !entry.consent.ok { + "consent_withdrawn" + } else if partner_uid.uid.is_empty() { "empty_uid" + } else if !partner_uid.is_usable(now) { + if now.is_some() { + "expired" + } else { + "expiry_unverifiable" + } } else { match registry.get(source_domain) { None => "not_in_registry", @@ -505,157 +506,14 @@ fn build_auction_view(registry: &PartnerRegistry, entry: &KvEntry) -> AuctionEid AuctionEidsView { eids, skipped } } -/// Admin EIDs echo payload. -#[derive(Debug, Serialize)] -struct AdminEidsResponse { - /// Whether a `ts-eids` cookie was present on the request. - cookie_present: bool, - /// EIDs parsed from the `ts-eids` cookie. Absent when the cookie is - /// missing or failed to parse. - #[serde(skip_serializing_if = "Option::is_none")] - eids: Option>, - /// Parse failure detail when the `ts-eids` cookie could not be decoded. - #[serde(skip_serializing_if = "Option::is_none")] - parse_error: Option, - /// Whether a `sharedId` cookie was present on the request. - sharedid_present: bool, - /// Number of partners configured in the registry. - partners_configured: usize, - /// Preview of what cookie ingestion would write into the EC entry's - /// `ids` map on a navigation carrying these cookies. - ingest: IngestPreview, -} - -/// What cookie ingestion would store, and what it would drop. -#[derive(Debug, Serialize)] -struct IngestPreview { - /// Cookie sources matched to a configured partner, with the UID that - /// would be stored (deduplicated exactly like the ingestion path). - matched: Vec, - /// `ts-eids` sources dropped on ingestion, with the reason. - unmatched: Vec, -} - -/// A cookie-derived partner UID that ingestion would store. -#[derive(Debug, Serialize)] -struct MatchedPartnerId { - /// Partner namespace key in the EC entry's `ids` map. - source_domain: String, - /// The UID that would be stored. - uid: String, -} - -#[derive(Debug, Serialize)] -struct DroppedEidSource { - /// EID source from the cookie. - source: String, - /// Why ingestion would drop the source. - reason: DroppedEidReason, -} - -#[derive(Debug, Serialize)] -#[serde(rename_all = "snake_case")] -enum DroppedEidReason { - NoPartner, - NoValidUid, -} - -/// Handles `GET /_ts/admin/eids`. -/// -/// Echoes the request's `ts-eids` and `sharedId` cookies: the parsed EID -/// list plus a preview of what cookie ingestion would write into the EC -/// entry's `ids` map given the configured partner registry. Pure request -/// inspection — no KV access — so it works on every adapter. -/// -/// Always responds `200 OK`; missing or malformed cookies are reported in -/// the payload instead of as errors. -/// -/// # Errors -/// -/// Returns [`TrustedServerError::Configuration`] only when the response -/// payload fails JSON serialization. -pub fn handle_admin_eids_lookup( - registry: &PartnerRegistry, - req: &Request, -) -> Result, Report> { - let eids_cookie = extract_cookie_value(req, COOKIE_TS_EIDS); - let sharedid_cookie = extract_cookie_value(req, COOKIE_SHAREDID); - - let (eids, parse_error, diagnostic_sources, mut updates) = match &eids_cookie { - None => (None, None, Vec::new(), Vec::new()), - Some(value) => match analyze_prebid_eids_cookie(value, registry) { - Ok(analysis) => ( - Some(analysis.eids), - None, - analysis.diagnostic_sources, - analysis.updates, - ), - Err(error) => ( - None, - Some(format!("failed to parse ts-eids cookie: {error}")), - Vec::new(), - Vec::new(), - ), - }, - }; - - // Collect matches from both cookies, then dedupe the same way as response - // finalization. The preview is cookie-only and ungated: it ignores the - // `/auction` request-body EIDs, the consent gate on identity-graph writes, - // and UIDs already stored in KV, so it lists candidate matches rather than - // exactly what a request would store. - if let Some(value) = &sharedid_cookie - && let Some(update) = collect_sharedid_update(value, registry) - { - updates.push(update); - } - let matched = dedupe_partner_updates(updates) - .into_iter() - .map(|update| MatchedPartnerId { - source_domain: update.partner_id, - uid: update.uid, - }) - .collect(); - - let mut source_has_valid_uid = BTreeMap::new(); - for diagnostic_source in diagnostic_sources { - let has_valid_uid = diagnostic_source - .uids - .iter() - .any(|uid| is_valid_eid_uid(uid)); - source_has_valid_uid - .entry(diagnostic_source.source) - .and_modify(|source_has_valid_uid| *source_has_valid_uid |= has_valid_uid) - .or_insert(has_valid_uid); - } - let unmatched = source_has_valid_uid - .into_iter() - .filter_map(|(source, has_valid_uid)| { - let reason = if registry.find_by_source_domain(&source).is_none() { - DroppedEidReason::NoPartner - } else if !has_valid_uid { - DroppedEidReason::NoValidUid - } else { - return None; - }; - Some(DroppedEidSource { source, reason }) - }) - .collect(); - - let payload = AdminEidsResponse { - cookie_present: eids_cookie.is_some(), - eids, - parse_error, - sharedid_present: sharedid_cookie.is_some(), - partners_configured: registry.len(), - ingest: IngestPreview { matched, unmatched }, - }; - - let body = - serde_json::to_string(&payload).change_context(TrustedServerError::Configuration { - message: "failed to serialize admin EIDs response".to_owned(), - })?; - Ok(json_response(StatusCode::OK, body)) +/// Locally denies the retired EID-cookie diagnostic after adapter authentication. +/// Never fall through to a publisher origin with admin credentials. +#[must_use] +pub fn admin_eids_lookup_retired() -> Response { + json_error( + StatusCode::GONE, + "EID cookie diagnostic retired; use /_ts/admin/ec", + ) } fn json_error(status: StatusCode, message: &str) -> Response { @@ -677,8 +535,6 @@ fn json_response(status: StatusCode, body: String) -> Response { mod tests { use std::time::Duration; - use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64}; - use crate::ec::kv::TombstoneOutcome; use super::*; @@ -694,6 +550,7 @@ mod tests { fn make_test_partner(source_domain: &str, bidstream_enabled: bool) -> EcPartner { EcPartner { + identity_owner: None, name: format!("Partner {source_domain}"), source_domain: source_domain.to_owned(), openrtb_atype: EcPartner::default_openrtb_atype(), @@ -780,12 +637,14 @@ mod tests { "disabled.example".to_owned(), KvPartnerId { uid: "uid-disabled".to_owned(), + ..Default::default() }, ); entry.ids.insert( "unknown.example".to_owned(), KvPartnerId { uid: "uid-unknown".to_owned(), + ..Default::default() }, ); entry @@ -1341,196 +1200,20 @@ mod tests { assert!(result.is_err(), "should propagate KV read failures"); } - fn eids_cookie_for(entries: &serde_json::Value) -> String { - BASE64.encode(entries.to_string()) - } - #[test] - fn eids_lookup_without_cookies_returns_empty_payload() { - let req = get_request("/_ts/admin/eids"); - - let response = - handle_admin_eids_lookup(&test_registry(), &req).expect("should handle eids lookup"); - - assert_eq!(response.status(), StatusCode::OK); - assert_eq!( - response.headers().get(header::X_CONTENT_TYPE_OPTIONS), - Some(&HeaderValue::from_static("nosniff")) - ); - let json = response_json(response); - assert_eq!(json["cookie_present"], false); - assert_eq!(json["sharedid_present"], false); - assert_eq!(json["partners_configured"], 2); - assert!( - json["ingest"]["matched"] - .as_array() - .expect("should have matched list") - .is_empty(), - "should preview no matches without cookies" - ); - } - - #[test] - fn eids_lookup_parses_cookie_and_previews_ingestion() { - let cookie = eids_cookie_for(&serde_json::json!([ - { - "source": "bidstream.example", - "uids": [{ "id": "uid-configured", "atype": 1 }] - }, - { - "source": "unknown.example", - "uids": [{ "id": "uid-unknown", "atype": 1 }] - } - ])); - let req = get_request_with_cookie("/_ts/admin/eids", &format!("ts-eids={cookie}")); - - let response = - handle_admin_eids_lookup(&test_registry(), &req).expect("should handle eids lookup"); - - assert_eq!(response.status(), StatusCode::OK); - let json = response_json(response); - assert_eq!(json["cookie_present"], true); - assert_eq!( - json["eids"] - .as_array() - .expect("should have parsed eids") - .len(), - 2, - "should echo both parsed EID sources" - ); - - let matched = json["ingest"]["matched"] - .as_array() - .expect("should have matched list"); - assert_eq!(matched.len(), 1, "should match only the configured partner"); - assert_eq!(matched[0]["source_domain"], "bidstream.example"); - assert_eq!(matched[0]["uid"], "uid-configured"); - - let unmatched = json["ingest"]["unmatched"] - .as_array() - .expect("should have unmatched list"); - assert_eq!(unmatched.len(), 1, "should report the unregistered source"); - assert_eq!(unmatched[0]["source"], "unknown.example"); - assert_eq!(unmatched[0]["reason"], "no_partner"); - } - - #[test] - fn eids_lookup_reports_configured_source_without_valid_uid() { - let oversized_uid = "x".repeat(513); - let cookie = eids_cookie_for(&serde_json::json!([{ - "source": "bidstream.example", - "uids": [ - { "id": "", "atype": 1 }, - { "id": " ", "atype": 1 }, - { "id": oversized_uid, "atype": 1 } - ] - }])); - let req = get_request_with_cookie("/_ts/admin/eids", &format!("ts-eids={cookie}")); - - let response = - handle_admin_eids_lookup(&test_registry(), &req).expect("should handle eids lookup"); - let json = response_json(response); - - assert!( - json["ingest"]["matched"] - .as_array() - .expect("should have matched list") - .is_empty(), - "invalid UIDs should not be matched" - ); - let unmatched = json["ingest"]["unmatched"] - .as_array() - .expect("should have unmatched list"); - assert_eq!(unmatched.len(), 1, "should report one dropped source"); - assert_eq!(unmatched[0]["source"], "bidstream.example"); - assert_eq!(unmatched[0]["reason"], "no_valid_uid"); - } - - #[test] - fn eids_lookup_does_not_drop_duplicate_source_with_valid_uid() { - let cookie = eids_cookie_for(&serde_json::json!([ - { - "source": "bidstream.example", - "uids": [{ "id": " ", "atype": 1 }] - }, - { - "source": "bidstream.example", - "uids": [{ "id": "uid-valid", "atype": 1 }] - } - ])); - let req = get_request_with_cookie("/_ts/admin/eids", &format!("ts-eids={cookie}")); - - let response = - handle_admin_eids_lookup(&test_registry(), &req).expect("should handle eids lookup"); - let json = response_json(response); - - let matched = json["ingest"]["matched"] - .as_array() - .expect("should have matched list"); - assert_eq!(matched.len(), 1, "should match the valid duplicate source"); - assert_eq!(matched[0]["uid"], "uid-valid"); - assert!( - json["ingest"]["unmatched"] - .as_array() - .expect("should have unmatched list") - .is_empty(), - "a valid duplicate should suppress no_valid_uid" - ); - } - - #[test] - fn eids_lookup_reports_parse_error() { - let req = get_request_with_cookie("/_ts/admin/eids", "ts-eids=!!!not-base64!!!"); - - let response = - handle_admin_eids_lookup(&test_registry(), &req).expect("should handle eids lookup"); - - assert_eq!( - response.status(), - StatusCode::OK, - "malformed cookies should be reported, not errored" - ); - let json = response_json(response); - assert_eq!(json["cookie_present"], true); - assert!( - json["parse_error"] - .as_str() - .expect("should have parse_error") - .contains("ts-eids"), - "should describe the parse failure" - ); - assert!(json.get("eids").is_none(), "should omit unparsed eids"); - assert!( - json["ingest"]["matched"] - .as_array() - .expect("should have matched list") - .is_empty(), - "unparseable cookie should preview no matches" - ); - } - - #[test] - fn eids_lookup_includes_sharedid_match() { - let registry = PartnerRegistry::from_config(&[ - make_test_partner("bidstream.example", true), - make_test_partner("sharedid.org", true), - ]) - .expect("should build sharedid test registry"); - let req = get_request_with_cookie("/_ts/admin/eids", "sharedId=shared-uid-123"); - - let response = - handle_admin_eids_lookup(®istry, &req).expect("should handle eids lookup"); - - assert_eq!(response.status(), StatusCode::OK); - let json = response_json(response); - assert_eq!(json["cookie_present"], false); - assert_eq!(json["sharedid_present"], true); - - let matched = json["ingest"]["matched"] - .as_array() - .expect("should have matched list"); - assert_eq!(matched.len(), 1, "should match the sharedid partner"); - assert_eq!(matched[0]["source_domain"], "sharedid.org"); - assert_eq!(matched[0]["uid"], "shared-uid-123"); + fn retired_eids_diagnostic_denies_locally_without_echoing_cookies() { + for cookie in ["ts-eids=secret; sharedId=secret", "ts-eids=!!!", ""] { + let _req = get_request_with_cookie("/_ts/admin/eids", cookie); + let response = admin_eids_lookup_retired(); + assert_eq!(response.status(), StatusCode::GONE); + assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); + assert_eq!( + response.headers()[header::X_CONTENT_TYPE_OPTIONS], + "nosniff" + ); + let payload = response_json(response); + assert!(payload.get("eids").is_none()); + assert!(!payload.to_string().contains("secret")); + } } } diff --git a/crates/trusted-server-core/src/ec/auth.rs b/crates/trusted-server-core/src/ec/auth.rs index b7db6f1fa..9f7f6bd7e 100644 --- a/crates/trusted-server-core/src/ec/auth.rs +++ b/crates/trusted-server-core/src/ec/auth.rs @@ -59,6 +59,7 @@ mod tests { pull_sync_allowed_domains: vec![], pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + identity_owner: None, ts_pull_token: None, } } diff --git a/crates/trusted-server-core/src/ec/batch_sync.rs b/crates/trusted-server-core/src/ec/batch_sync.rs index 4903589b2..598efeb31 100644 --- a/crates/trusted-server-core/src/ec/batch_sync.rs +++ b/crates/trusted-server-core/src/ec/batch_sync.rs @@ -131,6 +131,14 @@ fn handle_batch_sync_with_writer( return Ok(error_response(StatusCode::UNAUTHORIZED, "invalid_token")); }; + // Managed ownership keeps this token read-only for identify. + if partner.identity_owner.is_some() { + return Ok(error_response( + StatusCode::FORBIDDEN, + "identity_source_owned", + )); + } + // 2. Rate limit (per-partner, per-minute via batch_rate_limit) let rate_key = format!("batch:{}", partner.source_domain); if rate_limiter.exceeded_per_minute(&rate_key, partner.batch_rate_limit)? { @@ -306,11 +314,16 @@ fn error_response(status: StatusCode, reason: &str) -> Response { #[cfg(test)] mod tests { use super::*; + use std::cell::Cell; use std::collections::VecDeque; + use crate::consent::jurisdiction::Jurisdiction; + use crate::consent::types::{ConsentContext, ConsentSource}; + use crate::ec::kv_types::KvEntry; use crate::error::TrustedServerError; use crate::redacted::Redacted; use crate::settings::EcPartner; + use crate::test_support::tests::create_test_settings; // EC ID validation tests are in generation.rs (is_valid_ec_id). // Verify the import works here with a basic smoke test. @@ -408,6 +421,7 @@ mod tests { pull_sync_allowed_domains: vec![], pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + identity_owner: None, ts_pull_token: None, } } @@ -1074,4 +1088,99 @@ mod tests { "should stop after the failing group and accept A's later duplicate" ); } + + #[test] + fn owned_partner_token_is_read_only_for_batch_sync_but_still_authenticates_identify() { + struct CountingRateLimiter(Cell); + + impl RateLimiter for CountingRateLimiter { + fn exceeded( + &self, + _key: &str, + _hourly_limit: u32, + ) -> Result> { + self.0.set(self.0.get() + 1); + Ok(false) + } + + fn exceeded_per_minute( + &self, + _key: &str, + _per_minute_limit: u32, + ) -> Result> { + self.0.set(self.0.get() + 1); + Ok(false) + } + } + + const TOKEN: &str = "test-token-32-bytes-minimum-value"; + let source = "owned-source.identity.test"; + let mut partner = make_test_partner(source, TOKEN); + partner.identity_owner = Some("lockr".to_owned()); + let registry = PartnerRegistry::from_config(&[partner]) + .expect("should retain an owned partner with an API token"); + + let writer = MockWriter::new(vec![]); + let limiter = CountingRateLimiter(Cell::new(0)); + let response = handle_batch_sync_with_writer( + &writer, + ®istry, + &limiter, + authorized_batch_request("not-json"), + ) + .expect("should return the owned-source denial"); + assert_eq!( + response.status(), + StatusCode::FORBIDDEN, + "an owned source token must not write through batch-sync" + ); + assert_eq!( + limiter.0.get(), + 0, + "ownership should reject before consuming rate-limit budget" + ); + assert!( + writer.calls().is_empty(), + "ownership should reject before KV work" + ); + + let settings = create_test_settings(); + let graph = KvIdentityGraph::in_memory("owned-identify-store"); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let entry = KvEntry::minimal(source, "owned-source-uid", 1_000); + graph.create(&ec_id, &entry).expect("should seed owned UID"); + let identify_request = Request::builder() + .method("GET") + .uri("https://publisher.example.com/_ts/api/v1/identify") + .header("authorization", format!("Bearer {TOKEN}")) + .body(EdgeBody::empty()) + .expect("should build identify request"); + let consent = ConsentContext { + jurisdiction: Jurisdiction::NonRegulated, + source: ConsentSource::Cookie, + ..ConsentContext::default() + }; + let ec_context = crate::ec::EcContext::new_for_test(Some(ec_id), consent); + + let identify_response = crate::ec::identify::handle_identify( + &settings, + &graph, + ®istry, + &identify_request, + &ec_context, + ) + .expect("should allow identify through the same owned partner token"); + assert_eq!( + identify_response.status(), + StatusCode::OK, + "ownership should preserve read-only identify authentication" + ); + let body = identify_response + .into_body() + .into_bytes() + .expect("should read identify response"); + let json: serde_json::Value = + serde_json::from_slice(&body).expect("should decode identify response"); + assert_eq!(json["uid"], "owned-source-uid"); + } } diff --git a/crates/trusted-server-core/src/ec/eids.rs b/crates/trusted-server-core/src/ec/eids.rs index 1359f7509..11735e052 100644 --- a/crates/trusted-server-core/src/ec/eids.rs +++ b/crates/trusted-server-core/src/ec/eids.rs @@ -1,24 +1,16 @@ //! Shared EID resolution and formatting helpers. //! //! Used by both `/_ts/api/v1/identify` and `/auction` to resolve source-domain -//! keyed IDs from KV entries, convert them to `OpenRTB` EID structures, and -//! build base64-encoded response headers. +//! keyed IDs from KV entries and convert them to `OpenRTB` EID structures. -use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64}; -use error_stack::{Report, ResultExt as _}; - -use crate::error::TrustedServerError; use crate::openrtb::{Eid, Uid}; use super::kv_types::KvEntry; use super::registry::PartnerRegistry; -/// Maximum size (in bytes) for the base64-encoded `x-ts-eids` header value. -pub const MAX_EIDS_HEADER_BYTES: usize = 4096; - /// A source-domain keyed ID resolved from a KV entry against the partner registry. /// -/// Only includes partners with `bidstream_enabled = true` and a non-empty UID. +/// Only includes usable records from live entries and bidstream-enabled partners. pub struct ResolvedPartnerId { /// The partner's identity source domain and EC KV `ids` key. pub source_domain: String, @@ -30,14 +22,21 @@ pub struct ResolvedPartnerId { /// Resolves source-domain keyed IDs from a KV entry against the partner registry. /// -/// Filters to partners with `bidstream_enabled = true` and non-empty UIDs, -/// sorted deterministically by source domain. +/// Filters to live, consented, bidstream-enabled records with a usable UID at +/// the supplied checked time, sorted deterministically by source domain. #[must_use] -pub fn resolve_partner_ids(registry: &PartnerRegistry, entry: &KvEntry) -> Vec { +pub fn resolve_partner_ids( + registry: &PartnerRegistry, + entry: &KvEntry, + now: Option, +) -> Vec { let mut resolved = Vec::new(); + if !entry.consent.ok { + return resolved; + } for (source_domain, partner_uid) in &entry.ids { - if partner_uid.uid.is_empty() { + if !partner_uid.is_usable(now) { continue; } @@ -65,6 +64,9 @@ pub fn to_eids(resolved: &[ResolvedPartnerId]) -> Vec { resolved .iter() .map(|item| Eid { + inserter: None, + matcher: None, + mm: None, source: item.source_domain.clone(), uids: vec![Uid { id: item.uid.clone(), @@ -75,73 +77,6 @@ pub fn to_eids(resolved: &[ResolvedPartnerId]) -> Vec { .collect() } -/// Builds a base64-encoded EID header value, truncating if needed. -/// -/// Returns `(encoded_value, was_truncated)`. If the full set of EIDs exceeds -/// [`MAX_EIDS_HEADER_BYTES`] after base64 encoding, partners are removed -/// from the end of the deterministic partner ordering until it fits. -/// -/// # Errors -/// -/// Returns an error if JSON serialization fails. -pub fn build_eids_header( - resolved: &[ResolvedPartnerId], -) -> Result<(String, bool), Report> { - let eids = to_eids(resolved); - encode_eids_header(&eids) -} - -/// Encodes a pre-built EID slice into a base64 header value with truncation. -/// -/// Like [`build_eids_header`] but operates on already-constructed `Eid` values -/// (e.g., from `UserInfo.eids` in the auction response path). -/// -/// Returns `(encoded_value, was_truncated)`. -/// -/// # Errors -/// -/// Returns an error if JSON serialization fails. -pub fn encode_eids_header(eids: &[Eid]) -> Result<(String, bool), Report> { - let try_encode = |size: usize| -> Result> { - let json = serde_json::to_vec(&eids[..size]).change_context( - TrustedServerError::Configuration { - message: "Failed to serialize eids header payload".to_owned(), - }, - )?; - Ok(BASE64.encode(json)) - }; - - // Fast path: try the full slice first (common case — no truncation). - let encoded = try_encode(eids.len())?; - if encoded.len() <= MAX_EIDS_HEADER_BYTES { - return Ok((encoded, false)); - } - - // Binary search for the largest count that fits within the limit. - // Invariant: lo always fits, hi never fits. - let mut lo: usize = 0; - let mut hi: usize = eids.len(); - - while lo + 1 < hi { - let mid = lo + (hi - lo) / 2; - let encoded = try_encode(mid)?; - if encoded.len() <= MAX_EIDS_HEADER_BYTES { - lo = mid; - } else { - hi = mid; - } - } - - // `lo` is the largest size that fits. Re-encode it for the final value. - if lo == 0 && !eids.is_empty() { - log::warn!( - "encode_eids_header: no EIDs fit within {MAX_EIDS_HEADER_BYTES}B; emitting empty truncated header" - ); - } - let encoded = try_encode(lo)?; - Ok((encoded, true)) -} - #[cfg(test)] mod tests { use super::*; @@ -150,6 +85,7 @@ mod tests { fn make_test_partner(source_domain: &str) -> EcPartner { EcPartner { + identity_owner: None, name: format!("Partner {source_domain}"), source_domain: source_domain.to_owned(), openrtb_atype: EcPartner::default_openrtb_atype(), @@ -167,6 +103,63 @@ mod tests { } } + #[test] + fn resolve_partner_ids_omits_expired_identity() { + let registry = PartnerRegistry::from_config(&[make_test_partner("ids.example.com")]) + .expect("should build registry"); + let mut entry = KvEntry::minimal("ids.example.com", "expired-uid", 1_000); + entry + .ids + .get_mut("ids.example.com") + .expect("should contain record") + .expires_at = Some(1); + + assert!( + resolve_partner_ids(®istry, &entry, Some(2)).is_empty(), + "should omit a known-expired identity without deleting its record" + ); + } + + #[test] + fn resolve_partner_ids_checks_clock_boundaries_and_retains_legacy_compatibility() { + let registry = PartnerRegistry::from_config(&[make_test_partner("ids.example.com")]) + .expect("should build registry"); + let mut entry = KvEntry::minimal("ids.example.com", "example-uid", 1_000); + entry + .ids + .get_mut("ids.example.com") + .expect("should contain record") + .expires_at = Some(2_000); + assert_eq!( + resolve_partner_ids(®istry, &entry, Some(1_999)).len(), + 1, + "should resolve before expiry" + ); + assert!( + resolve_partner_ids(®istry, &entry, Some(2_000)).is_empty(), + "should omit at expiry" + ); + assert!( + resolve_partner_ids(®istry, &entry, None).is_empty(), + "should fail closed for known expiry when time is unavailable" + ); + entry + .ids + .get_mut("ids.example.com") + .expect("should contain record") + .expires_at = None; + assert_eq!( + resolve_partner_ids(®istry, &entry, None).len(), + 1, + "should retain unknown-expiry compatibility" + ); + entry.consent.ok = false; + assert!( + resolve_partner_ids(®istry, &entry, Some(1_000)).is_empty(), + "should never deliver a withdrawn row's records" + ); + } + #[test] fn resolve_partner_ids_sorts_by_source_domain() { let partners = vec![ @@ -181,16 +174,18 @@ mod tests { "zeta.example.com".to_owned(), super::super::kv_types::KvPartnerId { uid: "uid-z".to_owned(), + ..Default::default() }, ); entry.ids.insert( "alpha.example.com".to_owned(), super::super::kv_types::KvPartnerId { uid: "uid-a".to_owned(), + ..Default::default() }, ); - let resolved = resolve_partner_ids(®istry, &entry); + let resolved = resolve_partner_ids(®istry, &entry, Some(1_000)); let source_domains: Vec<&str> = resolved .iter() .map(|item| item.source_domain.as_str()) @@ -240,40 +235,4 @@ mod tests { "should preserve PAIR vendor-specific atype" ); } - - #[test] - fn build_eids_header_truncates_when_too_large() { - let mut resolved = Vec::new(); - for idx in 0..64 { - resolved.push(ResolvedPartnerId { - uid: format!("uid_{}", "x".repeat(100)), - source_domain: format!("partner-{idx}.example.com"), - openrtb_atype: 3, - }); - } - - let (encoded, truncated) = - build_eids_header(&resolved).expect("should build truncated header"); - - assert!(truncated, "should report truncation for large payload"); - assert!( - encoded.len() <= MAX_EIDS_HEADER_BYTES, - "should cap encoded header bytes" - ); - } - - #[test] - fn build_eids_header_fits_without_truncation() { - let resolved = vec![ResolvedPartnerId { - uid: "u1".to_owned(), - source_domain: "ssp.com".to_owned(), - openrtb_atype: 3, - }]; - - let (encoded, truncated) = - build_eids_header(&resolved).expect("should build header without truncation"); - - assert!(!truncated, "should not truncate small payload"); - assert!(!encoded.is_empty(), "should produce non-empty value"); - } } diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index 019ef2aa4..4d735cc7a 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -20,7 +20,7 @@ use super::kv::{ apply_partner_id_updates, }; use super::kv_types::KvEntry; -use super::prebid_eids::collect_eid_updates; +use super::prebid_eids::collect_browser_eid_updates; use super::pull_sync_marker::{expire_marker, reconcile_marker}; use super::registry::PartnerRegistry; use super::{EcKvSnapshot, EidSyncSource, current_timestamp, log_id}; @@ -42,20 +42,23 @@ const EC_RESPONSE_HEADERS: &[&str] = &[ /// immediately and the EC identity-graph KV tombstone is the authoritative /// revocation marker. There is no separate consent KV store to clean up. /// -/// `eids_cookie` should be the raw value of the `ts-eids` cookie extracted -/// from the request *before* routing consumes it. +/// Staged auction body updates and sharedId are combined into one browser mutation. pub fn ec_finalize_response( settings: &Settings, ec_context: &mut EcContext, kv: Option<&KvIdentityGraph>, registry: &PartnerRegistry, - eids_cookie: Option<&str>, sharedid_cookie: Option<&str>, response: &mut Response, ) { ec_context.validate_pull_sync_marker(settings, registry); - let consent_allows_ec = ec_consent_granted(ec_context.consent()); - let consent_withdrawn = ec_consent_withdrawn(ec_context.consent()); + let captured_withdrawal = response + .extensions() + .get::() + .is_some_and(|effects| effects.withdraws(ec_context)); + let consent_allows_ec = ec_consent_granted(ec_context.consent()) && !captured_withdrawal; + let consent_withdrawn = ec_consent_withdrawn(ec_context.consent()) || captured_withdrawal; + let body_updates = ec_context.take_browser_eid_updates(); if !consent_allows_ec { // Expire the request-local marker independently of the EC cookie: a @@ -81,12 +84,7 @@ pub fn ec_finalize_response( let source = ec_context.eid_sync_source(); let collected = source .map(|_| { - collect_eid_updates( - eids_cookie, - sharedid_cookie, - ec_context.client_eids(), - registry, - ) + collect_browser_eid_updates(body_updates.clone(), sharedid_cookie, registry) }) .unwrap_or_default(); let had_eid_updates = !collected.is_empty(); @@ -139,12 +137,7 @@ pub fn ec_finalize_response( }; let updates = gate_eid_updates_by_consent( - collect_eid_updates( - eids_cookie, - sharedid_cookie, - ec_context.client_eids(), - registry, - ), + collect_browser_eid_updates(body_updates, sharedid_cookie, registry), ec_context.consent(), ); sync_eid_cookie_updates(graph, ec_context, &ec_id, &updates, EidSyncSource::NewEc); @@ -526,12 +519,9 @@ where mod tests { use http::HeaderValue; - use base64::Engine as _; - use super::*; use crate::consent::jurisdiction::Jurisdiction; use crate::consent::types::{ConsentContext, ConsentSource}; - use crate::openrtb::{Eid, Uid}; use crate::redacted::Redacted; use crate::settings::EcPartner; use crate::test_support::tests::{create_test_settings, purpose_one_only_consent}; @@ -603,6 +593,7 @@ mod tests { fn make_partner(source_domain: &str) -> EcPartner { EcPartner { + identity_owner: None, name: format!("Partner {source_domain}"), source_domain: source_domain.to_owned(), openrtb_atype: EcPartner::default_openrtb_atype(), @@ -620,6 +611,131 @@ mod tests { } } + #[test] + fn captured_withdrawal_precedes_browser_enrichment_and_is_retained_for_after_send() { + let settings = create_test_settings(); + let ec_id = sample_ec_id("captwd"); + let graph = KvIdentityGraph::in_memory("captured-withdrawal"); + let mut live = KvEntry::minimal("id5-sync.com", "stored-id", current_timestamp()); + live.consent.ok = true; + graph.create(&ec_id, &live).expect("should seed live row"); + let registry = PartnerRegistry::from_config(&[ + make_partner("id5-sync.com"), + make_partner("sharedid.org"), + ]) + .expect("should build registry"); + let mut context = make_context( + Some(&ec_id), + Some(&ec_id), + true, + false, + Jurisdiction::NonRegulated, + ); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + context.set_eid_sync_source(EidSyncSource::Auction); + context.stage_browser_eid_updates(vec![PartnerIdUpdate::new("id5-sync.com", "body-id")]); + let effects = super::super::identity::stage_capture( + "lockr", + &["id5-sync.com"], + super::super::identity::IdentityObservation { + outcomes: vec![super::super::identity::IdentityOutcome::ConsentWithdrawn], + consent: Default::default(), + }, + &context, + ®istry, + &settings, + ) + .expect("should stage approved global withdrawal"); + let mut response = empty_response(); + response.extensions_mut().insert(effects.clone()); + set_header(&mut response, "x-ts-ec", &ec_id); + set_header(&mut response, "x-ts-eids", "retired-value"); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + ®istry, + Some("shared-id"), + &mut response, + ); + assert!( + response.headers()[http::header::SET_COOKIE] + .to_str() + .expect("cookie header") + .contains("Max-Age=0") + ); + assert!(response.headers().get("x-ts-ec").is_none()); + assert!(response.headers().get("x-ts-eids").is_none()); + let snapshot = context + .kv_snapshot() + .entry_for(&ec_id) + .expect("should retain withdrawn snapshot"); + assert!(!snapshot.consent.ok); + assert!( + snapshot.ids.is_empty(), + "withdrawal cannot seed body or sharedId" + ); + let (_, generation) = graph + .get(&ec_id) + .expect("should read tombstone") + .expect("tombstone exists"); + effects.apply(&graph, ®istry, &mut context); + let (stored, after_generation) = graph + .get(&ec_id) + .expect("should read tombstone") + .expect("tombstone exists"); + assert_eq!( + generation, after_generation, + "after-send must not duplicate finalization tombstone" + ); + assert!(stored.ids.is_empty()); + } + + #[test] + fn captured_withdrawal_for_another_full_ec_id_is_ignored() { + let settings = create_test_settings(); + let ec_id = sample_ec_id("captwd"); + let other_id = sample_ec_id("other1"); + let registry = PartnerRegistry::empty(); + let old_context = make_context( + Some(&ec_id), + Some(&ec_id), + true, + false, + Jurisdiction::NonRegulated, + ); + let effects = super::super::identity::stage_capture( + "lockr", + &[], + super::super::identity::IdentityObservation { + outcomes: vec![super::super::identity::IdentityOutcome::ConsentWithdrawn], + consent: Default::default(), + }, + &old_context, + ®istry, + &settings, + ) + .expect("should stage withdrawal"); + let mut context = make_context( + Some(&other_id), + Some(&other_id), + true, + false, + Jurisdiction::NonRegulated, + ); + let mut response = empty_response(); + response.extensions_mut().insert(effects); + ec_finalize_response( + &settings, + &mut context, + None, + ®istry, + None, + &mut response, + ); + assert!(response.headers().get(http::header::SET_COOKIE).is_none()); + } + #[test] fn withdrawal_ec_ids_returns_cookie_ec_only_when_active_missing() { let cookie_ec = sample_ec_id("cook1e"); @@ -764,7 +880,6 @@ mod tests { Some(&kv), ®istry, None, - None, &mut response, ); @@ -816,7 +931,6 @@ mod tests { Some(&kv), ®istry, None, - None, &mut response, ); @@ -867,7 +981,6 @@ mod tests { Some(&kv), ®istry, None, - None, &mut response, ); @@ -912,7 +1025,6 @@ mod tests { None, &test_registry, None, - None, &mut response, ); @@ -964,7 +1076,6 @@ mod tests { None, &test_registry, None, - None, &mut response, ); @@ -998,7 +1109,6 @@ mod tests { None, &test_registry, None, - None, &mut response, ); @@ -1032,7 +1142,6 @@ mod tests { None, &test_registry, None, - None, &mut response, ); @@ -1073,7 +1182,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -1115,14 +1223,10 @@ mod tests { ); ec_context.set_recovery_eligible(true); ec_context.set_eid_sync_source(EidSyncSource::Navigation); - ec_context.set_client_eids(vec![Eid { - source: "id5-sync.com".to_owned(), - uids: vec![Uid { - id: "ID5_should_not_persist".to_owned(), - atype: Some(1), - ext: None, - }], - }]); + ec_context.stage_browser_eid_updates(vec![PartnerIdUpdate::new( + "id5-sync.com", + "ID5_should_not_persist", + )]); let partners = vec![make_partner("id5-sync.com")]; let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); let graph = KvIdentityGraph::in_memory("test_store"); @@ -1134,7 +1238,6 @@ mod tests { Some(&graph), ®istry, None, - None, &mut response, ); @@ -1189,7 +1292,6 @@ mod tests { Some(&graph), ®istry, None, - None, &mut response, ); @@ -1232,7 +1334,6 @@ mod tests { &mut ec_context, Some(&graph), ®istry, - None, Some("shared-cookie-id"), &mut response, ); @@ -1276,7 +1377,6 @@ mod tests { &mut ec_context, Some(&graph), ®istry, - None, Some(cookie_id), &mut response, ); @@ -1319,7 +1419,6 @@ mod tests { &mut ec_context, Some(&graph), ®istry, - None, Some("generated-cookie-id"), &mut response, ); @@ -1464,7 +1563,6 @@ mod tests { &mut ec_context, Some(&graph), ®istry, - None, Some("shared-cookie-id"), &mut response, ); @@ -1482,12 +1580,9 @@ mod tests { } #[test] - fn finalize_persists_every_configured_partner_from_client_eids_over_a_trimmed_cookie() { - // Regression test for #1184: the `ts-eids` cookie only carries what - // the browser could fit under its size cap, but `/auction` also - // hands finalization the full EID set from the request body via - // `EcContext::set_client_eids`. That full set must land in KV even - // when the cookie alone would have dropped a configured partner. + fn finalize_persists_every_configured_partner_from_staged_body_eids() { + // Regression test for #1184: persistence must use the full auction body, + // without depending on the retired size-capped EID cookie. let settings = create_test_settings(); let ec_id = sample_ec_id("cleids1"); let graph = KvIdentityGraph::in_memory("test_store"); @@ -1508,31 +1603,9 @@ mod tests { false, ); ec_context.set_eid_sync_source(EidSyncSource::Auction); - // Only `id5-sync.com` "fit" in the (simulated) trimmed cookie; - // `liveramp.com` was dropped by the browser's size cap. - let eids_cookie = base64::engine::general_purpose::STANDARD.encode( - serde_json::to_vec(&serde_json::json!([ - {"source": "id5-sync.com", "uids": [{"id": "ID5_from_cookie", "atype": 1}]} - ])) - .expect("should serialize test cookie payload"), - ); - ec_context.set_client_eids(vec![ - Eid { - source: "id5-sync.com".to_owned(), - uids: vec![Uid { - id: "ID5_from_body".to_owned(), - atype: Some(1), - ext: None, - }], - }, - Eid { - source: "liveramp.com".to_owned(), - uids: vec![Uid { - id: "LR_from_body".to_owned(), - atype: Some(3), - ext: None, - }], - }, + ec_context.stage_browser_eid_updates(vec![ + PartnerIdUpdate::new("id5-sync.com", "ID5_from_body"), + PartnerIdUpdate::new("liveramp.com", "LR_from_body"), ]); let partners = vec![make_partner("id5-sync.com"), make_partner("liveramp.com")]; let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); @@ -1543,7 +1616,6 @@ mod tests { &mut ec_context, Some(&graph), ®istry, - Some(&eids_cookie), None, &mut response, ); @@ -1555,12 +1627,12 @@ mod tests { assert_eq!( stored.ids.get("id5-sync.com").map(|id| id.uid.as_str()), Some("ID5_from_body"), - "the request body's EID should win over the cookie's stale value" + "the request body's EID should persist without an EID cookie" ); assert_eq!( stored.ids.get("liveramp.com").map(|id| id.uid.as_str()), Some("LR_from_body"), - "a partner the cookie trimmed must still be ingested from the body" + "every configured partner must be ingested from the body" ); } @@ -1595,14 +1667,10 @@ mod tests { ec_id: ec_id.clone(), }); ec_context.set_eid_sync_source(EidSyncSource::Auction); - ec_context.set_client_eids(vec![Eid { - source: "id5-sync.com".to_owned(), - uids: vec![Uid { - id: "ID5_should_not_persist".to_owned(), - atype: Some(1), - ext: None, - }], - }]); + ec_context.stage_browser_eid_updates(vec![PartnerIdUpdate::new( + "id5-sync.com", + "ID5_should_not_persist", + )]); let partners = vec![make_partner("id5-sync.com")]; let registry = PartnerRegistry::from_config(&partners).expect("should build registry"); let mut response = empty_response(); @@ -1613,7 +1681,6 @@ mod tests { Some(&graph), ®istry, None, - None, &mut response, ); @@ -1652,7 +1719,6 @@ mod tests { &mut ec_context, Some(&graph), ®istry, - None, Some("shared-cookie-id"), &mut response, ); @@ -1697,7 +1763,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -1744,7 +1809,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -1801,7 +1865,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -1848,7 +1911,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -1878,7 +1940,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -1901,7 +1962,6 @@ mod tests { None, &test_registry, None, - None, &mut response, ); @@ -1937,7 +1997,6 @@ mod tests { None, &test_registry, None, - None, &mut response, ); @@ -2009,7 +2068,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -2036,7 +2094,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -2062,7 +2119,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -2090,7 +2146,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -2129,7 +2184,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -2182,7 +2236,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -2218,7 +2271,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut repeated_response, ); @@ -2266,7 +2318,6 @@ mod tests { Some(&graph), &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -2317,6 +2368,7 @@ mod tests { "ssp.example.com".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "partner-uid".to_owned(), + ..Default::default() }, ); ec_context.set_kv_snapshot(EcKvSnapshot::Present { @@ -2332,7 +2384,6 @@ mod tests { None, ®istry, None, - None, &mut response, ); @@ -2368,7 +2419,6 @@ mod tests { None, &PartnerRegistry::empty(), None, - None, &mut response, ); @@ -2399,7 +2449,6 @@ mod tests { None, &PartnerRegistry::empty(), None, - None, &mut response, ); diff --git a/crates/trusted-server-core/src/ec/identify.rs b/crates/trusted-server-core/src/ec/identify.rs index 94464d1a5..3c1153eff 100644 --- a/crates/trusted-server-core/src/ec/identify.rs +++ b/crates/trusted-server-core/src/ec/identify.rs @@ -97,7 +97,7 @@ pub fn handle_identify( } else { // Extract only this partner's UID. if let Some(partner_uid) = entry.ids.get(&partner.source_domain) - && !partner_uid.uid.is_empty() + && partner_uid.is_usable(super::checked_current_timestamp()) { uid = Some(partner_uid.uid.clone()); } @@ -125,6 +125,9 @@ pub fn handle_identify( } let eid = uid.as_ref().map(|u| Eid { + inserter: None, + matcher: None, + mm: None, source: partner.source_domain.clone(), uids: vec![Uid { id: u.clone(), @@ -374,6 +377,7 @@ mod tests { pull_sync_allowed_domains: vec![], pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + identity_owner: None, ts_pull_token: None, } } @@ -586,6 +590,70 @@ mod tests { assert_no_store(&response); } + #[test] + fn handle_identify_omits_expired_identity() { + let settings = create_test_settings(); + let graph = KvIdentityGraph::in_memory("identify-store"); + let registry = + PartnerRegistry::from_config(&[make_test_partner("ids.example.com", VALID_API_TOKEN)]) + .expect("should build registry"); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let mut entry = + crate::ec::kv_types::KvEntry::minimal("ids.example.com", "expired-uid", 1_000); + entry + .ids + .get_mut("ids.example.com") + .expect("should contain record") + .expires_at = Some(1); + entry.network = Some(crate::ec::kv_types::KvNetwork { + cluster_size: Some(1), + }); + graph.create(&ec_id, &entry).expect("should seed identity"); + let (_, generation_before) = graph + .get(&ec_id) + .expect("should read identity") + .expect("should find identity"); + let req = Request::builder() + .method("GET") + .uri("https://publisher.example.com/_ts/api/v1/identify") + .header("authorization", format!("Bearer {VALID_API_TOKEN}")) + .body(EdgeBody::empty()) + .expect("should build identify request"); + let ec_context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + + let response = handle_identify(&settings, &graph, ®istry, &req, &ec_context) + .expect("should build identify response"); + assert_eq!( + response.status(), + StatusCode::OK, + "should retain the identify response contract" + ); + assert_no_store(&response); + let body: serde_json::Value = serde_json::from_slice( + &response + .into_body() + .into_bytes() + .expect("should read response"), + ) + .expect("should decode identify response"); + assert!( + body.get("uid").is_none() && body.get("eid").is_none(), + "should omit the expired UID and EID" + ); + let (retained, generation_after) = graph + .get(&ec_id) + .expect("should read retained identity") + .expect("should retain identity"); + assert_eq!( + retained, entry, + "should retain lifecycle state without cleanup" + ); + assert_eq!( + generation_after, generation_before, + "should not write merely to omit an expired ID" + ); + } + #[test] fn handle_identify_kv_failure_sets_degraded_true() { let settings = create_test_settings(); diff --git a/crates/trusted-server-core/src/ec/identity.rs b/crates/trusted-server-core/src/ec/identity.rs new file mode 100644 index 000000000..ac587d68e --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity.rs @@ -0,0 +1,941 @@ +//! Shared policy for framework-associated identity observations. +//! +//! Integrations interpret provider data; this module validates authority and +//! stages effects. Only the adapter's post-send executor applies capture writes. + +use std::collections::BTreeMap; + +use crate::consent::{RawConsentSignals, identity_capture_consent_allowed}; +use crate::settings::Settings; + +use super::kv::KvIdentityGraph; +use super::kv_types::MAX_UID_LENGTH; +use super::partner::normalize_partner_source_domain; +use super::registry::PartnerRegistry; +use super::{EcContext, EcKvSnapshot, checked_current_timestamp, is_valid_ec_id}; + +/// Maximum distinct sources contributed by one observation. +const MAX_CAPTURE_SOURCES: usize = 64; + +/// Provider interpretation. Browser submissions cannot construct trusted effects. +pub enum IdentityOutcome { + /// A provider issued a UID, optionally with expiry in Unix seconds. + Issued { + /// Registered identity namespace. + source: String, + /// Opaque provider UID. + uid: String, + /// Supported provider expiry, when supplied. + expires_at: Option, + }, + /// No new token, not invalidation or withdrawal. + NoChange, + /// Explicit approved publisher-wide withdrawal, independent of token gating. + ConsentWithdrawn, +} + +/// Body consent signals that may restrict, but never grant request permission. +#[derive(Clone, Default)] +pub struct IdentityConsentSignals { + /// Populated TCF string. + pub tcf: Option, + /// Populated GPP string. + pub gpp: Option, + /// Populated US Privacy string. + pub usp: Option, + /// Malformed body or signal shape must fail acquisition closed. + pub malformed: bool, +} + +/// Untrusted interpretation returned by a registered capture capability. +pub struct IdentityObservation { + /// Interpreted token or withdrawal outcomes. + pub outcomes: Vec, + /// Optional body restrictions, checked by the shared consent policy. + pub consent: IdentityConsentSignals, +} + +/// Validated capture update. Writer and revisions are assigned by EC, not JSON. +#[derive(Clone)] +pub(crate) struct ManagedPartnerIdUpdate { + pub(crate) source: String, + pub(crate) uid: String, + pub(crate) expires_at: Option, +} + +/// Request-scoped effects sealed by the framework's proxy dispatch boundary. +/// +/// This type intentionally has neither deserialization nor a public constructor. +/// It contains only bounded UID data and the existing full EC-ID-bound snapshot. +#[derive(Clone)] +pub struct IdentityEffects { + module: &'static str, + ec_id: String, + snapshot: EcKvSnapshot, + action: IdentityAction, +} + +#[derive(Clone)] +enum IdentityAction { + Enrich(Vec), + Withdraw, +} + +impl IdentityEffects { + /// Whether this effect requests withdrawal for the current full EC identity. + #[must_use] + pub fn withdraws(&self, context: &EcContext) -> bool { + matches!(self.action, IdentityAction::Withdraw) + && context.ec_value() == Some(self.ec_id.as_str()) + } + + /// Applies once after send, then supplies persisted state to later pull work. + /// + /// Missing, failed, mismatched and withdrawn roots never authorize enrichment. + /// A withdrawal already completed by finalization does not write again. + pub fn apply( + self, + graph: &KvIdentityGraph, + partners: &PartnerRegistry, + context: &mut EcContext, + ) { + if context.ec_value() != Some(self.ec_id.as_str()) { + return; + } + let snapshot = if context.kv_snapshot().belongs_to(&self.ec_id) { + context.kv_snapshot().clone() + } else { + self.snapshot + }; + let result = match self.action { + IdentityAction::Withdraw => { + graph.tombstone_existing_from_snapshot(&self.ec_id, snapshot) + } + IdentityAction::Enrich(updates) if context.ec_allowed() => graph + .upsert_managed_ids_from_snapshot( + &self.ec_id, + self.module, + partners, + &updates, + snapshot, + ), + IdentityAction::Enrich(_) => return, + }; + context.set_kv_snapshot(result); + } +} + +/// Sends exactly once before opening capture storage or applying effects. +/// +/// Returns false if capture storage is unavailable, suppressing later disclosure. +/// No effect means no capture lookup. Adapters must finalize withdrawal cookies +/// before handing their actual response sender to this function. +pub fn send_then_apply_identity( + effects: Option, + context: &mut EcContext, + partners: &PartnerRegistry, + send: F, + graph_factory: G, +) -> bool +where + F: FnOnce(), + G: FnOnce() -> Option, +{ + send(); + if let Some(effects) = effects { + let Some(graph) = graph_factory() else { + return false; + }; + effects.apply(&graph, partners, context); + } + true +} + +/// Seals observations from the actual registered proxy response, never headers. +pub(crate) fn stage_capture( + module: &'static str, + claims: &[&str], + observation: IdentityObservation, + context: &EcContext, + partners: &PartnerRegistry, + settings: &Settings, +) -> Option { + let ec_id = context.ec_value().filter(|id| is_valid_ec_id(id))?; + if !context.ec_was_present() || context.ec_generated() { + return None; + } + if observation + .outcomes + .iter() + .any(|outcome| matches!(outcome, IdentityOutcome::ConsentWithdrawn)) + { + return Some(IdentityEffects { + module, + ec_id: ec_id.to_owned(), + snapshot: context.kv_snapshot().clone(), + action: IdentityAction::Withdraw, + }); + } + let signals = RawConsentSignals { + raw_tc_string: observation.consent.tcf, + raw_gpp_string: observation.consent.gpp, + raw_us_privacy: observation.consent.usp, + ..Default::default() + }; + if observation.consent.malformed + || !context.ec_allowed() + || !identity_capture_consent_allowed(context.consent(), &signals, &settings.consent) + { + return None; + } + let now = checked_current_timestamp(); + let mut updates = BTreeMap::new(); + for outcome in observation.outcomes { + let IdentityOutcome::Issued { + source, + uid, + expires_at, + } = outcome + else { + continue; + }; + let Ok(source) = normalize_partner_source_domain(&source) else { + continue; + }; + let Some(partner) = partners.get(&source) else { + continue; + }; + if partner.identity_owner.as_deref() != Some(module) + || !claims.iter().any(|claim| { + normalize_partner_source_domain(claim).ok().as_deref() == Some(&source) + }) + || uid.trim().is_empty() + || uid.len() > MAX_UID_LENGTH + || expires_at.is_some_and(|expiry| now.is_none_or(|now| now >= expiry)) + { + continue; + } + if !updates.contains_key(&source) && updates.len() >= MAX_CAPTURE_SOURCES { + continue; + } + updates.insert( + source.clone(), + ManagedPartnerIdUpdate { + source, + uid, + expires_at, + }, + ); + } + if updates.is_empty() { + return None; + } + Some(IdentityEffects { + module, + ec_id: ec_id.to_owned(), + snapshot: context.kv_snapshot().clone(), + action: IdentityAction::Enrich(updates.into_values().collect()), + }) +} + +#[cfg(test)] +mod tests { + use std::cell::{Cell, RefCell}; + + use super::*; + use crate::consent::jurisdiction::Jurisdiction; + use crate::consent::{ConsentContext, ConsentSource}; + use crate::consent_config::ConsentMode; + use crate::ec::kv_types::{KvEntry, KvPartnerId}; + use crate::settings::EcPartner; + use crate::test_support::tests::create_test_settings; + + const SOURCE: &str = "ids.example.com"; + const MODULE: &str = "example_identity"; + const CLAIMS: &[&str] = &[SOURCE]; + + fn ec_id(suffix: &str) -> String { + format!("{}.{}", "a".repeat(64), &suffix[..6]) + } + + fn context(id: &str) -> EcContext { + EcContext::new_for_test( + Some(id.to_owned()), + ConsentContext { + jurisdiction: Jurisdiction::NonRegulated, + source: ConsentSource::Cookie, + ..Default::default() + }, + ) + } + + fn registry(owner: Option<&str>) -> PartnerRegistry { + PartnerRegistry::from_config(&[EcPartner { + name: "Synthetic ID source".to_owned(), + source_domain: SOURCE.to_owned(), + identity_owner: owner.map(str::to_owned), + openrtb_atype: 1, + bidstream_enabled: true, + api_token: None, + batch_rate_limit: EcPartner::default_batch_rate_limit(), + pull_sync_enabled: false, + pull_sync_url: None, + pull_sync_allowed_domains: vec![], + pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), + pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + ts_pull_token: None, + }]) + .expect("should build synthetic registry") + } + + fn issued(source: &str, uid: &str, expiry: Option) -> IdentityObservation { + IdentityObservation { + outcomes: vec![IdentityOutcome::Issued { + source: source.to_owned(), + uid: uid.to_owned(), + expires_at: expiry, + }], + consent: IdentityConsentSignals::default(), + } + } + + fn live_graph(id: &str) -> KvIdentityGraph { + let graph = KvIdentityGraph::in_memory("identity-capture-tests"); + let mut entry = KvEntry::minimal( + "other.example.com", + "untouched", + super::super::current_timestamp(), + ); + entry.consent.ok = true; + graph + .create(id, &entry) + .expect("should seed consenting EC row"); + graph + } + + fn capture_and_apply( + graph: &KvIdentityGraph, + context: &mut EcContext, + partners: &PartnerRegistry, + settings: &Settings, + observation: IdentityObservation, + claims: &[&str], + ) -> bool { + let Some(effect) = stage_capture(MODULE, claims, observation, context, partners, settings) + else { + return false; + }; + effect.apply(graph, partners, context); + true + } + + fn stored(graph: &KvIdentityGraph, id: &str) -> (KvEntry, u64) { + graph + .get(id) + .expect("should read KV") + .expect("should find existing EC row") + } + + #[test] + fn capture_rejects_unclaimed_unowned_and_unknown_sources_without_mutation() { + let id = ec_id("source01"); + let graph = live_graph(&id); + let settings = create_test_settings(); + let owned = registry(Some(MODULE)); + let unowned = registry(None); + let original = stored(&graph, &id); + for (partners, claims, source) in [ + (&owned, &[][..], SOURCE), + (&unowned, CLAIMS, SOURCE), + (&owned, CLAIMS, "unknown.example.com"), + ] { + let mut request = context(&id); + request.set_kv_snapshot(graph.load_snapshot(&id)); + assert!(!capture_and_apply( + &graph, + &mut request, + partners, + &settings, + issued(source, "synthetic-uid", None), + claims + )); + assert_eq!( + stored(&graph, &id), + original, + "rejected source cannot write" + ); + } + let mut request = context(&id); + for bad in [ + issued("https://ids.example.com/path", "synthetic-uid", None), + issued(SOURCE, &"x".repeat(MAX_UID_LENGTH + 1), None), + ] { + assert!(!capture_and_apply( + &graph, + &mut request, + &owned, + &settings, + bad, + CLAIMS + )); + assert_eq!(stored(&graph, &id), original); + } + } + + #[test] + fn trusted_writer_and_revision_are_assigned_by_ec_not_observation() { + let id = ec_id("writer01"); + let graph = KvIdentityGraph::in_memory("identity-writer-test"); + let mut entry = KvEntry::minimal( + "other.example.com", + "untouched", + super::super::current_timestamp(), + ); + entry.consent.ok = true; + entry.ids.insert( + SOURCE.to_owned(), + KvPartnerId { + uid: "synthetic-uid".to_owned(), + expires_at: None, + writer: Some("browser".to_owned()), + revision: Some("untrusted-label".to_owned()), + }, + ); + graph + .create(&id, &entry) + .expect("should seed non-authoritative source metadata"); + let partners = registry(Some(MODULE)); + let settings = create_test_settings(); + let mut request = context(&id); + request.set_kv_snapshot(graph.load_snapshot(&id)); + assert!(capture_and_apply( + &graph, + &mut request, + &partners, + &settings, + issued("IDS.EXAMPLE.COM", "synthetic-uid", None), + CLAIMS + )); + let (entry, _) = stored(&graph, &id); + let record = &entry.ids[SOURCE]; + assert_eq!(record.uid, "synthetic-uid"); + assert_eq!(record.writer.as_deref(), Some(MODULE)); + assert!( + record + .revision + .as_ref() + .is_some_and(|revision| !revision.is_empty() && revision != "untrusted-label") + ); + assert!(record.expires_at.is_none()); + assert_eq!(entry.ids["other.example.com"].uid, "untouched"); + } + + #[test] + fn restrictive_body_signals_fail_closed_even_when_jurisdiction_or_mode_is_permissive() { + let id = ec_id("consent1"); + let graph = live_graph(&id); + let partners = registry(Some(MODULE)); + for mode in [ConsentMode::Proxy, ConsentMode::Interpreter] { + let mut settings = create_test_settings(); + settings.consent.mode = mode; + for consent in [ + IdentityConsentSignals { + tcf: Some("not-tcf".to_owned()), + ..Default::default() + }, + IdentityConsentSignals { + gpp: Some("not-gpp".to_owned()), + ..Default::default() + }, + IdentityConsentSignals { + usp: Some("not-usp".to_owned()), + ..Default::default() + }, + IdentityConsentSignals { + usp: Some("1YYN".to_owned()), + ..Default::default() + }, + IdentityConsentSignals { + malformed: true, + ..Default::default() + }, + ] { + let mut observation = issued(SOURCE, "synthetic-uid", None); + observation.consent = consent; + let mut request = context(&id); + assert!( + stage_capture(MODULE, CLAIMS, observation, &request, &partners, &settings) + .is_none() + ); + assert_eq!(stored(&graph, &id).0.ids.get(SOURCE), None); + request.set_kv_snapshot(graph.load_snapshot(&id)); + } + } + } + + #[test] + fn request_denial_wins_over_permissive_body_and_body_optout_over_cookie_permission() { + let id = ec_id("consent2"); + let partners = registry(Some(MODULE)); + let settings = create_test_settings(); + let graph = live_graph(&id); + for (raw_usp, gpc) in [(Some("1YYN"), false), (None, true)] { + let mut denied = context(&id); + denied.consent.raw_us_privacy = raw_usp.map(str::to_owned); + denied.consent.gpc = gpc; + assert!( + stage_capture( + MODULE, + CLAIMS, + issued(SOURCE, "synthetic-uid", None), + &denied, + &partners, + &settings + ) + .is_none() + ); + } + let mut allowed = context(&id); + let mut observation = issued(SOURCE, "synthetic-uid", None); + observation.consent.usp = Some("1YYN".to_owned()); + assert!(!capture_and_apply( + &graph, + &mut allowed, + &partners, + &settings, + observation, + CLAIMS + )); + assert!(!stored(&graph, &id).0.ids.contains_key(SOURCE)); + } + + #[test] + fn no_change_and_absent_identity_do_not_create_or_mutate_kv_rows() { + let id = ec_id("nochange"); + let graph = live_graph(&id); + let partners = registry(Some(MODULE)); + let settings = create_test_settings(); + let mut request = context(&id); + request.set_kv_snapshot(graph.load_snapshot(&id)); + let original = stored(&graph, &id); + assert!(!capture_and_apply( + &graph, + &mut request, + &partners, + &settings, + IdentityObservation { + outcomes: vec![IdentityOutcome::NoChange], + consent: Default::default() + }, + CLAIMS + )); + assert_eq!(stored(&graph, &id), original); + + let missing_id = ec_id("missing1"); + let mut missing = context(&missing_id); + missing.set_kv_snapshot(graph.load_snapshot(&missing_id)); + assert!(capture_and_apply( + &graph, + &mut missing, + &partners, + &settings, + issued(SOURCE, "synthetic-uid", None), + CLAIMS + )); + assert!(graph.get(&missing_id).expect("should read KV").is_none()); + assert!(matches!( + missing.kv_snapshot(), + EcKvSnapshot::Missing { .. } + )); + assert_eq!(stored(&graph, &id), original); + + let no_token = EcContext::new_for_test(None, context(&id).consent().clone()); + assert!( + stage_capture( + MODULE, + CLAIMS, + issued(SOURCE, "synthetic-uid", None), + &no_token, + &partners, + &settings + ) + .is_none() + ); + let generated = EcContext::new_for_test_with_cookie( + Some(ec_id("newborn")), + None, + false, + true, + context(&id).consent().clone(), + ); + assert!( + stage_capture( + MODULE, + CLAIMS, + issued(SOURCE, "synthetic-uid", None), + &generated, + &partners, + &settings + ) + .is_none() + ); + } + + #[test] + fn unreadable_or_tombstoned_roots_cannot_be_revived() { + let id = ec_id("unread1"); + let graph = live_graph(&id); + let partners = registry(Some(MODULE)); + let settings = create_test_settings(); + let original = stored(&graph, &id); + let mut failed = context(&id); + failed.set_kv_snapshot(EcKvSnapshot::Failed { ec_id: id.clone() }); + assert!(capture_and_apply( + &graph, + &mut failed, + &partners, + &settings, + issued(SOURCE, "synthetic-uid", None), + CLAIMS + )); + assert!(matches!(failed.kv_snapshot(), EcKvSnapshot::Failed { .. })); + assert_eq!(stored(&graph, &id), original); + + let mut stale = context(&id); + stale.set_kv_snapshot(graph.load_snapshot(&id)); + let late_effect = stage_capture( + MODULE, + CLAIMS, + issued(SOURCE, "stale-uid", None), + &stale, + &partners, + &settings, + ) + .expect("should stage while live"); + graph.tombstone_existing_from_snapshot(&id, graph.load_snapshot(&id)); + late_effect.apply(&graph, &partners, &mut stale); + let (_, before_generation) = stored(&graph, &id); + let mut withdrawn = context(&id); + withdrawn.set_kv_snapshot(graph.load_snapshot(&id)); + assert!(capture_and_apply( + &graph, + &mut withdrawn, + &partners, + &settings, + issued(SOURCE, "late-uid", None), + CLAIMS + )); + let (after, after_generation) = stored(&graph, &id); + assert!(!after.consent.ok); + assert!(after.ids.is_empty()); + assert_eq!(before_generation, after_generation); + } + + #[test] + fn managed_same_uid_retains_expiry_extends_it_and_never_restores_an_old_revision() { + let id = ec_id("revision"); + let graph = live_graph(&id); + let partners = registry(Some(MODULE)); + let settings = create_test_settings(); + let initial_expiry = checked_current_timestamp().expect("should read test clock") + 3600; + let extended_expiry = initial_expiry + 3600; + let mut request = context(&id); + let mut versions: Vec<(Option, u64)> = Vec::new(); + for (uid, expiry, expected_expiry, should_change) in [ + ( + "synthetic-uid", + Some(initial_expiry), + Some(initial_expiry), + true, + ), + ("synthetic-uid", None, Some(initial_expiry), false), + ( + "synthetic-uid", + Some(extended_expiry), + Some(extended_expiry), + true, + ), + ("other-uid", None, None, true), + ( + "synthetic-uid", + Some(initial_expiry), + Some(initial_expiry), + true, + ), + ] { + request.set_kv_snapshot(graph.load_snapshot(&id)); + assert!(capture_and_apply( + &graph, + &mut request, + &partners, + &settings, + issued(SOURCE, uid, expiry), + CLAIMS + )); + let (entry, generation) = stored(&graph, &id); + let record = &entry.ids[SOURCE]; + assert_eq!(record.uid, uid); + assert_eq!(record.expires_at, expected_expiry); + assert_eq!(record.writer.as_deref(), Some(MODULE)); + if let Some((earlier_revision, earlier_generation)) = versions.last() { + assert_eq!( + generation != *earlier_generation, + should_change, + "only material changes should write" + ); + assert_eq!( + record.revision.as_ref() != earlier_revision.as_ref(), + should_change, + "revision must change only for material updates" + ); + } + versions.push((record.revision.clone(), generation)); + } + let (before, generation) = stored(&graph, &id); + assert_ne!( + versions[0].0, versions[4].0, + "ABA must not restore an older revision" + ); + request.set_kv_snapshot(graph.load_snapshot(&id)); + assert!(capture_and_apply( + &graph, + &mut request, + &partners, + &settings, + issued(SOURCE, "synthetic-uid", None), + CLAIMS + )); + assert_eq!( + stored(&graph, &id), + (before, generation), + "same UID without expiry cannot clear or bump metadata" + ); + } + + #[test] + fn expiry_is_rechecked_when_the_deferred_effect_is_applied() { + let id = ec_id("expire01"); + let graph = live_graph(&id); + let partners = registry(Some(MODULE)); + let mut request = context(&id); + request.set_kv_snapshot(graph.load_snapshot(&id)); + let original = stored(&graph, &id); + // Construct a sealed effect inside the owning module to model an ID + // that was valid at capture but expired while the response was sent. + let effect = IdentityEffects { + module: MODULE, + ec_id: id.clone(), + snapshot: request.kv_snapshot().clone(), + action: IdentityAction::Enrich(vec![ManagedPartnerIdUpdate { + source: SOURCE.to_owned(), + uid: "expired-uid".to_owned(), + expires_at: Some(1), + }]), + }; + effect.apply(&graph, &partners, &mut request); + assert_eq!( + stored(&graph, &id), + original, + "expired token cannot be written" + ); + } + + #[test] + fn capture_effects_are_bound_to_the_full_ec_id() { + let id = ec_id("original"); + let rotated = ec_id("rotated1"); + let graph = live_graph(&id); + let partners = registry(Some(MODULE)); + let settings = create_test_settings(); + let first = context(&id); + let effect = stage_capture( + MODULE, + CLAIMS, + issued(SOURCE, "synthetic-uid", None), + &first, + &partners, + &settings, + ) + .expect("should stage eligible capture"); + let mut later = context(&rotated); + assert!(!effect.withdraws(&later)); + effect.apply(&graph, &partners, &mut later); + assert!(!stored(&graph, &id).0.ids.contains_key(SOURCE)); + assert!(graph.get(&rotated).expect("should read KV").is_none()); + } + + #[test] + fn finalized_withdrawal_skips_duplicate_tombstone_and_late_enrichment() { + let id = ec_id("withdraw"); + let graph = live_graph(&id); + let partners = registry(Some(MODULE)); + let settings = create_test_settings(); + let mut request = context(&id); + request.set_kv_snapshot(graph.load_snapshot(&id)); + let withdrawal = stage_capture( + MODULE, + CLAIMS, + IdentityObservation { + outcomes: vec![IdentityOutcome::ConsentWithdrawn], + consent: Default::default(), + }, + &request, + &partners, + &settings, + ) + .expect("should stage explicit withdrawal"); + assert!(withdrawal.withdraws(&request)); + let finalized = graph.tombstone_existing_from_snapshot(&id, request.kv_snapshot().clone()); + request.set_kv_snapshot(finalized); + let (first, generation) = stored(&graph, &id); + assert!(!first.consent.ok); + assert!(first.ids.is_empty()); + withdrawal.apply(&graph, &partners, &mut request); + let (second, after_generation) = stored(&graph, &id); + assert_eq!( + after_generation, generation, + "post-send must not duplicate finalization" + ); + assert_eq!(second, first); + assert!(capture_and_apply( + &graph, + &mut request, + &partners, + &settings, + issued(SOURCE, "late-uid", None), + CLAIMS + )); + assert_eq!( + stored(&graph, &id), + (first, generation), + "late result cannot revive tombstone" + ); + } + + #[test] + fn send_boundary_persists_staged_effect_only_after_the_response() { + let id = ec_id("send01"); + let graph = live_graph(&id); + let partners = registry(Some(MODULE)); + let settings = create_test_settings(); + let mut request = context(&id); + request.set_kv_snapshot(graph.load_snapshot(&id)); + let before = stored(&graph, &id); + let expires_at = checked_current_timestamp().expect("should read test clock") + 3600; + let effect = stage_capture( + MODULE, + CLAIMS, + issued(SOURCE, "synthetic-uid", Some(expires_at)), + &request, + &partners, + &settings, + ) + .expect("should stage a real validated effect"); + let order = RefCell::new(Vec::new()); + let handled = send_then_apply_identity( + Some(effect), + &mut request, + &partners, + || { + assert_eq!( + stored(&graph, &id), + before, + "send cannot observe capture mutation" + ); + order.borrow_mut().push("sent"); + }, + || { + assert_eq!( + order.borrow().as_slice(), + &["sent"], + "open graph only after send" + ); + assert_eq!(stored(&graph, &id), before); + order.borrow_mut().push("opened graph"); + Some(graph.clone()) + }, + ); + assert!(handled); + assert_eq!(order.borrow().as_slice(), &["sent", "opened graph"]); + let (persisted, generation) = stored(&graph, &id); + assert!( + generation > before.1, + "managed update must reach storage after send" + ); + let record = &persisted.ids[SOURCE]; + assert_eq!(record.uid, "synthetic-uid"); + assert_eq!(record.expires_at, Some(expires_at)); + assert_eq!(record.writer.as_deref(), Some(MODULE)); + assert!(record.revision.is_some()); + let handed_to_later_pull = request + .kv_snapshot() + .entry_for(&id) + .expect("should keep persisted identity state in post-send context"); + assert_eq!( + handed_to_later_pull, &persisted, + "later pull must receive persisted outcome, not the pre-send snapshot" + ); + } + + #[test] + fn send_boundary_skips_storage_without_effects_and_fails_closed_when_unavailable() { + let id = ec_id("send02"); + let graph = live_graph(&id); + let partners = registry(Some(MODULE)); + let settings = create_test_settings(); + let mut request = context(&id); + request.set_kv_snapshot(graph.load_snapshot(&id)); + let before = stored(&graph, &id); + let sends = Cell::new(0); + let factories = Cell::new(0); + assert!(send_then_apply_identity( + None, + &mut request, + &partners, + || sends.set(sends.get() + 1), + || { + factories.set(factories.get() + 1); + Some(graph.clone()) + } + )); + assert_eq!(sends.get(), 1); + assert_eq!(factories.get(), 0, "no token must not open capture storage"); + assert_eq!(stored(&graph, &id), before); + + let effect = stage_capture( + MODULE, + CLAIMS, + issued(SOURCE, "synthetic-uid", None), + &request, + &partners, + &settings, + ) + .expect("should stage real effect"); + let available = send_then_apply_identity( + Some(effect), + &mut request, + &partners, + || { + assert_eq!(stored(&graph, &id), before); + sends.set(sends.get() + 1); + }, + || { + assert_eq!(sends.get(), 2, "factory must follow response send"); + factories.set(factories.get() + 1); + None + }, + ); + assert!( + !available, + "unavailable graph suppresses post-send identity work" + ); + assert_eq!(sends.get(), 2, "each invocation sends exactly once"); + assert_eq!(factories.get(), 1); + assert_eq!(stored(&graph, &id), before); + assert_eq!( + request.kv_snapshot().entry_for(&id), + Some(&before.0), + "failed capture must not claim a persisted UID" + ); + } +} diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index b0eb83d3f..f47ae74e2 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -159,6 +159,7 @@ fn apply_cookie_partner_id_updates( partner_id.to_owned(), super::kv_types::KvPartnerId { uid: uid.to_owned(), + ..Default::default() }, ); changed = true; @@ -187,6 +188,30 @@ fn partner_id_updates_match(entry: &KvEntry, updates: &[PartnerIdUpdate]) -> boo }) } +/// Applies a UID-only legacy update without retaining another UID's lifecycle metadata. +fn apply_legacy_partner_id_update(entry: &mut KvEntry, partner_id: &str, uid: &str) -> bool { + let previous = entry.ids.get(partner_id); + if previous.is_some_and(|record| record.uid == uid) { + return false; + } + let had_metadata = previous.is_some_and(|record| { + record.expires_at.is_some() || record.writer.is_some() || record.revision.is_some() + }); + let mut replacement = super::kv_types::KvPartnerId { + uid: uid.to_owned(), + ..Default::default() + }; + // Keep legacy-only rows UID-only during the reader/writer rollout. Once a + // record has lifecycle metadata, replacement must change its version and + // writer without applying the old token's expiry to the replacement UID. + if had_metadata { + replacement.writer = Some("legacy".to_owned()); + replacement.revision = Some(uuid::Uuid::new_v4().to_string()); + } + entry.ids.insert(partner_id.to_owned(), replacement); + true +} + pub(crate) fn apply_partner_id_updates(entry: &mut KvEntry, updates: &[PartnerIdUpdate]) -> bool { let mut latest_updates = BTreeMap::new(); for update in updates { @@ -195,21 +220,7 @@ pub(crate) fn apply_partner_id_updates(entry: &mut KvEntry, updates: &[PartnerId let mut changed = false; for (partner_id, uid) in latest_updates { - if entry - .ids - .get(partner_id) - .is_some_and(|existing| existing.uid == uid) - { - continue; - } - - entry.ids.insert( - partner_id.to_owned(), - super::kv_types::KvPartnerId { - uid: uid.to_owned(), - }, - ); - changed = true; + changed |= apply_legacy_partner_id_update(entry, partner_id, uid); } changed @@ -837,6 +848,124 @@ impl KvIdentityGraph { } } + /// Applies validated owned capture records using bounded existing-root CAS. + /// + /// Missing and failed snapshots are never promoted to existence proof. + /// Ownership, root consent and effective expiry are rechecked at mutation. + pub(crate) fn upsert_managed_ids_from_snapshot( + &self, + ec_id: &str, + module: &str, + partners: &super::registry::PartnerRegistry, + updates: &[super::identity::ManagedPartnerIdUpdate], + snapshot: EcKvSnapshot, + ) -> EcKvSnapshot { + if updates.is_empty() { + return snapshot; + } + let mut current = match snapshot { + EcKvSnapshot::NotRead => self.load_snapshot(ec_id), + EcKvSnapshot::Present { + ec_id: ref id, + generation: None, + .. + } if id == ec_id => self.load_snapshot(ec_id), + state if state.belongs_to(ec_id) => state, + _ => { + return EcKvSnapshot::Failed { + ec_id: ec_id.to_owned(), + }; + } + }; + for _ in 0..MAX_CAS_RETRIES { + let EcKvSnapshot::Present { + ec_id: ref id, + ref entry, + generation: Some(generation), + } = current + else { + return current; + }; + if id != ec_id || !entry.consent.ok { + return current; + } + let now = checked_current_timestamp(); + let mut candidate = entry.as_ref().clone(); + let mut changed = false; + for update in updates { + if partners + .get(&update.source) + .and_then(|partner| partner.identity_owner.as_deref()) + != Some(module) + { + continue; + } + let previous = candidate.ids.get(&update.source); + let same_uid = previous.is_some_and(|record| record.uid == update.uid); + let expiry = update.expires_at.or_else(|| { + if same_uid { + previous.and_then(|record| record.expires_at) + } else { + None + } + }); + if expiry.is_some_and(|expiry| now.is_none_or(|now| now >= expiry)) { + continue; + } + if previous.is_some_and(|record| { + same_uid + && record.expires_at == expiry + && record.writer.as_deref() == Some(module) + && record.revision.is_some() + }) { + continue; + } + candidate.ids.insert( + update.source.clone(), + super::kv_types::KvPartnerId { + uid: update.uid.clone(), + expires_at: expiry, + writer: Some(module.to_owned()), + revision: Some(uuid::Uuid::new_v4().to_string()), + }, + ); + changed = true; + } + if !changed { + return current; + } + let Ok((body, metadata)) = Self::serialize_entry(&candidate, self.store_name()) else { + return EcKvSnapshot::Failed { + ec_id: ec_id.to_owned(), + }; + }; + match self.write_entry( + ec_id, + &body, + &metadata, + ENTRY_TTL, + EcKvWriteMode::IfGenerationMatch(generation), + ) { + Ok(EcKvWriteOutcome::Written) => { + return EcKvSnapshot::Present { + ec_id: ec_id.to_owned(), + entry: Box::new(candidate), + generation: None, + }; + } + Ok(EcKvWriteOutcome::PreconditionFailed) => current = self.load_snapshot(ec_id), + Err(_) => { + return EcKvSnapshot::Failed { + ec_id: ec_id.to_owned(), + }; + } + } + } + EcKvSnapshot::Failed { + ec_id: ec_id.to_owned(), + } + } + /// Returns `proven` instead of a read outcome that cannot disprove it. /// /// Partner-ID enrichment is best effort. When the caller already held proof @@ -910,21 +1039,10 @@ impl KvIdentityGraph { ))); } - if entry - .ids - .get(partner_id) - .is_some_and(|existing| existing.uid == uid) - { + if !apply_legacy_partner_id_update(&mut entry, partner_id, uid) { return Ok(()); } - entry.ids.insert( - partner_id.to_owned(), - super::kv_types::KvPartnerId { - uid: uid.to_owned(), - }, - ); - let (body, meta_str) = Self::serialize_entry(&entry, self.store_name())?; match self.write_entry( @@ -982,21 +1100,10 @@ impl KvIdentityGraph { return Ok(UpsertResult::ConsentWithdrawn); } - if entry - .ids - .get(partner_id) - .is_some_and(|existing| existing.uid == uid) - { + if !apply_legacy_partner_id_update(&mut entry, partner_id, uid) { return Ok(UpsertResult::Unchanged); } - entry.ids.insert( - partner_id.to_owned(), - super::kv_types::KvPartnerId { - uid: uid.to_owned(), - }, - ); - let (body, meta_str) = Self::serialize_entry(&entry, self.store_name())?; match self.write_entry( @@ -1739,6 +1846,7 @@ mod tests { "ssp_x".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "x".repeat(crate::ec::kv_types::MAX_UID_LENGTH + 1), + ..Default::default() }, ); let body = serde_json::to_vec(&entry).expect("should serialize invalid entry payload"); @@ -1774,6 +1882,7 @@ mod tests { "ssp_x".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "x".repeat(crate::ec::kv_types::MAX_UID_LENGTH + 1), + ..Default::default() }, ); @@ -1792,12 +1901,251 @@ mod tests { entry } + fn lifecycle_entry() -> KvEntry { + let mut entry = live_entry(); + for source in ["ids.example.com", "other.example.com"] { + entry.ids.insert( + source.to_owned(), + super::super::kv_types::KvPartnerId { + uid: format!("uid-{source}"), + expires_at: Some(2_000), + writer: Some("example_identity".to_owned()), + revision: Some(format!("revision-{source}")), + }, + ); + } + entry + } + + #[test] + fn lifecycle_metadata_survives_all_existing_enrichment_writers() { + for mode in ["browser", "bulk", "single", "batch", "cluster"] { + let graph = KvIdentityGraph::in_memory("lifecycle-store"); + let ec_id = snapshot_ec_id(); + let entry = lifecycle_entry(); + assert_eq!( + graph + .create_if_absent(&ec_id, &entry) + .expect("should create identity"), + CreateIfAbsentOutcome::Written + ); + match mode { + "browser" => { + let (snapshot, outcome) = graph.sync_eid_cookie_updates_from_snapshot( + &ec_id, + &[ + PartnerIdUpdate::new("ids.example.com", "different-browser-uid"), + PartnerIdUpdate::new("new.example.com", "new-uid"), + ], + graph.load_snapshot(&ec_id), + ); + assert_eq!(outcome, EidCookieSyncOutcome::WrittenWithDeferredFreshness); + assert!( + snapshot.entry_for(&ec_id).is_some(), + "should return the persisted snapshot" + ); + } + "bulk" => { + let snapshot = graph.upsert_partner_ids_from_snapshot( + &ec_id, + &[PartnerIdUpdate::new("new.example.com", "new-uid")], + graph.load_snapshot(&ec_id), + ); + assert!( + snapshot.entry_for(&ec_id).is_some(), + "should return the persisted snapshot" + ); + } + "single" => graph + .upsert_partner_id(&ec_id, "new.example.com", "new-uid") + .expect("should update legacy source"), + "batch" => assert_eq!( + graph + .upsert_partner_id_if_exists(&ec_id, "new.example.com", "new-uid") + .expect("should update legacy source"), + UpsertResult::Written + ), + "cluster" => { + let (stored, generation) = graph + .get(&ec_id) + .expect("should read identity") + .expect("should find identity"); + graph + .evaluate_cluster(&ec_id, &stored, generation) + .expect("should update network metadata"); + } + _ => unreachable!("should use a listed writer"), + } + let (retained, _) = graph + .get(&ec_id) + .expect("should read stored state") + .expect("should retain identity"); + for source in entry.ids.keys() { + assert_eq!( + retained.ids.get(source), + entry.ids.get(source), + "should preserve unrelated lifecycle state through {mode}" + ); + } + if mode != "cluster" { + assert_eq!( + retained.ids["new.example.com"].uid, "new-uid", + "should actually persist the additional source" + ); + } + } + } + + #[test] + fn lifecycle_legacy_same_uid_is_a_noop_for_every_update_path() { + for mode in ["browser", "bulk", "single", "batch"] { + let graph = KvIdentityGraph::in_memory("lifecycle-store"); + let ec_id = snapshot_ec_id(); + let entry = lifecycle_entry(); + graph.create(&ec_id, &entry).expect("should seed identity"); + let (_, before) = graph + .get(&ec_id) + .expect("should read identity") + .expect("should find identity"); + let uid = &entry.ids["ids.example.com"].uid; + match mode { + "browser" => { + let (_, outcome) = graph.sync_eid_cookie_updates_from_snapshot( + &ec_id, + &[PartnerIdUpdate::new("ids.example.com", uid)], + graph.load_snapshot(&ec_id), + ); + assert_eq!(outcome, EidCookieSyncOutcome::AlreadyMatched); + } + "bulk" => { + graph.upsert_partner_ids_from_snapshot( + &ec_id, + &[PartnerIdUpdate::new("ids.example.com", uid)], + graph.load_snapshot(&ec_id), + ); + } + "single" => graph + .upsert_partner_id(&ec_id, "ids.example.com", uid) + .expect("should accept unchanged UID"), + "batch" => assert_eq!( + graph + .upsert_partner_id_if_exists(&ec_id, "ids.example.com", uid) + .expect("should accept unchanged UID"), + UpsertResult::Unchanged + ), + _ => unreachable!("should use a listed writer"), + } + let (retained, after) = graph + .get(&ec_id) + .expect("should read retained identity") + .expect("should retain identity"); + assert_eq!( + retained, entry, + "should retain metadata omitted by UID-only input" + ); + assert_eq!( + after, before, + "should not write merely to observe the same UID" + ); + } + } + + #[test] + fn lifecycle_legacy_replacement_resets_expiry_and_never_reuses_revision() { + for mode in ["bulk", "single", "batch"] { + let graph = KvIdentityGraph::in_memory("lifecycle-store"); + let ec_id = snapshot_ec_id(); + let entry = lifecycle_entry(); + graph.create(&ec_id, &entry).expect("should seed identity"); + let mut previous_revision = entry.ids["ids.example.com"].revision.clone(); + for uid in ["replacement-uid", entry.ids["ids.example.com"].uid.as_str()] { + match mode { + "bulk" => { + graph.upsert_partner_ids_from_snapshot( + &ec_id, + &[PartnerIdUpdate::new("ids.example.com", uid)], + graph.load_snapshot(&ec_id), + ); + } + "single" => graph + .upsert_partner_id(&ec_id, "ids.example.com", uid) + .expect("should replace legacy UID"), + "batch" => assert_eq!( + graph + .upsert_partner_id_if_exists(&ec_id, "ids.example.com", uid) + .expect("should replace legacy UID"), + UpsertResult::Written + ), + _ => unreachable!("should use a listed writer"), + } + let (retained, _) = graph + .get(&ec_id) + .expect("should read retained identity") + .expect("should retain identity"); + let record = &retained.ids["ids.example.com"]; + assert_eq!(record.uid, uid, "should persist replacement UID"); + assert_eq!( + record.expires_at, None, + "should not apply another token's expiry to the replacement" + ); + assert_eq!( + record.writer.as_deref(), + Some("legacy"), + "should not retain another writer's provenance" + ); + assert!( + record.revision.is_some(), + "should version the lifecycle change" + ); + assert_ne!( + record.revision, previous_revision, + "should not reuse the previous revision" + ); + assert_ne!( + record.revision, entry.ids["ids.example.com"].revision, + "should not restore a historical revision when restoring a UID" + ); + assert_eq!( + retained.ids["other.example.com"], entry.ids["other.example.com"], + "should preserve unrelated sources" + ); + previous_revision.clone_from(&record.revision); + } + } + } + + #[test] + fn lifecycle_withdrawal_removes_all_identity_payload() { + let graph = KvIdentityGraph::in_memory("lifecycle-store"); + let ec_id = snapshot_ec_id(); + graph + .create(&ec_id, &lifecycle_entry()) + .expect("should seed identity"); + let snapshot = graph.tombstone_existing_from_snapshot(&ec_id, graph.load_snapshot(&ec_id)); + assert!( + snapshot + .entry_for(&ec_id) + .is_some_and(|entry| !entry.consent.ok), + "should return the persisted tombstone" + ); + let (retained, _) = graph + .get(&ec_id) + .expect("should read tombstone") + .expect("should retain tombstone"); + assert!(!retained.consent.ok, "should preserve withdrawal authority"); + assert!( + retained.ids.is_empty(), + "should remove UIDs and all lifecycle metadata" + ); + } + fn concurrent_live_entry() -> KvEntry { let mut entry = live_entry(); entry.ids.insert( "concurrent.example.com".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "concurrent-uid".to_owned(), + ..Default::default() }, ); entry @@ -2133,6 +2481,7 @@ mod tests { "ssp_x".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "uid-1".to_owned(), + ..Default::default() }, ); let updates = vec![PartnerIdUpdate::new("ssp_x", "uid-1")]; @@ -2161,6 +2510,7 @@ mod tests { "ssp_x".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "old-uid".to_owned(), + ..Default::default() }, ); let updates = vec![PartnerIdUpdate::new("ssp_x", "new-uid")]; @@ -2193,6 +2543,7 @@ mod tests { "ssp_x".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "original".to_owned(), + ..Default::default() }, ); let updates = vec![ @@ -2217,6 +2568,7 @@ mod tests { "ssp_x".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "desired-uid".to_owned(), + ..Default::default() }, ); let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); @@ -2335,6 +2687,7 @@ mod tests { "ssp_x".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "latest-uid".to_owned(), + ..Default::default() }, ); let updates = [ @@ -2356,6 +2709,7 @@ mod tests { "ssp_x".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "concurrent-uid".to_owned(), + ..Default::default() }, ); let lookups = std::sync::Arc::new(std::sync::atomic::AtomicUsize::new(0)); @@ -2441,6 +2795,7 @@ mod tests { "ssp_x".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "protected-uid".to_owned(), + ..Default::default() }, ); graph.create(&ec_id, &entry).expect("should seed entry"); @@ -2468,6 +2823,7 @@ mod tests { "ssp_x".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "protected-uid".to_owned(), + ..Default::default() }, ); graph.create(&ec_id, &entry).expect("should seed entry"); diff --git a/crates/trusted-server-core/src/ec/kv_types.rs b/crates/trusted-server-core/src/ec/kv_types.rs index 589f280c1..1c8ba0abf 100644 --- a/crates/trusted-server-core/src/ec/kv_types.rs +++ b/crates/trusted-server-core/src/ec/kv_types.rs @@ -34,10 +34,13 @@ pub const MAX_STORED_DOMAIN_LENGTH: usize = 255; /// validation. pub const MAX_UID_LENGTH: usize = 512; +/// Maximum byte length for a stored writer label or opaque record revision. +pub const MAX_LIFECYCLE_LABEL_BYTES: usize = 128; + /// Full KV entry stored as the body of an EC identity graph record. /// /// **KV key:** Full EC ID (`{64hex}.{6alnum}`). -/// **KV value:** JSON-serialized `KvEntry` (max ~5KB). +/// **KV value:** JSON-serialized `KvEntry`. #[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] pub struct KvEntry { /// Schema version — always [`SCHEMA_VERSION`]. @@ -97,11 +100,34 @@ pub struct KvGeo { pub dma: Option, } -/// A partner user ID within a KV entry. -#[derive(Debug, Clone, PartialEq, Serialize, Deserialize)] +/// A partner user ID and its optional lifecycle metadata within a KV entry. +#[derive(Debug, Clone, Default, PartialEq, Eq, Serialize, Deserialize)] pub struct KvPartnerId { /// The partner's user identifier. pub uid: String, + /// Provider expiry in Unix seconds, when known. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub expires_at: Option, + /// Framework-assigned writer identity, not `OpenRTB` provenance. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub writer: Option, + /// Opaque version of a material record change. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub revision: Option, +} + +impl KvPartnerId { + /// Returns whether this record can be delivered at the supplied checked time. + /// + /// Unknown-expiry records retain legacy behavior. A known expiry requires + /// usable time and is exclusive: the record is unusable at its expiry. + #[must_use] + pub fn is_usable(&self, now: Option) -> bool { + !self.uid.is_empty() + && self + .expires_at + .is_none_or(|expiry| now.is_some_and(|now| now < expiry)) + } } /// Publisher property metadata captured when an EC entry is created. @@ -307,6 +333,7 @@ impl KvEntry { partner_id.to_owned(), KvPartnerId { uid: uid.to_owned(), + ..Default::default() }, ); Self { @@ -381,6 +408,18 @@ impl KvEntry { } for (partner_id, partner_uid) in &self.ids { + for (field, label) in [ + ("writer", partner_uid.writer.as_deref()), + ("revision", partner_uid.revision.as_deref()), + ] { + if let Some(label) = label + && (label.trim().is_empty() || label.len() > MAX_LIFECYCLE_LABEL_BYTES) + { + return Err(format!( + "partner ID '{partner_id}' has invalid {field} metadata" + )); + } + } if partner_uid.uid.len() > MAX_UID_LENGTH { return Err(format!( "partner ID '{partner_id}' exceeds MAX_UID_LENGTH ({})", @@ -485,6 +524,149 @@ mod tests { } } + #[test] + fn lifecycle_labels_are_bounded_without_logging_their_values() { + for label in [ + String::new(), + " ".to_owned(), + "x".repeat(MAX_LIFECYCLE_LABEL_BYTES + 1), + ] { + for field in ["writer", "revision"] { + let mut entry = KvEntry::minimal("ids.example.com", "example-uid", 1_000); + let record = entry + .ids + .get_mut("ids.example.com") + .expect("should contain record"); + if field == "writer" { + record.writer = Some(label.clone()); + } else { + record.revision = Some(label.clone()); + } + let error = entry + .validate() + .expect_err("should reject invalid lifecycle labels"); + assert_eq!( + error, + format!("partner ID 'ids.example.com' has invalid {field} metadata"), + "should report only the field, not its value" + ); + } + } + } + + #[test] + fn partner_usability_respects_expiry_boundaries_and_clock_failure() { + let mut record = KvPartnerId { + uid: "example-uid".to_owned(), + expires_at: Some(2_000), + ..Default::default() + }; + assert!( + record.is_usable(Some(1_999)), + "should allow a record before expiry" + ); + assert!( + !record.is_usable(Some(2_000)), + "should omit a record at expiry" + ); + assert!( + !record.is_usable(Some(2_001)), + "should omit a record after expiry" + ); + assert!( + !record.is_usable(None), + "should not prove known expiry usable without a clock" + ); + record.expires_at = None; + assert!( + record.is_usable(None), + "should retain unknown-expiry legacy behavior" + ); + record.uid.clear(); + assert!(!record.is_usable(Some(1_999)), "should omit an empty UID"); + } + + #[test] + fn legacy_partner_records_keep_their_uid_only_shape() { + let record = serde_json::json!({ "uid": "example-uid" }); + let partner: KvPartnerId = + serde_json::from_value(record.clone()).expect("should decode a legacy partner record"); + + assert_eq!(partner.expires_at, None, "should retain unknown expiry"); + assert_eq!( + partner.writer, None, + "should not invent a historical writer" + ); + assert_eq!( + partner.revision, None, + "should not invent a historical revision" + ); + assert_eq!( + serde_json::to_value(partner).expect("should serialize a legacy partner record"), + record, + "should omit absent lifecycle fields" + ); + } + + #[test] + fn partner_record_equality_includes_lifecycle_metadata() { + let original = KvPartnerId { + uid: "example-uid".to_owned(), + expires_at: Some(2_000), + writer: Some("example_identity".to_owned()), + revision: Some("example-revision".to_owned()), + }; + let mut changed = original.clone(); + changed.expires_at = Some(3_000); + assert_ne!( + original, changed, + "should distinguish same-UID expiry changes" + ); + changed = original.clone(); + changed.writer = Some("legacy".to_owned()); + assert_ne!(original, changed, "should distinguish writer changes"); + changed = original.clone(); + changed.revision = Some("next-revision".to_owned()); + assert_ne!(original, changed, "should distinguish revision changes"); + assert_eq!( + original, + original.clone(), + "should match unchanged metadata" + ); + } + + #[test] + fn malformed_partner_expiry_fails_deserialization() { + for expiry in [serde_json::json!(-1), serde_json::json!("invalid")] { + assert!( + serde_json::from_value::(serde_json::json!({ + "uid": "example-uid", + "expires_at": expiry + })) + .is_err(), + "should not downgrade malformed known expiry to unknown expiry" + ); + } + } + + #[test] + fn partner_lifecycle_metadata_round_trips() { + let record = serde_json::json!({ + "uid": "example-uid", + "expires_at": 2_000, + "writer": "example_identity", + "revision": "example-revision" + }); + let partner: KvPartnerId = serde_json::from_value(record.clone()) + .expect("should deserialize partner lifecycle metadata"); + + assert_eq!( + serde_json::to_value(partner).expect("should serialize partner lifecycle metadata"), + record, + "should preserve lifecycle metadata instead of silently dropping it" + ); + } + #[test] fn entry_serialization_roundtrip() { let geo = sample_geo_info(); @@ -494,6 +676,7 @@ mod tests { "liveramp".to_owned(), KvPartnerId { uid: "LR_xyz".to_owned(), + ..Default::default() }, ); @@ -739,6 +922,7 @@ mod tests { "ssp_x".to_owned(), KvPartnerId { uid: "x".repeat(MAX_UID_LENGTH + 1), + ..Default::default() }, ); diff --git a/crates/trusted-server-core/src/ec/mod.rs b/crates/trusted-server-core/src/ec/mod.rs index 31a57fece..44b827fd3 100644 --- a/crates/trusted-server-core/src/ec/mod.rs +++ b/crates/trusted-server-core/src/ec/mod.rs @@ -40,6 +40,7 @@ pub mod eids; pub mod finalize; pub mod generation; pub mod identify; +pub mod identity; pub mod kv; pub mod kv_backend; pub mod kv_types; @@ -78,7 +79,6 @@ use crate::cookies::handle_request_cookies; use crate::ec::cookies::ec_id_has_only_allowed_chars; use crate::error::TrustedServerError; use crate::geo::GeoInfo; -use crate::openrtb::Eid; use crate::platform::RuntimeServices; use crate::settings::Settings; use device::DeviceSignals; @@ -87,7 +87,7 @@ use self::kv::{CreateIfAbsentOutcome, KvIdentityGraph}; use self::kv_types::KvEntry; use self::pull_sync_marker::{PullSyncMarkerState, validate_marker_state}; -/// Bounded request classifications that may persist browser EID cookies. +/// Bounded request classifications that may persist browser EIDs. /// /// Adapters classify publisher navigations and `POST /auction` only after /// pre-route filters allow dispatch. The shared page-bids handler classifies an @@ -272,16 +272,12 @@ pub struct EcContext { kv_snapshot: EcKvSnapshot, /// Whether this request may rotate an orphaned EC identity. recovery_eligible: bool, - /// EIDs the route resolved from the current request: the `/auction` JSON - /// body, or the `ts-eids` cookie when the body carried none. Carried to - /// response finalization so KV ingestion can use the full set the client - /// sent instead of being limited to whatever fits in the size-capped - /// `ts-eids` cookie. - client_eids: Option>, /// Browser-carried proof of recent pull-partner completeness. pull_sync_marker: PullSyncMarkerState, /// Allowed returning-user EID persistence source, assigned only after request filters pass. eid_sync_source: Option, + /// Normalized browser body updates retained even when auction dispatch fails. + browser_eid_updates: Vec, } impl EcContext { @@ -361,9 +357,9 @@ impl EcContext { device_signals: None, kv_snapshot: EcKvSnapshot::NotRead, recovery_eligible: false, - client_eids: None, pull_sync_marker: PullSyncMarkerState::from_cookie(parsed.pull_sync_marker), eid_sync_source: None, + browser_eid_updates: Vec::new(), }) } @@ -547,11 +543,21 @@ impl EcContext { self.recovery_eligible = eligible; } - /// Allows returning-user EID cookie persistence for this request source. + /// Allows returning-user browser EID persistence for this request source. pub fn set_eid_sync_source(&mut self, source: EidSyncSource) { self.eid_sync_source = Some(source); } + /// Stages registered browser body updates before dispatch. No root is created. + pub(crate) fn stage_browser_eid_updates(&mut self, updates: Vec) { + self.browser_eid_updates = updates; + } + + /// Takes the request's browser updates for one finalization attempt. + pub(crate) fn take_browser_eid_updates(&mut self) -> Vec { + std::mem::take(&mut self.browser_eid_updates) + } + /// Returns the allowed returning-user EID persistence source. #[must_use] pub fn eid_sync_source(&self) -> Option { @@ -564,18 +570,6 @@ impl EcContext { self.recovery_eligible } - /// Records the current request's own EIDs (e.g. an `/auction` body) for - /// use by response finalization's KV ingestion. - pub(crate) fn set_client_eids(&mut self, eids: Vec) { - self.client_eids = Some(eids); - } - - /// Returns the current request's own EIDs, if the route captured any. - #[must_use] - pub(crate) fn client_eids(&self) -> Option<&[Eid]> { - self.client_eids.as_deref() - } - /// Validates a browser completeness marker against the active EC and partner set. pub(crate) fn validate_pull_sync_marker( &mut self, @@ -652,6 +646,7 @@ impl EcContext { #[must_use] pub fn new_for_test(ec_value: Option, consent: ConsentContext) -> Self { Self { + browser_eid_updates: Vec::new(), ec_was_present: ec_value.is_some(), cookie_ec_value: ec_value.clone(), ec_value, @@ -662,7 +657,6 @@ impl EcContext { device_signals: None, kv_snapshot: EcKvSnapshot::NotRead, recovery_eligible: false, - client_eids: None, pull_sync_marker: PullSyncMarkerState::Absent, eid_sync_source: None, } @@ -677,6 +671,7 @@ impl EcContext { client_ip: Option, ) -> Self { Self { + browser_eid_updates: Vec::new(), ec_was_present: ec_value.is_some(), cookie_ec_value: ec_value.clone(), ec_value, @@ -687,7 +682,6 @@ impl EcContext { device_signals: None, kv_snapshot: EcKvSnapshot::NotRead, recovery_eligible: false, - client_eids: None, pull_sync_marker: PullSyncMarkerState::Absent, eid_sync_source: None, } @@ -705,6 +699,7 @@ impl EcContext { consent: ConsentContext, ) -> Self { Self { + browser_eid_updates: Vec::new(), ec_value, cookie_ec_value, ec_was_present, @@ -715,7 +710,6 @@ impl EcContext { device_signals: None, kv_snapshot: EcKvSnapshot::NotRead, recovery_eligible: false, - client_eids: None, pull_sync_marker: PullSyncMarkerState::Absent, eid_sync_source: None, } diff --git a/crates/trusted-server-core/src/ec/prebid_eids.rs b/crates/trusted-server-core/src/ec/prebid_eids.rs index d823cb32f..8fe71c882 100644 --- a/crates/trusted-server-core/src/ec/prebid_eids.rs +++ b/crates/trusted-server-core/src/ec/prebid_eids.rs @@ -1,16 +1,7 @@ -//! Prebid EID cookie ingestion. +//! Browser EID normalization for existing-root enrichment. //! -//! Parses the `ts-eids` cookie written by the TSJS Prebid integration and -//! syncs matched partner UIDs to the KV identity graph. -//! -//! The current cookie format stores a base64-encoded JSON array of full -//! OpenRTB-style `Eid` objects (`{source, uids:[...]}`). For rollout -//! compatibility we also accept the earlier flattened payload shape -//! (`{source, id, atype}` per entry). - -use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64}; -use serde::Deserialize; -use serde_json::Value as JsonValue; +//! Auction bodies supply current-request EIDs. Only registered sources produce +//! UID-only storage updates; provenance and lifecycle metadata are never stored. use crate::openrtb::{Eid, Uid}; @@ -18,140 +9,29 @@ use super::kv::PartnerIdUpdate; use super::kv_types::MAX_UID_LENGTH; use super::registry::PartnerRegistry; -/// Maximum raw `ts-eids` cookie size accepted before base64 decode. -const MAX_EIDS_COOKIE_BYTES: usize = 8 * 1024; - -/// Legacy flattened `ts-eids` cookie entry. -#[derive(Debug, Deserialize)] -struct LegacyCookieEid { - source: String, - id: String, - atype: i32, -} - -/// OpenRTB-style `ts-eids` cookie entry. -#[derive(Debug, Deserialize)] -struct StructuredCookieEid { - source: String, - #[serde(default)] - uids: Vec, -} - -#[derive(Debug, Deserialize)] -struct StructuredCookieUid { - id: String, - #[serde(default)] - atype: Option, - #[serde(default)] - ext: Option, -} - -enum DecodedCookieEids { - Legacy(Vec), - Structured(Vec), -} - -pub(crate) struct DiagnosticEidSource { - pub(crate) source: String, - pub(crate) uids: Vec, -} - -pub(crate) struct PrebidEidAnalysis { - pub(crate) eids: Vec, - pub(crate) diagnostic_sources: Vec, - pub(crate) updates: Vec, -} - -/// Parses a `ts-eids` cookie value into OpenRTB-style `Eid` entries. -/// -/// Accepts both the current structured cookie format and the earlier legacy -/// flattened format for backwards compatibility. -/// -/// # Errors -/// -/// Returns an error when the cookie exceeds the raw size limit, is not valid -/// base64, or does not contain either supported JSON payload shape. -pub fn parse_prebid_eids_cookie(cookie_value: &str) -> Result, String> { - decode_prebid_eids_cookie(cookie_value).map(DecodedCookieEids::into_openrtb) -} - -fn decode_prebid_eids_cookie(cookie_value: &str) -> Result { - if eids_cookie_exceeds_size_limit(cookie_value) { - return Err(format!( - "ts-eids cookie too large ({} bytes)", - cookie_value.len() - )); - } - - let bytes = BASE64 - .decode(cookie_value) - .map_err(|e| format!("base64 decode failed: {e}"))?; - - if let Ok(eids) = serde_json::from_slice::>(&bytes) { - return Ok(DecodedCookieEids::Legacy(eids)); - } - - let structured = serde_json::from_slice::>(&bytes) - .map_err(|e| format!("JSON parse failed: {e}"))?; - Ok(DecodedCookieEids::Structured(structured)) -} - -impl DecodedCookieEids { - fn diagnostic_sources(&self) -> Vec { - match self { - Self::Legacy(entries) => entries - .iter() - .filter(|entry| !entry.source.is_empty()) - .map(|entry| DiagnosticEidSource { - source: entry.source.clone(), - uids: vec![entry.id.clone()], - }) - .collect(), - Self::Structured(entries) => entries - .iter() - .filter(|entry| !entry.source.is_empty()) - .map(|entry| DiagnosticEidSource { - source: entry.source.clone(), - uids: entry.uids.iter().map(|uid| uid.id.clone()).collect(), - }) - .collect(), - } - } - - fn into_openrtb(self) -> Vec { - match self { - Self::Legacy(entries) => legacy_cookie_eids_to_openrtb(entries), - Self::Structured(entries) => structured_cookie_eids_to_openrtb(entries), - } - } +/// Collects one valid UID per registered source without KV I/O. +pub(crate) fn collect_prebid_eid_updates_from_eids( + eids: &[Eid], + registry: &PartnerRegistry, +) -> Vec { + let updates = eids + .iter() + .filter_map(|eid| { + let partner = registry.find_by_source_domain(&eid.source)?; + let uid = first_valid_uid(&eid.uids)?; + Some(PartnerIdUpdate::new(&partner.source_domain, &uid.id)) + }) + .collect(); + dedupe_partner_updates(updates) } -/// Collects validated request-local partner updates without performing KV I/O. -/// -/// `body_eids` is the current request's own EIDs (e.g. an `/auction` JSON -/// body), when the route captured any. It is applied after the `ts-eids` -/// cookie so it wins on conflicts: the cookie is size-capped by the browser -/// (see `MAX_EID_COOKIE_BYTES` in the TSJS Prebid integration) and may be -/// missing partners the request body still carries in full. The `sharedId` -/// cookie is applied last, unaffected by `body_eids`, preserving its existing -/// override behavior. -pub(crate) fn collect_eid_updates( - eids_cookie: Option<&str>, +/// Combines staged body updates with the separate sharedId cookie once. +/// The direct sharedId cookie retains its legacy precedence over body input. +pub(crate) fn collect_browser_eid_updates( + mut updates: Vec, sharedid_cookie: Option<&str>, - body_eids: Option<&[Eid]>, registry: &PartnerRegistry, ) -> Vec { - if registry.is_empty() { - return Vec::new(); - } - - let mut updates = Vec::new(); - if let Some(cookie) = eids_cookie { - updates.extend(collect_prebid_eid_updates(cookie, registry)); - } - if let Some(eids) = body_eids { - updates.extend(collect_prebid_eid_updates_from_eids(eids, registry)); - } if let Some(cookie) = sharedid_cookie && let Some(update) = collect_sharedid_update(cookie, registry) { @@ -160,69 +40,11 @@ pub(crate) fn collect_eid_updates( dedupe_partner_updates(updates) } -pub(crate) fn collect_prebid_eid_updates( - cookie_value: &str, - registry: &PartnerRegistry, -) -> Vec { - let Ok(analysis) = analyze_prebid_eids_cookie(cookie_value, registry) else { - log::trace!("Prebid EIDs: failed to decode ts-eids cookie; dropping"); - return Vec::new(); - }; - - analysis.updates -} - -pub(crate) fn analyze_prebid_eids_cookie( - cookie_value: &str, - registry: &PartnerRegistry, -) -> Result { - let decoded = decode_prebid_eids_cookie(cookie_value)?; - let diagnostic_sources = decoded.diagnostic_sources(); - let eids = decoded.into_openrtb(); - let updates = collect_prebid_eid_updates_from_eids(&eids, registry); - - Ok(PrebidEidAnalysis { - eids, - diagnostic_sources, - updates, - }) -} - -fn collect_prebid_eid_updates_from_eids( - eids: &[Eid], - registry: &PartnerRegistry, -) -> Vec { - let mut updates = Vec::new(); - for eid in eids { - let Some(partner) = registry.find_by_source_domain(&eid.source) else { - log::debug!("Prebid EIDs: no partner for source '{}'", eid.source); - continue; - }; - - // KV stores one UID per partner. Preserve the previous cookie-ingestion - // behavior by syncing the first valid UID under each source, while - // skipping malformed candidates instead of dropping the whole source. - let Some(uid) = first_valid_uid(&eid.uids) else { - log::debug!( - "Prebid EIDs: no valid uid for source_domain '{}' from source '{}'", - partner.source_domain, - eid.source, - ); - continue; - }; - - updates.push(PartnerIdUpdate::new(&partner.source_domain, &uid.id)); - } - - updates -} - pub(crate) fn dedupe_partner_updates(updates: Vec) -> Vec { let mut latest = std::collections::BTreeMap::new(); for update in updates { latest.insert(update.partner_id, update.uid); } - latest .into_iter() .map(|(partner_id, uid)| PartnerIdUpdate::new(partner_id, uid)) @@ -234,127 +56,35 @@ fn first_valid_uid(uids: &[Uid]) -> Option<&Uid> { } pub(crate) fn is_valid_eid_uid(uid: &str) -> bool { - !uid.trim().is_empty() && !eid_id_exceeds_size_limit(uid) + !uid.trim().is_empty() && uid.len() <= MAX_UID_LENGTH } -/// `SharedID` EID source domain used for partner registry lookup. -const SHAREDID_SOURCE_DOMAIN: &str = "sharedid.org"; - pub(crate) fn collect_sharedid_update( cookie_value: &str, registry: &PartnerRegistry, ) -> Option { let cookie_value = cookie_value.trim(); - if cookie_value.is_empty() { - return None; - } - - if sharedid_cookie_exceeds_size_limit(cookie_value) { - log::debug!( - "SharedID: cookie exceeds MAX_UID_LENGTH ({} bytes)", - cookie_value.len() - ); + if !is_valid_eid_uid(cookie_value) { return None; } - - let Some(partner) = registry.find_by_source_domain(SHAREDID_SOURCE_DOMAIN) else { - log::debug!("SharedID: no partner configured for source '{SHAREDID_SOURCE_DOMAIN}'"); - return None; - }; - + let partner = registry.find_by_source_domain("sharedid.org")?; Some(PartnerIdUpdate::new(&partner.source_domain, cookie_value)) } -fn eids_cookie_exceeds_size_limit(cookie_value: &str) -> bool { - cookie_value.len() > MAX_EIDS_COOKIE_BYTES -} - -fn eid_id_exceeds_size_limit(uid: &str) -> bool { - uid.len() > MAX_UID_LENGTH -} - -fn sharedid_cookie_exceeds_size_limit(cookie_value: &str) -> bool { - cookie_value.len() > MAX_UID_LENGTH -} - -fn structured_cookie_eids_to_openrtb(entries: Vec) -> Vec { - let mut eids = Vec::new(); - - for entry in entries { - if entry.source.is_empty() { - continue; - } - - let uids: Vec<_> = entry - .uids - .into_iter() - .filter_map(structured_cookie_uid_to_openrtb) - .collect(); - if uids.is_empty() { - continue; - } - - eids.push(Eid { - source: entry.source, - uids, - }); - } - - eids -} - -fn structured_cookie_uid_to_openrtb(uid: StructuredCookieUid) -> Option { - if uid.id.is_empty() { - return None; - } - - let atype = uid.atype.filter(|atype| *atype >= 0); - let ext = match uid.ext { - Some(JsonValue::Object(_)) => uid.ext, - _ => None, - }; - - Some(Uid { - id: uid.id, - atype, - ext, - }) -} - -fn legacy_cookie_eids_to_openrtb(entries: Vec) -> Vec { - entries - .into_iter() - .filter(|entry| !entry.source.is_empty() && !entry.id.is_empty()) - .map(|entry| Eid { - source: entry.source, - uids: vec![Uid { - id: entry.id, - atype: (entry.atype >= 0).then_some(entry.atype), - ext: None, - }], - }) - .collect() -} - #[cfg(test)] mod tests { use super::*; - use base64::engine::general_purpose::STANDARD as BASE64; - use serde_json::json; - - use crate::ec::registry::PartnerRegistry; - use crate::redacted::Redacted; use crate::settings::EcPartner; + use serde_json::json; - fn make_test_partner(_id: &str, source_domain: &str) -> EcPartner { - EcPartner { - name: format!("Partner {source_domain}"), - source_domain: source_domain.to_owned(), + fn registry() -> PartnerRegistry { + let partners = ["id5-sync.com", "sharedid.org"].map(|source| EcPartner { + name: source.to_owned(), + source_domain: source.to_owned(), openrtb_atype: EcPartner::default_openrtb_atype(), bidstream_enabled: true, - api_token: Some(Redacted::new(format!( - "token-{source_domain}-32-bytes-minimum-value" - ))), + identity_owner: None, + api_token: None, batch_rate_limit: EcPartner::default_batch_rate_limit(), pull_sync_enabled: false, pull_sync_url: None, @@ -362,420 +92,45 @@ mod tests { pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), ts_pull_token: None, - } - } - - fn make_registry(partners: Vec<(&str, &str)>) -> PartnerRegistry { - let partners: Vec<_> = partners - .into_iter() - .map(|(id, source_domain)| make_test_partner(id, source_domain)) - .collect(); - PartnerRegistry::from_config(&partners).expect("should build partner registry") - } - - fn encode_json(value: &serde_json::Value) -> String { - BASE64.encode(serde_json::to_vec(value).expect("should serialize EID JSON")) - } - - #[test] - fn parse_prebid_eids_cookie_parses_legacy_flat_payload() { - let eids = vec![ - json!({"source": "id5-sync.com", "id": "ID5_abc", "atype": 1}), - json!({"source": "liveramp.com", "id": "LR_xyz", "atype": 3}), - json!({"source": "google.com", "id": "pair-id", "atype": 571187}), - ]; - let encoded = BASE64.encode(serde_json::to_vec(&eids).expect("should serialize")); - - let decoded = parse_prebid_eids_cookie(&encoded).expect("should decode valid payload"); - assert_eq!(decoded.len(), 3, "should parse all EIDs"); - assert_eq!(decoded[0].source, "id5-sync.com"); - assert_eq!(decoded[0].uids[0].id, "ID5_abc"); - assert_eq!(decoded[1].source, "liveramp.com"); - assert_eq!(decoded[1].uids[0].id, "LR_xyz"); - assert_eq!( - decoded[2].source, "google.com", - "should preserve PAIR source" - ); - assert_eq!( - decoded[2].uids[0].atype, - Some(571187), - "should preserve PAIR vendor-specific atype" - ); - } - - #[test] - fn parse_prebid_eids_cookie_parses_structured_payload() { - let eids = vec![json!({ - "source": "sharedid.org", - "uids": [ - {"id": "shared_123", "atype": 3}, - {"id": "shared_456", "ext": {"provider": "example"}}, - {"id": "shared_invalid", "atype": -1} - ] - })]; - let encoded = BASE64.encode(serde_json::to_vec(&eids).expect("should serialize")); - - let decoded = parse_prebid_eids_cookie(&encoded).expect("should decode valid payload"); - assert_eq!(decoded.len(), 1, "should parse one structured EID entry"); - assert_eq!(decoded[0].source, "sharedid.org"); - assert_eq!(decoded[0].uids.len(), 3, "should preserve multiple UIDs"); - assert_eq!(decoded[0].uids[0].id, "shared_123"); - assert_eq!(decoded[0].uids[0].atype, Some(3)); - assert_eq!( - decoded[0].uids[1].ext, - Some(json!({"provider": "example"})), - "should preserve UID ext objects" - ); - assert_eq!( - decoded[0].uids[2].atype, None, - "should drop negative atype values" - ); - } - - #[test] - fn parse_prebid_eids_cookie_preserves_pair_atype() { - let encoded = encode_json(&json!([ - { - "source": "google.com", - "uids": [{ "id": "pair-id", "atype": 571187 }] - } - ])); - - let decoded = parse_prebid_eids_cookie(&encoded).expect("should decode PAIR EID"); - - assert_eq!( - decoded[0].uids[0].atype, - Some(571187), - "should preserve PAIR's vendor-specific atype" - ); - } - - #[test] - fn parse_prebid_eids_cookie_rejects_invalid_base64() { - let result = parse_prebid_eids_cookie("not-valid-base64!!!"); - assert!(result.is_err(), "should reject invalid base64"); - } - - #[test] - fn parse_prebid_eids_cookie_rejects_invalid_json() { - let encoded = BASE64.encode(b"not json"); - let result = parse_prebid_eids_cookie(&encoded); - assert!(result.is_err(), "should reject invalid JSON"); - } - - #[test] - fn ts_eids_cookie_rejects_oversized_payloads() { - let oversized = "x".repeat(MAX_EIDS_COOKIE_BYTES + 1); - let exact_limit = "x".repeat(MAX_EIDS_COOKIE_BYTES); - - assert!( - eids_cookie_exceeds_size_limit(&oversized), - "should reject cookies larger than the raw size cap" - ); - assert!( - !eids_cookie_exceeds_size_limit(&exact_limit), - "should allow cookies exactly at the raw size cap" - ); - } - - #[test] - fn sharedid_cookie_rejects_values_larger_than_uid_limit() { - let oversized = "x".repeat(MAX_UID_LENGTH + 1); - let exact_limit = "x".repeat(MAX_UID_LENGTH); - - assert!( - sharedid_cookie_exceeds_size_limit(&oversized), - "should reject sharedId values larger than MAX_UID_LENGTH" - ); - assert!( - !sharedid_cookie_exceeds_size_limit(&exact_limit), - "should allow sharedId values exactly at MAX_UID_LENGTH" - ); - } - - #[test] - fn prebid_eid_uid_rejects_values_larger_than_uid_limit() { - let oversized = "x".repeat(MAX_UID_LENGTH + 1); - let exact_limit = "x".repeat(MAX_UID_LENGTH); - - assert!( - eid_id_exceeds_size_limit(&oversized), - "should reject EID values larger than MAX_UID_LENGTH" - ); - assert!( - !eid_id_exceeds_size_limit(&exact_limit), - "should allow EID values exactly at MAX_UID_LENGTH" - ); - } - - #[test] - fn first_valid_uid_skips_oversized_and_uses_later_valid_uid() { - let oversized = "x".repeat(MAX_UID_LENGTH + 1); - let uids = vec![ - Uid { - id: oversized, - atype: Some(1), - ext: None, - }, - Uid { - id: "valid-uid".to_owned(), - atype: Some(1), - ext: None, - }, - ]; - - let uid = first_valid_uid(&uids).expect("should find later valid UID"); - assert_eq!(uid.id, "valid-uid", "should skip oversized first UID"); - } - - #[test] - fn first_valid_uid_rejects_whitespace_only_ids() { - let uids = vec![Uid { - id: " ".to_owned(), - atype: Some(1), - ext: None, - }]; - - assert!( - first_valid_uid(&uids).is_none(), - "should reject whitespace UID" - ); - } - - #[test] - fn collect_prebid_eid_updates_collects_multiple_partner_matches() { - let registry = make_registry(vec![("id5", "id5-sync.com"), ("liveramp", "liveramp.com")]); - let cookie = encode_json(&json!([ - {"source": "id5-sync.com", "uids": [{"id": "ID5_abc", "atype": 1}]}, - {"source": "liveramp.com", "uids": [{"id": "LR_xyz", "atype": 3}]} - ])); - - let updates = collect_prebid_eid_updates(&cookie, ®istry); - - assert_eq!(updates.len(), 2, "should collect both partner matches"); - assert_eq!(updates[0], PartnerIdUpdate::new("id5-sync.com", "ID5_abc")); - assert_eq!(updates[1], PartnerIdUpdate::new("liveramp.com", "LR_xyz")); - } - - #[test] - fn collect_prebid_eid_updates_skips_unknown_sources() { - let registry = make_registry(vec![("id5", "id5-sync.com")]); - let cookie = encode_json(&json!([ - {"source": "unknown.example", "uids": [{"id": "unknown", "atype": 1}]} - ])); - - let updates = collect_prebid_eid_updates(&cookie, ®istry); - - assert!( - updates.is_empty(), - "should skip EIDs without configured source-domain partners" - ); - } - - #[test] - fn collect_prebid_eid_updates_uses_later_valid_uid_candidate() { - let registry = make_registry(vec![("id5", "id5-sync.com")]); - let oversized = "x".repeat(MAX_UID_LENGTH + 1); - let cookie = encode_json(&json!([ - { - "source": "id5-sync.com", - "uids": [ - {"id": oversized, "atype": 1}, - {"id": "ID5_valid", "atype": 1} - ] - } - ])); - - let updates = collect_prebid_eid_updates(&cookie, ®istry); - - assert_eq!( - updates, - vec![PartnerIdUpdate::new("id5-sync.com", "ID5_valid")] - ); - } - - #[test] - fn collect_sharedid_update_maps_configured_sharedid_partner() { - let registry = make_registry(vec![("sharedid", "sharedid.org")]); - - let update = collect_sharedid_update(" shared-cookie-id ", ®istry) - .expect("should collect sharedId update"); - - assert_eq!( - update, - PartnerIdUpdate::new("sharedid.org", "shared-cookie-id") - ); - } - - #[test] - fn collect_eid_updates_merges_prebid_and_sharedid_without_kv() { - let registry = make_registry(vec![("id5", "id5-sync.com"), ("sharedid", "sharedid.org")]); - let eids_cookie = encode_json(&json!([ - {"source": "id5-sync.com", "uids": [{"id": "ID5_abc", "atype": 1}]} - ])); - - let updates = collect_eid_updates(Some(&eids_cookie), Some(" shared-1 "), None, ®istry); - - assert_eq!( - updates.len(), - 2, - "should collect prebid and sharedId matches" - ); - assert!(updates.contains(&PartnerIdUpdate::new("id5-sync.com", "ID5_abc"))); - assert!(updates.contains(&PartnerIdUpdate::new("sharedid.org", "shared-1"))); + }); + PartnerRegistry::from_config(&partners).expect("should build registry") } #[test] - fn collect_eid_updates_body_eids_ingest_partners_the_cookie_trimmed() { - // Simulates a `ts-eids` cookie that the browser trimmed to fit - // `MAX_EID_COOKIE_BYTES`, dropping the `liveramp.com` source, while - // the `/auction` request body still carried it (and the newer, - // updated `id5-sync.com` uid) in full. - let registry = make_registry(vec![("id5", "id5-sync.com"), ("liveramp", "liveramp.com")]); - let eids_cookie = encode_json(&json!([ - {"source": "id5-sync.com", "uids": [{"id": "ID5_stale", "atype": 1}]} - ])); - let body_eids = vec![ - Eid { - source: "id5-sync.com".to_owned(), - uids: vec![Uid { - id: "ID5_fresh".to_owned(), - atype: Some(1), - ext: None, - }], - }, - Eid { - source: "liveramp.com".to_owned(), - uids: vec![Uid { - id: "LR_xyz".to_owned(), - atype: Some(3), - ext: None, - }], - }, - ]; - - let updates = collect_eid_updates(Some(&eids_cookie), None, Some(&body_eids), ®istry); - + fn body_updates_are_registered_uid_only_and_skip_invalid_candidates() { + let eids: Vec = serde_json::from_value(json!([ + {"source":"ID5-SYNC.COM", "inserter":"forged-owner", "matcher":"", "mm":0, + "uids":[{"id":" ", "atype":1}, {"id":"opaque-envelope", "ext":{"writer":"forged"}}]}, + {"source":"unknown.example", "uids":[{"id":"current-request-only"}]} + ])) + .expect("should parse EIDs"); assert_eq!( - updates, - vec![ - PartnerIdUpdate::new("id5-sync.com", "ID5_fresh"), - PartnerIdUpdate::new("liveramp.com", "LR_xyz"), - ], - "body EIDs should ingest every configured partner present in the \ - request, including the one the cookie trimmed, and should win \ - over a stale cookie value for a partner both carry" - ); - } - - #[test] - fn collect_eid_updates_empty_registry_returns_no_updates() { - let registry = PartnerRegistry::empty(); - let eids_cookie = encode_json(&json!([ - {"source": "id5-sync.com", "uids": [{"id": "ID5_abc", "atype": 1}]} - ])); - - let updates = collect_eid_updates(Some(&eids_cookie), Some("shared-1"), None, ®istry); - - assert!( - updates.is_empty(), - "an empty registry matches no partners and touches no KV" + collect_prebid_eid_updates_from_eids(&eids, ®istry()), + vec![PartnerIdUpdate::new("id5-sync.com", "opaque-envelope")] ); } #[test] - fn dedupe_partner_updates_uses_last_partner_value() { + fn sharedid_combines_with_body_and_retains_direct_cookie_precedence() { let updates = vec![ - PartnerIdUpdate::new("sharedid.org", "prebid-shared"), - PartnerIdUpdate::new("id5-sync.com", "id5-uid"), - PartnerIdUpdate::new("sharedid.org", "cookie-shared"), + PartnerIdUpdate::new("sharedid.org", "body-shared"), + PartnerIdUpdate::new("id5-sync.com", "id5"), ]; - - let deduped = dedupe_partner_updates(updates); - - assert_eq!(deduped.len(), 2, "should keep one update per partner"); - assert_eq!( - deduped, - vec![ - PartnerIdUpdate::new("id5-sync.com", "id5-uid"), - PartnerIdUpdate::new("sharedid.org", "cookie-shared"), - ], - "should keep the last value for duplicate partners" - ); - } - - #[test] - fn collect_eid_updates_combines_cookie_sources() { - let registry = make_registry(vec![ - ("id5", "id5-sync.com"), - ("liveramp", "liveramp.com"), - ("sharedid", "sharedid.org"), - ]); - let cookie = encode_json(&json!([ - {"source": "id5-sync.com", "uids": [{"id": "ID5_abc", "atype": 1}]}, - {"source": "liveramp.com", "uids": [{"id": "LR_xyz", "atype": 3}]} - ])); - - let updates = collect_eid_updates(Some(&cookie), Some("shared-cookie-id"), None, ®istry); - assert_eq!( - updates, + collect_browser_eid_updates(updates, Some(" cookie-shared "), ®istry()), vec![ - PartnerIdUpdate::new("id5-sync.com", "ID5_abc"), - PartnerIdUpdate::new("liveramp.com", "LR_xyz"), - PartnerIdUpdate::new("sharedid.org", "shared-cookie-id"), - ], - "should collect every matched update in one batch" - ); - } - - #[test] - fn collect_liveramp_eid_cookie_updates_preserves_the_opaque_envelope() { - let registry = make_registry(vec![("liveramp", "liveramp.com")]); - let cookie = encode_json(&json!([ - { - "source": "liveramp.com", - "uids": [{"id": "opaque-test-envelope", "atype": 3}] - } - ])); - - let updates = collect_eid_updates(Some(&cookie), None, None, ®istry); - - assert_eq!( - updates, - vec![PartnerIdUpdate::new("liveramp.com", "opaque-test-envelope")], - "should preserve the opaque envelope without decoding it" - ); - } - - #[test] - fn collect_eid_updates_prefers_direct_sharedid_cookie() { - let registry = make_registry(vec![("sharedid", "sharedid.org")]); - let cookie = encode_json(&json!([ - {"source": "sharedid.org", "uids": [{"id": "prebid-shared", "atype": 3}]} - ])); - - let updates = collect_eid_updates(Some(&cookie), Some("cookie-shared"), None, ®istry); - - assert_eq!( - updates, - vec![PartnerIdUpdate::new("sharedid.org", "cookie-shared")], - "should apply the direct sharedId cookie after Prebid EIDs" + PartnerIdUpdate::new("id5-sync.com", "id5"), + PartnerIdUpdate::new("sharedid.org", "cookie-shared") + ] ); } #[test] - fn collect_eid_updates_ignores_unknown_sources() { - let registry = make_registry(vec![("id5", "id5-sync.com")]); - let cookie = encode_json(&json!([ - {"source": "unknown.example", "uids": [{"id": "unknown", "atype": 1}]} - ])); - - let updates = collect_eid_updates(Some(&cookie), None, None, ®istry); - - assert!( - updates.is_empty(), - "should ignore unmatched partner updates" - ); + fn uid_limits_are_byte_bounded() { + assert!(is_valid_eid_uid(&"x".repeat(MAX_UID_LENGTH))); + assert!(!is_valid_eid_uid(&"x".repeat(MAX_UID_LENGTH + 1))); + assert!(!is_valid_eid_uid(" \t")); + assert!(collect_sharedid_update(&"x".repeat(MAX_UID_LENGTH + 1), ®istry()).is_none()); + assert!(collect_sharedid_update("valid", &PartnerRegistry::empty()).is_none()); } } diff --git a/crates/trusted-server-core/src/ec/pull_sync.rs b/crates/trusted-server-core/src/ec/pull_sync.rs index 191e6c6d2..1e700a99a 100644 --- a/crates/trusted-server-core/src/ec/pull_sync.rs +++ b/crates/trusted-server-core/src/ec/pull_sync.rs @@ -310,9 +310,10 @@ pub(crate) fn entry_is_pull_complete(entry: &KvEntry, registry: &PartnerRegistry } fn is_partner_pull_eligible(partner: &PartnerConfig, kv_entry: Option<&KvEntry>) -> bool { - kv_entry - .and_then(|entry| entry.ids.get(&partner.source_domain)) - .is_none() + partner.identity_owner.is_none() + && kv_entry + .and_then(|entry| entry.ids.get(&partner.source_domain)) + .is_none() } fn validated_pull_sync_url(partner: &PartnerConfig) -> Option { @@ -559,6 +560,7 @@ mod tests { pull_sync_allowed_domains: vec!["sync.partner.test".to_owned()], pull_sync_ttl_sec: ttl_sec, pull_sync_rate_limit: 20, + identity_owner: None, ts_pull_token: Some(Redacted::new("token".to_owned())), } } @@ -654,6 +656,7 @@ mod tests { "ssp.example.com".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "uid".to_owned(), + ..Default::default() }, ); } @@ -686,6 +689,26 @@ mod tests { ); } + #[test] + fn expired_identity_does_not_trigger_legacy_pull_refresh() { + let registry = PartnerRegistry::from_config(&[pull_enabled_ec_partner("ssp.example.com")]) + .expect("should build registry"); + let mut entry = KvEntry::minimal("ssp.example.com", "expired-uid", 1_000); + entry + .ids + .get_mut("ssp.example.com") + .expect("should contain record") + .expires_at = Some(1); + assert!( + entry_is_pull_complete(&entry, ®istry), + "legacy completeness tracks stored presence, not auction usability" + ); + assert!( + !is_partner_pull_eligible(&pull_partner(3_600), Some(&entry)), + "expiry must not introduce an unapproved generic refresh" + ); + } + #[test] fn completeness_requires_all_pull_partner_ids() { let registry = PartnerRegistry::from_config(&[ @@ -704,6 +727,7 @@ mod tests { "b.example.com".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "uid-b".to_owned(), + ..Default::default() }, ); @@ -928,6 +952,7 @@ mod tests { pull_sync_allowed_domains: vec![source_domain.to_owned()], pull_sync_ttl_sec: 3600, pull_sync_rate_limit: 100, + identity_owner: None, ts_pull_token: Some(Redacted::new("outbound-token".to_owned())), } } @@ -1215,6 +1240,7 @@ mod tests { "alpha.example.com".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "already-known".to_owned(), + ..Default::default() }, ); graph diff --git a/crates/trusted-server-core/src/ec/pull_sync_marker.rs b/crates/trusted-server-core/src/ec/pull_sync_marker.rs index f2a2fe601..17e763c34 100644 --- a/crates/trusted-server-core/src/ec/pull_sync_marker.rs +++ b/crates/trusted-server-core/src/ec/pull_sync_marker.rs @@ -350,6 +350,7 @@ mod tests { pull_sync_allowed_domains: vec![format!("sync.{source_domain}")], pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + identity_owner: None, ts_pull_token: Some(Redacted::new("pull-token".to_owned())), } } @@ -377,6 +378,7 @@ mod tests { (*domain).to_owned(), crate::ec::kv_types::KvPartnerId { uid: format!("uid-{domain}"), + ..Default::default() }, ); } @@ -512,6 +514,7 @@ mod tests { "a.example.com".to_owned(), crate::ec::kv_types::KvPartnerId { uid: "stale".to_owned(), + ..Default::default() }, ); let snapshots = [ @@ -690,4 +693,64 @@ mod tests { ); assert!(!cookie.contains("Domain="), "marker should be host-only"); } + + #[test] + fn ownership_change_updates_effective_pull_fingerprint() { + let (settings, _) = settings_and_registry(&[]); + let unowned = PartnerRegistry::from_config(&[ + pull_partner("owned-source.example.com"), + pull_partner("legacy-source.example.com"), + ]) + .expect("should build unowned pull registry"); + let mut owned_source = pull_partner("owned-source.example.com"); + owned_source.identity_owner = Some("identity_module".to_owned()); + let owned = PartnerRegistry::from_config(&[ + owned_source, + pull_partner("legacy-source.example.com"), + ]) + .expect("should build registry with one owned source"); + + let before = partner_set_fingerprint(&unowned); + let after = partner_set_fingerprint(&owned); + assert_ne!( + before, after, + "moving a source to an owner should change the effective pull fingerprint" + ); + let marker = create_marker(&settings, &unowned, EC_ID, 4_600) + .expect("should create marker for the original effective pull set"); + assert!( + validate_marker(&marker, &settings, &owned, EC_ID, 1_000).is_none(), + "ownership change should invalidate a marker signed for the old pull set" + ); + } + + #[test] + fn owned_missing_source_is_skipped_and_expired_unowned_presence_is_complete() { + let mut owned_source = pull_partner("owned-source.example.com"); + owned_source.identity_owner = Some("identity_module".to_owned()); + let legacy_source = pull_partner("legacy-source.example.com"); + let registry = PartnerRegistry::from_config(&[owned_source, legacy_source]) + .expect("should build mixed owned and legacy pull registry"); + + let mut entry = KvEntry::tombstone(1_000); + entry.consent.ok = true; + entry.ids.insert( + "legacy-source.example.com".to_owned(), + crate::ec::kv_types::KvPartnerId { + uid: "expired-legacy-uid".to_owned(), + expires_at: Some(1), + ..Default::default() + }, + ); + assert!( + entry_is_pull_complete(&entry, ®istry), + "owned absence should not make legacy acquisition incomplete, and expired unowned presence remains fill-missing complete" + ); + + entry.ids.remove("legacy-source.example.com"); + assert!( + !entry_is_pull_complete(&entry, ®istry), + "a missing unowned source should still make legacy acquisition incomplete" + ); + } } diff --git a/crates/trusted-server-core/src/ec/registry.rs b/crates/trusted-server-core/src/ec/registry.rs index c4637b431..01780c623 100644 --- a/crates/trusted-server-core/src/ec/registry.rs +++ b/crates/trusted-server-core/src/ec/registry.rs @@ -28,6 +28,8 @@ pub struct PartnerConfig { pub openrtb_atype: i32, /// Whether this partner's UIDs appear in auction `user.eids`. pub bidstream_enabled: bool, + /// Selected authoritative integration, when managed acquisition owns this source. + pub identity_owner: Option, /// SHA-256 hex of the partner's API token, when inbound API access is enabled. pub api_key_hash: Option, /// Max batch sync API requests per partner per minute. @@ -239,7 +241,7 @@ impl PartnerRegistry { pub fn pull_enabled_partners(&self) -> Vec<&PartnerConfig> { self.by_source_domain .values() - .filter(|p| p.pull_sync_enabled) + .filter(|p| p.pull_sync_enabled && p.identity_owner.is_none()) .collect() } @@ -304,6 +306,7 @@ fn build_partner_config( pull_sync_allowed_domains: partner.pull_sync_allowed_domains.clone(), pull_sync_ttl_sec: partner.pull_sync_ttl_sec, pull_sync_rate_limit: partner.pull_sync_rate_limit, + identity_owner: partner.identity_owner.clone(), ts_pull_token: partner.ts_pull_token.clone(), } } @@ -431,6 +434,7 @@ mod tests { pull_sync_allowed_domains: vec![], pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + identity_owner: None, ts_pull_token: None, } } diff --git a/crates/trusted-server-core/src/integrations/identity.rs b/crates/trusted-server-core/src/integrations/identity.rs new file mode 100644 index 000000000..3b6e7ee30 --- /dev/null +++ b/crates/trusted-server-core/src/integrations/identity.rs @@ -0,0 +1,447 @@ +//! Shared, policy-free response inspection for registered identity capture modules. +//! Original response parts and bytes are never decoded or rewritten for the client. + +use std::io::Read as _; + +use bytes::Bytes; +use edgezero_core::body::Body as EdgeBody; +use flate2::read::MultiGzDecoder; +use futures::{StreamExt as _, stream}; +use http::{Method, Response, StatusCode, header}; + +const CAPTURE_WIRE_LIMIT: usize = 64 * 1024; +const CAPTURE_DECODED_LIMIT: usize = 64 * 1024; + +/// An integration's response capture capability. +/// +/// The registry selects matching upstream routes and supplies only bounded, +/// successfully decoded response bytes to [`Self::observe`]. +pub trait IdentityCapture: Send + Sync { + /// Return the finite EID sources claimed by this integration. + fn sources(&self) -> &'static [&'static str]; + /// Return whether response observation is active, without removing claims. + fn enabled(&self) -> bool; + /// Select an exact method and proxy path for inspection. + fn matches(&self, method: &Method, path: &str) -> bool; + /// Interpret a selected upstream response without changing forwarding. + fn observe( + &self, + input: IdentityCaptureInput<'_>, + ) -> Option; +} + +/// Selected upstream response and already-buffered forwarding request bytes. +pub struct IdentityCaptureInput<'a> { + /// Actual proxy request path used to distinguish token and revoke routes. + pub path: &'a str, + /// Borrowed POST forwarding buffer; inspection never reads it twice. + pub request_body: &'a [u8], + /// Upstream HTTP response status. + pub status: StatusCode, + /// Decoded response body, bounded separately from original wire bytes. + pub decoded_body: &'a [u8], +} + +/// Shared forwarding bytes, attached only to selected capture requests. +#[derive(Clone)] +pub(crate) struct IdentityRequestBody(pub Bytes); + +/// Inspect a selected upstream response without changing what the client sees. +/// +/// A stream is read only while its wire prefix fits the limit. On overflow or +/// upstream read error, the copied prefix precedes the untouched oversized +/// chunk or error and unread tail. An inspected stream can be returned buffered; +/// the response bytes remain identical. +/// `Content-Length` is deliberately not trusted for either bound. +pub(crate) async fn inspect_response( + response: Response, +) -> (Response, Option>) { + let (parts, body) = response.into_parts(); + let mut encodings = parts.headers.get_all(header::CONTENT_ENCODING).iter(); + let encoding = match (encodings.next(), encodings.next()) { + (None, _) => Some(Encoding::Identity), + (Some(value), None) if value.as_bytes().eq_ignore_ascii_case(b"identity") => { + Some(Encoding::Identity) + } + (Some(value), None) if value.as_bytes().eq_ignore_ascii_case(b"gzip") => { + Some(Encoding::Gzip) + } + _ => None, + }; + let Some(encoding) = encoding else { + return (Response::from_parts(parts, body), None); + }; + + match body { + EdgeBody::Once(bytes) => { + let decoded = (bytes.len() <= CAPTURE_WIRE_LIMIT) + .then(|| decode(&bytes, encoding)) + .flatten(); + (Response::from_parts(parts, EdgeBody::Once(bytes)), decoded) + } + EdgeBody::Stream(mut upstream) => { + // Only copy chunks that fit. A single oversized chunk stays in + // its original Bytes allocation and is replayed untouched. + let mut prefix = Vec::new(); + while let Some(item) = upstream.next().await { + match item { + Ok(chunk) => { + if chunk.len() > CAPTURE_WIRE_LIMIT - prefix.len() { + let replay = stream::once(async { Ok(Bytes::from(prefix)) }) + .chain(stream::once(async { Ok(chunk) })) + .chain(upstream); + return ( + Response::from_parts(parts, EdgeBody::from_stream(replay)), + None, + ); + } + prefix.extend_from_slice(&chunk); + } + Err(error) => { + let replay = stream::once(async { Ok(Bytes::from(prefix)) }) + .chain(stream::once(async { Err(error) })) + .chain(upstream); + return ( + Response::from_parts(parts, EdgeBody::from_stream(replay)), + None, + ); + } + } + } + + let decoded = decode(&prefix, encoding); + (Response::from_parts(parts, EdgeBody::from(prefix)), decoded) + } + } +} + +#[derive(Clone, Copy)] +enum Encoding { + Identity, + Gzip, +} + +fn decode(wire: &[u8], encoding: Encoding) -> Option> { + match encoding { + Encoding::Identity => (wire.len() <= CAPTURE_DECODED_LIMIT).then(|| wire.to_vec()), + Encoding::Gzip => { + // MultiGzDecoder also checks concatenated members instead of + // accepting only the first gzip member and ignoring trailing data. + let mut decoder = MultiGzDecoder::new(wire); + let mut decoded = Vec::new(); + let mut scratch = [0_u8; 4096]; + loop { + // One byte past the cap distinguishes an exact-size response + // from one with more decoded data without allocating it. + let remaining = CAPTURE_DECODED_LIMIT + 1 - decoded.len(); + let window = scratch.len().min(remaining); + let count = decoder.read(&mut scratch[..window]).ok()?; + if count == 0 { + return Some(decoded); + } + if count > CAPTURE_DECODED_LIMIT - decoded.len() { + return None; + } + decoded.extend_from_slice(&scratch[..count]); + } + } + } +} + +#[cfg(test)] +mod tests { + use std::io::Write as _; + + use bytes::Bytes; + use flate2::{Compression, write::GzEncoder}; + use futures::{StreamExt as _, stream}; + use http::{Response, StatusCode, header}; + + use super::{ + CAPTURE_DECODED_LIMIT, CAPTURE_WIRE_LIMIT, EdgeBody, IdentityCaptureInput, inspect_response, + }; + + #[test] + fn capture_input_exposes_the_actual_route() { + let input = IdentityCaptureInput { + path: "/publisher/app/v2/identityLockr/revoke-consent", + request_body: b"{}", + status: StatusCode::OK, + decoded_body: b"{}", + }; + assert_eq!(input.path, "/publisher/app/v2/identityLockr/revoke-consent"); + } + + fn response( + body: EdgeBody, + encoding: Option<&str>, + content_length: &str, + ) -> Response { + let mut builder = Response::builder() + .status(StatusCode::CREATED) + .header(header::CONTENT_LENGTH, content_length) + .header("x-provider-header", "original"); + if let Some(encoding) = encoding { + builder = builder.header(header::CONTENT_ENCODING, encoding); + } + builder.body(body).expect("should build test response") + } + + async fn collect(body: EdgeBody) -> Vec { + match body { + EdgeBody::Once(bytes) => bytes.to_vec(), + EdgeBody::Stream(mut chunks) => { + let mut result = Vec::new(); + while let Some(chunk) = chunks.next().await { + result.extend_from_slice(&chunk.expect("should read replayed body chunk")); + } + result + } + } + } + + fn gzip(input: &[u8]) -> Vec { + let mut encoder = GzEncoder::new(Vec::new(), Compression::default()); + encoder.write_all(input).expect("should encode gzip input"); + encoder.finish().expect("should finish gzip encoding") + } + + #[tokio::test] + async fn wire_exact_limit_and_one_more_preserve_original_parts() { + for size in [CAPTURE_WIRE_LIMIT, CAPTURE_WIRE_LIMIT + 1] { + let bytes = vec![b'a'; size]; + let original = response(EdgeBody::from(bytes.clone()), None, "1"); + let (returned, decoded) = inspect_response(original).await; + assert_eq!(returned.status(), StatusCode::CREATED); + assert_eq!(returned.headers()[header::CONTENT_LENGTH], "1"); + assert_eq!(returned.headers()["x-provider-header"], "original"); + assert_eq!( + decoded, + (size == CAPTURE_WIRE_LIMIT).then_some(bytes.clone()) + ); + assert_eq!(collect(returned.into_body()).await, bytes); + } + } + + #[tokio::test] + async fn absent_length_and_duplicate_encoding_are_safe() { + let plain = b"synthetic"; + let response = Response::builder() + .status(StatusCode::ACCEPTED) + .body(EdgeBody::from(plain.to_vec())) + .expect("should build lengthless response"); + let (returned, decoded) = inspect_response(response).await; + assert_eq!(returned.status(), StatusCode::ACCEPTED); + assert!(!returned.headers().contains_key(header::CONTENT_LENGTH)); + assert_eq!(decoded.as_deref(), Some(plain.as_slice())); + assert_eq!(collect(returned.into_body()).await, plain); + + let response = Response::builder() + .header(header::CONTENT_ENCODING, "gzip") + .header(header::CONTENT_ENCODING, "identity") + .body(EdgeBody::from(gzip(plain))) + .expect("should build duplicate-encoding response"); + let (returned, decoded) = inspect_response(response).await; + assert!(decoded.is_none()); + assert_eq!( + returned + .headers() + .get_all(header::CONTENT_ENCODING) + .iter() + .count(), + 2 + ); + assert_eq!(collect(returned.into_body()).await, gzip(plain)); + } + + #[tokio::test] + async fn decoded_exact_limit_and_one_more_with_compressed_expansion() { + for size in [CAPTURE_DECODED_LIMIT, CAPTURE_DECODED_LIMIT + 1] { + let plain = vec![b'a'; size]; + let wire = gzip(&plain); + assert!(wire.len() < CAPTURE_WIRE_LIMIT); + let (returned, decoded) = inspect_response(response( + EdgeBody::from(wire.clone()), + Some("gzip"), + "999999", + )) + .await; + assert_eq!(decoded, (size == CAPTURE_DECODED_LIMIT).then_some(plain)); + assert_eq!(returned.headers()[header::CONTENT_ENCODING], "gzip"); + assert_eq!(returned.headers()[header::CONTENT_LENGTH], "999999"); + assert_eq!(collect(returned.into_body()).await, wire); + } + } + + #[tokio::test] + async fn streaming_gzip_crosses_chunk_boundary_and_retains_headers() { + let plain = b"a valid synthetic token"; + let wire = gzip(plain); + let chunks = vec![ + Bytes::copy_from_slice(&wire[..7]), + Bytes::copy_from_slice(&wire[7..]), + ]; + let body = EdgeBody::stream(stream::iter(chunks)); + let (returned, decoded) = inspect_response(response(body, Some("gzip"), "1")).await; + assert_eq!(decoded.as_deref(), Some(plain.as_slice())); + assert_eq!(returned.headers()[header::CONTENT_LENGTH], "1"); + assert_eq!(collect(returned.into_body()).await, wire); + } + + #[tokio::test] + async fn unsupported_and_broken_encodings_do_not_change_bytes() { + for (encoding, bytes) in [ + ("br", b"not gzip".as_slice()), + ("gzip", b"broken gzip".as_slice()), + ("gzip, br", b"not gzip".as_slice()), + ] { + let (returned, decoded) = inspect_response(response( + EdgeBody::from(bytes.to_vec()), + Some(encoding), + "0", + )) + .await; + assert!(decoded.is_none()); + assert_eq!(returned.headers()[header::CONTENT_ENCODING], encoding); + assert_eq!(returned.headers()[header::CONTENT_LENGTH], "0"); + assert_eq!(collect(returned.into_body()).await, bytes); + } + } + + #[tokio::test] + async fn streaming_oversized_chunk_replays_prefix_and_tail_without_copy() { + let prefix = Bytes::from_static(b"prefix"); + let oversized = Bytes::from(vec![b'X'; CAPTURE_WIRE_LIMIT + 1]); + let pointer = oversized.as_ptr(); + let tail = Bytes::from_static(b"tail"); + let body = EdgeBody::stream(stream::iter(vec![ + prefix.clone(), + oversized.clone(), + tail.clone(), + ])); + let (returned, decoded) = inspect_response(response(body, None, "2")).await; + assert!(decoded.is_none()); + assert_eq!(returned.headers()[header::CONTENT_LENGTH], "2"); + let EdgeBody::Stream(mut replay) = returned.into_body() else { + panic!("expected stream"); + }; + assert_eq!( + replay + .next() + .await + .expect("should replay prefix") + .expect("should read prefix"), + prefix + ); + let replayed_large = replay + .next() + .await + .expect("should replay oversized chunk") + .expect("should read oversized chunk"); + assert_eq!(replayed_large.as_ptr(), pointer); + assert_eq!(replayed_large, oversized); + assert_eq!( + replay + .next() + .await + .expect("should replay tail") + .expect("should read tail"), + tail + ); + assert!(replay.next().await.is_none()); + } + + #[tokio::test] + async fn many_single_byte_chunks_use_one_prefix_and_return_original_bytes() { + let chunks = (0..CAPTURE_WIRE_LIMIT).map(|_| Bytes::from_static(b"x")); + let body = EdgeBody::stream(stream::iter(chunks)); + let (returned, decoded) = inspect_response(response(body, None, "0")).await; + let expected = vec![b'x'; CAPTURE_WIRE_LIMIT]; + assert_eq!(decoded.as_deref(), Some(expected.as_slice())); + assert!(matches!(returned.body(), EdgeBody::Once(_))); + assert_eq!(collect(returned.into_body()).await, expected); + } + + #[tokio::test] + async fn streaming_total_overflow_at_plus_one_replays_every_byte() { + let first = Bytes::from(vec![b'X'; CAPTURE_WIRE_LIMIT]); + let tail = Bytes::from_static(b"Y"); + let body = EdgeBody::stream(stream::iter(vec![first.clone(), tail.clone()])); + let (returned, decoded) = inspect_response(response(body, None, "0")).await; + assert!(decoded.is_none()); + let EdgeBody::Stream(mut replay) = returned.into_body() else { + panic!("expected stream"); + }; + assert_eq!( + replay + .next() + .await + .expect("should replay prefix") + .expect("should read prefix"), + first + ); + assert_eq!( + replay + .next() + .await + .expect("should replay overflowing byte") + .expect("should read overflowing byte"), + tail + ); + assert!(replay.next().await.is_none()); + } + + #[tokio::test] + async fn streaming_read_error_is_replayed_in_order_with_remaining_tail() { + let body = EdgeBody::from_stream(stream::iter(vec![ + Ok(Bytes::from_static(b"before")), + Err(std::io::Error::other("upstream read failed")), + Ok(Bytes::from_static(b"after")), + ])); + let (returned, decoded) = inspect_response(response(body, None, "0")).await; + assert!(decoded.is_none()); + let EdgeBody::Stream(mut replay) = returned.into_body() else { + panic!("expected stream"); + }; + assert_eq!( + replay + .next() + .await + .expect("should replay prefix") + .expect("should read prefix"), + "before" + ); + assert!( + replay + .next() + .await + .expect("should replay upstream error") + .expect_err("should retain upstream error") + .to_string() + .contains("upstream read failed") + ); + assert_eq!( + replay + .next() + .await + .expect("should replay tail") + .expect("should read tail"), + "after" + ); + assert!(replay.next().await.is_none()); + } + + #[tokio::test] + async fn truncated_gzip_and_trailing_garbage_skip_capture() { + let mut truncated = gzip(b"test"); + truncated.pop(); + let mut trailing = gzip(b"test"); + trailing.extend_from_slice(b"garbage"); + for wire in [truncated, trailing] { + let (returned, decoded) = + inspect_response(response(EdgeBody::from(wire.clone()), Some("gzip"), "0")).await; + assert!(decoded.is_none()); + assert_eq!(collect(returned.into_body()).await, wire); + } + } +} diff --git a/crates/trusted-server-core/src/integrations/lockr.rs b/crates/trusted-server-core/src/integrations/lockr.rs index 002182bd1..cbe5e2f80 100644 --- a/crates/trusted-server-core/src/integrations/lockr.rs +++ b/crates/trusted-server-core/src/integrations/lockr.rs @@ -10,13 +10,18 @@ use std::sync::Arc; use async_trait::async_trait; +use bytes::Bytes; use edgezero_core::body::Body as EdgeBody; use error_stack::{Report, ResultExt}; use http::header::{self, HeaderMap, HeaderValue}; use http::{Method, StatusCode}; use serde::Deserialize; +use serde_json::Value; use validator::Validate; +use crate::ec::identity::{IdentityConsentSignals, IdentityObservation, IdentityOutcome}; +use crate::integrations::identity::{IdentityCapture, IdentityCaptureInput, IdentityRequestBody}; + use crate::constants::INTERNAL_HEADERS; use crate::error::TrustedServerError; use crate::integrations::{ @@ -29,6 +34,15 @@ use crate::platform::{PlatformHttpRequest, RuntimeServices}; use crate::settings::{IntegrationConfig, Settings}; const LOCKR_INTEGRATION_ID: &str = "lockr"; +const ID5_SOURCE: &str = "id5-sync.com"; +const LOCKR_API_PREFIX: &str = "/integrations/lockr/api"; +const LOCKR_TOKEN_PATHS: [&str; 4] = [ + "/publisher/app/v2/identityLockr/page-view", + "/publisher/app/v2/identityLockr/generate-tokens", + "/publisher/app/v2/identityLockr/refresh-tokens", + "/publisher/app/v2/identityLockr/sync-no-hem-ids", +]; +const LOCKR_REVOKE_PATH: &str = "/publisher/app/v2/identityLockr/revoke-consent"; /// Configuration for Lockr integration. #[derive(Debug, Deserialize, Validate)] @@ -37,6 +51,14 @@ pub struct LockrConfig { #[serde(default = "default_enabled")] pub enabled: bool, + /// Kill switch for token observation; does not remove the source claim. + #[serde(default = "default_enabled")] + pub capture_identity: bool, + + /// Separately approved global EC withdrawal from successful revoke responses. + #[serde(default)] + pub capture_global_withdrawal: bool, + /// Lockr app ID (from meta tag lockr-signin-app_id) #[validate(length(min = 1))] pub app_id: String, @@ -111,7 +133,7 @@ impl LockrIntegration { services: &RuntimeServices, ) -> Result, Report> { let sdk_url = &self.config.sdk_url; - log::info!("Fetching Lockr SDK from {}", sdk_url); + log::info!("Fetching Lockr SDK"); // TODO: Check KV store cache first (future enhancement) @@ -121,19 +143,16 @@ impl LockrIntegration { .header(header::USER_AGENT, "TrustedServer/1.0") .header(header::ACCEPT, "application/javascript, */*") .body(EdgeBody::empty()) - .change_context(Self::error("Failed to build Lockr SDK request"))?; + .map_err(|_| Report::new(Self::error("Failed to build Lockr SDK request")))?; let backend_name = Self::backend_name_for_url(services, sdk_url) - .change_context(Self::error("Failed to determine backend for SDK fetch"))?; + .map_err(|_| Report::new(Self::error("Failed to determine backend for SDK fetch")))?; let lockr_response = services .http_client() .send(PlatformHttpRequest::new(lockr_req, backend_name)) .await - .change_context(Self::error(format!( - "Failed to fetch Lockr SDK from {}", - sdk_url - )))? + .map_err(|_| Report::new(Self::error("Failed to fetch Lockr SDK")))? .response; if !lockr_response.status().is_success() { @@ -170,7 +189,6 @@ impl LockrIntegration { ) .header("X-Lockr-SDK-Proxy", "true") .header("X-Lockr-SDK-Mode", "trust-server") - .header("X-SDK-Source", sdk_url) .body(EdgeBody::from(sdk_body)) .change_context(Self::error("Failed to build Lockr SDK response")) } @@ -186,13 +204,13 @@ impl LockrIntegration { let original_path = parts.uri.path().to_string(); let method = parts.method.clone(); - log::info!("Proxying Lockr API request: {} {}", method, original_path); + log::info!("Proxying Lockr API request: {}", method); // Extract path after /integrations/lockr/api and pass through directly. // This allows the Lockr SDK to use any API endpoint without hardcoded mappings. let target_path = original_path .strip_prefix("/integrations/lockr/api") - .ok_or_else(|| Self::error(format!("Invalid Lockr API path: {}", original_path)))?; + .ok_or_else(|| Self::error("Invalid Lockr API path"))?; let query = parts .uri @@ -201,39 +219,43 @@ impl LockrIntegration { .unwrap_or_default(); let target_url = format!("{}{}{}", self.config.api_endpoint, target_path, query); - log::info!("Forwarding to Lockr API: {}", target_url); - - let request_body = if method == Method::POST { - let bytes = + let capture_selected = self.matches(&method, &original_path); + // The same allocation backs the forwarded POST and request-local capture signals. + // Never read the browser request a second time for capture. + let (request_body, capture_body) = if method == Method::POST { + let bytes = Bytes::from( collect_body_bounded(body, INTEGRATION_MAX_BODY_BYTES, LOCKR_INTEGRATION_ID) - .await?; - EdgeBody::from(bytes) + .await?, + ); + let capture_body = capture_selected.then(|| IdentityRequestBody(bytes.clone())); + (EdgeBody::from_bytes(bytes), capture_body) } else { - EdgeBody::empty() + (EdgeBody::empty(), None) }; let mut target_req = http::Request::builder() .method(method.clone()) .uri(&target_url) .body(request_body) - .change_context(Self::error("Failed to build Lockr API proxy request"))?; + .map_err(|_| Report::new(Self::error("Failed to build Lockr API proxy request")))?; self.copy_request_headers(&parts.headers, target_req.headers_mut())?; let backend_name = Self::backend_name_for_url(services, &self.config.api_endpoint) - .change_context(Self::error("Failed to determine backend for API proxy"))?; + .map_err(|_| Report::new(Self::error("Failed to determine backend for API proxy")))?; let response = services .http_client() .send(PlatformHttpRequest::new(target_req, backend_name)) .await - .change_context(Self::error(format!( - "Failed to forward request to {}", - target_url - )))? + .map_err(|_| Report::new(Self::error("Failed to forward Lockr API request")))? .response; log::info!("Lockr API responded with status {}", response.status()); + let mut response = response; + if let Some(body) = capture_body { + response.extensions_mut().insert(body); + } Ok(response) } @@ -280,8 +302,8 @@ impl LockrIntegration { Ok(value) => { to.insert(header::ORIGIN, value); } - Err(error) => { - log::warn!("Skipping invalid Lockr origin header value '{origin}': {error}"); + Err(_) => { + log::warn!("Skipping invalid Lockr origin header"); } } } @@ -304,6 +326,95 @@ impl LockrIntegration { } } +// Only the confirmed ID5 token schema is recognized. Unknown keys do not become +// source claims, even when an EID in the same response names a registered source. +fn id5_token(token: &Value, ids: Option<&Value>, now: Option) -> Option { + if token.get("key_name")?.as_str()? != "id5id" { + return None; + } + let encoded = token.get("advertising_token")?.as_str()?; + // A malformed percent escape must not silently become a different UID. + let bytes = encoded.as_bytes(); + for (i, byte) in bytes.iter().enumerate() { + if *byte == b'%' + && (!bytes.get(i + 1).is_some_and(u8::is_ascii_hexdigit) + || !bytes.get(i + 2).is_some_and(u8::is_ascii_hexdigit)) + { + return None; + } + } + let decoded = urlencoding::decode(encoded).ok()?; + let uid = serde_json::from_str::(&decoded) + .ok()? + .get("universal_uid")? + .as_str()? + .to_string(); + if uid.is_empty() || uid.len() > 512 { + return None; + } + + // An associated EID is corroboration, not a source of new authority. + if let Some(ids) = ids { + let eid = ids.as_object()?.get("id5id"); + if let Some(eid) = eid { + if eid.get("source")?.as_str()? != ID5_SOURCE { + return None; + } + let uids = eid.get("uids")?.as_array()?; + if uids.len() != 1 + || uids[0].get("id")?.as_str()? != uid + || uids[0].get("atype")?.as_u64()? != 1 + { + return None; + } + } + } + + let expires_at = match token.get("identity_expires") { + None => None, + Some(value) => { + let millis = value.as_u64()?; + let seconds = millis / 1000; + // Without a working clock we cannot prove that a dated token is live. + if seconds <= now? { + return None; + } + Some(seconds) + } + }; + Some(IdentityOutcome::Issued { + source: ID5_SOURCE.to_string(), + uid, + expires_at, + }) +} + +fn request_consent(body: &[u8]) -> IdentityConsentSignals { + let mut signals = IdentityConsentSignals::default(); + let Ok(value) = serde_json::from_slice::(body) else { + signals.malformed = true; + return signals; + }; + let Some(object) = value.as_object() else { + signals.malformed = true; + return signals; + }; + for (field, target) in [ + ("consentString", &mut signals.tcf), + ("gppString", &mut signals.gpp), + ("ccpaString", &mut signals.usp), + ] { + if let Some(value) = object.get(field) { + match value.as_str() { + Some("") => {} + Some(text) => *target = Some(text.to_string()), + None => signals.malformed = true, + } + } + } + signals +} + fn build(settings: &Settings) -> Result>, Report> { let Some(config) = settings.integration_config::(LOCKR_INTEGRATION_ID)? else { return Ok(None); @@ -339,6 +450,7 @@ pub fn register( Ok(Some( IntegrationRegistration::builder(LOCKR_INTEGRATION_ID) .with_proxy(integration.clone()) + .with_identity_capture(integration.clone()) .with_attribute_rewriter(integration) .build(), )) @@ -367,14 +479,89 @@ impl IntegrationProxy for LockrIntegration { } else if path.starts_with("/integrations/lockr/api/") { self.handle_api_proxy(settings, services, req).await } else { - Err(Report::new(Self::error(format!( - "Unknown Lockr route: {}", - path - )))) + Err(Report::new(Self::error("Unknown Lockr route"))) } } } +impl IdentityCapture for LockrIntegration { + fn sources(&self) -> &'static [&'static str] { + &[ID5_SOURCE] + } + + // The trait's enabled switch controls capture, not the entire integration proxy. + #[allow(clippy::misnamed_getters)] + fn enabled(&self) -> bool { + self.config.capture_identity + } + + fn matches(&self, method: &Method, path: &str) -> bool { + if *method != Method::POST || !self.enabled() { + return false; + } + let Some(path) = path.strip_prefix(LOCKR_API_PREFIX) else { + return false; + }; + LOCKR_TOKEN_PATHS.contains(&path) + || (self.config.capture_global_withdrawal && path == LOCKR_REVOKE_PATH) + } + + fn observe(&self, input: IdentityCaptureInput<'_>) -> Option { + if !self.config.capture_identity || !input.status.is_success() { + return None; + } + let response: Value = serde_json::from_slice(input.decoded_body).ok()?; + if !response.get("success")?.as_bool()? { + return None; + } + let path = input.path.strip_prefix(LOCKR_API_PREFIX)?; + if path == LOCKR_REVOKE_PATH { + return self + .config + .capture_global_withdrawal + .then(|| IdentityObservation { + outcomes: vec![IdentityOutcome::ConsentWithdrawn], + consent: IdentityConsentSignals::default(), + }); + } + if !LOCKR_TOKEN_PATHS.contains(&path) { + return None; + } + let tokens = match response.get("aimTokens") { + None => &[][..], + Some(tokens) => tokens.as_array()?.as_slice(), + }; + let ids = response.get("ids"); + if ids.is_some_and(|value| !value.is_object()) { + return None; + } + let now = crate::ec::checked_current_timestamp(); + let outcomes = tokens + .iter() + .filter_map(|token| id5_token(token, ids, now)) + .collect::>(); + // The provider has not defined ordering between overlapping tokens. + // Never choose an arbitrary winner for this one-source mapping. + if outcomes.len() > 1 { + return None; + } + let consent = request_consent(input.request_body); + // Bad request JSON or signal shapes may not acquire an ID, even if the + // upstream returned one. Revocation above is deliberately independent. + if consent.malformed { + return None; + } + Some(IdentityObservation { + outcomes: if outcomes.is_empty() { + vec![IdentityOutcome::NoChange] + } else { + outcomes + }, + consent, + }) + } +} + impl IntegrationAttributeRewriter for LockrIntegration { fn integration_id(&self) -> &'static str { LOCKR_INTEGRATION_ID @@ -443,6 +630,8 @@ mod tests { fn test_config() -> LockrConfig { LockrConfig { enabled: true, + capture_identity: true, + capture_global_withdrawal: false, app_id: "test-app-id".to_string(), api_endpoint: default_api_endpoint(), sdk_url: default_sdk_url(), @@ -609,6 +798,7 @@ mod tests { let response = futures::executor::block_on(integration.handle(&settings, &services, req)) .expect("should proxy request"); assert_eq!(response.status(), http::StatusCode::OK, "should return OK"); + assert!(response.extensions().get::().is_none()); let bodies = stub.recorded_request_bodies(); assert_eq!( @@ -699,6 +889,288 @@ mod tests { ); } + const TOKEN_PATH: &str = "/integrations/lockr/api/publisher/app/v2/identityLockr/page-view"; + const REVOKE_PATH: &str = + "/integrations/lockr/api/publisher/app/v2/identityLockr/revoke-consent"; + + fn sample_token() -> Value { + json!({ + "success": true, + "aimTokens": [{ + "key_name": "id5id", + "advertising_token": "%7B%22universal_uid%22%3A%22ID5*synthetic%22%7D", + "identity_expires": 4102444800000_u64 + }] + }) + } + + fn observed( + integration: &LockrIntegration, + path: &str, + response: &Value, + request: &[u8], + ) -> Option { + let decoded = serde_json::to_vec(response).expect("should serialize fixture"); + integration.observe(IdentityCaptureInput { + path, + request_body: request, + status: StatusCode::OK, + decoded_body: &decoded, + }) + } + + #[test] + fn capture_route_and_kill_switch() { + let integration = LockrIntegration::new(test_config()); + for route in LOCKR_TOKEN_PATHS { + let path = format!("{LOCKR_API_PREFIX}{route}"); + assert!(integration.matches(&Method::POST, &path)); + assert!(!integration.matches(&Method::GET, &path)); + assert!(!integration.matches(&Method::POST, &format!("{path}/extra"))); + } + assert!(!integration.matches(&Method::POST, REVOKE_PATH)); + assert!(!integration.matches(&Method::POST, "/integrations/lockr/sdk")); + assert!(!integration.matches( + &Method::POST, + "/integrations/lockr/api/publisher/app/v2/identityLockr/settings" + )); + let killed = LockrIntegration::new(LockrConfig { + capture_identity: false, + ..test_config() + }); + assert_eq!(killed.sources(), &[ID5_SOURCE]); + assert!(!killed.enabled()); + assert!(!killed.matches(&Method::POST, TOKEN_PATH)); + } + + #[test] + fn id5_capture_validates_associated_eid_expiry_and_uid() { + let integration = LockrIntegration::new(test_config()); + let request = br#"{"consentString":"","gppString":"","ccpaString":""}"#; + let result = observed(&integration, TOKEN_PATH, &sample_token(), request) + .expect("should capture synthetic ID5 token"); + assert!( + matches!(&result.outcomes[..], [IdentityOutcome::Issued { source, uid, expires_at: Some(4102444800) }] if source == ID5_SOURCE && uid == "ID5*synthetic") + ); + assert!(result.consent.tcf.is_none()); + let mut with_eid = sample_token(); + with_eid["ids"] = json!({"id5id":{"eid":{"source":"id5-sync.com","uids":[{"id":"ID5*synthetic","atype":1}]}}}); + assert_eq!( + observed(&integration, TOKEN_PATH, &with_eid, request) + .expect("should capture matching EID") + .outcomes + .len(), + 1 + ); + with_eid["ids"]["id5id"]["eid"]["uids"][0]["id"] = json!("different"); + assert!(matches!( + &observed(&integration, TOKEN_PATH, &with_eid, request) + .expect("should treat mismatched EID as no change") + .outcomes[..], + [IdentityOutcome::NoChange] + )); + let mut changed_expiry = sample_token(); + changed_expiry["aimTokens"][0]["identity_expires"] = json!(4102444801000_u64); + assert!(matches!( + &observed(&integration, TOKEN_PATH, &changed_expiry, request) + .expect("should observe extended expiry") + .outcomes[..], + [IdentityOutcome::Issued { uid, expires_at: Some(4102444801), .. }] if uid == "ID5*synthetic" + )); + let mut mismatch = sample_token(); + mismatch["ids"] = json!({"id5id":{"eid":{"source":"other.example","uids":[{"id":"ID5*synthetic","atype":1}]}}}); + assert!(matches!( + &observed(&integration, TOKEN_PATH, &mismatch, request) + .expect("should reject mismatched source") + .outcomes[..], + [IdentityOutcome::NoChange] + )); + let mut expired = sample_token(); + expired["aimTokens"][0]["identity_expires"] = json!(1000); + assert!(matches!( + &observed(&integration, TOKEN_PATH, &expired, request) + .expect("should reject expired token") + .outcomes[..], + [IdentityOutcome::NoChange] + )); + let mut missing = sample_token(); + missing["aimTokens"][0] + .as_object_mut() + .expect("should access fixture token") + .remove("identity_expires"); + assert!(matches!( + &observed(&integration, TOKEN_PATH, &missing, request) + .expect("should capture token without expiry") + .outcomes[..], + [IdentityOutcome::Issued { + expires_at: None, + .. + }] + )); + let mut unknown = sample_token(); + unknown["aimTokens"][0]["key_name"] = json!("liveramp"); + assert!(matches!( + &observed(&integration, TOKEN_PATH, &unknown, request) + .expect("should ignore unknown provider key") + .outcomes[..], + [IdentityOutcome::NoChange] + )); + let mut huge = sample_token(); + huge["aimTokens"][0]["advertising_token"] = json!( + urlencoding::encode(&json!({"universal_uid":"x".repeat(513)}).to_string()).to_string() + ); + assert!(matches!( + &observed(&integration, TOKEN_PATH, &huge, request) + .expect("should reject oversized UID") + .outcomes[..], + [IdentityOutcome::NoChange] + )); + } + + #[test] + fn acquisition_requires_valid_request_json_and_signal_shapes() { + let integration = LockrIntegration::new(test_config()); + assert!(observed(&integration, TOKEN_PATH, &sample_token(), b"{").is_none()); + assert!( + observed( + &integration, + TOKEN_PATH, + &sample_token(), + br#"{"gppString":4}"# + ) + .is_none() + ); + let signals = observed( + &integration, + TOKEN_PATH, + &sample_token(), + br#"{"consentString":"opt-out","gppString":"gpp","ccpaString":"usp"}"#, + ) + .expect("should parse populated consent signals") + .consent; + assert_eq!(signals.tcf.as_deref(), Some("opt-out")); + assert_eq!(signals.gpp.as_deref(), Some("gpp")); + assert_eq!(signals.usp.as_deref(), Some("usp")); + assert!( + observed( + &integration, + TOKEN_PATH, + &json!({"success":true,"aimTokens":[]}), + b"{}" + ) + .is_some() + ); + assert!( + observed( + &integration, + TOKEN_PATH, + &json!({"success":false,"aimTokens":[]}), + b"{}" + ) + .is_none() + ); + assert!( + observed( + &integration, + TOKEN_PATH, + &json!({"success":true,"aimTokens":{}}), + b"{}" + ) + .is_none() + ); + } + + #[test] + fn revoke_needs_explicit_global_approval_and_application_success() { + let integration = LockrIntegration::new(LockrConfig { + capture_global_withdrawal: true, + ..test_config() + }); + assert!(integration.matches(&Method::POST, REVOKE_PATH)); + assert!(matches!( + &observed(&integration, REVOKE_PATH, &json!({"success":true}), b"{") + .expect("should observe approved explicit withdrawal") + .outcomes[..], + [IdentityOutcome::ConsentWithdrawn] + )); + assert!(observed(&integration, REVOKE_PATH, &json!({"success":false}), b"{}").is_none()); + let default = LockrIntegration::new(test_config()); + assert!(observed(&default, REVOKE_PATH, &json!({"success":true}), b"{}").is_none()); + } + + #[test] + fn selected_proxy_reuses_forwarded_body_and_preserves_response() { + let stub = Arc::new(StubHttpClient::new()); + stub.push_response(200, b"synthetic upstream".to_vec()); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let integration = LockrIntegration::new(test_config()); + let payload = br#"{"consentString":"synthetic"}"#; + let req = http::Request::builder() + .method(Method::POST) + .uri(format!("https://publisher.example{TOKEN_PATH}?id=private")) + .header(header::COOKIE, "session=private") + .header(header::AUTHORIZATION, "Bearer private") + .body(EdgeBody::from(payload.to_vec())) + .expect("should build selected proxy request"); + let response = futures::executor::block_on(integration.handle( + &create_test_settings(), + &services, + req, + )) + .expect("should forward selected proxy request"); + assert_eq!(stub.recorded_request_bodies(), vec![payload.to_vec()]); + let headers = stub.recorded_request_headers(); + assert!( + !headers[0] + .iter() + .any(|(key, _)| key == "cookie" || key == "authorization") + ); + assert_eq!( + response + .extensions() + .get::() + .expect("should attach shared forwarding bytes") + .0 + .as_ref(), + payload + ); + assert!( + matches!(response.into_body(), EdgeBody::Once(bytes) if bytes.as_ref() == b"synthetic upstream") + ); + } + + #[test] + fn malformed_or_failed_upstream_does_not_acquire_or_revoke() { + let integration = LockrIntegration::new(LockrConfig { + capture_global_withdrawal: true, + ..test_config() + }); + for path in [TOKEN_PATH, REVOKE_PATH] { + assert!( + integration + .observe(IdentityCaptureInput { + path, + request_body: b"{}", + status: StatusCode::BAD_REQUEST, + decoded_body: br#"{"success":true}"#, + }) + .is_none() + ); + assert!( + integration + .observe(IdentityCaptureInput { + path, + request_body: b"{}", + status: StatusCode::OK, + decoded_body: b"{", + }) + .is_none() + ); + } + } + #[test] fn disabled_invalid_config_does_not_error() { let mut settings = create_test_settings(); diff --git a/crates/trusted-server-core/src/integrations/mod.rs b/crates/trusted-server-core/src/integrations/mod.rs index 820e2c24b..25eedc0c6 100644 --- a/crates/trusted-server-core/src/integrations/mod.rs +++ b/crates/trusted-server-core/src/integrations/mod.rs @@ -18,6 +18,7 @@ pub mod didomi; pub mod google_tag_manager; pub mod gpt; pub mod gpt_diagnostics; +pub mod identity; pub mod js_asset_proxy; pub mod lockr; pub mod nextjs; diff --git a/crates/trusted-server-core/src/integrations/prebid.rs b/crates/trusted-server-core/src/integrations/prebid.rs index f1d1d0b50..6cf87aec2 100644 --- a/crates/trusted-server-core/src/integrations/prebid.rs +++ b/crates/trusted-server-core/src/integrations/prebid.rs @@ -6864,6 +6864,9 @@ external_bundle_sri = "sha384-AAAA" let mut auction_request = create_test_auction_request(); auction_request.user.eids = Some(vec![ crate::openrtb::Eid { + inserter: None, + matcher: None, + mm: None, source: "liveramp.com".to_owned(), uids: vec![crate::openrtb::Uid { id: "LR_xyz".to_owned(), @@ -6872,6 +6875,9 @@ external_bundle_sri = "sha384-AAAA" }], }, crate::openrtb::Eid { + inserter: None, + matcher: None, + mm: None, source: "id5-sync.com".to_owned(), uids: vec![crate::openrtb::Uid { id: "ID5_abc".to_owned(), @@ -6880,6 +6886,9 @@ external_bundle_sri = "sha384-AAAA" }], }, crate::openrtb::Eid { + inserter: None, + matcher: None, + mm: None, source: "google.com".to_owned(), uids: vec![crate::openrtb::Uid { id: "pair-id".to_owned(), diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index d3c09cd7d..f0c1bd643 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -8,10 +8,16 @@ use error_stack::Report; use http::{Method, Request, Response}; use matchit::Router; +use super::identity::{ + IdentityCapture, IdentityCaptureInput, IdentityRequestBody, inspect_response, +}; use crate::auction::AuctionPlan; use crate::constants::HEADER_X_TS_EC; use crate::ec::EcContext; +use crate::ec::identity::stage_capture; use crate::ec::kv::KvIdentityGraph; +use crate::ec::partner::normalize_partner_source_domain; +use crate::ec::registry::PartnerRegistry; use crate::error::TrustedServerError; use crate::geo::GeoInfo; use crate::http_util::is_navigation_request; @@ -629,6 +635,8 @@ pub struct IntegrationRegistration { pub html_stream_processors: Vec>, pub head_injectors: Vec>, pub request_filters: Vec>, + /// Capture capability and finite ownership descriptor, including when disabled. + pub identity_capture: Option>, } impl IntegrationRegistration { @@ -655,6 +663,7 @@ impl IntegrationRegistrationBuilder { html_stream_processors: Vec::new(), head_injectors: Vec::new(), request_filters: Vec::new(), + identity_capture: None, }, } } @@ -665,6 +674,13 @@ impl IntegrationRegistrationBuilder { self } + /// Registers capture and ownership claims in the existing integration registry. + #[must_use] + pub fn with_identity_capture(mut self, capture: Arc) -> Self { + self.registration.identity_capture = Some(capture); + self + } + #[must_use] pub fn with_attribute_rewriter( mut self, @@ -745,6 +761,7 @@ struct IntegrationRegistryInner { html_stream_processors: Vec>, head_injectors: Vec>, request_filters: Vec>, + identity_captures: BTreeMap<&'static str, Arc>, } impl Default for IntegrationRegistryInner { @@ -766,6 +783,7 @@ impl Default for IntegrationRegistryInner { html_stream_processors: Vec::new(), head_injectors: Vec::new(), request_filters: Vec::new(), + identity_captures: BTreeMap::new(), } } } @@ -863,6 +881,11 @@ impl IntegrationRegistry { } for registration in registrations { + if let Some(capture) = registration.identity_capture { + inner + .identity_captures + .insert(registration.integration_id, capture); + } inner .enabled_integration_ids .push(registration.integration_id); @@ -931,12 +954,71 @@ impl IntegrationRegistry { } } + Self::check_identity_owners(settings, &inner.identity_captures)?; Ok(Self { inner: Arc::new(inner), plan: Some(plan), }) } + /// Validates source ownership through existing builders, including disabled capture. + /// + /// Only selected owner builders are constructed, so deploy validation does + /// not accidentally run startup checks on unrelated unresolved secrets. + /// + /// # Errors + /// + /// Returns a configuration error for an unavailable or non-claiming owner. + pub fn validate_identity_owners(settings: &Settings) -> Result<(), Report> { + let mut captures = BTreeMap::new(); + for partner in &settings.ec.partners { + let Some(owner) = partner.identity_owner.as_deref() else { + continue; + }; + if captures.contains_key(owner) { + continue; + } + if let Some(builder) = crate::integrations::builders() + .iter() + .find(|builder| builder.id == owner) + && let Some(registration) = (builder.build)(settings)? + && let Some(capture) = registration.identity_capture + { + captures.insert(registration.integration_id, capture); + } + } + Self::check_identity_owners(settings, &captures) + } + + fn check_identity_owners( + settings: &Settings, + captures: &BTreeMap<&'static str, Arc>, + ) -> Result<(), Report> { + for partner in &settings.ec.partners { + let Some(owner) = partner.identity_owner.as_deref() else { + continue; + }; + let source = normalize_partner_source_domain(&partner.source_domain).map_err(|_| { + Report::new(TrustedServerError::Configuration { + message: "identity source is invalid".to_owned(), + }) + })?; + let claims_source = captures.get(owner).is_some_and(|capture| { + capture.sources().iter().any(|claim| { + normalize_partner_source_domain(claim).ok().as_deref() == Some(source.as_str()) + }) + }); + if !claims_source || partner.pull_sync_enabled { + return Err(Report::new(TrustedServerError::Configuration { + message: format!( + "identity owner for '{source}' must be a registered claiming integration with legacy pull disabled" + ), + })); + } + } + Ok(()) + } + /// Return whether this registry and another consumer share the same plan allocation. #[must_use] pub fn shares_plan(&self, plan: &Arc) -> bool { @@ -1041,7 +1123,7 @@ impl IntegrationRegistry { services, mut req, } = input; - if let Some((proxy, _)) = self.find_route(method, path) { + if let Some((proxy, integration_id)) = self.find_route(method, path) { // Organic proxy handler: generate if needed (best effort). // Only generate for document navigations — subresource requests // may lack consent signals such as the Sec-GPC header. @@ -1058,7 +1140,38 @@ impl IntegrationRegistry { // Remove any caller-supplied EC header rather than forwarding it. req.headers_mut().remove(HEADER_X_TS_EC.clone()); - Some(proxy.handle(settings, services, req).await) + let result = proxy.handle(settings, services, req).await; + let Ok(mut response) = result else { + return Some(result); + }; + let request_body = response.extensions_mut().remove::(); + if let Some(capture) = self.inner.identity_captures.get(integration_id) + && capture.enabled() + && capture.matches(method, path) + { + let (original, decoded) = inspect_response(response).await; + response = original; + if let Some(decoded) = decoded + && let Some(observation) = capture.observe(IdentityCaptureInput { + path, + request_body: request_body.as_ref().map_or(&[], |body| body.0.as_ref()), + status: response.status(), + decoded_body: &decoded, + }) + && let Ok(partners) = PartnerRegistry::from_config(&settings.ec.partners) + && let Some(effects) = stage_capture( + integration_id, + capture.sources(), + observation, + ec_context, + &partners, + settings, + ) + { + response.extensions_mut().insert(effects); + } + } + Some(Ok(response)) } else { None } @@ -1270,6 +1383,7 @@ impl IntegrationRegistry { ) -> Self { Self { inner: Arc::new(IntegrationRegistryInner { + identity_captures: BTreeMap::new(), get_router: Router::new(), post_router: Router::new(), put_router: Router::new(), @@ -1300,6 +1414,7 @@ impl IntegrationRegistry { ) -> Self { Self { inner: Arc::new(IntegrationRegistryInner { + identity_captures: BTreeMap::new(), get_router: Router::new(), post_router: Router::new(), put_router: Router::new(), @@ -1326,6 +1441,7 @@ impl IntegrationRegistry { pub fn from_request_filters(request_filters: Vec>) -> Self { Self { inner: Arc::new(IntegrationRegistryInner { + identity_captures: BTreeMap::new(), get_router: Router::new(), post_router: Router::new(), put_router: Router::new(), @@ -1392,6 +1508,7 @@ impl IntegrationRegistry { Self { inner: Arc::new(IntegrationRegistryInner { + identity_captures: BTreeMap::new(), get_router, post_router, put_router, diff --git a/crates/trusted-server-core/src/openrtb.rs b/crates/trusted-server-core/src/openrtb.rs index 4aded7488..193a5c14e 100644 --- a/crates/trusted-server-core/src/openrtb.rs +++ b/crates/trusted-server-core/src/openrtb.rs @@ -65,11 +65,20 @@ pub struct ConsentedProvidersSettings { } /// An Extended User ID entry from an identity provider. -#[derive(Debug, Clone, Serialize, Deserialize)] +#[derive(Debug, Clone, Default, Serialize, Deserialize)] #[serde(deny_unknown_fields)] pub struct Eid { /// Identity provider domain (e.g. `"id5-sync.com"`). pub source: String, + /// Entity that inserted this EID into the request. + #[serde(skip_serializing_if = "Option::is_none")] + pub inserter: Option, + /// Entity that matched this EID. An empty supplied string is preserved. + #[serde(skip_serializing_if = "Option::is_none")] + pub matcher: Option, + /// `OpenRTB` match method. Zero is a supplied value, not absence. + #[serde(skip_serializing_if = "Option::is_none")] + pub mm: Option, /// One or more user IDs from this provider. pub uids: Vec, } @@ -394,6 +403,9 @@ mod tests { #[test] fn eid_serializes_correctly() { let eid = Eid { + inserter: None, + matcher: None, + mm: None, source: "id5-sync.com".to_owned(), uids: vec![Uid { id: "ID5-abc123".to_owned(), @@ -418,6 +430,9 @@ mod tests { #[test] fn eid_serializes_vendor_specific_atype() { let eid = Eid { + inserter: None, + matcher: None, + mm: None, source: "google.com".to_owned(), uids: vec![Uid { id: "pair-id".to_owned(), diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index d8c217076..f82184b4e 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -27,7 +27,6 @@ use std::time::{Duration, Instant, SystemTime}; use brotli::Decompressor; use brotli::enc::BrotliEncoderParams; use brotli::enc::writer::CompressorWriter; -use cookie::CookieJar; use edgezero_core::body::Body as EdgeBody; use error_stack::{Report, ResultExt}; use flate2::read::ZlibDecoder; @@ -36,9 +35,7 @@ use futures::StreamExt as _; use http::{HeaderValue, Method, Request, Response, StatusCode, Uri, header}; use sha2::Digest as _; -use crate::auction::endpoints::{ - merge_auction_eids, resolve_auction_eids, resolve_client_auction_eids, -}; +use crate::auction::endpoints::resolve_and_merge_auction_eids; use crate::auction::formats::sanitize_publisher_page_url; use crate::auction::orchestrator::{ AuctionOrchestrator, DispatchAuctionOutcome, DispatchedAuction, ERROR_TYPE_ALL, @@ -56,7 +53,7 @@ use crate::cache_policy::{ CachePolicy, EdgeCacheHeader, cache_control_headers_are_private_or_no_store, }; use crate::consent::{consent_allows_server_side_auction, gate_eids_by_consent}; -use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EIDS, HEADER_X_COMPRESS_HINT}; +use crate::constants::{COOKIE_SHAREDID, HEADER_X_COMPRESS_HINT}; use crate::cookies::handle_request_cookies; use crate::cookies::template_cache_policy::{TemplateCookieDecision, evaluate_cookie_policy}; use crate::creative_opportunities::{ @@ -4812,7 +4809,7 @@ pub async fn handle_publisher_request( let mut origin_request = Some(req); let eid_cookie_may_need_persistence = cookie_jar .as_ref() - .is_some_and(|jar| jar.get(COOKIE_TS_EIDS).is_some() || jar.get(COOKIE_SHAREDID).is_some()); + .is_some_and(|jar| jar.get(COOKIE_SHAREDID).is_some()); // This decision also gates the concurrent origin send below. A marker skip // cannot currently delay the origin behind an auction because marker // validation requires a non-empty registry, and every such auction sets @@ -4932,8 +4929,6 @@ pub async fn handle_publisher_request( apply_auction_eids_and_device( &mut auction_request, &AuctionEidTargeting { - cookie_jar: cookie_jar.as_ref(), - ec_id, kv_snapshot: ec_context.kv_snapshot(), partner_registry: auction.registry, ec_context, @@ -5511,8 +5506,6 @@ pub(crate) struct MatchedSlotsContext<'a> { /// Borrowed inputs for [`apply_auction_eids_and_device`], bundled to keep the /// helper within the project's 7-argument cap. struct AuctionEidTargeting<'a> { - cookie_jar: Option<&'a CookieJar>, - ec_id: Option<&'a str>, kv_snapshot: &'a crate::ec::EcKvSnapshot, partner_registry: Option<&'a PartnerRegistry>, ec_context: &'a EcContext, @@ -5532,21 +5525,12 @@ fn apply_auction_eids_and_device( auction_request: &mut AuctionRequest, targeting: &AuctionEidTargeting<'_>, ) { - let ts_eids_value = targeting - .cookie_jar - .and_then(|j| j.get(COOKIE_TS_EIDS)) - .map(|c| c.value().to_owned()); - let client_eids = if targeting.ec_id.is_some() { - resolve_client_auction_eids(None, ts_eids_value.as_deref()) - } else { - None - }; - let kv_eids = resolve_auction_eids( + let merged_eids = resolve_and_merge_auction_eids( + None, targeting.kv_snapshot, targeting.partner_registry, targeting.ec_context, ); - let merged_eids = merge_auction_eids(client_eids, kv_eids); let had_eids = merged_eids.as_ref().is_some_and(|v| !v.is_empty()); auction_request.user.eids = gate_eids_by_consent(merged_eids, auction_request.user.consent.as_ref()); @@ -6877,8 +6861,8 @@ fn normalize_page_bids_path(raw: &str) -> String { /// /// `kv` enriches the bid request with server-side EIDs from the EC identity /// graph. Only the Fastly adapter has a KV identity store, so Axum, Cloudflare, -/// and Spin pass `None`; client cookie EIDs are still resolved and consent-gated -/// on every adapter, so no adapter forwards unconsented EIDs. +/// and Spin pass `None` and cannot resolve server-side EIDs. Retired EID cookies +/// are ignored on every adapter. /// /// # Errors /// @@ -6998,7 +6982,6 @@ pub async fn handle_page_bids( .map(str::to_owned); let consent_context = ec_context.consent().clone(); let geo = ec_context.geo_info().cloned(); - let cookie_jar = handle_request_cookies(&req)?; // Same fail-closed jurisdiction-aware gate the publisher navigation path // uses — relies on the adapter's geo-aware EC context. @@ -7072,7 +7055,7 @@ pub async fn handle_page_bids( }; // Hand the loaded row to the request context so response // finalization — which runs on an EC context the adapter owns, - // after this handler returns — ingests `ts-eids`/`sharedId` updates + // after this handler returns — ingests sharedId updates // from this read instead of paying for a second lookup. if !matches!(page_bids_kv_snapshot, crate::ec::EcKvSnapshot::NotRead) { ec_context.set_kv_snapshot(page_bids_kv_snapshot.clone()); @@ -7088,8 +7071,6 @@ pub async fn handle_page_bids( apply_auction_eids_and_device( &mut auction_request, &AuctionEidTargeting { - cookie_jar: cookie_jar.as_ref(), - ec_id: ec_id.as_deref(), kv_snapshot: &page_bids_kv_snapshot, partner_registry: auction.registry, ec_context, @@ -7260,6 +7241,7 @@ mod tests { use brotli::Decompressor; use brotli::enc::writer::CompressorWriter; + use cookie::CookieJar; use flate2::read::GzDecoder; use flate2::write::GzEncoder; @@ -7867,6 +7849,7 @@ mod tests { fn marker_partner(source_domain: &str) -> crate::settings::EcPartner { crate::settings::EcPartner { + identity_owner: None, name: format!("Marker partner {source_domain}"), source_domain: source_domain.to_owned(), openrtb_atype: crate::settings::EcPartner::default_openrtb_atype(), @@ -7979,6 +7962,90 @@ mod tests { lookups.load(Ordering::SeqCst) } + #[test] + fn publisher_and_page_bids_ignore_retired_cookie_ids() { + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let context = EcContext::new_for_test( + Some(ec_id.clone()), + crate::consent::ConsentContext { + jurisdiction: crate::consent::jurisdiction::Jurisdiction::NonRegulated, + ..Default::default() + }, + ); + let mut entry = + crate::ec::kv_types::KvEntry::minimal("ids.example.com", "expired-uid", 1_000); + entry + .ids + .get_mut("ids.example.com") + .expect("should contain record") + .expires_at = Some(1); + let snapshot = crate::ec::EcKvSnapshot::Present { + ec_id: ec_id.clone(), + entry: Box::new(entry.clone()), + generation: Some(1), + }; + let registry = PartnerRegistry::from_config(&[marker_partner("ids.example.com")]) + .expect("should build registry"); + let services = crate::platform::test_support::noop_services(); + for path_label in ["Server-side", "Page-bids"] { + for uid in ["expired-uid", "different-uid"] { + let encoded = base64::Engine::encode( + &base64::engine::general_purpose::STANDARD, + serde_json::to_vec( + &serde_json::json!([{"source": "ids.example.com", "uids": [{"id": uid}]}]), + ) + .expect("should encode cookie EIDs"), + ); + let mut jar = CookieJar::new(); + jar.add(cookie::Cookie::new("ts-eids", encoded)); + let mut request = AuctionRequest { + id: "example-auction".to_owned(), + slots: Vec::new(), + publisher: crate::auction::types::PublisherInfo { + domain: "publisher.example.com".to_owned(), + page_url: None, + }, + user: crate::auction::types::UserInfo { + id: Some(ec_id.clone()), + consent: Some(context.consent().clone()), + eids: None, + }, + device: None, + site: None, + context: Default::default(), + }; + apply_auction_eids_and_device( + &mut request, + &AuctionEidTargeting { + kv_snapshot: &snapshot, + partner_registry: Some(®istry), + ec_context: &context, + services: &services, + geo: None, + path_label, + }, + ); + let uids: Vec<&str> = request + .user + .eids + .iter() + .flatten() + .flat_map(|eid| &eid.uids) + .map(|uid| uid.id.as_str()) + .collect(); + assert!( + uids.is_empty(), + "publisher and SPA must not consume retired EID cookies" + ); + assert_eq!( + snapshot.entry_for(&ec_id), + Some(&entry), + "should not mutate retained lifecycle state" + ); + } + } + } + #[tokio::test] async fn valid_completeness_marker_skips_pull_only_snapshot_lookup() { assert_eq!( @@ -8004,10 +8071,10 @@ mod tests { } #[tokio::test] - async fn valid_marker_does_not_skip_eid_cookie_persistence_lookup() { + async fn valid_marker_ignores_retired_eid_cookie() { assert_eq!( run_marker_lookup_probe(MarkerProbe::Valid, true, false).await, - 1 + 0 ); } @@ -14123,7 +14190,6 @@ mod tests { Some(&graph), &partners, None, - None, &mut response, ); assert_eq!( diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 0c20b2b1f..0063bea4b 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -400,6 +400,12 @@ pub struct EcPartner { /// Whether this partner's UIDs appear in auction `user.eids`. #[serde(default, deserialize_with = "from_value_or_str")] pub bidstream_enabled: bool, + /// Integration selected as the authoritative identity writer for this source. + /// + /// An owned source retains token-authenticated read access, rejects legacy + /// batch writes and cannot enable legacy pull acquisition. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub identity_owner: Option, /// Plaintext API token used by inbound batch sync and identify requests. /// /// When present, the token is hashed at startup for auth lookups. Omitting @@ -3988,6 +3994,7 @@ mod tests { pull_sync_allowed_domains: Vec::new(), pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + identity_owner: None, ts_pull_token: Some(Redacted::new(CANARY_EC_PARTNER_TS_PULL_TOKEN.to_string())), }]; @@ -5800,6 +5807,7 @@ source_domain = "partner.example.com" pull_sync_allowed_domains: vec!["partner.example.com".to_owned()], pull_sync_ttl_sec: EcPartner::default_pull_sync_ttl_sec(), pull_sync_rate_limit: EcPartner::default_pull_sync_rate_limit(), + identity_owner: None, ts_pull_token: Some(Redacted::new(ts_pull_token.to_owned())), } } diff --git a/crates/trusted-server-js/lib/src/core/auction.ts b/crates/trusted-server-js/lib/src/core/auction.ts index a02684362..cb22a32a0 100644 --- a/crates/trusted-server-js/lib/src/core/auction.ts +++ b/crates/trusted-server-js/lib/src/core/auction.ts @@ -30,6 +30,9 @@ export interface AuctionUid { /** An auction-level EID entry forwarded to the server. */ export interface AuctionEid { source: string; + inserter?: string; + matcher?: string; + mm?: number; uids: AuctionUid[]; } diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index 82c0e8c6b..020d3909f 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -1011,6 +1011,9 @@ type TrustedServerRequest = { type PrebidUserIdEid = { source?: unknown; + inserter?: unknown; + matcher?: unknown; + mm?: unknown; uids?: Array<{ id?: unknown; atype?: unknown; ext?: unknown }>; }; @@ -1907,7 +1910,18 @@ function collectAuctionEids(): AuctionEid[] | undefined { continue; } - eids.push({ source: eid.source, uids }); + const sanitized: AuctionEid = { source: eid.source, uids }; + if (typeof eid.inserter === 'string') sanitized.inserter = eid.inserter; + if (typeof eid.matcher === 'string') sanitized.matcher = eid.matcher; + if ( + typeof eid.mm === 'number' && + Number.isInteger(eid.mm) && + eid.mm >= -2147483648 && + eid.mm <= 2147483647 + ) { + sanitized.mm = eid.mm; + } + eids.push(sanitized); } return eids.length > 0 ? eids : undefined; @@ -1987,6 +2001,7 @@ function installApsBidResponseRegistry(): void { } export function installPrebidNpm(config?: Partial): typeof pbjs { + expireRetiredPrebidEidsCookie(); // The prebid integration requires the external Prebid.js bundle // (integrations.prebid.external_bundle_url). When it failed to load (network // error, SRI mismatch) window.pbjs is still the head-injected stub with no @@ -2357,11 +2372,7 @@ export function installPrebidNpm(config?: Partial): typeof pbjs buildRequests(validBidRequests: TrustedServerBidRequest[]): TrustedServerRequest { log.debug('[tsjs-prebid] buildRequests', { count: validBidRequests.length }); const requestScopedBidRequests = [...validBidRequests]; - const hasUserIdApi = typeof pbjs.getUserIdsAsEids === 'function'; const auctionEids = collectAuctionEids(); - if (hasUserIdApi && !auctionEids) { - clearPrebidEidsCookie(); - } const payload = buildAdRequest(validBidRequests, { eids: auctionEids }); return { method: 'POST', @@ -2538,7 +2549,6 @@ export function installPrebidNpm(config?: Partial): typeof pbjs // and persist them as a cookie for backend sync. const originalBidsBack = opts.bidsBackHandler; opts.bidsBackHandler = function (...args: unknown[]) { - syncPrebidEidsCookie(); const registrationId = isSyntheticRefresh ? undefined : registerPendingPublisherBids(publisherAdUnitCodes, args[0], firstImpressionTokens); @@ -2897,114 +2907,14 @@ export function installUserIdModules(): void { } } -// --------------------------------------------------------------------------- -// Prebid EID cookie sync -// --------------------------------------------------------------------------- - -/** Maximum cookie payload size in bytes (leave room for other cookies). */ -const MAX_EID_COOKIE_BYTES = 3072; - -/** Cookie name for persisted Prebid EIDs. */ -const EID_COOKIE_NAME = 'ts-eids'; - -/** Cookie max-age in seconds (1 day). */ -const EID_COOKIE_MAX_AGE = 86400; - -/** Clears any previously persisted Prebid EIDs cookie. */ -function clearPrebidEidsCookie(): void { - document.cookie = `${EID_COOKIE_NAME}=; Path=/; Secure; SameSite=Lax; Max-Age=0`; -} - /** - * Trims an EID payload so its base64-encoded cookie fits `MAX_EID_COOKIE_BYTES`, - * dropping UIDs then whole sources from the tail. `/auction` requests still - * forward the untrimmed set in the request body (see `buildAdRequest`); this - * cap only bounds what the `ts-eids` cookie carries for routes without a body - * (e.g. `GET /_ts/page-bids`) and for backend ingestion at response finalize. + * Transition-only cleanup for the retired host-only Path=/ EID cookie. + * Remove once bundles that write it have aged past both their deployment cache + * lifetime and the old cookie's one-day max-age. New servers ignore those writes. */ -function fitAuctionEidsToCookie(eids: AuctionEid[]): AuctionEid[] | undefined { - const payload = eids.map((eid) => ({ source: eid.source, uids: [...eid.uids] })); - const droppedSources = new Set(); - const trimmedSources = new Set(); - - while (payload.length > 0) { - const encoded = btoa(JSON.stringify(payload)); - if (encoded.length <= MAX_EID_COOKIE_BYTES) { - logTrimmedEidSources(droppedSources, trimmedSources); - return payload; - } - - const last = payload[payload.length - 1]; - if (last && last.uids.length > 1) { - last.uids = last.uids.slice(0, last.uids.length - 1); - trimmedSources.add(last.source); - continue; - } - - const dropped = payload.pop(); - if (dropped) { - droppedSources.add(dropped.source); - // A source trimmed and then dropped is reported once, as dropped. - trimmedSources.delete(dropped.source); - } - } - - logTrimmedEidSources(droppedSources, trimmedSources); - return undefined; -} - -/** - * Logs which EID sources `fitAuctionEidsToCookie` had to drop or trim UIDs - * from, if any. Debug level: `/auction` sends the untrimmed set in the request - * body, so a trimmed cookie is expected steady state rather than a warning. - */ -function logTrimmedEidSources(droppedSources: Set, trimmedSources: Set): void { - if (droppedSources.size === 0 && trimmedSources.size === 0) { - return; - } - const parts: string[] = []; - if (droppedSources.size > 0) { - parts.push(`dropped sources: ${[...droppedSources].join(', ')}`); - } - if (trimmedSources.size > 0) { - parts.push(`trimmed uids from sources: ${[...trimmedSources].join(', ')}`); - } - log.debug( - `[tsjs-prebid] ts-eids cookie exceeded ${MAX_EID_COOKIE_BYTES} bytes; ${parts.join('; ')}` - ); -} - -/** - * Collects EIDs from Prebid's User ID Module and writes them as a - * base64-encoded OpenRTB-style JSON cookie (`ts-eids`) for backend ingestion - * and auction fallback on later requests. - */ -function syncPrebidEidsCookie(): void { - try { - if (typeof pbjs.getUserIdsAsEids !== 'function') { - // Without Prebid EIDs to forward, stale auction fallback IDs must not persist. - clearPrebidEidsCookie(); - return; - } - - const eids = collectAuctionEids(); - if (!eids) { - clearPrebidEidsCookie(); - return; - } - - const payload = fitAuctionEidsToCookie(eids); - if (!payload) { - clearPrebidEidsCookie(); - return; - } - - const encoded = btoa(JSON.stringify(payload)); - document.cookie = `${EID_COOKIE_NAME}=${encoded}; Path=/; Secure; SameSite=Lax; Max-Age=${EID_COOKIE_MAX_AGE}`; - - log.debug(`[tsjs-prebid] synced ${payload.length} EID sources to cookie`); - } catch (err) { - log.warn('[tsjs-prebid] failed to sync EIDs cookie', err); +function expireRetiredPrebidEidsCookie(): void { + if (typeof document !== 'undefined') { + document.cookie = 'ts-eids=; Path=/; Secure; SameSite=Lax; Max-Age=0'; } } diff --git a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts index 0c209b89a..1351206c1 100644 --- a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts @@ -2543,9 +2543,9 @@ describe('prebid/installPrebidNpm', () => { expect(logged.some((value) => JSON.stringify(value).includes(sentinel))).toBe(false); }); - it('buildRequests clears stale ts-eids cookie when current Prebid EIDs are absent', () => { - const spec = getAdapterSpec(); + it('installation expires the retired ts-eids cookie when EIDs are absent', () => { document.cookie = 'ts-eids=stale-value'; + const spec = getAdapterSpec(); mockGetUserIdsAsEids.mockReturnValue([]); spec.buildRequests([ @@ -2615,6 +2615,45 @@ describe('prebid/installPrebidNpm', () => { ]); }); + it('buildRequests preserves supplied EID provenance including empty matcher and zero mm', () => { + const spec = getAdapterSpec(); + mockGetUserIdsAsEids.mockReturnValue([ + { + source: 'id5-sync.com', + inserter: 'Lockr-For-Publishers', + matcher: '', + mm: 0, + uids: [{ id: 'opaque-id', atype: 1, ext: { linkType: 1 } }], + }, + { + source: 'other.example', + inserter: 1, + matcher: null, + mm: 1.5, + uids: [{ id: 'other-id' }], + }, + ]); + const result = spec.buildRequests([ + { + adUnitCode: 'slot', + bidder: 'trustedServer', + mediaTypes: { banner: { sizes: [[300, 250]] } }, + params: {}, + }, + ]); + expect(JSON.parse(result.data).eids).toEqual([ + { + source: 'id5-sync.com', + inserter: 'Lockr-For-Publishers', + matcher: '', + mm: 0, + uids: [{ id: 'opaque-id', atype: 1, ext: { linkType: 1 } }], + }, + { source: 'other.example', uids: [{ id: 'other-id' }] }, + ]); + expect(document.cookie).not.toMatch(/(?:^|; )ts-eids=/); + }); + it('buildRequests uses custom endpoint when configured', () => { mockRegisterBidAdapter.mockClear(); installPrebidNpm({ endpoint: '/custom/auction' }); @@ -3157,7 +3196,7 @@ describe('prebid/installPrebidNpm', () => { expect(hasTsBidder).toBe(true); }); - it('syncs a structured ts-eids cookie after bidsBackHandler', () => { + it('never writes a ts-eids cookie after bidsBackHandler', () => { mockRequestBids.mockImplementation((opts?: { bidsBackHandler?: () => void }) => { opts?.bidsBackHandler?.(); }); @@ -3176,20 +3215,10 @@ describe('prebid/installPrebidNpm', () => { adUnits: [{ bids: [{ bidder: 'appnexus', params: {} }] }], } as unknown as RequestBidsArg); - const cookieValue = document.cookie.match(/(?:^|; )ts-eids=([^;]+)/)?.[1]; - expect(cookieValue).toBeDefined(); - expect(JSON.parse(atob(cookieValue!))).toEqual([ - { - source: 'sharedid.org', - uids: [ - { id: 'shared_123', atype: 3 }, - { id: 'shared_456', ext: { provider: 'example' } }, - ], - }, - ]); + expect(document.cookie).not.toMatch(/(?:^|; )ts-eids=/); }); - it('preserves an opaque LiveRamp envelope in the ts-eids cookie', () => { + it('does not persist an opaque LiveRamp envelope in a cookie', () => { mockRequestBids.mockImplementation((opts?: { bidsBackHandler?: () => void }) => { opts?.bidsBackHandler?.(); }); @@ -3205,140 +3234,10 @@ describe('prebid/installPrebidNpm', () => { adUnits: [{ bids: [{ bidder: 'appnexus', params: {} }] }], } as unknown as RequestBidsArg); - const cookieValue = document.cookie.match(/(?:^|; )ts-eids=([^;]+)/)?.[1]; - expect(cookieValue).toBeDefined(); - expect(JSON.parse(atob(cookieValue!))).toEqual([ - { - source: 'liveramp.com', - uids: [{ id: 'opaque-test-envelope', atype: 3 }], - }, - ]); - }); - - it('trims an oversized ts-eids cookie deterministically and logs which sources were dropped', () => { - const debugSpy = vi.spyOn(log, 'debug').mockImplementation(() => {}); - const warnSpy = vi.spyOn(log, 'warn').mockImplementation(() => {}); - mockRequestBids.mockImplementation((opts?: { bidsBackHandler?: () => void }) => { - opts?.bidsBackHandler?.(); - }); - // A long uid value per source makes the encoded payload exceed - // MAX_EID_COOKIE_BYTES (3072) once several sources are present, the - // same way a real page with many ID modules would. - const longUid = 'x'.repeat(400); - mockGetUserIdsAsEids.mockReturnValue([ - { source: 'id5-sync.com', uids: [{ id: longUid, atype: 1 }] }, - { source: 'liveramp.com', uids: [{ id: longUid, atype: 3 }] }, - { source: 'criteo.com', uids: [{ id: longUid, atype: 1 }] }, - { source: 'uidapi.com', uids: [{ id: longUid, atype: 3 }] }, - { source: 'pubcid.org', uids: [{ id: longUid, atype: 1 }] }, - { source: 'sharedid.org', uids: [{ id: longUid, atype: 1 }] }, - { source: 'dropped.example', uids: [{ id: longUid, atype: 1 }] }, - ]); - - const pbjs = installPrebidNpm(); - pbjs.requestBids({ - adUnits: [{ bids: [{ bidder: 'appnexus', params: {} }] }], - } as unknown as RequestBidsArg); - - const cookieValue = document.cookie.match(/(?:^|; )ts-eids=([^;]+)/)?.[1]; - expect(cookieValue).toBeDefined(); - expect(cookieValue!.length).toBeLessThanOrEqual(3072); - - const persistedSources = (JSON.parse(atob(cookieValue!)) as Array<{ source: string }>).map( - (eid) => eid.source - ); - expect(persistedSources).not.toContain('dropped.example'); - expect(persistedSources).toContain('id5-sync.com'); - - const loggedMessage = debugSpy.mock.calls - .map((call) => call.at(-1)) - .find( - (arg): arg is string => typeof arg === 'string' && arg.includes('ts-eids cookie exceeded') - ); - expect(loggedMessage).toBeDefined(); - expect(loggedMessage).toContain('dropped.example'); - expect(warnSpy).not.toHaveBeenCalledWith(expect.stringContaining('ts-eids cookie exceeded')); - }); - - it('logs a source whose UIDs were partially trimmed but retained', () => { - const debugSpy = vi.spyOn(log, 'debug').mockImplementation(() => {}); - mockRequestBids.mockImplementation((opts?: { bidsBackHandler?: () => void }) => { - opts?.bidsBackHandler?.(); - }); - // A single source with many UIDs is trimmed down to fit rather than - // dropped outright — it stays in the payload, but with fewer UIDs. - const longUid = 'x'.repeat(400); - mockGetUserIdsAsEids.mockReturnValue([ - { - source: 'id5-sync.com', - uids: Array.from({ length: 10 }, (_, i) => ({ id: `${longUid}${i}`, atype: 1 })), - }, - ]); - - const pbjs = installPrebidNpm(); - pbjs.requestBids({ - adUnits: [{ bids: [{ bidder: 'appnexus', params: {} }] }], - } as unknown as RequestBidsArg); - - const cookieValue = document.cookie.match(/(?:^|; )ts-eids=([^;]+)/)?.[1]; - expect(cookieValue).toBeDefined(); - - const persisted = JSON.parse(atob(cookieValue!)) as Array<{ - source: string; - uids: unknown[]; - }>; - expect(persisted.map((eid) => eid.source)).toContain('id5-sync.com'); - const persistedSource = persisted.find((eid) => eid.source === 'id5-sync.com'); - expect(persistedSource!.uids.length).toBeLessThan(10); - - const loggedMessage = debugSpy.mock.calls - .map((call) => call.at(-1)) - .find( - (arg): arg is string => typeof arg === 'string' && arg.includes('ts-eids cookie exceeded') - ); - expect(loggedMessage).toBeDefined(); - expect(loggedMessage).toContain('trimmed uids from sources: id5-sync.com'); - }); - - it('reports a source trimmed and then dropped only as dropped', () => { - const debugSpy = vi.spyOn(log, 'debug').mockImplementation(() => {}); - mockRequestBids.mockImplementation((opts?: { bidsBackHandler?: () => void }) => { - opts?.bidsBackHandler?.(); - }); - // The tail source has several UIDs that are each too large to fit: the - // trimmer first removes UIDs from it, then drops it entirely. - const hugeUid = 'y'.repeat(2500); - mockGetUserIdsAsEids.mockReturnValue([ - { source: 'id5-sync.com', uids: [{ id: 'kept', atype: 1 }] }, - { - source: 'dropped.example', - uids: Array.from({ length: 3 }, (_, i) => ({ id: `${hugeUid}${i}`, atype: 1 })), - }, - ]); - - const pbjs = installPrebidNpm(); - pbjs.requestBids({ - adUnits: [{ bids: [{ bidder: 'appnexus', params: {} }] }], - } as unknown as RequestBidsArg); - - const cookieValue = document.cookie.match(/(?:^|; )ts-eids=([^;]+)/)?.[1]; - expect(cookieValue).toBeDefined(); - const persistedSources = (JSON.parse(atob(cookieValue!)) as Array<{ source: string }>).map( - (eid) => eid.source - ); - expect(persistedSources).toEqual(['id5-sync.com']); - - const loggedMessage = debugSpy.mock.calls - .map((call) => call.at(-1)) - .find( - (arg): arg is string => typeof arg === 'string' && arg.includes('ts-eids cookie exceeded') - ); - expect(loggedMessage).toBeDefined(); - expect(loggedMessage).toContain('dropped sources: dropped.example'); - expect(loggedMessage).not.toContain('trimmed uids from sources'); + expect(document.cookie).not.toMatch(/(?:^|; )ts-eids=/); }); - it('clears ts-eids cookie after bidsBackHandler when no current EIDs remain', () => { + it('expires the retired cookie on installation before any auction', () => { document.cookie = `ts-eids=${btoa(JSON.stringify([{ source: 'sharedid.org', uids: [{ id: 'stale' }] }]))}`; mockRequestBids.mockImplementation((opts?: { bidsBackHandler?: () => void }) => { opts?.bidsBackHandler?.(); diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index a9dcaad1f..6d4e645b3 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -657,6 +657,25 @@ the partner calls the inbound identify or batch-sync APIs. A partner without `api_token` remains available for source-domain lookup, bidstream EIDs, and outbound pull sync, but cannot authenticate to those inbound APIs. +Optional `identity_owner` selects one registered integration whose capture +capability claims the source. Deploy validation, config push and startup reject +missing or non-claiming owners and owned sources with legacy pull enabled. +For owned sources, `api_token` still permits read-only `identify`; `batch-sync` +returns 403 before accessing storage. Ownership also excludes the source from +legacy pull dispatch, completeness and marker fingerprints. Disable capture +without removing ownership when stopping acquisition. Changing owners requires +an operator migration and old-invocation drain, not automatic fallback. + +Provider expiry, writer and opaque revision are optional schema-v1 KV fields. +Known-expired IDs are omitted from auctions and `identify` without cleanup +writes. Unowned legacy pull remains fill-missing-only even for expired records. +Drain metadata-unaware readers and writers before enabling managed capture. +`/auction` body EIDs enrich existing consenting rows conservatively; the retired +`ts-eids` cookie and EID diagnostic response headers are no longer identity +transport. Inbound diagnostic headers remain stripped. See the +[Prebid transport migration](./integrations/prebid.md) and +[Lockr capture gates](./integrations/lockr.md). + **Example**: ```toml diff --git a/docs/guide/integrations/lockr.md b/docs/guide/integrations/lockr.md index ef9e57b0f..ac6c78154 100644 --- a/docs/guide/integrations/lockr.md +++ b/docs/guide/integrations/lockr.md @@ -22,18 +22,14 @@ Lockr is an identity resolution and privacy platform that helps publishers manag ## How It Works -``` -┌──────────────────────────────────────────────────┐ -│ User Visit │ -│ ↓ │ -│ Trusted Server generates EC ID │ -│ ↓ │ -│ Lockr Integration maps to Lockr Identity │ -│ ↓ │ -│ Identity synchronized (with consent) │ -│ ↓ │ -│ Lockr ID added to bid requests (user.ext.eids) │ -└──────────────────────────────────────────────────┘ +```mermaid +flowchart TD + A[Browser loads first-party Lockr SDK] --> B[SDK calls first-party API proxy] + B --> C[Lockr returns provider tokens] + C --> D[Capture validates ID5, consent and source ownership] + D --> E[Original response sent to browser] + E --> F[Eligible EC record updated after send on Fastly] + F --> G[Later auctions read usable IDs from KV] ``` ## Configuration @@ -43,39 +39,48 @@ Add Lockr configuration to `trusted-server.toml`: ```toml [integrations.lockr] enabled = true -api_endpoint = "https://api.lockr.io" -organization_id = "your-org-id" -project_id = "your-project-id" +app_id = "example-app-id" +# Leave capture off until provider, consent, ownership and rollout gates pass. +capture_identity = false +capture_global_withdrawal = false + +[[ec.partners]] +name = "ID5" +source_domain = "id5-sync.com" +openrtb_atype = 1 +bidstream_enabled = true +identity_owner = "lockr" ``` ### Configuration Options -| Field | Type | Required | Description | -| ----------------- | ------- | -------- | --------------------------------------------- | -| `enabled` | boolean | No | Enable/disable integration (default: `false`) | -| `api_endpoint` | string | Yes | Lockr API endpoint URL | -| `organization_id` | string | Yes | Your Lockr organization ID | -| `project_id` | string | Yes | Your Lockr project ID | +| Field | Type | Required | Description | +| --------------------------- | ------- | -------- | ------------------------------------------------------------------------------ | +| `enabled` | boolean | No | Enable/disable the integration (default: `true` when configured) | +| `app_id` | string | Yes | Lockr application ID | +| `capture_identity` | boolean | No | Token-capture kill switch (default: `true`; set `false` for rollout) | +| `capture_global_withdrawal` | boolean | No | Approved global EC withdrawal on application-success revoke (default: `false`) | +| `api_endpoint` | URL | No | Override the Lockr API origin | +| `sdk_url` | URL | No | Override the SDK URL | +| `cache_ttl_seconds` | integer | No | SDK response cache lifetime in seconds (default: 3600) | +| `rewrite_sdk` | boolean | No | Rewrite matching SDK URLs to the first-party route (default: `true`) | +| `origin_override` | URL | No | Origin header override for local or approved domains | + +`rewrite_sdk_host` is deprecated and ignored. An absent Lockr configuration does not register the integration. ### Environment Variables ```bash TRUSTED_SERVER__INTEGRATIONS__LOCKR__ENABLED=true -TRUSTED_SERVER__INTEGRATIONS__LOCKR__API_ENDPOINT=https://api.lockr.io -TRUSTED_SERVER__INTEGRATIONS__LOCKR__ORGANIZATION_ID=your-org-id -TRUSTED_SERVER__INTEGRATIONS__LOCKR__PROJECT_ID=your-project-id +TRUSTED_SERVER__INTEGRATIONS__LOCKR__APP_ID=example-app-id +TRUSTED_SERVER__INTEGRATIONS__LOCKR__CAPTURE_IDENTITY=false ``` ## Features ### Identity Synchronization -Lockr integration automatically syncs Trusted Server EC IDs with Lockr's identity vault: - -1. User visits site (Trusted Server generates EC ID) -2. If GDPR consent granted, sync with Lockr -3. Lockr returns unified identity -4. Identity available for bid requests and analytics +The SDK requests identities from Lockr through the first-party proxy. Trusted Server does not call Lockr to resolve identifiers independently. With approved capture enabled, valid ID5 tokens enrich an existing consenting EC record after the proxy response has been sent on Fastly. Later auctions can use that ID once the KV write is visible. Neither the first auction nor a failed post-send write has a guaranteed repair. ### Privacy Vault @@ -88,27 +93,7 @@ User data is stored securely in Lockr's privacy vault: ### Extended ID (EID) Support -Lockr identities are injected into OpenRTB bid requests as Extended Identifiers: - -```json -{ - "user": { - "ext": { - "eids": [ - { - "source": "lockr.io", - "uids": [ - { - "id": "lockr-user-id-123", - "atype": 1 - } - ] - } - ] - } - } -} -``` +A later server-side auction can include a usable registered ID5 record as an `id5-sync.com` EID with `atype=1`. Trusted Server does not store provider `ext` or OpenRTB `inserter`/`matcher` metadata in KV; browser-supplied provenance remains current-request-only. Other provider token mappings are not enabled by the ID5 parser. ## Use Cases @@ -126,7 +111,7 @@ Lockr identities are injected into OpenRTB bid requests as Extended Identifiers: **Solution**: Lockr's identity graph links devices to a unified user profile. -**Benefit**: Better audience targeting and frequency capping across devices. +**Benefit**: Provider-side linking may support cross-device recognition. Trusted Server does not link EC IDs across browsers or devices through capture. ### 3. Consent-Based Data Sharing @@ -150,55 +135,45 @@ The Lockr integration is implemented in [crates/trusted-server-core/src/integrat **ID Mapping**: -- Maps Trusted Server EC ID → Lockr unified ID -- Cached for performance -- Respects consent status +- The only supported capture mapping is Lockr `id5id` to registered `id5-sync.com` with `atype=1`. +- The EC identity graph retains the source, UID and lifecycle metadata, including known provider expiry. It does not store provider `ext` or additional UIDs. +- Publisher SDK storage remains the SDK's responsibility. Capture does not add an EID cookie. **Consent Validation**: -- Checks consent signals before syncing -- Integrates with CMP (e.g., Didomi) -- Respects user withdrawal of consent +- Request consent and populated body `consentString`, `gppString` and `ccpaString` pass shared acquisition policy. Malformed populated signals fail closed. +- Body signals can restrict but cannot override denied request consent, including opt-out and GPC. +- An explicitly approved successful revoke is separate from acquisition gating. -## Best Practices +### Capture and rollout -### 1. Consent First +Capture inspects only exact POST `page-view`, `generate-tokens`, `refresh-tokens` and `sync-no-hem-ids` routes under `/publisher/app/v2/identityLockr/`. It decodes URL-encoded `advertising_token.universal_uid` from `aimTokens`, verifies an associated `ids.id5id.eid` when present, and converts `identity_expires` milliseconds to Unix seconds. Unknown provider keys, mismatched source or UID, expired tokens and malformed responses do not change stored identity. SDK, settings and unlisted API requests stay on the original proxy path. The proxy still strips publisher `Cookie` and `Authorization` headers. -Always validate consent before initiating Lockr sync: +Original and decoded capture responses have separate 64 KiB limits. Inspection failure leaves upstream response bytes and headers intact. With capture enabled, Fastly sends the SDK response before shared capture persistence against the existing consenting EC record. Missing, unreadable or tombstoned roots cannot be created or revived by acquisition. There is no native resolver, extra EID cookie, first-auction guarantee or durable retry queue. -```rust -if !has_gdpr_consent() { - return; // Skip Lockr sync -} -``` +`capture_identity` defaults to `true` but must be explicitly set to `false` during rollout until sanitized provider fixtures, consent requirements, source ownership, metadata-aware binary drain and an approved publisher test route have been verified. Disabling capture keeps the registered ID5 source claim and already-stored metadata. Do not roll back to a metadata-unaware binary after managed writes without an approved compatibility procedure. Reject conflicting legacy push or pull writers before assigning `identity_owner = "lockr"`. -### 2. Cache Lockr IDs +`capture_global_withdrawal` remains `false` until Lockr confirms that a 2xx `revoke-consent` response with `"success": true` means global withdrawal for this EC identity. Only then may operators opt in: the normal finalizer expires `ts-ec` and the existing-key-only tombstone prevents later enrichment. Empty token arrays, network failures and other responses never imply withdrawal. Overlapping SDK calls have no provider sequence contract; CAS follows observation order. + +Use synthetic or sanitized source/count/status evidence for an approved test route. Verify unchanged proxy response, delayed KV read-back of UID/source/expiry/writer/revision without exporting values, later auction source presence, same-UID expiry extension, expired omission and confirmed opt-out/revoke. No live IDs, consent strings, credentials or query URLs belong in test records. -Cache Lockr identity mappings to reduce API calls: +## Best Practices -- TTL: 24 hours recommended -- Invalidate on consent withdrawal -- Use KV store for persistence +### 1. Consent First -### 3. Monitor Sync Rate +The publisher should gate SDK activity according to its consent policy. Trusted Server also checks EC consent and any populated request-body signals before ID5 capture. A denied token acquisition does not block an approved explicit revocation. -Track Lockr sync success/failure rates: +### 2. Cache Lockr IDs -- Alert on elevated failures -- Monitor API latency -- Track consent decline impact +The SDK response can be cached for the configured `cache_ttl_seconds`. Trusted Server stores accepted provider IDs in the EC identity graph with known provider expiry where supplied; expiry is not an instruction to refresh or call Lockr. Consent withdrawal tombstones the existing EC row. -### 4. Test Identity Flow +### 3. Monitor Sync Rate -Validate end-to-end identity flow: +Track the proxy success rate and source-only counts for accepted or skipped capture, without logging IDs, consent strings, credentials or provider query URLs. Monitor post-send KV failures separately from the SDK's response status. -```bash -# 1. Generate EC ID -curl https://edge.example.com/ +### 4. Test Identity Flow -# 2. Verify Lockr sync (check logs) -# 3. Validate EID in bid request -``` +On an approved synthetic test route, first confirm the SDK request and original response. After post-send persistence becomes visible, inspect the EC row without exporting identifiers and check for the registered EID source in a later auction. A no-token SDK response must not create an identity. ## Troubleshooting @@ -211,10 +186,10 @@ curl https://edge.example.com/ **Solutions**: -- Verify `organization_id` and `project_id` are correct -- Check Lockr API endpoint is reachable -- Ensure GDPR consent is granted -- Review Lockr API credentials +- Verify `app_id`, the SDK URL and the API endpoint configuration +- Check that the first-party proxy reaches the configured Lockr API origin +- Check publisher consent gating and whether `capture_identity` is still disabled for rollout +- Review upstream status codes without logging response tokens or request query URLs ### Missing EID in Bid Requests @@ -224,40 +199,36 @@ curl https://edge.example.com/ **Solutions**: -- Verify OpenRTB request builder includes `user.ext.eids` -- Check integration is registered in IntegrationRegistry -- Ensure `contribute_eids()` method is implemented +- Confirm `id5-sync.com` is registered with `identity_owner = "lockr"` and bidstream enabled +- Confirm the response contained a valid, unexpired ID5 token with allowed consent +- Wait for the Fastly post-send KV update, then inspect the EC row and a later auction +- Check whether the EC ID rotated, the record expired, or withdrawal tombstoned its row ## Performance -### Typical Latency +### Response timing -- Identity sync: 50-100ms -- Cached lookup: <5ms -- API timeout: 500ms (configurable) +The SDK call still waits for the upstream Lockr API response. Fastly runs eligible KV capture updates after sending that response; later auctions see a UID only after the write is visible. The SDK route uses `cache_ttl_seconds` for its own response. No separate Lockr fetch loop or refresh timeout is configured by capture. ### Optimization Tips -- Enable caching for repeat visitors -- Batch sync operations when possible -- Use async/non-blocking sync -- Monitor API quota usage +- Set the SDK cache lifetime to suit the publisher's deployment +- Monitor upstream proxy latency and capture persistence separately +- Avoid assuming an ID will be present in the first auction ## Security Considerations ### API Credentials -- Store Lockr credentials in secret management system -- Rotate credentials periodically -- Never commit credentials to git -- Use environment variables in production +- Keep provider configuration and any credentials in approved secret management +- Never commit credentials to git or forward publisher `Authorization` and `Cookie` headers to Lockr +- Do not log provider query URLs, body tokens, UID values or consent strings ### Data Privacy -- Only sync with explicit user consent -- Implement right to erasure -- Audit data access logs -- Encrypt data in transit (HTTPS) +- Apply publisher consent policy before SDK activity and shared acquisition consent policy before KV writes +- Handle right-to-erasure requests through the established EC deletion process, not a token response +- Use HTTPS and record status/count evidence without exporting private identifiers ## Next Steps diff --git a/docs/guide/integrations/prebid.md b/docs/guide/integrations/prebid.md index 18db20e23..ebdd4179d 100644 --- a/docs/guide/integrations/prebid.md +++ b/docs/guide/integrations/prebid.md @@ -849,12 +849,9 @@ page or auction. When available, the opaque value follows the standard path: 2. The current `/auction` request includes that entry. 3. Trusted Server merges and consent-gates it, then forwards it to Prebid Server as `user.ext.eids`. -4. After the response, the same `/auction` request ingests the body entry into - an EC/KV partner configured with `source_domain = "liveramp.com"`, when that - partner UID is not already stored. -5. The browser also persists the opaque value in the bounded `ts-eids` cookie, - which lets requests without an EID body, such as `GET /_ts/page-bids` and - page navigations, ingest it. +4. Response finalization conservatively stores the body UID on an existing, + consenting EC row for a partner configured with `source_domain = "liveramp.com"`. +5. Later auctions resolve the stored ID from KV. No EID cookie is used. Trusted Server treats the RampID envelope as an opaque string. Do not log, decode, publish, or dimension metrics by the value. Source names, counts, @@ -922,8 +919,8 @@ domain, booleans, source names, counts, and status codes: recording its value. 5. Confirm a controlled Prebid Server request contains that source in `user.ext.eids`. -6. Confirm a later request ingests the source into the configured - `liveramp.com` EC partner. +6. Confirm the body submission persists the source into the existing + `liveramp.com` EC partner, then confirm a later auction resolves it from KV. 7. Repeat with denied consent and confirm the envelope endpoint is not called, `idl_env` is not written, and no LiveRamp EID is forwarded. 8. Repeat on an unapproved origin and confirm identity resolution degrades @@ -934,48 +931,68 @@ path. LiveRamp ATS Direct audience segments, including `_lr_atsDirect` storage and GAM or Prebid segment activation, require a separate integration and are not passed by this implementation. -## Identity Forwarding +## Identity forwarding -Trusted Server uses a **hybrid EID forwarding model** for Prebid-routed auctions: +Prebid.js sends current EIDs from `pbjs.getUserIdsAsEids()` in the `/auction` +body. Trusted Server merges them with usable registered IDs from the current +EC/KV row, applies consent gating, and forwards `user.ext.eids` to Prebid Server. -1. **Current-request EIDs from Prebid.js** are read from `pbjs.getUserIdsAsEids()` in the browser and sent in the `/auction` request body. -2. **Server-side EIDs from the EC/KV identity graph** are resolved on the edge from the current EC ID. -3. Trusted Server **merges and deduplicates** both sets before calling Prebid Server. -4. The merged result is forwarded downstream as `user.ext.eids` in the OpenRTB request. -5. After the response, Trusted Server adds matched partner UIDs that are missing from the EC identity graph, reading the `ts-eids` cookie, then the `/auction` request-body EIDs, then the `sharedId` cookie. Because the body carries the untrimmed EID set, `/auction` ingestion is not limited by the size-capped `ts-eids` cookie. +Registered body sources are staged before provider dispatch. Response finalization +combines them with the separate `sharedId` cookie in one conservative mutation, +using the auction's KV snapshot. No bid and handled provider-error responses still +finalize these updates. Browser input fills absent sources only. It cannot replace +stored IDs, forge lifecycle metadata, create a root, recover a missing root, or +revive a withdrawal tombstone. Unknown sources remain current-request-only. -The `ts-eids` cookie still matters for requests without an EID body, such as `GET /_ts/page-bids` and page navigations, where it is the source for both EID fallback and identity-graph ingestion. Identity-graph EID writes follow the same consent rule as bidstream forwarding: under GDPR, TCF Purpose 1 and Purpose 4 must both be consented. - -This means Prebid auctions get same-request transparency for browser-resolved IDs without giving up the durability of the server-managed EC identity graph. +This persistence requires EC KV support. Fastly supports it; adapters without an +EC identity graph can forward permitted body EIDs but do not persist them. +Publisher and SPA page-bid auctions use KV IDs only. An ID arriving late needs +another `/auction` body submission; there is no page-exit beacon or cookie repair. ### Identity flow ```mermaid -sequenceDiagram - participant B as Browser / Prebid.js - participant T as Trusted Server /auction - participant K as EC + KV identity graph - participant P as Prebid Server - - B->>B: User ID modules resolve EIDs - B->>T: POST /auction\n(adUnits + current-request eids) - T->>K: Resolve EC-backed source-domain IDs - K-->>T: KV-derived EIDs - T->>T: Merge + dedupe client + KV EIDs - T->>T: Apply consent gating - T->>P: OpenRTB request\nuser.ext.eids = merged set - P-->>T: OpenRTB bid response - T-->>B: Auction response - T->>K: Ingest body eids + ts-eids/sharedId cookies\n(TCF Purpose 1 + 4 under GDPR) +flowchart TD + A[Prebid User ID modules resolve EIDs] --> B[POST auction with current body EIDs] + B --> C[Stage registered UID-only updates on EC context] + C --> D[Read current EC row and merge usable KV EIDs] + D --> E[Consent-gated OpenRTB request to Prebid Server] + E --> F[Bid, no-bid, or handled error response] + F --> G[Finalize body plus sharedId on existing consenting root] + G --> H[Send response] + G --> I[Later auctions read stored IDs from KV] ``` +### Cookie and diagnostic retirement + +New servers ignore `ts-eids`, including writes by cached older bundles. No TS EID +cookie replaces it. The transition JS bundle expires the old host-only `Path=/` +cookie once installed. Remove that cleanup only after writing bundles have aged +past both the old cookie's one-day max-age and the deployment cache lifetime. +The server does not emit unconditional cleanup `Set-Cookie` headers. + +`x-ts-eids` and `x-ts-eids-truncated` are no longer emitted and remain stripped +from inbound requests. Check external diagnostic consumers before rollout. +`/_ts/admin/eids` remains authenticated and returns a local `410 Gone` with +`Cache-Control: no-store`, never publisher fallback. Use `/_ts/admin/ec` to inspect +persisted state on adapters with EC KV support. EC and consent headers remain. + +Deploy metadata-preserving readers and writers everywhere and drain old invocations +before activating body writer/revision metadata. Old binaries can erase metadata +or ignore expiry; rolling back to them after activation is unsafe. A passing test +on a new binary is not deployment-drain evidence. + ### Merge and deduplication rules - Client-request EIDs and KV-resolved EIDs are merged by `source` - UIDs are deduplicated by `source + id` - If the same UID appears in both places, it is sent only once downstream - Distinct UIDs under the same source are preserved -- Consent gating is applied to the **merged** set before forwarding +- Supplied `inserter`, `matcher`, and integer `mm` remain current-request provenance, + including empty `matcher` and zero `mm`; they are not stored in KV +- UID `ext` remains current-request-only; existing UID merge precedence is unchanged +- Known-expired KV IDs are omitted, including an identical body UID for that source +- Consent gating is applied to the merged set before forwarding ### What reaches Prebid Server @@ -988,7 +1005,7 @@ The downstream Prebid Server request includes: In practice, this gives operators both: - **same-request identity transparency** for Prebid User ID Module output, and -- **future-request continuity** through cookie ingestion and KV-backed partner resolution. +- future-request continuity through body persistence and KV-backed partner resolution. ## Endpoints diff --git a/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md b/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md new file mode 100644 index 000000000..718cd88d5 --- /dev/null +++ b/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md @@ -0,0 +1,115 @@ +# Server-side identity foundation and Lockr capture implementation plan + +**Status:** Milestones 1–5 implemented in the draft change. Local verification is recorded below. Provider activation and deployment compatibility remain gated; this is not live acceptance. + +**Specs:** [Foundation](../specs/2026-10-09-server-side-identity-foundation-design.md), [Lockr capture](../specs/2026-10-08-lockr-identity-capture-design.md), [KV snapshot and recovery](../specs/2026-07-10-kv-eid-request-snapshot-ec-recovery-design.md). The [existing LiveRamp browser design](../specs/2026-08-21-liveramp-integration-design.md) does not approve a native resolver. + +**Goal:** Persist registered browser auction EIDs and trusted Lockr proxy observations on an existing, consenting EC row, retain expiry and writer metadata through every mutation, and expose only usable records to later auctions. Keep the response intact and send it before capture writes on Fastly. No extra EID cookie, root creation by acquisition, or new provider fetch loop. + +```mermaid +flowchart TD + A[Record compatibility and expiry-aware readers] --> B[Ownership and mutation policy] + B --> C[Auction body ingestion] + C --> D[Retire EID cookie and diagnostic headers] + B --> E[Bounded capture and Fastly after-send effects] + D --> F[Lockr capture consumer] + E --> F + F --> G[Approved fixture and deployment gates] +``` + +## Boundaries and starting code (pre-implementation baseline) + +- Core EC owns record and mutation policy: `ec/kv_types.rs` stores schema-v1 `KvPartnerId { uid }`; `ec/kv.rs` has browser `sync_eid_cookie_updates_from_snapshot` with one CAS plus one conflict read, and authoritative `apply_partner_id_updates`, `upsert_partner_ids_from_snapshot`, `upsert_partner_id`, `upsert_partner_id_if_exists`. `ec/batch_sync.rs` calls the last method; `ec/pull_sync.rs` aggregates fill-missing responses. `ec/generation.rs`, `ec/finalize.rs` and `ec/admin.rs` participate in creation, recovery, withdrawal and administrative reads. `ec/identify.rs` separately returns partner UIDs without using `resolve_partner_ids`; include it in expiry-aware reader coverage. Audit every write/serialize site before enabling metadata. Tombstoning deliberately removes identity payload; ordinary mutations must not. +- `ec/registry.rs` builds `PartnerRegistry` from `settings.rs` `EcPartner`; `config.rs::validate_settings_for_deploy` and `validate_settings_for_runtime` are the shared CLI push/validate and startup validation boundaries. `integrations/registry.rs::IntegrationRegistration` and `IntegrationRegistry::with_plan` own capability discovery. Do not introduce a parallel vendor list. +- `auction/endpoints.rs::handle_auction` parses body EIDs, falls back to `ts-eids`, loads an `EcKvSnapshot` and hands it to finalization; its orchestration error path currently returns `Err`. `ec/finalize.rs::ec_finalize_response` currently ingests cookie EIDs plus `sharedId`. `ec/prebid_eids.rs` already normalizes EIDs into source-keyed updates; `ec/eids.rs::resolve_partner_ids` supplies registered KV IDs. `openrtb.rs::Eid` currently has only `source` and `uids`. +- `integrations/lockr.rs::handle_api_proxy` buffers POST bytes once, forwards them and returns the upstream response uninspected. It currently logs a full target URL, including query, which must be redacted before captured traffic is enabled. `integrations/mod.rs::collect_response_bounded` is destructive on overflow and cannot be used for capture pass-through. Fastly `app.rs::EcFinalizeState` carries request context, while `main.rs::edgezero_main` removes extensions before conversion and sends before `run_edgezero_pull_sync_after_send`. +- `auction/formats.rs` emits EID diagnostic headers; `ec/finalize.rs` clears them on withdrawal; `constants.rs::INTERNAL_HEADERS` strips inbound variants. The JS writer is `trusted-server-js/lib/src/integrations/prebid/index.ts::syncPrebidEidsCookie`; `ec/admin.rs` has an EID-cookie diagnostic endpoint. `cookies/template_cache_policy.rs` and `publisher.rs` still recognize the cookie. Inventory these consumers before deleting a constant or admin diagnostic. See also `adapter-fastly/src/app.rs` cookie extraction and `auction/types.rs` stale cookie documentation. + +## Check discipline + +Each numbered milestone contains smaller red/green steps. Finish and check one step before starting the next; do not change every listed file at once. Add the regression, observe failure, implement the step, then rerun it and adjacent tests. Compatibility tests for already-correct behavior must stay green rather than be forced to fail. + +Use these focused commands from the repository root, replacing the example filter with the affected module or regression: + +```bash +cargo test-fastly ec::kv_types::tests +cargo test-fastly ec::kv::tests +cargo test-fastly auction::endpoints::tests +cargo test-fastly integrations::lockr::tests +cargo test-axum +HOST_TARGET="$(rustc -vV | awk '/host:/ { print $2 }')" +cargo test --package trusted-server-cli --target "$HOST_TARGET" +(cd crates/trusted-server-js/lib && npx vitest run test/integrations/prebid/index.test.ts) +``` + +`scripts/test-cli.sh` takes a host target, not a test filter, and runs the full CLI suite plus browser fixtures. Reserve it for the final integration gate. Verify each selected filter actually executes its intended tests. A compile-only check does not prove persistence. Reuse passing evidence while relevant source, fixtures, config and environment remain unchanged. Run broader suites and target-matched lint at stable integration checkpoints, not after each edit. + +### 1. Land compatible lifecycle records and reads, without managed writes + +**Files:** `crates/trusted-server-core/src/ec/{mod.rs,kv_types.rs,kv.rs,eids.rs,identify.rs,admin.rs,prebid_eids.rs,finalize.rs,generation.rs,batch_sync.rs,pull_sync.rs}`, `crates/trusted-server-core/src/auction/endpoints.rs`, plus existing tests in those files. Change an existing writer only where the audit finds UID-only reconstruction or equality, stale metadata inherited by a replacement UID, or a mutation-policy bypass. Keep schema version 1 and optional/defaulted `expires_at`, `writer`, `revision` fields. A legacy UID-only record has unknown expiry and no inferred provider provenance. Do not implement `updated_at`, `invalidated_at`, attempt state or a refresh scheduler without a current consumer; reserve invalidation semantics for the later provider delivery. + +- Red: decode old schema-v1 fixtures; reject unsupported schema; serialize a record with metadata across browser same-UID/different-UID paths, batch push, single partner push, bulk pull, generation/recovery and admin-related round trips. Demonstrate the current UID-only replacement strips metadata. Add equal-UID/different-expiry and equal-UID/equal-metadata cases. Check withdrawal removes payload, not a partial record. +- Green: preserve the existing record where the UID is unchanged, and give legacy authoritative UID replacements an explicit policy that does not inherit a different UID's old expiry. Source ownership will later reject conflicting writes. Assign a non-reused opaque revision for every material managed record change; do not bump on no-op. Test a change that later restores the earlier UID/expiry: it must not restore the earlier revision. Store a known expiry in seconds, reject already-expired `Issued`, retain it when a same-UID observation omits expiry. Do not mistake the existing `KvConsent.updated` for provider freshness. +- Red then green: centralize record usability and test `now < expires_at` versus `now >= expires_at` through `ec/eids.rs::resolve_partner_ids`, `ec/identify.rs` and publisher, `/auction`, SPA `/_ts/page-bids` callers. Obtain usable time through `ec::checked_current_timestamp`, not the epoch-zero failure fallback in `current_timestamp`. Test unavailable time explicitly; a known-expiry record cannot be declared usable without a usable clock. Keep unknown-expiry compatibility unchanged. Suppress an auction-body UID identical to the unusable stored UID before merge, using normalized source matching; a different valid body UID remains current-request-only. No cleanup write on reads. Admin inspection may show retained expired records and their metadata; it is not a delivery API. +- Check: filtered `cargo test-fastly` record, CAS, resolution, publisher and auction tests. Read back serialized KV state in storage tests. **Gate:** do not activate metadata-producing writers until all deployed readers and _all_ old writers have drained. Old binaries discard unknown fields and old readers forward expired UIDs. A passing new-binary test is not a deployment-compatibility proof. + +### 2. Validate source owners and separate mutation classes + +**Files:** `crates/trusted-server-core/src/settings.rs`, `config.rs`, `ec/{mod.rs,registry.rs,kv.rs,pull_sync.rs,pull_sync_marker.rs,batch_sync.rs}`, `integrations/{registry.rs,mod.rs}`, `crates/trusted-server-cli/src/run.rs` validation tests, `trusted-server.example.toml`; config, registry and KV tests nearby. Add `ec/identity.rs` for common outcomes, trusted update types and the shared acquisition service, and `integrations/identity.rs` for capability contracts and later bounded inspection. These are proposed new paths, not existing modules. Add `identity_owner` to source policy and a capture descriptor with finite source claims to integration registration. Keep disabled acquisition descriptors discoverable, so a `capture_identity = false` kill switch does not invalidate config. No production resolver, named identifier intake, synthetic resolver HTTP protocol or rate-limit state now. + +- Red: duplicate/missing/non-claiming owner, disabled owner descriptor, owner with enabled pull or push API conflict, unauthorized integration claiming a source and forged browser metadata. Exercise all three boundaries: `ts config validate`, `ts config push`, and startup `validate_settings_for_runtime`/`IntegrationRegistry::with_plan`. Ensure deploy validation works on unresolved secret references. Existing unowned sources retain legacy push/pull behavior; owned sources must reject legacy authoritative push and skip legacy pull completeness and dispatch, not merely suppress one write after making an outbound request. Resolve the legacy-write configuration decision below before implementing its acceptance rules. `api_token` authenticates read-only `identify` as well as writable `batch-sync`; token presence alone does not prove a conflicting writer. Tests must cover the approved read/write policy without accidentally removing identify access. Update the effective pull-partner set used by completeness, dispatch and `pull_sync_marker.rs` fingerprinting together; an ownership migration must not leave a stale completeness cookie suppressing an eligible legacy pull. +- Green: descriptor claims and runtime registry are one source of truth; validate source policy against claims in the shared deploy/runtime validation functions. Shared service validates trusted origin, registered source, owner, UID length/source count, consent and full EC-ID-bound snapshot. Implement the capture-used `Issued`, `NoChange` and approved withdrawal actions; reserve the remaining outcome contracts without an unused active resolver. Test expiry again at mutation time, because a token can expire between capture and post-send persistence. Browser update cannot set authority or lifecycle metadata. Keep `ec/kv.rs` browser one-CAS/one-conflict-read/no-retry policy distinct from authoritative bounded CAS/re-merge. Authoritative replacement preserves unrelated sources and never creates a missing, failed or tombstoned root. Same UID with changed expiry is a write; unchanged UID and lifecycle metadata is not. A stale result tied to a prior revision must be rejected when revision-conditional mutation becomes necessary; do not expose an unused resolver callback merely to test an imagined provider. +- Check: filtered `cargo test-fastly` config, registry, push/pull and KV tests; host CLI validation tests. Inspect log assertions for redaction. + +### 3. Persist browser body once, then retire cookie/header transport + +**Files:** `crates/trusted-server-core/src/{auction/endpoints.rs,auction/formats.rs,auction/types.rs,openrtb.rs,ec/prebid_eids.rs,ec/finalize.rs,ec/admin.rs,ec/eids.rs,constants.rs,cookies/template_cache_policy.rs,publisher.rs}`, `crates/trusted-server-adapter-fastly/src/{app.rs,main.rs}`, `crates/trusted-server-js/lib/src/integrations/prebid/index.ts` and its existing tests; update operator docs at `docs/guide/integrations/prebid.md` and `docs/guide/configuration.md` for changed transport. Limit other adapters to compile/behavior parity; they currently have no EC KV graph. + +- Red: registered body EIDs enrich an existing root even on no-bid or handled auction-provider error; unknown sources remain current-request-only; no-token/consent-denied/missing/failed/tombstone roots do not write or recover. Prove one combined browser finalization attempt with `sharedId`, using the auction snapshot rather than a second unconditional KV read. Directly reuse the EID-to-update normalization in `ec/prebid_eids.rs`, not a fake cookie. Stage normalized updates in `EcContext` or a typed response effect before dispatch so `Err` paths retain them through the adapter's error conversion; inspect `app.rs` response-finalization behavior rather than assuming an error response keeps extensions. +- Green: keep normal EC finalization timing and publisher-only orphan recovery, including no seeding from the retired cookie. Extend `Eid` with optional string `inserter`/`matcher` and an OpenRTB-compatible integer `mm`, omitted only when absent. Update both the manual parser in `parse_client_auction_eids` and `merge_auction_eids`, not only serde derives. Body-supplied provenance wins over missing KV provenance; preserve empty `matcher` and supplied zero-valued `mm`. Keep existing UID merge precedence and retain UID `ext` for the current auction. Update affected struct constructors and serialization tests. KV-derived EIDs remain registry-`atype` only. Preserve source/UID caps and consent gates. Filter the matching known-expired body UID before merging. +- Only after body tests pass: remove `/auction` cookie fallback, server cookie ingest, JS writer, header emission and old cookie-dependent diagnostics. If retiring an admin route, keep it locally denied with its existing authentication/no-store behavior; do not remove its route and forward admin credentials to publisher fallback. Apply that protection to the Axum and Spin routes as well as Fastly. Keep `x-ts-eids` and `x-ts-eids-truncated` on the inbound strip list; keep EC/consent headers. Keep one transition JS release that expires the old host-only `Path=/` cookie, with a named removal condition: cached bundles that write it have aged beyond the existing one-day max-age and deployment cache lifetime. New servers ignore old bundles' writes. No unconditional server `Set-Cookie` cleanup. Agree an admin diagnostic replacement/removal and identify external header consumers before this deletion. +- Check: filtered `cargo test-fastly` auction/finalize/admin/publisher tests, JS Prebid tests, `rg -n 'ts-eids|x-ts-eids'` review of remaining intentional compatibility references. Verify before/after request cookies and response cache privacy through Fastly tests. + +### 4. Add bounded shared capture and one Fastly post-send executor + +**Files:** `crates/trusted-server-core/src/integrations/{identity.rs,registry.rs,mod.rs}`, `crates/trusted-server-core/src/ec/{identity.rs,mod.rs,kv.rs,finalize.rs}`, `crates/trusted-server-adapter-fastly/src/{app.rs,main.rs}`; tests beside the boundary. Start with a synthetic capture fixture, not Lockr. A capability declares exact method/path/source claims and interprets a bounded decoded response. The framework associates it with the actual proxied upstream result; inbound client headers/JSON cannot select trusted acquisition. + +- Red: 64 KiB original **and separate** 64 KiB decoded JSON cap at exact boundary and +1, compressed expansion, unsupported encoding, decoder failure, malformed capture, streamed oversized response with original prefix replayed, absent or misleading `Content-Length`, and byte-for-byte response/header preservation including encoding and length. Include an oversized single stream chunk and upstream read failure; replay consumed bytes and preserve the underlying stream error rather than manufacturing a capture error response. `collect_response_bounded` cannot be reused because it consumes oversize streams. Bound added capture buffers and decoder working memory; do not copy an oversized incoming chunk into an inspection buffer. Account for retained stream chunks and parser overhead too. No full decompression first. +- Green: inspect only selected proxy responses. A capture failure is a no-op, not a 502 or rewritten body. Stage cloneable effects holding full EC ID, source-validated trusted origin and `EcKvSnapshot`; consume them once before response conversion drops extensions. Before send, expose an approved captured withdrawal to normal finalization so it expires `ts-ec`, removes identity headers and suppresses enrichment. After send, run its existing-key-only tombstone work unless request finalization already handled that identity. A tombstone-only after-send handler cannot change the cookie already sent to the browser. Keep one shared Fastly executor for captured work, with no Lockr-specific branch. Apply withdrawal before capture enrichment and hand **persisted** resulting snapshots to legacy pull sync. `NotRead` may fetch after send; missing and failed never authorize creation. CAS conflicts may refresh, but no unbounded retry or zero-read claim. No-token queues no mutation. If finalization already tombstoned the root, suppress a second tombstone; do not turn a provider timeout into withdrawal. +- Check: filtered `cargo test-fastly` registry/capture/KV/entrypoint tests with a recording send and KV backend. Prove send precedes capture write, no response-byte changes, full-ID association across rotation, failed-write snapshot stays failed, no-token no capture KV work and resulting snapshot changes pull eligibility. Exercise settings-snapshot/reload, finalization-failure and fallback send branches. Failed or withdrawn state must suppress subsequent pull disclosure, not reuse the original live snapshot. Test Axum/Cloudflare/Spin behavior only where an EC KV graph exists; document before-send or unsupported capture on adapters lacking a post-send hook rather than promising persistence. + +### 5. Implement Lockr capture behind explicit approval gates + +**Files:** `crates/trusted-server-core/src/integrations/lockr.rs`, shared acquisition consent handling in `crates/trusted-server-core/src/{consent/mod.rs,consent/types.rs,ec/consent.rs,ec/identity.rs}`, its tests and sanitized fixtures under its existing test module or a dedicated `crates/trusted-server-core/tests/fixtures/lockr/` directory; `trusted-server.example.toml` and `docs/guide/integrations/lockr.md` for confirmed configuration and operator behavior. Register capture only, not a resolver. Add default-true `capture_identity` kill switch without removing the ownership descriptor. Keep SDK, settings and unlisted paths on the existing proxy path; inspect exact POST `page-view`, `generate-tokens`, `refresh-tokens`, `sync-no-hem-ids`, `revoke-consent` paths. Reuse the already buffered POST bytes for request consent without changing forwarding. + +- Fixture gate before enabling each mapping: sanitized SDK `tokenMappings` and `tokenSourceMappings` **and** response examples prove token shape, key/source/type, expiry unit and consent contract. Start with ID5, using `id5id`, `id5-sync.com` and `atype=1`, after its fixture and consent gates pass; decode URL-encoded JSON `advertising_token.universal_uid`, associate `ids..eid` when present, reject mismatched source/oversized UID, convert `identity_expires` milliseconds to seconds and reject expired issue. Do not infer LiveRamp, UID2, EUID, First ID, Yahoo, Criteo or Epsilon token decoding from ID5; leave each unmapped until its own fixture passes. Panorama source/type remains disabled until Lockr confirms it. No provider provenance or `ext` is stored in KV. +- Red: exact route and method selection, aimTokens-only and associated EID, unknown key, source mismatch, non-2xx/malformed/no-token no invalidation, missing/extended expiry, repeat equal observation no write, original/decoded overflow with unchanged proxy response. Consent tests include populated opt-out, malformed non-empty strings, empty optional strings, cookie/body conflict, GPC and otherwise permissive/non-regulated or proxy-mode request policy. Map Lockr's buffered `gppString`, `consentString` and `ccpaString` into named shared consent inputs. Reuse shared decoders and request jurisdiction/expiration policy. Require explicit parse success for populated signals; the existing decoder's absent decoded value or a permissive jurisdiction must not turn malformed body consent into permission. Empty fields impose no positive permission, and body consent can restrict but never override denied request consent. Keep this acquisition check separate from approved explicit withdrawal handling. Never log UID, query-bearing URL, consent string or token; audit shared consent display and error logs as well as Lockr logs. Remove the current full-target-URL logging/error attachment in `lockr.rs` on captured paths, and audit shared error formatting. +- Withdrawal is gated separately: only after Lockr confirms 2xx application-success semantics **and** whether revoke is global EC withdrawal or per-source invalidation. Test revoke despite denied acquisition consent, finalization precedence, cookie expiry and existing-key-only tombstone if global scope is approved. Until then, leave revoke capture disabled and let ordinary request consent withdrawal continue; never infer withdrawal from empty tokens or a network error. Clarify ordering if overlapping SDK calls can return older tokens after newer ones; absent a provider sequence, CAS order is observation order only. +- Check: filtered `cargo test-fastly` Lockr parser/proxy/post-send tests with synthetic responses; no live IDs in fixtures. Then run approved publisher test-route acceptance: consent, unchanged original response, delayed KV readback of UID/source/expiry/writer/revision without exporting values, later bidder payload source presence, same-UID extension, known-expired omission including matching body UID, negative consent and approved revoke. Record statuses, counts and booleans only. Capture stays off for a source until its fixture, consent, ownership and compatibility gates pass. + +## Release and rollback + +1. Ship optional schema-v1 metadata-aware readers and **every** metadata-preserving writer first, with managed acquisition off. Verify mixed-record fixtures. Obtain deployment propagation and old-invocation drain evidence; old UID-only readers ignore expiry and old writers erase metadata. Confirm operator procedure before the first managed write. +2. Treat browser writer/revision output as metadata activation too; complete the compatibility gate before enabling any path that produces lifecycle metadata. Activate body persistence in a coordinated server release; prove body/no-cookie behavior before the transition JS writer removal. Check diagnostic consumers before header retirement. Roll back the JS transition only with a server that still understands body submission; old cached JS may write the ignored cookie for its remaining lifetime. +3. Deploy synthetic capture framework inactive and verify Fastly send ordering. Although the Lockr setting defaults to true, explicitly deploy `capture_identity = false` until approval and compatibility gates pass; ownership validation must still work. On an approved route, add one confirmed Lockr source with owner and `atype` after resolving conflicting legacy writes or pull configuration. Enable capture, measure source-only counts, KV readback and later auction visibility. Canary first; do not claim first-auction guarantees or automatic retry. +4. Safe immediate mitigation after activation is `capture_identity = false` or removing ownership through an **approved migration**; disabling capture alone does not erase stored expiry. Rolling back to a UID-only binary is unsafe until managed writes stop and expiry-aware read/metadata compatibility is explicitly resolved. Do not automatically transfer owner on module failure. DSR deletion and global consent withdrawal remain authoritative over acquired records. +5. At intermediate checkpoints, run only suites affected by the completed milestone. At the final stable implementation checkpoint, run `cargo fmt --all -- --check`, `cargo test-fastly`, `cargo test-fastly-reuse`, `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin`, `./scripts/test-cli.sh`, JS tests/build/format, integration parity and affected target-matched clippy aliases per `AGENTS.md`. Reuse unchanged passing results rather than repeat this matrix at every checkpoint. Confirm CI and approved live-route evidence separately. Core tests are not a substitute for an observed post-send KV write and later auction. + +## Decisions and remaining approval gates + +- The foundation requests revision-conditional invalidation tests but defers `invalidated_at`, invalidation storage and active resolver to later work. Does the first capture delivery need an invalidation marker at all? This plan avoids dead state and blocks source-specific revoke until scope is approved. Native LiveRamp and its intake/refresh questions remain deferred in [#1245](https://github.com/IABTechLab/trusted-server/issues/1245). Cross-browser linkage needs a separate approved design; #1245 is not authorization to build it. +- Implemented ownership policy: reject owned sources with legacy pull enabled. Keep `api_token` read-only for `identify`; reject owned-source batch writes with 403 before limiter/storage. Ownership is validated through deploy/config-push/runtime/startup paths, even with acquisition disabled. Operator ownership migration and invocation drain still need confirmation. +- Implemented `/_ts/admin/eids` retirement as authenticated local 410/no-store/nosniff on all four adapters; unknown admin subpaths remain locally denied. Cookie readers/writer and diagnostic emission are retired; inbound strips remain. External diagnostic consumers must be checked before deployment. +- Confirm deployment drain/rollback for mixed binary versions, external uses of `x-ts-eids` headers, Lockr fixtures and each enabled provider mapping, Lockr consent requirements and revoke semantics/scope. Clarify overlapping SDK response ordering. None is settled by local tests. + +## Local implementation verification (2026-10-09) + +- `cargo test-fastly`: 219 adapter tests and 2,924 core tests passed, with six core tests ignored; shared-crate and doc tests also passed. `cargo test-fastly-reuse`: 219 passed. +- `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin`: passed, including authenticated retirement and fallback protection routes. These adapters do not claim post-send KV capture persistence. +- `./scripts/test-cli.sh`: host CLI suite and all four required ignored browser-fixture filters passed. Native `template_build_digest`: ten tests passed. +- Full JS Vitest suite: 48 files, 1,186 tests passed, no type errors. Focused Prebid transport suite: 288 passed. +- Target-matched Clippy passed for Fastly, Axum, Cloudflare native/WASM, Spin native/WASM, CLI and codegen, plus native build-digest tests. Fastly, Axum and Cloudflare development builds passed. +- Recording tests use the actual shared send executor: send sees unchanged KV; factory runs after send; persisted lifecycle state matches the context snapshot consumed by later pull. No effects never opens storage; unavailable storage sends once and leaves KV/context unchanged. Transport tests preserve response parts/bytes and stream errors across independent wire/decoded caps and gzip failures. +- Ownership/config, restrictive consent, missing/failed/withdrawn/full-ID isolation, CAS/no-op/ABA/expiry, browser one-finalization, expiry bypass and withdrawal tests passed. Independent bounded source review found no additional verified privacy/concurrency defect. + +Not verified: real provider acceptance, sanitized provider-approved mappings/fixtures, consent/revoke scope approval, overlapping response ordering, deployment propagation/drain, remote CI or external diagnostic consumers. Synthetic fixtures are not provider approval. Keep both Lockr capture switches explicitly false until their independent gates pass. Native LiveRamp, refresh/invalidation scheduling, durable retry and cross-browser linking remain deferred. diff --git a/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md b/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md index a51b336ce..70f0016d3 100644 --- a/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md +++ b/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md @@ -1,57 +1,66 @@ # Proxy-side identity capture for lockr-issued IDs -Status: Design proposed, awaiting engineering review. +Status: ID5-only capture implemented in the draft change, awaiting code review. +Provider fixtures, consent approval, revocation semantics and live acceptance +remain gated; rollout configuration explicitly disables capture. Depends on the +[server-side identity foundation](2026-10-09-server-side-identity-foundation-design.md). +See the [implementation plan and verification](../plans/2026-10-09-server-side-identity-foundation-and-lockr.md). Issue: [#1246](https://github.com/IABTechLab/trusted-server/issues/1246). -Related: [2026-07-10 KV EID request snapshot and EC recovery](2026-07-10-kv-eid-request-snapshot-ec-recovery-design.md) -defines the snapshot and post-send primitives this design reuses. +The foundation owns capability registration, common identity outcomes, source +ownership, lifecycle records, consent/mutation policy and adapter execution. +It also owns auction-body ingestion and retirement of `ts-eids` and the EID +diagnostic headers. This spec defines Lockr's capture behavior using those +contracts; it does not introduce a separate identity service or KV writer. -## 1. Purpose +Related: [KV EID request snapshot and EC recovery](2026-07-10-kv-eid-request-snapshot-ec-recovery-design.md) +defines existing snapshot and root safety rules retained by the foundation. -The lockr SDK obtains partner IDs (ID5 today; RampID, UID2, EUID, FirstID, -Yahoo ConnectID, PanoramaID, Criteo and Epsilon by configuration) and hands -them to Prebid through `pbjs.setConfig({ortb2: {user: {ext: {eids}}}})`. -Trusted Server collects auction EIDs from `pbjs.getUserIdsAsEids()`, which only -reports the Prebid User ID module. The lockr IDs therefore reach client-side -bidders and never reach the server-side auction, the `ts-eids` cookie, or the -EC identity graph. This was measured on the production publisher route on -2026-10-06 (Safari) and reconfirmed on 2026-10-08 (Chrome, direct versus TS -matched arm): six captured `/auction` payloads contained no `id5-sync.com` -entry while the same page carried the ID in `ortb2.user.ext.eids`. +## 1. Purpose and agreed direction -The fix proposed in the issue thread (merge configured `ortb2` EIDs into the -JavaScript collector) corrects the symptom but keeps three dependencies this -design removes: +The lockr SDK obtains partner IDs, ID5 today and other providers by +configuration, and hands them to Prebid through +`pbjs.setConfig({ortb2: {user: {ext: {eids}}}})`. Trusted Server collects auction +EIDs from `pbjs.getUserIdsAsEids()`, which only reports the Prebid User ID +module. The lockr IDs therefore reach client-side bidders but do not reach the +server-side auction or the EC identity graph. This was measured on the +production publisher route on 2026-10-06 in Safari and reconfirmed on +2026-10-08 in Chrome, direct versus TS matched arm. Six captured `/auction` +payloads contained no `id5-sync.com` entry while the same page carried the ID +in `ortb2.user.ext.eids`. -1. The ID still travels through Prebid configuration. -2. The ID still depends on the tsjs prebid module running in the page. -3. The `ts-eids` cookie is written in `bidsBackHandler`, after the first - auction request has left, and KV ingestion happens on a later eligible - navigation. +Capture those IDs in the first-party proxy response from lockr's API. Submit +common identity outcomes to the shared service, which persists them under the +visitor's existing EC ID after sending the response on Fastly. -This design captures the IDs where Trusted Server already sees them, in the -first-party proxy response from lockr's API, and writes them to a server-owned -first-party cookie immediately and to KV after the response has been sent. +The lockr SDK still runs in the browser. Capture does not depend on Prebid +configuration or the TS JavaScript EID collector. There is no `ts-eids-s` or +replacement EID cookie. `ts-ec` remains the browser's identity pointer. + +Delayed availability is accepted: a later auction sees an ID only after the +deferred write completes and its KV read can see the record. The first auction +may miss it. There is no cookie fallback or durable capture retry queue. ## 2. Observed lockr flow -All lockr traffic on a TS page goes through `/integrations/lockr/api/*`, proxied -by `handle_api_proxy` in `crates/trusted-server-core/src/integrations/lockr.rs` -to `https://identity.loc.kr`. The trust-server SDK build hard-codes +All lockr API traffic on a TS page goes through `/integrations/lockr/api/*`, +proxied by `handle_api_proxy` in +`crates/trusted-server-core/src/integrations/lockr.rs` to +`https://identity.loc.kr`. The trust-server SDK build hard-codes `host: "/integrations/lockr/api"`, so no client shim is involved in routing. Measured call sequence after the publisher's consent gate opens: -| Order | Request | Carries | -| ----- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| 1 | `GET /integrations/lockr/sdk` | SDK body, 118,542 bytes | -| 2 | `POST .../publisher/app/v2/identityLockr/settings` | Request body `{appID}`. Response: publisher flags, `rampJSClientId`, client IP and geo, `hashedUserAgent`, `vvFlag` | -| 3 | `POST .../publisher/app/v2/identityLockr/page-view` | Request body: `LTID`, existing `tokens` keyed by provider, `noGenerate`, consent strings, URL, referrer. Response on first issue: `aimTokens[]` and `ids..eid` | -| 4 | `POST .../refresh-tokens`, `.../generate-tokens`, `.../sync-no-hem-ids` | Later token updates, same shapes | -| 5 | `POST .../revoke-consent` | `LTID` and `optOutReason` | +| Order | Request | Carries | +| ----- | ----------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | +| 1 | `GET /integrations/lockr/sdk` | SDK body, 118,542 bytes | +| 2 | `POST .../publisher/app/v2/identityLockr/settings` | Request `{appID}`; response publisher settings | +| 3 | `POST .../publisher/app/v2/identityLockr/page-view` | Request `LTID`, existing provider tokens, consent strings and page context; first-issue response `aimTokens[]` and `ids..eid` | +| 4 | `POST .../refresh-tokens`, `.../generate-tokens`, `.../sync-no-hem-ids` | Later token updates | +| 5 | `POST .../revoke-consent` | `LTID` and `optOutReason` | -Measured `page-view` response on first issue (identifier values elided): +Measured `page-view` response on first issue, with identifier values elided: ```json { @@ -83,298 +92,209 @@ Measured `page-view` response on first issue (identifier values elided): } ``` -The response is 603 bytes. The SDK then writes cookie and localStorage `id5id`, -and exposes the EID to Prebid with `inserter: "Lockr-For-Publishers"`, -`matcher: ""`, `mm: 2`. - -The request arrives at the proxy as a same-origin XHR, so it carries the -`ts-ec`, `__gpp`, `__gpp_sid` and `us_privacy` cookies, and its body carries -lockr's own `gppString`, `consentString` and `ccpaString`. The integration route -already builds an `EcContext` from the request and attaches `EcFinalizeState` -to the response (`dispatch_fallback` in -`crates/trusted-server-adapter-fastly/src/app.rs`). The Fastly entry point -already runs post-send work after `send_edgezero_response` (pull sync in -`crates/trusted-server-adapter-fastly/src/main.rs`). - -Nothing reads the proxied response body today. `handle_api_proxy` returns the -upstream response as-is. - -## 3. Requirements - -1. Lockr-issued IDs reach the server-side auction without the Prebid User ID - module and without any client-side JavaScript module. -2. The IDs are written to a first-party cookie by the server on the same - response that delivered them, and to the EC identity graph without a KV - write before response bytes are sent. -3. The IDs are available to the next auction on the same page load through the - cookie, and to later navigations through KV, with the cookie as fallback. -4. Token refresh and consent revocation from lockr are reflected without a new - mechanism. -5. OpenRTB 2.6 EID provenance (`inserter`, `matcher`, `mm`) survives into the - auction payload. -6. Existing consent gating, size caps and fail-closed KV behavior are - unchanged. - -Out of scope: changing the JavaScript EID collector, server-to-server calls to -lockr, direct proxies for ID5 or LiveRamp, and storing provenance metadata in -KV. - -## 4. Design - -### 4.1 Capture hook on the integration proxy - -Add an optional trait to the integration registry: - -```rust -/// Identity tokens an integration proxy observed in an upstream response. -pub struct CapturedEid { - /// OpenRTB `source` domain, for example `id5-sync.com`. - pub source: String, - pub uid: String, - pub atype: Option, - /// Unix seconds; absent when the provider gave no expiry. - pub expires_at: Option, - pub inserter: Option, - pub matcher: Option, - pub mm: Option, -} - -/// Outcome of inspecting one proxied response. -pub enum IdentityCapture { - None, - Tokens(Vec), - ConsentRevoked, -} - -pub trait IntegrationIdentityCapture { - /// Paths (relative to the integration's `/api` prefix) whose responses - /// should be buffered and inspected. - fn capture_paths(&self) -> &'static [&'static str]; - - /// Parses a buffered upstream response body. - fn capture(&self, path: &str, status: StatusCode, body: &[u8]) -> IdentityCapture; -} +The response is 603 bytes. The SDK writes provider cookie and localStorage +`id5id`, and exposes the EID to Prebid with `inserter: "Lockr-For-Publishers"`, +`matcher: ""`, `mm: 2`. Those provider-owned browser stores are not removed. + +The integration XHR carries `ts-ec` and publisher consent cookies to TS. The +proxy does not forward publisher cookies or authorization to lockr. Its body +carries lockr's own consent strings. The route already builds an `EcContext` +and attaches `EcFinalizeState`; Fastly already has post-send execution. + +Today the upstream response body is not inspected. `/auction` body EIDs are +used for that auction, while cookie EIDs are persisted by finalization. The +foundation changes the latter contract to direct auction-body ingestion before +retiring the cookie path. + +## 3. Requirements and non-goals + +1. Capture lockr-issued IDs without requiring the Prebid User ID module or + a TS browser capture module. +2. Implement the foundation's optional capture capability only. Lockr has no + active server-side resolver in this delivery. +3. Return original proxy responses intact, with no added EID cookie and no + Fastly KV enrichment write before send. +4. Submit authoritative `Issued` outcomes for owned, registered sources; + subsequent SDK token responses use the same path. +5. Preserve known provider expiry through the foundation's lifecycle record. + Suppress expired IDs on later auctions without a hot-path cleanup write. +6. Translate explicit consent revocation separately from token acquisition, + including when current request consent no longer allows identity use. +7. Retain consent, size, UID, root and tombstone safety requirements from the + foundation. A failed capture must not break the SDK call. + +Out of scope: server-to-server calls to lockr, identifier intake, cross-browser +linking, changing Prebid provider discovery, direct ID5 or LiveRamp proxies, +OpenRTB provenance/provider `ext` storage, and client-side delivery of KV IDs. + +## 4. Capture implementation + +### 4.1 Registration and response handling + +Register `LockrIntegration` through `.with_identity_capture()` alongside its +existing proxy and attribute rewriter. Its identity descriptor claims the +confirmed source mappings in section 4.2. It registers no resolver capability. +`capture_identity = false` disables observation while preserving the descriptor +needed to validate configured source ownership. + +The capture paths are exact POST paths relative to `/integrations/lockr/api`: + +- `/publisher/app/v2/identityLockr/page-view` +- `/publisher/app/v2/identityLockr/generate-tokens` +- `/publisher/app/v2/identityLockr/refresh-tokens` +- `/publisher/app/v2/identityLockr/sync-no-hem-ids` +- `/publisher/app/v2/identityLockr/revoke-consent` + +The foundation handles bounded inspection and response preservation. Apply its +independent 64 KiB original-body and 64 KiB decoded-JSON limits. Compressed +responses must be decoded incrementally; overflow, unsupported encoding or +decode failure skips capture and forwards original bytes and headers unchanged. +The SDK and unlisted API paths retain existing handling. + +Extract request-body consent signals from the POST bytes already buffered for +forwarding. Supply them to the shared service without consuming the request +again or modifying the forwarded payload. The integration does not write KV, +set identity cookies or run its own background dispatch. + +### 4.2 Token normalization + +For token endpoints, read `aimTokens[]` and associated +`ids..eid`. Prefer the EID's source and UID, validating the source +against the mapping for that known key. When only `aimTokens[]` is present, +map `key_name` using the SDK's `tokenMappings` and `tokenSourceMappings`: + +| `key_name` | `source` | Expected `atype` | +| -------------------------- | --------------------------- | ---------------- | +| `id5id` | `id5-sync.com` | 1 | +| `_lr_env` | `liveramp.com` | 3 | +| `__uid2_advertising_token` | `uidapi.com` | 3 | +| `__euid_advertising_token` | `euid.eu` | 3 | +| `firstid` | `first-id.fr` | 1 | +| `connectId` | `yahooinc.com` | 3 | +| `panoramaId` | `panorama.com`, unconfirmed | 1 | +| `cto_bidid` | `criteo.com` | 3 | +| `_publink` | `epsilon.com` | 3 | + +Do not enable the unconfirmed Panorama mapping until lockr confirms the source +and type. Unknown keys and mismatched sources are dropped with redacted debug +logging. An arbitrary EID source cannot expand Lockr's registered claims. + +For ID5, decode the URL-encoded JSON `advertising_token` and extract +`universal_uid`. Submit one valid UID per source using the existing non-empty, +512-byte UID limit. Source and aggregate caps are enforced by the foundation. +Decode all supported provider token shapes using sanitized SDK/response +fixtures before enabling their mappings; the ID5 shape alone is not evidence +that every provider uses the same token encoding. + +Convert associated `identity_expires` from milliseconds to Unix seconds and +include it in `Issued`. Drop a token already expired at capture time. The +foundation stores known expiry and filters expired values on every later +auction path. Missing expiry remains unknown; it does not erase a known expiry +for the same UID. + +Unlike the earlier standalone draft, expiry is not discarded after parsing. +This depends on the foundation's metadata-compatible storage rollout, not a +Lockr-specific KV schema or timer. Lockr does not actively refresh records: +new expiry or replacement tokens arrive through later SDK responses. + +A response with valid tokens produces `Issued`; a valid response with no new +tokens produces `NoChange`. Non-2xx or malformed token responses skip capture. +Neither an empty token response nor a fetch failure invalidates stored IDs. + +### 4.3 Consent and ownership + +The shared service requires allowed EC consent, a valid existing `ts-ec`, +a registered source owned by `lockr`, and a live consenting root at mutation. +Integration XHRs never generate or recover an identity. + +Lockr supplies its request-body `gppString`, `consentString` and `ccpaString` +when populated. Empty optional fields are absent signals, not positive consent. +Malformed non-empty signals fail closed; valid opt-outs cannot be overridden +by allowed cookie consent. The shared consent layer interprets these signals, +not a second Lockr-specific consent engine. Confirm any additional broker or +provider-purpose requirements before enabling the relevant mappings. + +Failed token gating leaves the response intact and queues no enrichment. +Browser submissions of the same source remain conservative foundation inputs; +they cannot choose the `lockr` writer identity or replace a stored owned record. + +### 4.4 Explicit withdrawal + +The current provider contract assumes a successful 2xx `revoke-consent` response +means explicit publisher-level identity withdrawal. Translate it into the +foundation's `ConsentWithdrawn` outcome with global EC scope. Confirm both the +application-success semantics and the scope with lockr before rollout; a +provider-specific opt-out must not be widened into a global withdrawal. + +This outcome does not pass through token-consent eligibility: revocation must +still work after consent becomes denied. The shared finalizer expires `ts-ec` +and the service applies the existing-key-only tombstone path. Normal request +withdrawal takes precedence and suppresses duplicate work. Lockr never clears +or recreates a root independently. + +### 4.5 Shared persistence and auction delivery + +The shared service stages trusted capture effects with the full EC ID and +snapshot. Fastly extracts them before response conversion, sends the response, +then applies them before legacy pull sync. Lockr has no private +`DeferredIdentityUpdates` type, adapter branch or mutation function. + +The service applies authoritative replacement for the configured owner, +preserving unrelated sources and rejecting withdrawn roots. Same UID and same +lifecycle metadata skip writes; a changed known expiry can require a write even +if the UID is unchanged. Pass the resulting persisted snapshot to later work. + +Integration requests can carry a `NotRead` snapshot. The first required lookup +then happens after send; generation refresh or CAS conflict can add reads. +Persistence is best-effort and costs execution time without holding Fastly +response bytes. Other adapters follow the foundation's supported timing and +KV capability rules. + +Auctions resolve usable registered records from KV, deriving `atype` from the +source registry. Set the registry type to the confirmed mapping. Do not stamp +`Lockr-For-Publishers` onto every KV ID for a source: another input may have +populated it. Provider `ext`, extra UIDs and OpenRTB provenance remain absent +from later KV-only auctions, as defined by the foundation. + +## 5. Data flow and freshness + +```mermaid +flowchart TD + A[Lockr SDK calls TS first-party proxy] --> B[Lockr API returns tokens] + B --> C[Lockr capture module normalizes IDs and expiry] + C --> D[Shared identity service validates consent and ownership] + D --> E[Original response sent without an EID cookie] + E --> F[Shared service applies authoritative updates after send] + F --> G[EC identity graph stores IDs and lifecycle metadata] + G --> H[Later auctions read visible unexpired IDs] ``` -`IntegrationRegistration::builder` gains `.with_identity_capture()`. The -registry's `handle_proxy` checks whether the matched proxy implements the trait -and whether the request path is in `capture_paths()`. If so it buffers the -upstream response with `collect_response_bounded` at a new -`IDENTITY_CAPTURE_MAX_RESPONSE_BYTES` of 64 KiB, calls `capture`, then rebuilds -the response from the buffered bytes. Every other path streams through -unchanged. Buffering is limited to the listed JSON endpoints; the SDK body and -unlisted paths are never buffered by this hook. - -### 4.2 Lockr implementation - -`LockrIntegration` implements the trait for `page-view`, `generate-tokens`, -`refresh-tokens`, `sync-no-hem-ids` and `revoke-consent`. - -For token endpoints the parser reads `aimTokens[]` and, when present, -`ids..eid`. The `eid` object is preferred because it already carries -`source`, `atype` and provenance. When only `aimTokens[]` is present the -`key_name` is mapped to a source domain with a static table taken from the -SDK's own `tokenMappings` and `tokenSourceMappings`: - -| `key_name` | `source` | `atype` | -| -------------------------- | ---------------------------------- | ------- | -| `id5id` | `id5-sync.com` | 1 | -| `_lr_env` | `liveramp.com` | 3 | -| `__uid2_advertising_token` | `uidapi.com` | 3 | -| `__euid_advertising_token` | `euid.eu` | 3 | -| `firstid` | `first-id.fr` | 1 | -| `connectId` | `yahooinc.com` | 3 | -| `panoramaId` | `panorama.com` (verify with lockr) | 1 | -| `cto_bidid` | `criteo.com` | 3 | -| `_publink` | `epsilon.com` | 3 | - -Unknown `key_name` values are logged at debug level and dropped. The ID5 -`advertising_token` is URL-encoded JSON `{"universal_uid": "..."}`; the parser -extracts `universal_uid`. Tokens are validated with the existing -`is_valid_eid_uid` rule (non-empty, at most `MAX_UID_LENGTH` of 512 bytes). -`identity_expires` is milliseconds and is converted to seconds. - -`revoke-consent` with a 2xx upstream status yields `ConsentRevoked`. - -Non-2xx upstream responses and unparseable bodies yield `None` and are passed -through to the browser unchanged. - -### 4.3 Consent gate - -Capture runs only when all of the following hold: - -1. `ec_context.consent()` passes `ec_consent_granted`, evaluated from the - request's consent cookies exactly as on publisher navigations. -2. The request carries a valid `ts-ec` cookie, so there is an EC ID to attach - the IDs to. Integration XHRs never generate a new EC ID; that rule is - unchanged. -3. The lockr request body's `gppString` or `consentString`, when present, does - not indicate an opt-out. The body is already buffered for POST forwarding by - `collect_body_bounded`; this adds a parse, not a read. - -When the gate fails the response is passed through and nothing is written. - -### 4.4 Server-owned first-party cookie - -A new cookie, `ts-eids-s`, is set on the proxy response that delivered the -tokens: - -- Value: base64 of the structured JSON array already accepted by - `parse_prebid_eids_cookie`, holding `source` and one `id` per source and - nothing else. No `atype`, no `ext`, no provenance. Those are re-attached on - the server from a static per-source table when the auction payload is built - (section 4.6). Example before encoding: - - ```json - [{ "source": "id5-sync.com", "uids": [{ "id": "ID5*..." }] }] - ``` - - The cookie rides on every same-origin request, not only auctions, so it is - kept as small as the ID itself. One ID5 token encodes to 184 bytes - (measured). All nine lockr-issued sources with representative token lengths - encode to 1,716 bytes (modeled with the measured ID5, RampID, UID2 and - Criteo values); UID2 and Criteo tokens account for most of that. - -- Attributes: `Path=/; Secure; HttpOnly; SameSite=Lax`, `Domain` as configured - for `ts-ec`, `Max-Age` = the smallest `expires_at` minus now, capped at 30 - days, defaulting to 30 days when no expiry was given. -- Merge policy: the new value is the union of the existing `ts-eids-s` cookie - on the request and the captured EIDs, keyed by `source`; captured values - replace existing values for the same source. Sources whose stored expiry has - passed are dropped at merge time. -- Size: encoded value capped at 2,048 bytes. When the merged value exceeds - the cap, sources are dropped oldest-expiry first until it fits; dropping is - logged. The cap is below the 3,072-byte `ts-eids` cap because this cookie - carries one ID per source and nothing else, and above the modeled full set - so no lockr-issued source is dropped in practice. -- `ConsentRevoked` sets `ts-eids-s` with `Max-Age=0`. - -The cookie is separate from `ts-eids` for two reasons. The tsjs prebid module -rewrites `ts-eids` on every auction from `getUserIdsAsEids()`, so a server -value stored there would be overwritten within seconds. And a server-set -`HttpOnly` cookie is not subject to the script-written cookie lifetime cap in -WebKit, and cannot be edited by page scripts. - -`enforce_set_cookie_cache_privacy` in the Fastly middleware already marks any -response carrying `Set-Cookie` as private and uncacheable, so no cache change -is needed. - -### 4.5 Readers of the new cookie - -Two readers change: - -- `resolve_client_auction_eids` in `crates/trusted-server-core/src/auction/endpoints.rs` - currently prefers the request body and falls back to `ts-eids`. It will - parse `ts-eids-s` and merge it with whichever of body or `ts-eids` applied, - through `merge_auction_eids`. Before merging, each `ts-eids-s` entry is - expanded with `atype` and provenance from the static table in section 4.6. - For the same `source`, the `ts-eids-s` entry is ordered first so its values - win under the existing "first value wins, fill missing fields" merge rule. -- `collect_eid_cookie_updates` in `crates/trusted-server-core/src/ec/prebid_eids.rs` - takes a third optional cookie value and feeds it through - `collect_prebid_eid_updates_from_eids`. `EcFinalizeState` and - `EcRequestState` carry the extra cookie value. Existing partner-registry - gating applies: a source without an `[[ec.partners]]` entry still reaches the - auction body but is not written to KV. - -The JavaScript collector is not changed by this design. Its `ts-eids` output -continues to cover User ID module providers. - -The `x-ts-eids` response header on `/auction` is unchanged. It is emitted -after the merge and consent gating from the final `user.eids`, is on the -`INTERNAL_HEADERS` strip list for inbound requests, and is read by nothing on -the server. It remains the diagnostic view of what was sent to bidders. - -### 4.6 EID provenance - -`Eid` in `crates/trusted-server-core/src/openrtb.rs` gains three optional -fields with `skip_serializing_if = "Option::is_none"`: `inserter: Option`, -`matcher: Option`, `mm: Option`. `deny_unknown_fields` stays. -`StructuredCookieEid` and `structured_cookie_eids_to_openrtb` carry the same -three fields. `parse_client_auction_eids` and `merge_auction_eids` preserve -them, with the conflict rule "first non-empty value wins", matching how `atype` -and `ext` are merged today. - -Provenance is not stored in the cookie or in KV. A static table in the lockr -integration, keyed by `source`, supplies `atype`, `inserter` -(`Lockr-For-Publishers`), `matcher` (empty) and `mm` (2) when a `ts-eids-s` -entry is expanded for the auction, matching what the SDK sets on -`ortb2.user.ext.eids`. The `atype` column is the one from the mapping table in -section 4.2. Provider `ext` payloads such as ID5's `pba` are not carried; they -are not stored today either and bidders that need them read the provider -cookie directly. - -Legacy flattened cookies decode as before. KV entries are unchanged: the -identity graph still stores one UID per partner source and no provenance. - -### 4.7 KV write after send - -Capture never writes KV before the response is sent. Instead the proxy handler -attaches a response extension: - -```rust -pub struct DeferredIdentityUpdates { - pub updates: Vec, - pub revoke: bool, -} -``` - -`PartnerIdUpdate` values are built with the existing partner-registry lookup, -so only configured sources are included. In the Fastly entry point, after -`send_edgezero_response` returns and before pull sync, a new -`run_identity_updates_after_send` applies them: - -- `updates` go through `KvIdentityGraph::upsert_partner_ids_from_snapshot` - with the request's `EcKvSnapshot`, which already skips unchanged UIDs, fails - closed on a missing or unreadable root row, and refuses to write when the - stored `consent.ok` is false. -- `revoke` calls `write_withdrawal_tombstone`, the same path the CMP - withdrawal flow uses. - -Both are best-effort and log on failure. The Wasm instance stays alive until -the handler returns, so this work costs nothing on the response path. On a -`page-view` that carries no new token (the common returning-visitor case) the -extension is absent and no KV operation runs. - -The Axum, Cloudflare and Spin adapters apply the same extension in their own -post-send positions where one exists; where an adapter has no post-send hook -the updates are applied before send, matching how those adapters already -handle pull sync. The hot-path constraint is a Fastly production constraint. - -### 4.8 Data flow summary - -``` -browser -> POST /integrations/lockr/api/.../page-view (ts-ec, consent cookies, body) - TS proxy -> identity.loc.kr -> 200 {aimTokens, ids} - TS: consent gate -> capture -> Set-Cookie ts-eids-s (HttpOnly) - TS: attach DeferredIdentityUpdates -> send response to browser - TS: after send -> KV upsert (CAS, skip-unchanged) or tombstone - -next /auction on this page -> reads ts-eids-s -> user.eids includes id5-sync.com -next navigation -> KV snapshot (hot-path read already present) - -> ts-eids-s fallback when the row is missing -``` - -## 5. Freshness - -| Event | Cookie | KV | Auction sees it | -| ------------------------------------------ | -------------------------------------- | ------------------------------------- | --------------------------- | -| First issue (`page-view` with `aimTokens`) | Same response | After that response is sent | Next `/auction` on the page | -| Lockr refresh (`refresh-tokens`) | Same response, new value and `Max-Age` | After send, only when the UID changed | Next `/auction` | -| Lockr revoke (`revoke-consent`) | Cleared on that response | Tombstone after send | Next request | -| Returning visitor, no new token | Untouched | No operation | From KV snapshot | - -Modeled cost per captured response: one JSON parse of at most 64 KiB and one -`Set-Cookie` of at most 2,048 bytes. Modeled cost on every later same-origin -request: the cookie bytes, 184 for ID5 alone and up to 1,716 for all nine -lockr-issued sources. No additional KV read is added; the deferred upsert reuses the -request snapshot and performs its own refresh read only on a CAS retry, as it -does today for `ts-eids` ingestion. +| Event | Persistence | Auction availability | +| ------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------ | +| First issue | Authoritative update after send | Once a later KV read sees the record | +| SDK refresh | Replacement or expiry update after send | Once new state is visible | +| Provider expiry | Stored record becomes unusable; no timer required | Omitted when request time reaches known expiry | +| Explicit global revoke | EC cookie expired; existing-key-only tombstone | Later requests stop using the identity; dispatched auctions cannot be recalled | +| Returning visitor, no new token | No capture mutation | Existing usable KV state only | + +Inspection can retain 64 KiB of original bytes and 64 KiB of decoded JSON, plus +bounded stream/decoder overhead. There are no additional TS EID cookie bytes +on later requests. + +There is no cookie-based repair of a failed capture write. A later token-bearing +SDK response, browser-body submission or separately authorized partner path can +attempt persistence again under the foundation's writer rules. A returning +`page-view` with no token does not repair a missed capture. Expiry-aware omission +also does not initiate a Lockr fetch. These limits and delayed KV visibility are +accepted for this draft. ## 6. Configuration -No new lockr options. KV persistence for a lockr-issued source requires the -existing partner registry entry, for example: +Add `capture_identity` under `[integrations.lockr]`, default `true`, as the +publisher capture kill switch. It does not disable browser-body ingestion, +expiry-aware reads or normal EC consent withdrawal. + +Capturing a source requires its foundation source policy, for example: ```toml [[ec.partners]] @@ -382,80 +302,105 @@ name = "ID5" source_domain = "id5-sync.com" openrtb_atype = 1 bidstream_enabled = true +identity_owner = "lockr" ``` -Without this entry the ID still reaches the auction body through the cookie -and is never written to KV, matching current `ts-eids` behavior. The deployed -registry for the affected publisher should be checked for this entry as part of -rollout. +`identity_owner` is a proposed foundation field, not an existing runtime option. +Check the publisher registry and disable conflicting legacy authoritative paths +before enabling the owner. Missing registration or another selected owner means +Lockr does not persist the source. There is no cookie to forward an unregistered +captured ID. Valid browser-body EIDs can still reach their current auction under +the foundation's existing forwarding rules. ## 7. Error handling -- Upstream non-2xx or malformed JSON: pass-through, no cookie, no KV, debug log. -- Response larger than 64 KiB on a capture path: pass-through with a warning; - the limit is an order of magnitude above the measured 603-byte response. -- Consent gate failure: pass-through, nothing written. -- Cookie over 2,048 bytes after merge: drop oldest-expiry sources, log. -- KV upsert failure after send: logged, cookie already delivered, next - navigation retries through normal cookie ingestion. -- Missing `ts-ec`: pass-through, nothing written; the SDK call still succeeds. - -## 8. Testing - -Rust, alongside the code: - -- `integrations/lockr.rs`: parser tests for `aimTokens` only, `ids.*.eid` - present, URL-encoded ID5 JSON, unknown `key_name`, oversized UID, non-2xx, - malformed body, `revoke-consent`. -- `integrations/registry.rs`: capture path buffering versus pass-through - streaming for unlisted paths; response bytes identical after buffering. -- `auction/endpoints.rs`: `ts-eids-s` merged with body, with `ts-eids`, alone; - expansion from the static table; provenance precedence; caps. -- `ec/prebid_eids.rs`: structured cookie with and without provenance fields; - legacy cookie unchanged; third cookie input. -- `ec/finalize.rs` and adapter entry point: deferred updates applied after send; - absent when no tokens; tombstone on revoke; skip-unchanged. -- Consent-denied regression coverage retained for every new path. - -JavaScript: no changes, existing suites must still pass. - -Live acceptance on the TS matched arm: - -1. Fresh profile, accept consent, wait for the `page-view` response. -2. The `page-view` response carries `Set-Cookie: ts-eids-s=...; HttpOnly`, - and the decoded value holds only `source` and `id`. -3. The next `/auction` request body decodes with `id5-sync.com` in - `user.eids`, with `inserter` and `mm` present. -4. `GET /_ts/admin/ec` for the session's EC ID shows the ID5 UID under - `ids["id5-sync.com"]` after one further eligible request. -5. Reload: same UID, no new KV write (admin generation unchanged). -6. Trigger `revoke-consent` from the lockr SDK: cookie cleared, tombstone - present. +Use the foundation's redaction, bounds and root safety policy: + +- Non-2xx, malformed JSON, unknown key, mismatched source or invalid token: + preserve the response, drop the affected acquisition, no raw-token log. +- Original or decoded size overflow, unsupported encoding or decode failure: + skip capture and replay the original response unchanged. +- Denied token consent, missing EC, missing registration or owner mismatch: + no enrichment; the SDK call still succeeds independently. +- Missing, unreadable or withdrawn root: no creation or resurrection. +- Post-send persistence failure: redacted log; no immediate retry, durable + queue or change to the already-sent response. +- No tokens: `NoChange`, not invalidation or consent withdrawal. + +## 8. Verification + +Foundation tests own shared registration, browser migration, metadata +compatibility, buffering limits, mutation rules and adapter timing. Lockr tests +prove that this consumer reaches those paths correctly: + +- Exact POST path selection; SDK, settings and unlisted paths unchanged. +- `aimTokens` only, associated `ids.*.eid`, URL-encoded ID5 JSON, each enabled + provider token shape, unknown keys, mismatched sources and oversized UIDs. +- Millisecond expiry conversion, already-expired rejection, missing expiry and + same-UID expiry extension; known expiry survives the persisted record. +- Original and decoded cap overflow, including compressed expansion, skips + Lockr effects while retaining response bytes and encoding headers. +- No-token response produces no token mutation; non-2xx/malformed token + response cannot invalidate existing records. +- Populated body opt-outs, malformed signals, empty optional fields and cookie + consent conflicts exercise shared consent policy without modifying forwarding. +- Explicit revoke works with denied consent, chooses the confirmed withdrawal + scope and does not duplicate request-finalization tombstones. +- Owned/registered sources persist; unregistered or differently owned sources + do not; browser input cannot impersonate Lockr authority. +- Capture effects execute after Fastly send through the foundation, not a + separate Lockr adapter implementation; persisted snapshot handoff is retained. +- No newly emitted TS EID cookie or diagnostic EID header; provider browser + stores and existing publisher credential stripping remain unchanged. + +Live acceptance on an approved publisher test route: + +1. Fresh profile, grant consent and wait for `page-view`. +2. Confirm original response bytes and no TS EID cookie from capture. +3. Allow for post-send work and KV visibility. Inspect the session's admin EC + record for the registered source and lifecycle metadata, without exporting IDs. +4. Trigger a later auction and inspect bidder-facing payload evidence. Do not + require the browser body to contain a server-added ID or rely on `x-ts-eids`. +5. Reload with no new tokens: unchanged UID and expiry do not cause a write. +6. Trigger SDK refresh: confirm changed UID or expiry persists after visibility. +7. Exercise known expiry in a controlled fixture and confirm omission, including + a stale matching UID resubmitted in a browser auction body. +8. Trigger confirmed global revoke: EC cookie expires, tombstone exists and + later enrichment cannot revive it. + +No live identifiers, placement settings or credentials belong in fixtures or +committed validation notes. Keep sanitized status/source/count evidence only. ## 9. Rollout -1. Land the `Eid` provenance fields and cookie parser changes first; they are - additive and safe on their own. -2. Land the capture hook with lockr behind a `capture_identity` boolean on - `[integrations.lockr]`, default `true`, so a publisher can switch it off - without a deploy. -3. Add the `id5-sync.com` partner entry on the affected publisher. -4. Verify with the live acceptance steps above. - -## 10. Open questions - -1. Should `ts-eids-s` also be exposed to client-side auctions, for example - through `/identify` or a response header, or is the server-side auction the - only consumer? This design assumes server-side only. -2. The `panoramaId` source domain and `atype` should be confirmed with lockr - before the mapping table ships. -3. Whether a captured `expires_at` should also bound the KV entry. The identity - graph has no per-partner expiry today, and this design leaves that as is. - -## 11. Future-state options - -- Register lockr as a pull-sync partner and resolve tokens server-to-server - from the `ltid` value, removing the SDK from the identity path entirely. - Requires an S2S contract with lockr that does not exist today. -- Apply the same capture hook to direct ID5 and LiveRamp proxies once those - endpoints are proxied first-party. +1. Deliver the foundation, including metadata-preserving writers/readers, + source ownership, browser-body migration and EID cookie/header retirement. +2. Confirm Lockr path, provider token shapes, consent and revoke contracts using + sanitized fixtures; leave unconfirmed source mappings disabled. +3. Register the capture consumer and its `capture_identity` kill switch. +4. Configure the affected registered sources with `identity_owner = "lockr"` + and their confirmed `atype`, after resolving conflicting authoritative paths. +5. Complete the foundation's deployment compatibility gate before activation. +6. Enable capture on an approved test route and run live acceptance, including + visibility delay, expiry extension and withdrawal. + +Disabling capture stops new observations; it does not make already stored +expiry metadata safe for an older UID-only binary. Follow the foundation's +rollback contract. No `ts-eids-s` is introduced at any stage. + +## 10. Open questions and future work + +1. Confirm the Panorama source/type and all enabled provider token shapes with + lockr. Do not infer them solely from the measured ID5 example. +2. Confirm request-body consent details and any additional broker/provider + permission requirements. +3. Confirm successful revoke response semantics and whether withdrawal is + global or source-specific before enabling that outcome. +4. Confirm token ordering when SDK requests overlap. The foundation does not + infer provider issuance order from edge wall clocks. + +Known expiry persistence is now a proposed foundation decision rather than an +unimplemented Lockr-local open question. OpenRTB provenance storage remains +outside this delivery. Active Lockr resolution from `ltid`, direct provider +capture and client-side consumers require separate follow-up specs; no current +S2S contract is assumed. diff --git a/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md b/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md new file mode 100644 index 000000000..a1eafd9c5 --- /dev/null +++ b/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md @@ -0,0 +1,473 @@ +# Server-side identity foundation + +Status: Architecture implemented in the draft change, awaiting code review. +See the [implementation plan and verification](../plans/2026-10-09-server-side-identity-foundation-and-lockr.md). +Provider activation and metadata-compatible deployment remain unverified. + +First consumer: [Lockr proxy-side identity capture](2026-10-08-lockr-identity-capture-design.md). + +Related: [KV EID request snapshot and EC recovery](2026-07-10-kv-eid-request-snapshot-ec-recovery-design.md) +defines the snapshot, root-creation and withdrawal rules retained here. +[The existing LiveRamp design](2026-08-21-liveramp-integration-design.md) +describes the browser path. A separate future spec must approve any native +resolution protocol and identifier intake. + +## 1. Purpose + +Give integrations a standard way to contribute server-side identities without +making each integration implement storage, consent handling and auction wiring. + +The ownership rule is: + +> Integrations acquire and interpret identities. A shared identity service +> coordinates lifecycle work. EC owns persistence, consent and auction delivery. + +Capture and active resolution are different capabilities. They share identity +outcomes and mutation policy, not a single provider HTTP protocol. + +```mermaid +flowchart TD + A[Browser IDs through auction body] --> D[Shared identity service] + B[Integration captures IDs from proxied traffic] --> D + C[Future integration resolves or refreshes IDs] --> D + D --> E[EC identity graph in KV] + E --> F[Server-side auctions read usable IDs] +``` + +This spec owns the shared contracts. The Lockr spec owns the first provider +implementation. A later LiveRamp spec can use the resolution contract without +changing the Lockr protocol. + +## 2. Starting system and gaps (pre-implementation baseline) + +- `integrations/registry.rs` registers optional proxies, request filters, + rewriters and head injectors, but no identity capabilities. +- `ec/registry.rs` builds the source policy and partner API registry from + `[[ec.partners]]`. An EID source is the key into the identity graph. +- `/auction` uses body EIDs for the current auction. Its EC finalization + currently persists request cookies, not those body EIDs. +- `ec/pull_sync.rs` implements a specific bearer-authenticated protocol: + send `ec_id`, receive `{"uid":"..."}`, and fill missing sources only. + It does not refresh existing UIDs; its refresh interval is currently unused. +- `KvPartnerId` stores only `uid`. It cannot express provider expiry, the + authoritative writer, or the version a refresh result was based on. + +Reuse the existing snapshot, conditional-write and post-send mechanisms. +Do not mistake the legacy pull-sync wire protocol for a universal resolver. + +The shared identity service belongs in core EC code and uses `KvIdentityGraph`; +it is not another store or vendor registry. The integration registry exposes +capabilities to that service. Platform adapters only supply execution timing +and services, including the point at which a response has been sent. + +## 3. Scope and invariants + +The first delivery covers integration capability registration, common outcomes, +source ownership, expiry-aware records, browser-body ingestion, bounded capture +and the Fastly post-send capture path. Resolution and refresh contracts are +specified now; a production active resolver is implemented only with an +approved provider spec. + +Invariants: + +1. No additional EID cookie. `ts-ec` remains the pointer to an existing EC row. +2. Integration capture and `/auction` enrichment never create or recover a + root. Existing eligible publisher-navigation creation and recovery remain. +3. Missing and unreadable rows are distinct. Neither authorizes enrichment. +4. A tombstone is never revived by identity acquisition. +5. Browser input is not an authoritative partner update. +6. A provider timeout or configuration failure does not revoke consent or + erase a still-valid ID. +7. Known-expired or invalidated records are not exposed to bidders. +8. Stored state and response extensions identify the full EC ID, not just its + IP-derived hash prefix. A rotated identity cannot reuse the old snapshot. +9. Logs and telemetry never contain UIDs, envelopes, hashes submitted for + resolution, consent strings, or query URLs carrying those values. +10. Fastly capture persistence and future active resolution run after send. + Browser-body persistence keeps existing EC finalization timing. + +Non-goals: LiveRamp API implementation, hashed-identifier collection or storage, +cross-browser linking, provider-specific authentication, audience segments, +multiple stored UIDs per source, arbitrary provider metadata in KV, durable +background queues, and guaranteed inclusion in the first auction. + +## 4. Registration and configuration + +### 4.1 Optional integration capabilities + +Extend the existing integration registration, not a separately maintained list +of identity vendors: + +```rust +IntegrationRegistration::builder("example_identity") + .with_identity_capture(capture) + .with_identity_resolver(resolver) + .build() +``` + +The sketch shows optional capabilities, not a requirement to implement both. +A capture-only module needs no resolver or dummy HTTP endpoint. A resolver-only +module needs no browser SDK or proxy route. Each capability declares the finite +set of source namespaces it can produce and its consent requirements. + +The initial implementation registers capture. Reserve the resolver contract +without enabling an unused production fetch loop. + +### 4.2 Integration IDs and source namespaces + +An integration ID identifies the acquisition module. An EID `source` identifies +the partner namespace stored in KV. They are not interchangeable: one broker +can issue several sources, and multiple modules can issue the same source. + +Keep provider protocol settings under `[integrations.]`. Keep generic +source policy, bidstream permission and existing partner API authentication +under `[[ec.partners]]`. + +Add an optional, vendor-neutral source policy field: + +```toml +[[ec.partners]] +name = "Example identity source" +source_domain = "ids.example.com" +openrtb_atype = 3 +bidstream_enabled = true +identity_owner = "example_identity" +``` + +`identity_owner` selects the one integration allowed to replace or invalidate +that source through managed acquisition. Validate that the configured module +has an identity descriptor claiming the source. Reject ambiguous ownership; +do not resolve it by registration order. A configured module can disable its +acquisition capability without losing its ownership descriptor or invalidating +the kill switch configuration. + +Without an owner, browser and legacy partner paths retain their existing source +policy. Managed integration updates require an explicit owner. Browser input +can still fill an absent source conservatively, but cannot replace a stored +record or confer provider authority. + +For an owned source, legacy push and pull paths must not bypass owner checks. +Disable conflicting authoritative paths or route an explicitly authorized +writer through the same policy. Do not silently allow a generic pull to replace +an integration-owned envelope. Changing the owner is an operator migration, +not an automatic fallback when a module fails. + +## 5. Acquisition contracts and common outcomes + +### 5.1 Capture + +A capture capability declares exact proxy response paths and interprets a +bounded decoded body. The framework supplies request-local EC and consent +context, method, status and response metadata. The module knows its provider +schema; the framework owns buffering, bounds and response preservation. + +If the provider needs consent values from a POST body, the integration extracts +them from its existing forwarding buffer and supplies normalized signals. +It must not consume the body twice or let a second parser modify forwarding. + +Only framework-associated upstream responses can produce authoritative capture +updates. An arbitrary browser body or inbound diagnostic header cannot select +this trust level or impersonate a module. + +### 5.2 Resolution and refresh + +The future resolver contract separates planning, HTTP execution and parsing: + +1. Given permitted inputs, current record and consent, the module plans no + work, reports missing inputs, or builds a resolution/refresh request. +2. The shared coordinator enforces eligibility, due time, consent, allowlists, + rate limits, timeout and response limits using platform HTTP services. +3. The module interprets the provider response into common identity outcomes. +4. EC applies those outcomes against the record version used to plan the work. + +Inputs must be approved by the provider spec and represented as named types. +No generic request header becomes a trusted hashed identifier by convention. +The framework does not grant modules access to all publisher cookies or +credentials; disclosure is limited to declared, permitted inputs. + +### 5.3 Outcomes + +| Outcome | Meaning and shared action | +| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | +| `Issued` | Validated source and UID, optional provider expiry; apply the appropriate update policy. A refresh may keep the UID while extending expiry. | +| `NoChange` | Nothing new was observed; do not manufacture a write. | +| `NoMatch` | Resolution produced no ID; it does not invalidate an existing ID or withdraw consent. | +| `Invalidated` | The provider rejected a particular existing record; make that record unusable only if its expected revision still matches. | +| `ConsentWithdrawn` | Explicit withdrawal with a provider-spec-defined scope; never inferred from a generic empty response. | +| `RetryableFailure` | Temporary network/provider failure; preserve valid state and apply bounded retry policy. | +| `ConfigurationFailure` | Approval, credentials or request configuration is wrong; preserve valid state and report a redacted operational error. | + +Capture decode/schema errors skip capture without changing the proxy response. +Module output is validated again at the shared trust boundary against source +claims, configured ownership, UID caps and allowed metadata. The service assigns +the writer identity and update class; neither comes from browser JSON. + +Per-source invalidation is not a global EC withdrawal. A provider spec must +explicitly justify a global withdrawal outcome. The shared finalizer gives +withdrawal precedence over enrichment and suppresses duplicate tombstones when +normal request consent finalization has already handled it. + +## 6. Stored records and freshness + +### 6.1 Proposed record extension + +Keep one UID per source, and add optional lifecycle fields to `KvPartnerId`: + +| Field | Purpose | +| ---------------- | --------------------------------------------------------------------------------- | +| `uid` | Existing opaque identifier. | +| `expires_at` | Provider expiry in Unix seconds, when known. | +| `writer` | Framework-assigned integration, browser or legacy writer identity. | +| `revision` | Opaque version changed on every material record mutation. | +| `updated_at` | Server time of a material persisted change, not proof of provider issuance order. | +| `invalidated_at` | Marks a rejected record unusable while retaining its revision. | + +The first delivery adds expiry, writer and revision support. The table also +reserves the semantics of update timestamps and per-source invalidation; do not +add unused runtime state or scheduling merely to fill out the future contract. +Those fields, refresh timing and attempt/backoff state are implemented when a +consumer needs them. Acquisition timestamps are not a substitute for +provider-owned ordering. + +Known provider expiry is preserved through KV and auction resolution. Reject an +already-expired issued token. Do not clear a known expiry merely because a +same-UID observation omitted it. A same-UID authoritative observation with a +changed, supported expiry is a material update; unchanged UID and lifecycle +metadata require no write. Do not write merely to update an observation clock. + +Legacy UID-only records and browser inputs have unknown expiry and retain their +existing row-lifetime behavior. Do not invent a historical expiry or falsely +label them as provider-resolved. This compatibility choice does not guarantee +that legacy IDs are still valid; tightening unknown-expiry policy is a separate +operator decision. + +The record extension does not store OpenRTB provenance, UID `ext`, or extra UIDs. +Writer identity is storage metadata, not proof that a provider requires a +particular `inserter` or `matcher` value in the bidstream. + +### 6.2 Mutation classes + +Browser enrichment keeps the current conservative policy: fill absent sources, +skip unchanged values, preserve different values, and perform at most one CAS +write followed by at most one conflict read with no retry write. Browser input +cannot set expiry, writer, revision or invalidation metadata. Expired and +invalidated records count as existing records, not empty slots that the browser +can revive. + +Authoritative acquisition can replace an owned source and update supported +lifecycle metadata. It retains bounded CAS refresh and re-merge, preserving +unrelated sources and rejecting a missing or withdrawn root. + +For future resolver updates and invalidation, carry the revision used to plan +the call. If another writer changed that record, discard the stale result; +do not delete or overwrite the replacement. Comparing UID alone is insufficient +because a same-UID refresh can change expiry. Source invalidation retains a +non-deliverable record marker so a stale browser submission cannot refill it. +DSR deletion and global withdrawal still remove identity payload through their +existing authoritative paths. + +Capture-only updates without a provider sequence retain observation/CAS ordering. +They do not promise chronological provider freshness across overlapping SDK +requests. A provider requiring stronger ordering must supply an ordering +contract in its integration spec, not rely on edge wall clocks. + +### 6.3 Auction reads + +Resolve only live, consented, bidstream-enabled records whose known expiry has +not passed and which are not invalidated. Retain expired records as lifecycle +state; auction omission does not require a hot-path cleanup write. + +When browser EIDs are merged with KV, a browser UID matching a known-expired or +invalidated record must also be omitted for that source. Otherwise request-body +precedence would bypass the usability rule. A different valid browser UID can +still be used in its current auction under existing consent rules without +becoming an authoritative replacement in KV. + +Body-supplied OpenRTB provenance remains current-request metadata. Add optional +`inserter`, `matcher` and `mm` to `Eid` parsing and merging; an empty `matcher` +is a supplied value. Later KV-derived EIDs use source registry `atype` and do +not synthesize provider provenance. The first delivery preserves existing caps +and consent gates across all auction paths. + +## 7. Consent and execution timing + +The shared service evaluates global EC consent and each module's additional +requirements. A global `consent.ok` flag alone does not prove vendor or purpose +permission. Future outbound disclosure rechecks the live consenting row just +before dispatch and respects current request privacy signals. Mutation rechecks +root consent through conditional-write semantics. + +```mermaid +flowchart TD + A[Request has existing EC and consent context] --> B[Integration captures identity outcomes] + B --> C[Shared service stages permitted effects] + C --> D[Normal response finalization handles withdrawal] + D --> E[Response sent to browser] + E --> F[Shared service applies captured updates] + F --> G[Persisted snapshot passed to later identity work] + G --> H[Future due resolver work or existing eligible pull sync] +``` + +For browser-body ingestion, stage updates before auction dispatch and carry them +through EC finalization, including handled provider-error responses. Preserve +current auction behavior when persistence fails. Denied identity use stages no +updates. Browser persistence does not require a successful bid or a winning bid. + +Fastly removes cloneable identity effects before response conversion drops +extensions. Shared effects carry the full EC ID, trusted acquisition origin and +snapshot; the adapter does not parse provider payloads. Run captured mutations +after send and before legacy pull sync. Pass only persisted outcome snapshots +forward, including changed generations and failed states. + +Capture routes can have `NotRead` snapshots. Their first required lookup occurs +after send. A reusable snapshot avoids that lookup, but unusable generations, +stale misses and CAS conflicts can require refresh reads. No zero-read claim is +made. No-token responses queue no token mutation. + +Best-effort after-send work adds execution time, not Fastly response latency. +There is no durable queue, guaranteed retry or first-auction availability. +Other adapters use the shared service only where EC KV support exists. Document +before-send behavior where no post-send execution hook exists; do not claim +persistence on unsupported adapters. + +## 8. Browser-body migration and EID cookie retirement + +Use validated `/auction` body EIDs both for current-request forwarding and for +registered-source browser enrichment. Expose the existing EID-to-update helper +without round-tripping through a fake cookie. Reuse the auction snapshot in +finalization instead of performing a second unconditional lookup. + +Remove the `ts-eids` writer and server readers only after body ingestion is +verified. Final behavior: + +- `/auction` gets current-request client EIDs from the body, with no EID-cookie + fallback. Unknown sources remain usable only in that current auction. +- Publisher and SPA page-bid auctions read registered IDs from KV. +- The separate `sharedId` cookie retains its existing ingestion behavior; + combine browser inputs into one finalization mutation attempt. +- Orphan recovery no longer seeds IDs from a `ts-eids` cookie. +- Late browser IDs need another auction submission for persistence. No new + page-exit beacon or cookie fallback is introduced. + +Retire `x-ts-eids` and `x-ts-eids-truncated` emission, checking external diagnostic +tooling first. Keep both names on the inbound internal-header strip list. +Do not remove EC or consent headers as part of this change. + +A transition JavaScript release expires the old host-only `Path=/` cookie; +its existing max-age is one day. New servers ignore it even if cached older +bundles still write it. Remove the cleanup shim once the transition artifacts +have aged out. Do not add unconditional server cookie cleanup that changes +cache privacy on every response. + +## 9. Bounded transport and failure policy + +Capture response inspection has independent 64 KiB limits for original body +bytes and decoded JSON bytes. Decode incrementally into bounded buffers; do not +allocate the full decompressed body before checking its size. Exceeding either +limit, unsupported encoding or decode failure skips capture and preserves the +original body and encoding/length headers. Oversized streaming bodies must +replay the collected original prefix and remaining stream, not fail after a +destructive collector consumes the prefix. + +Memory accounting includes both buffers and bounded stream/decoder overhead; +the total budget is not 64 KiB. Future resolver HTTP uses equivalent independent +wire and decoded limits, though it has no browser response to replay. + +Retain UID length and source-count caps. Provider request building must use +HTTPS, an explicit destination allowlist and narrowly allowed headers. Do not +forward publisher credentials by default. Disable or revalidate redirects +before any request carrying identity input leaves the approved destination. + +Failures are best-effort and redacted. No-match and failed attempts need bounded +negative-cache/backoff policy when an active resolver is introduced; define its +state lifetime and key in that provider delivery. Do not introduce nested, +unbounded retries in modules and the coordinator. + +## 10. Resolution lifecycle reserved for later delivery + +A future resolver distinguishes missing, usable, refresh-due, expired and +invalidated state. Missing approved inputs means no call. Expired or invalidated +values are not auction-usable, even if refresh or re-resolution cannot run. + +Refresh is initially activity-driven: a qualifying request can trigger due work +after send. There is no promise that a task runs while a visitor is inactive. +A provider spec defines resolution inputs, refresh inputs, interval, failure +backoff and whether a particular response invalidates the old record. + +Reuse post-send orchestration without changing the legacy protocol's +fill-missing-only behavior for unowned sources. Skip integration-owned sources +in legacy completeness and dispatch decisions. Rate-limit keys must define +whether their identity scope is the full EC ID or the IP-derived EC hash; +the existing generic pull limiter uses the hash and is not a per-cookie limit. + +No module can create cross-browser linkage by storing a UID in KV. Chrome and +Safari ordinarily have different full EC IDs. A later linking design must +establish an approved association; a common IP prefix is not proof of a person. + +## 11. Compatibility and delivery sequence + +The proposed record fields are additive and optional within schema version 1. +Legacy records decode as before. Unsupported schema versions still fail closed. +Do not bump the schema merely to make old records unreadable. + +However, older writers would deserialize and reserialize away unknown metadata, +and older readers would ignore expiry. Deploy metadata-preserving readers and +all mutation paths before activating managed writes. This includes browser, +push, pull, withdrawal, recovery and admin-related mutations. Test their +round-trips and equality rules; a UID-only overwrite must not strip expiry. + +Activation requires deployment propagation and old invocation drain. Rolling +back to a UID-only binary after activation is unsafe unless managed writes are +stopped and record compatibility is explicitly handled. Disabling acquisition +is safe but does not remove expiry-aware reader requirements. + +Delivery stages: + +1. Add defaulted record fields, usability checks and metadata-preserving + mutation plumbing with managed acquisition inactive. +2. Add common outcomes, source ownership and capture registration. Keep + authoritative and browser mutation classes distinct. +3. Add browser-body ingestion, verify it, then retire EID cookie/header paths. +4. Add bounded capture handling and shared Fastly effect execution, using + synthetic integration fixtures before enabling Lockr. +5. Implement and enable the separate Lockr consumer after ownership and + provider validation checks pass. +6. Implement active resolution and refresh only with its later provider spec. + +## 12. Verification and open decisions + +Foundation tests must cover: + +- Optional capability registration and source-ownership validation, including + a capture kill switch without invalidating the configured owner. +- Legacy record decoding and every writer preserving lifecycle metadata. +- Browser fill-missing and conflict bounds; rejection of forged authority and + metadata; no browser resurrection of expired/invalidated state. +- Authoritative replacement and same-UID expiry changes; unchanged metadata + skips writes; unrelated sources survive CAS conflicts. +- Known expiry boundaries across publisher, `/auction` and SPA auction paths, + including an expired UID resubmitted in the auction body. +- Revision-conditional invalidation and stale resolver-result rejection, + with same-UID/new-expiry and concurrent-withdrawal cases. +- Missing versus failed roots, no-create enrichment, snapshot identity + association, withdrawal precedence and duplicate-tombstone suppression. +- Independent original/decoded size limits, compressed expansion, unsupported + encodings, decode failure and byte-identical capture pass-through. +- Browser updates surviving handled auction-provider failures; one combined + browser finalization mutation with `sharedId`; body-only migration behavior. +- Fastly send-before-capture-write ordering and resulting snapshot handoff to + pull sync; no-token and failed-update paths; supported adapter behavior. +- Redacted errors and no private data in logs, including provider query URLs. + +Resolve before implementation activation: + +1. Confirm deployment propagation/drain and rollback procedures for additive + lifecycle metadata. Passing new-reader tests does not prove old-writer safety. +2. Confirm capture ordering expectations when an SDK issues overlapping token + requests; no provider sequence is assumed by this foundation. +3. Identify external consumers of the diagnostic EID headers before retirement. + +Provider expiry, writer and revision storage are proposed foundation decisions, +not already implemented capabilities. LiveRamp protocol approval, identifier +intake, refresh response semantics and country-specific TTL remain for its +future spec. OpenRTB provenance storage remains outside this first delivery. diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 066ad659e..5eea58fdb 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -586,6 +586,8 @@ geo_query_parameters = false # sdk_url = "https://identity.example.com/trusted-server.js" # cache_ttl_seconds = 3600 # rewrite_sdk = true +# capture_identity = false # require provider approval and old-binary drain +# capture_global_withdrawal = false # require confirmed global revoke semantics # DataDome bot protection. Proxies tags.js + signal-collection API first-party. # Kept active but disabled so `ts audit` can flip `enabled` when detected. From f22c4ad1146d80728fdef28c22133edeb0204f83 Mon Sep 17 00:00:00 2001 From: Christian Date: Fri, 9 Oct 2026 16:42:07 -0500 Subject: [PATCH 3/7] Use official ECR mirror for CI framework images --- .github/actions/setup-integration-test-env/action.yml | 6 +++++- .../fixtures/frameworks/nextjs/Dockerfile | 5 +++-- .../fixtures/frameworks/wordpress/Dockerfile | 3 ++- 3 files changed, 10 insertions(+), 4 deletions(-) diff --git a/.github/actions/setup-integration-test-env/action.yml b/.github/actions/setup-integration-test-env/action.yml index 841d8d5bd..10d030331 100644 --- a/.github/actions/setup-integration-test-env/action.yml +++ b/.github/actions/setup-integration-test-env/action.yml @@ -96,8 +96,11 @@ runs: - name: Build WordPress test container if: ${{ inputs.build-test-images == 'true' }} shell: bash + # Docker's official ECR mirror avoids Docker Hub anonymous pull limits. run: | - docker build -t test-wordpress:latest \ + docker build \ + --build-arg DOCKER_LIBRARY=public.ecr.aws/docker/library \ + -t test-wordpress:latest \ crates/trusted-server-integration-tests/fixtures/frameworks/wordpress/ - name: Build Next.js test container @@ -106,6 +109,7 @@ runs: run: | docker build \ --build-arg NODE_VERSION=${{ steps.node-version.outputs.node-version }} \ + --build-arg DOCKER_LIBRARY=public.ecr.aws/docker/library \ -t test-nextjs:latest \ crates/trusted-server-integration-tests/fixtures/frameworks/nextjs/ diff --git a/crates/trusted-server-integration-tests/fixtures/frameworks/nextjs/Dockerfile b/crates/trusted-server-integration-tests/fixtures/frameworks/nextjs/Dockerfile index 6a6dc7d71..1e3e52cdc 100644 --- a/crates/trusted-server-integration-tests/fixtures/frameworks/nextjs/Dockerfile +++ b/crates/trusted-server-integration-tests/fixtures/frameworks/nextjs/Dockerfile @@ -14,7 +14,8 @@ # --- Build stage --- ARG NODE_VERSION -FROM node:${NODE_VERSION}-alpine AS builder +ARG DOCKER_LIBRARY=docker.io/library +FROM ${DOCKER_LIBRARY}/node:${NODE_VERSION}-alpine AS builder WORKDIR /app @@ -27,7 +28,7 @@ COPY app/ ./app/ RUN npm run build # --- Runtime stage --- -FROM node:${NODE_VERSION}-alpine AS runner +FROM ${DOCKER_LIBRARY}/node:${NODE_VERSION}-alpine AS runner WORKDIR /app diff --git a/crates/trusted-server-integration-tests/fixtures/frameworks/wordpress/Dockerfile b/crates/trusted-server-integration-tests/fixtures/frameworks/wordpress/Dockerfile index 513b7a8e5..71d6c0012 100644 --- a/crates/trusted-server-integration-tests/fixtures/frameworks/wordpress/Dockerfile +++ b/crates/trusted-server-integration-tests/fixtures/frameworks/wordpress/Dockerfile @@ -9,7 +9,8 @@ # Run manually: # docker run -p 8080:80 test-wordpress:latest -FROM php:8.3-cli-alpine +ARG DOCKER_LIBRARY=docker.io/library +FROM ${DOCKER_LIBRARY}/php:8.3-cli-alpine WORKDIR /var/www/html From bb7dbf23a80894e73fa6c5b5af7251c0e84c28cf Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 12 Oct 2026 10:00:24 -0500 Subject: [PATCH 4/7] Fix Lockr EID capture and GPP USP consent Read the associated ID5 EID from ids.id5id.eid and assert the issued identity rather than only counting capture outcomes. Normalize legacy USP sections in shared GPP decoding while preserving opt-out precedence and failing acquisition closed on malformed signals. Add parser, mixed-signal and capture persistence regressions. --- crates/trusted-server-core/src/consent/gpp.rs | 107 +++++++++++++++--- crates/trusted-server-core/src/consent/mod.rs | 1 + .../trusted-server-core/src/consent/types.rs | 10 +- crates/trusted-server-core/src/ec/identity.rs | 60 +++++++++- .../src/integrations/lockr.rs | 69 +++++++---- 5 files changed, 206 insertions(+), 41 deletions(-) diff --git a/crates/trusted-server-core/src/consent/gpp.rs b/crates/trusted-server-core/src/consent/gpp.rs index 59e39613a..2e8fc3c49 100644 --- a/crates/trusted-server-core/src/consent/gpp.rs +++ b/crates/trusted-server-core/src/consent/gpp.rs @@ -23,7 +23,7 @@ use error_stack::Report; -use super::types::{ConsentDecodeError, GppConsent, TcfConsent}; +use super::types::{ConsentDecodeError, GppConsent, PrivacyFlag, TcfConsent}; /// Maximum length of a raw GPP string before parsing. /// @@ -35,7 +35,8 @@ const MAX_GPP_STRING_LEN: usize = 8192; /// Decodes a GPP string into a [`GppConsent`] struct. /// /// Parses the raw `__gpp` cookie value, extracts section IDs, and optionally -/// decodes the EU TCF v2.2 section if present. +/// decodes the EU TCF v2.2 section if present. Normalizes sale opt-outs across +/// legacy US Privacy and US state/national sections. /// /// # Arguments /// @@ -44,7 +45,8 @@ const MAX_GPP_STRING_LEN: usize = 8192; /// # Errors /// /// - [`ConsentDecodeError::InvalidGppString`] if the string exceeds -/// `MAX_GPP_STRING_LEN` or the `iab_gpp` parser fails. +/// `MAX_GPP_STRING_LEN`, the `iab_gpp` parser fails, or a present US Privacy +/// section is malformed without an independently decoded US sale opt-out. pub fn decode_gpp_string(gpp_string: &str) -> Result> { if gpp_string.len() > MAX_GPP_STRING_LEN { return Err(Report::new(ConsentDecodeError::InvalidGppString { @@ -71,7 +73,7 @@ pub fn decode_gpp_string(gpp_string: &str) -> Result = 7..=23; /// Extracts the `sale_opt_out` signal across all US sections in a parsed GPP /// string. /// -/// Iterates through section IDs looking for any in the US range (7–23), -/// decodes each US section, and aggregates the result conservatively: +/// Aggregates US state/national sections (7–23) and a legacy US Privacy section +/// (6), interpreted with the shared USP decoder: /// -/// - `Some(true)` if any decodable US section says the user opted out of sale -/// - `Some(false)` if at least one decodable US section says they did not opt -/// out and none say they opted out -/// - `None` if no US section is present or no decodable US section yields a -/// usable `sale_opt_out` signal -fn decode_us_sale_opt_out(parsed: &iab_gpp::v1::GPPString) -> Option { +/// - `Some(true)` if any decoded US section says the user opted out of sale +/// - `Some(false)` if at least one decoded US section has no sale opt-out +/// - `None` if no decoded US section yields a usable sale signal +/// +/// A valid sale opt-out is retained even if another section is malformed. +/// Otherwise, malformed legacy USP fails decoding rather than being masked by +/// another section's no-opt-out signal. +fn decode_us_sale_opt_out( + parsed: &iab_gpp::v1::GPPString, +) -> Result, Report> { let mut result = None; for us_section_id in parsed @@ -131,7 +137,7 @@ fn decode_us_sale_opt_out(parsed: &iab_gpp::v1::GPPString) -> Option { { match parsed.decode_section(*us_section_id) { Ok(section) => match us_sale_opt_out_from_section(§ion) { - Some(true) => return Some(true), + Some(true) => return Ok(Some(true)), Some(false) => result = Some(false), None => {} }, @@ -141,7 +147,16 @@ fn decode_us_sale_opt_out(parsed: &iab_gpp::v1::GPPString) -> Option { } } - result + if let Some(raw) = parsed.section(iab_gpp::sections::SectionId::UspV1) { + let usp = super::us_privacy::decode_us_privacy(raw).map_err(|_| { + Report::new(ConsentDecodeError::InvalidGppString { + reason: "malformed US Privacy section".to_owned(), + }) + })?; + result = Some(usp.opt_out_sale == PrivacyFlag::Yes); + } + + Ok(result) } fn us_sale_opt_out_from_section(section: &iab_gpp::sections::Section) -> Option { @@ -231,6 +246,65 @@ mod tests { result.eu_tcf.is_none(), "should not have TCF section in USP-only string" ); + assert_eq!( + result.us_sale_opt_out, + Some(false), + "should normalize the USP section's no-opt-out signal" + ); + } + + #[test] + fn decodes_usp_section_sale_opt_out() { + for (raw, expected) in [ + ("1YYN", true), + ("1YNN", false), + ("1---", false), + ("1ynn", false), + ] { + let gpp = gpp_with_sections(&[(6, raw)]); + let result = decode_gpp_string(&gpp).expect("should decode a valid USP section"); + assert_eq!(result.us_sale_opt_out, Some(expected)); + } + } + + #[test] + fn rejects_malformed_usp_sections_even_with_a_valid_us_section() { + for raw in ["1XNN", "2YNN", "1YN", "1YNNN"] { + for sections in [vec![(6, raw)], vec![(6, raw), (7, "BVQqAAAAAgA.QA")]] { + assert!( + decode_gpp_string(&gpp_with_sections(§ions)).is_err(), + "should not turn a malformed USP section into permission" + ); + } + } + } + + #[test] + fn malformed_usp_does_not_hide_a_valid_us_state_opt_out() { + let result = decode_gpp_string(&gpp_with_sections(&[(6, "1XNN"), (9, "BVVVVVVVVWA.AA")])) + .expect("should retain an independently decoded sale opt-out"); + assert_eq!( + result.us_sale_opt_out, + Some(true), + "should not discard a valid opt-out because another section is malformed" + ); + } + + #[test] + fn usp_and_us_state_sections_keep_any_opt_out() { + for (usp, state, expected) in [ + ("1YYN", "BVgVVVVVVWA.AA", true), + ("1YNN", "BVVVVVVVVWA.AA", true), + ("1YNN", "BVgVVVVVVWA.AA", false), + ] { + let result = decode_gpp_string(&gpp_with_sections(&[(6, usp), (9, state)])) + .expect("should decode USP and US-state sections"); + assert_eq!( + result.us_sale_opt_out, + Some(expected), + "should preserve opt-out precedence across legacy and modern sections" + ); + } } #[test] @@ -395,10 +469,11 @@ mod tests { #[test] fn no_us_section_returns_none() { - let result = decode_gpp_string(GPP_TCF_AND_USP).expect("should decode GPP"); + let gpp = gpp_with_sections(&[(2, "CPXxRfAPXxRfAAfKABENB-CgAAAAAAAAAAYgAAAAAAAA")]); + let result = decode_gpp_string(&gpp).expect("should decode GPP without US sections"); assert_eq!( result.us_sale_opt_out, None, - "should return None when no US section (7-23) is present" + "should return None when neither USP nor US-state sections are present" ); } diff --git a/crates/trusted-server-core/src/consent/mod.rs b/crates/trusted-server-core/src/consent/mod.rs index 72c694a32..4597d2469 100644 --- a/crates/trusted-server-core/src/consent/mod.rs +++ b/crates/trusted-server-core/src/consent/mod.rs @@ -468,6 +468,7 @@ fn restrictive_identity_signals_allow(signals: &RawConsentSignals, config: &Cons }; if gpp.us_sale_opt_out == Some(true) || gpp.eu_tcf.as_ref().is_some_and(|tcf| !tcf_allowed(tcf)) + || (gpp.section_ids.contains(&GPP_SECTION_ID_TCF_EU_V2) && gpp.eu_tcf.is_none()) || (gpp.us_sale_opt_out.is_none() && gpp.eu_tcf.is_none()) { return false; diff --git a/crates/trusted-server-core/src/consent/types.rs b/crates/trusted-server-core/src/consent/types.rs index 7d1b1902a..89bae77a0 100644 --- a/crates/trusted-server-core/src/consent/types.rs +++ b/crates/trusted-server-core/src/consent/types.rs @@ -301,12 +301,12 @@ pub struct GppConsent { pub section_ids: Vec, /// Decoded EU TCF v2.2 section (if present in GPP, section ID 2). pub eu_tcf: Option, - /// Whether the user opted out of sale of personal information via a US GPP - /// section (IDs 7–23). + /// Whether the user opted out of sale via a legacy US Privacy section (6) + /// or a US state/national GPP section (7–23). /// - /// - `Some(true)` — a US section is present and `sale_opt_out == OptedOut` - /// - `Some(false)` — a US section is present and user did not opt out - /// - `None` — no US section exists in the GPP string + /// - `Some(true)` — any decoded US section signals a sale opt-out + /// - `Some(false)` — a decoded US section has no sale opt-out + /// - `None` — no decoded US section yields a usable sale signal pub us_sale_opt_out: Option, } diff --git a/crates/trusted-server-core/src/ec/identity.rs b/crates/trusted-server-core/src/ec/identity.rs index ac587d68e..056c17b7d 100644 --- a/crates/trusted-server-core/src/ec/identity.rs +++ b/crates/trusted-server-core/src/ec/identity.rs @@ -430,6 +430,46 @@ mod tests { assert_eq!(entry.ids["other.example.com"].uid, "untouched"); } + #[test] + fn usp_section_in_gpp_allows_capture_persistence_from_request_or_body() { + let id = ec_id("gppusp01"); + let partners = registry(Some(MODULE)); + for mode in [ConsentMode::Proxy, ConsentMode::Interpreter] { + let mut settings = create_test_settings(); + settings.consent.mode = mode; + for in_body in [false, true] { + let graph = live_graph(&id); + let mut request = context(&id); + let mut observation = issued(SOURCE, "synthetic-uid", None); + if in_body { + observation.consent.gpp = Some("DBABTA~1YNN".to_owned()); + } else { + request.consent = + crate::consent::build_context_from_signals(&RawConsentSignals { + raw_gpp_string: Some("DBABTA~1YNN".to_owned()), + ..Default::default() + }); + request.consent.jurisdiction = Jurisdiction::NonRegulated; + } + + assert!( + capture_and_apply( + &graph, + &mut request, + &partners, + &settings, + observation, + CLAIMS, + ), + "should accept a valid no-opt-out USP section in request or body GPP" + ); + let (entry, _) = stored(&graph, &id); + assert_eq!(entry.ids[SOURCE].uid, "synthetic-uid"); + assert_eq!(entry.ids[SOURCE].writer.as_deref(), Some(MODULE)); + } + } + } + #[test] fn restrictive_body_signals_fail_closed_even_when_jurisdiction_or_mode_is_permissive() { let id = ec_id("consent1"); @@ -447,6 +487,18 @@ mod tests { gpp: Some("not-gpp".to_owned()), ..Default::default() }, + IdentityConsentSignals { + gpp: Some("DBABTA~1YYN".to_owned()), + ..Default::default() + }, + IdentityConsentSignals { + gpp: Some("DBABTA~1XNN".to_owned()), + ..Default::default() + }, + IdentityConsentSignals { + gpp: Some("DBACNY~not-a-valid-tcf~1YNN".to_owned()), + ..Default::default() + }, IdentityConsentSignals { usp: Some("not-usp".to_owned()), ..Default::default() @@ -479,9 +531,15 @@ mod tests { let partners = registry(Some(MODULE)); let settings = create_test_settings(); let graph = live_graph(&id); - for (raw_usp, gpc) in [(Some("1YYN"), false), (None, true)] { + for (raw_usp, raw_gpp, gpc) in [ + (Some("1YYN"), None, false), + (None, Some("DBABTA~1YYN"), false), + (None, Some("DBABTA~1XNN"), false), + (None, None, true), + ] { let mut denied = context(&id); denied.consent.raw_us_privacy = raw_usp.map(str::to_owned); + denied.consent.raw_gpp_string = raw_gpp.map(str::to_owned); denied.consent.gpc = gpc; assert!( stage_capture( diff --git a/crates/trusted-server-core/src/integrations/lockr.rs b/crates/trusted-server-core/src/integrations/lockr.rs index cbe5e2f80..953819228 100644 --- a/crates/trusted-server-core/src/integrations/lockr.rs +++ b/crates/trusted-server-core/src/integrations/lockr.rs @@ -354,19 +354,19 @@ fn id5_token(token: &Value, ids: Option<&Value>, now: Option) -> Option Date: Mon, 12 Oct 2026 10:36:27 -0500 Subject: [PATCH 5/7] Reconstruct provenance for Lockr-captured ID5 IDs Use the persisted acquisition writer and source to restore Lockr's fixed EID provenance without changing the KV schema or capture activation. Keep other writers unattributed and suppress reconstructed provenance when browser input adds mixed-origin UIDs for the same source. Update the specs, guide and verification plan, with regressions for writer eligibility, browser precedence and the KV-to-bidder path. --- .../src/auction/endpoints.rs | 174 +++++++++++++++++- crates/trusted-server-core/src/ec/eids.rs | 76 ++++++-- .../src/integrations/lockr.rs | 13 ++ docs/guide/integrations/lockr.md | 6 +- ...rver-side-identity-foundation-and-lockr.md | 12 +- ...026-10-08-lockr-identity-capture-design.md | 47 ++++- ...-server-side-identity-foundation-design.md | 29 ++- 7 files changed, 322 insertions(+), 35 deletions(-) diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index 26ccce8eb..8aff7f3c2 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -576,8 +576,25 @@ fn parse_client_auction_uid(raw: &JsonValue) -> Option { pub(crate) fn merge_auction_eids( client_eids: Option>, - resolved_eids: Option>, + mut resolved_eids: Option>, ) -> Option> { + // EID-level provenance applies to all UIDs for the source. Do not extend + // a stored writer's provenance to additional browser UIDs of unknown origin. + for resolved in resolved_eids.iter_mut().flatten() { + let has_other_browser_uid = client_eids.iter().flatten().any(|client| { + client.source == resolved.source + && client.uids.iter().any(|uid| { + !uid.id.trim().is_empty() + && uid.id.len() <= MAX_UID_LENGTH + && !resolved.uids.iter().any(|stored| stored.id == uid.id) + }) + }); + if has_other_browser_uid { + resolved.inserter = None; + resolved.matcher = None; + resolved.mm = None; + } + } let mut merged = Vec::new(); for eid in resolved_eids @@ -604,8 +621,8 @@ pub(crate) fn merge_auction_eids( } }; - // Browser provenance wins over missing KV provenance. Empty matcher - // and zero-valued mm are both meaningful supplied values. + // Explicit browser provenance wins over reconstructed KV provenance. + // Empty matcher and zero-valued mm are meaningful supplied values. if eid.inserter.is_some() { merged[source_index].inserter = eid.inserter; } @@ -1700,6 +1717,79 @@ mod tests { } } + #[tokio::test] + async fn auction_delivers_lockr_provenance_from_persisted_writer() { + let settings = create_test_settings(); + let mut orchestrator = AuctionOrchestrator::new(AuctionConfig { + enabled: true, + providers: AuctionConfig::legacy_provider_map(&["eid_capturing_provider"]), + ..Default::default() + }); + let requests = Arc::new(Mutex::new(Vec::new())); + orchestrator.register_provider(Arc::new(EidCapturingProvider { + requests: Arc::clone(&requests), + })); + let mut partner = counting_test_partner("id5-sync.com"); + partner.openrtb_atype = 1; + let registry = + PartnerRegistry::from_config(&[partner]).expect("should build source registry"); + let graph = KvIdentityGraph::in_memory("lockr-provenance-auction"); + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let now = crate::ec::current_timestamp(); + let mut entry = crate::ec::kv_types::KvEntry::minimal("id5-sync.com", "synthetic-uid", now); + let record = entry + .ids + .get_mut("id5-sync.com") + .expect("should contain ID5"); + record.writer = Some("lockr".to_owned()); + record.expires_at = Some(now + 3_600); + record.revision = Some("synthetic-revision".to_owned()); + graph + .create(&ec_id, &entry) + .expect("should persist identity"); + let stored_before = graph.get(&ec_id).expect("should read identity"); + let mut context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + let request = Request::builder() + .method("POST") + .uri("https://publisher.example.com/auction") + .body(EdgeBody::from( + serde_json::to_vec(&json!({"adUnits": [{ + "code": "example-slot", + "mediaTypes": {"banner": {"sizes": [[300, 250]]}} + }]})) + .expect("should serialize auction"), + )) + .expect("should build auction request"); + // This provider deliberately fails after recording its bidder-facing request. + let _ = handle_auction( + &settings, + &orchestrator, + Some(&graph), + Some(®istry), + &mut context, + &noop_services(), + request, + ) + .await; + let captured = requests.lock().expect("should lock bidder requests"); + let sent = captured.last().expect("should dispatch bidder request"); + let payload = serde_json::to_value(sent.user.eids.as_ref().expect("should deliver ID5")) + .expect("should serialize bidder EIDs"); + assert_eq!( + payload, + json!([{ + "source": "id5-sync.com", "inserter": "Lockr-For-Publishers", + "matcher": "", "mm": 2, + "uids": [{"id": "synthetic-uid", "atype": 1}] + }]) + ); + assert_eq!( + graph.get(&ec_id).expect("should read identity"), + stored_before, + "should reconstruct provenance without changing persisted state" + ); + } + #[test] fn resolve_auction_eids_returns_empty_without_snapshot() { let registry = PartnerRegistry::empty(); @@ -1820,23 +1910,95 @@ mod tests { ); } + #[test] + fn lockr_provenance_does_not_label_other_browser_uids() { + let ec_id = format!("{}.ABC123", "a".repeat(64)); + let now = crate::ec::current_timestamp(); + let mut entry = crate::ec::kv_types::KvEntry::minimal("id5-sync.com", "captured-uid", now); + let record = entry + .ids + .get_mut("id5-sync.com") + .expect("should contain ID5"); + record.writer = Some("lockr".to_owned()); + record.expires_at = Some(now + 3_600); + let snapshot = EcKvSnapshot::Present { + ec_id: ec_id.clone(), + entry: Box::new(entry), + generation: Some(1), + }; + let registry = PartnerRegistry::from_config(&[counting_test_partner("id5-sync.com")]) + .expect("should build source registry"); + let context = make_ec_context(Jurisdiction::NonRegulated, Some(&ec_id)); + for (uids, body_provenance, captured_provenance) in [ + (vec!["captured-uid"], false, true), + (vec!["browser-uid"], false, false), + (vec!["captured-uid", "browser-uid"], false, false), + (vec!["browser-uid"], true, false), + ] { + let mut raw = json!([{"source": "id5-sync.com", + "uids": uids.iter().map(|id| json!({"id": id})).collect::>() }]); + if body_provenance { + raw[0]["inserter"] = json!("Browser-Integration"); + raw[0]["matcher"] = json!(""); + raw[0]["mm"] = json!(0); + } + let client = parse_client_auction_eids(Some(&raw)); + let merged = + resolve_and_merge_auction_eids(client, &snapshot, Some(®istry), &context) + .expect("should merge captured and browser EIDs"); + let payload = serde_json::to_value(&merged).expect("should serialize EIDs"); + let eid = &payload[0]; + if body_provenance { + assert_eq!(eid["inserter"], "Browser-Integration"); + assert_eq!(eid["matcher"], ""); + assert_eq!(eid["mm"], 0); + } else if captured_provenance { + assert_eq!(eid["inserter"], "Lockr-For-Publishers"); + assert_eq!(eid["matcher"], ""); + assert_eq!(eid["mm"], 2); + } else { + for field in ["inserter", "matcher", "mm"] { + assert!( + eid.get(field).is_none(), + "should not attribute a mixed-origin EID to Lockr" + ); + } + } + let mut expected_uids = vec!["captured-uid"]; + if uids.contains(&"browser-uid") { + expected_uids.push("browser-uid"); + } + assert_eq!( + merged[0] + .uids + .iter() + .map(|uid| uid.id.as_str()) + .collect::>(), + expected_uids, + "should preserve existing UID merge behavior" + ); + } + } + #[test] fn body_provenance_survives_parse_merge_and_serialization() { - let raw = json!([{"source":"id5-sync.com", "inserter":"Lockr-For-Publishers", + let raw = json!([{"source":"id5-sync.com", "inserter":"Browser-Integration", "matcher":"", "mm":0, "uids":[{"id":"uid", "ext":{"linkType":1}}]}]); let client = parse_client_auction_eids(Some(&raw)); let resolved = Some(vec![Eid { source: "id5-sync.com".to_owned(), + inserter: Some("Lockr-For-Publishers".to_owned()), + matcher: Some("stored-matcher".to_owned()), + mm: Some(2), uids: vec![Uid { id: "uid".to_owned(), atype: Some(1), ext: None, }], - ..Eid::default() }]); let merged = merge_auction_eids(client, resolved).expect("should merge body and KV"); let output = serde_json::to_value(&merged).expect("should serialize"); - assert_eq!(output[0]["inserter"], "Lockr-For-Publishers"); + assert_eq!(output[0]["inserter"], "Browser-Integration"); assert_eq!(output[0]["matcher"], ""); assert_eq!(output[0]["mm"], 0); assert_eq!(output[0]["uids"][0]["atype"], 1); diff --git a/crates/trusted-server-core/src/ec/eids.rs b/crates/trusted-server-core/src/ec/eids.rs index 11735e052..a1a380200 100644 --- a/crates/trusted-server-core/src/ec/eids.rs +++ b/crates/trusted-server-core/src/ec/eids.rs @@ -16,6 +16,8 @@ pub struct ResolvedPartnerId { pub source_domain: String, /// The synced user ID value. pub uid: String, + /// Persisted acquisition writer, never inferred from configured ownership. + pub writer: Option, /// `OpenRTB` agent type for this partner's identifiers. pub openrtb_atype: i32, } @@ -50,6 +52,7 @@ pub fn resolve_partner_ids( resolved.push(ResolvedPartnerId { source_domain: partner.source_domain.clone(), uid: partner_uid.uid.clone(), + writer: partner_uid.writer.clone(), openrtb_atype: partner.openrtb_atype, }); } @@ -58,21 +61,25 @@ pub fn resolve_partner_ids( resolved } -/// Converts resolved partner IDs to `OpenRTB` `Eid` entries. +/// Converts resolved partner IDs to `OpenRTB` EIDs with supported writer provenance. +/// +/// Lockr owns the fixed ID5 mapping. Other writers retain absent provenance. #[must_use] pub fn to_eids(resolved: &[ResolvedPartnerId]) -> Vec { resolved .iter() - .map(|item| Eid { - inserter: None, - matcher: None, - mm: None, - source: item.source_domain.clone(), - uids: vec![Uid { - id: item.uid.clone(), - atype: Some(item.openrtb_atype), - ext: None, - }], + .map(|item| { + let mut eid = Eid { + source: item.source_domain.clone(), + uids: vec![Uid { + id: item.uid.clone(), + atype: Some(item.openrtb_atype), + ext: None, + }], + ..Eid::default() + }; + crate::integrations::lockr::apply_eid_provenance(&mut eid, item.writer.as_deref()); + eid }) .collect() } @@ -198,21 +205,68 @@ mod tests { ); } + #[test] + fn lockr_provenance_requires_the_stored_writer_and_id5_source() { + for source in ["id5-sync.com", "ids.example.com"] { + let mut partner = make_test_partner(source); + partner.identity_owner = Some("lockr".to_owned()); + let registry = + PartnerRegistry::from_config(&[partner]).expect("should build source registry"); + for writer in [ + None, + Some("browser"), + Some("legacy_pull"), + Some("other"), + Some("lockr"), + ] { + let mut entry = KvEntry::minimal(source, "synthetic-uid", 1_000); + let record = entry.ids.get_mut(source).expect("should contain source"); + record.writer = writer.map(str::to_owned); + record.expires_at = Some(2_000); + let resolved = resolve_partner_ids(®istry, &entry, Some(1_999)); + let payload = serde_json::to_value(to_eids(&resolved)) + .expect("should serialize resolved EIDs"); + let eid = &payload[0]; + assert_eq!(eid["source"], source); + assert_eq!(eid["uids"][0]["id"], "synthetic-uid"); + if source == "id5-sync.com" && writer == Some("lockr") { + assert_eq!(eid["inserter"], "Lockr-For-Publishers"); + assert_eq!(eid["matcher"], ""); + assert_eq!(eid["mm"], 2); + } else { + for field in ["inserter", "matcher", "mm"] { + assert!( + eid.get(field).is_none(), + "should not infer provenance from owner or source alone" + ); + } + } + assert!(resolve_partner_ids(®istry, &entry, Some(2_000)).is_empty()); + assert!(resolve_partner_ids(®istry, &entry, None).is_empty()); + entry.consent.ok = false; + assert!(resolve_partner_ids(®istry, &entry, Some(1_999)).is_empty()); + } + } + } + #[test] fn to_eids_maps_resolved_ids_correctly() { let resolved = vec![ ResolvedPartnerId { uid: "LR_xyz".to_owned(), + writer: None, source_domain: "liveramp.com".to_owned(), openrtb_atype: 3, }, ResolvedPartnerId { uid: "ID5_abc".to_owned(), + writer: None, source_domain: "id5-sync.com".to_owned(), openrtb_atype: 1, }, ResolvedPartnerId { uid: "pair-id".to_owned(), + writer: None, source_domain: "google.com".to_owned(), openrtb_atype: 571187, }, diff --git a/crates/trusted-server-core/src/integrations/lockr.rs b/crates/trusted-server-core/src/integrations/lockr.rs index 953819228..46f1a704f 100644 --- a/crates/trusted-server-core/src/integrations/lockr.rs +++ b/crates/trusted-server-core/src/integrations/lockr.rs @@ -30,6 +30,7 @@ use crate::integrations::{ UPSTREAM_SDK_MAX_RESPONSE_BYTES, collect_body_bounded, collect_response_bounded, ensure_integration_backend, }; +use crate::openrtb::Eid; use crate::platform::{PlatformHttpRequest, RuntimeServices}; use crate::settings::{IntegrationConfig, Settings}; @@ -44,6 +45,18 @@ const LOCKR_TOKEN_PATHS: [&str; 4] = [ ]; const LOCKR_REVOKE_PATH: &str = "/publisher/app/v2/identityLockr/revoke-consent"; +/// Reconstructs Lockr's fixed ID5 provenance from the persisted acquisition writer. +/// +/// Source or configured ownership alone cannot establish Lockr origin. This +/// mapping remains valid for stored captures even when new capture is disabled. +pub(crate) fn apply_eid_provenance(eid: &mut Eid, writer: Option<&str>) { + if eid.source == ID5_SOURCE && writer == Some(LOCKR_INTEGRATION_ID) { + eid.inserter = Some("Lockr-For-Publishers".to_owned()); + eid.matcher = Some(String::new()); + eid.mm = Some(2); + } +} + /// Configuration for Lockr integration. #[derive(Debug, Deserialize, Validate)] pub struct LockrConfig { diff --git a/docs/guide/integrations/lockr.md b/docs/guide/integrations/lockr.md index ac6c78154..ca796228b 100644 --- a/docs/guide/integrations/lockr.md +++ b/docs/guide/integrations/lockr.md @@ -93,7 +93,9 @@ User data is stored securely in Lockr's privacy vault: ### Extended ID (EID) Support -A later server-side auction can include a usable registered ID5 record as an `id5-sync.com` EID with `atype=1`. Trusted Server does not store provider `ext` or OpenRTB `inserter`/`matcher` metadata in KV; browser-supplied provenance remains current-request-only. Other provider token mappings are not enabled by the ID5 parser. +A later server-side auction can include a usable registered ID5 record as an `id5-sync.com` EID with the registry's `atype=1`. When the persisted record's writer is `lockr`, Trusted Server reconstructs `inserter: "Lockr-For-Publishers"`, `matcher: ""` and `mm: 2`. These fixed values require no extra KV fields and remain available if new capture is disabled. Source or configured ownership alone does not establish Lockr origin; browser, legacy and unknown-writer records are not labeled as Lockr. + +Explicit browser-supplied provenance retains current-request precedence. If the browser adds a different UID for the same source, the merged EID omits reconstructed provenance to avoid attributing every UID to Lockr. Trusted Server does not store arbitrary provenance, provider `ext` or extra UIDs in KV. Other provider token mappings are not enabled by the ID5 parser. ## Use Cases @@ -136,7 +138,7 @@ The Lockr integration is implemented in [crates/trusted-server-core/src/integrat **ID Mapping**: - The only supported capture mapping is Lockr `id5id` to registered `id5-sync.com` with `atype=1`. -- The EC identity graph retains the source, UID and lifecycle metadata, including known provider expiry. It does not store provider `ext` or additional UIDs. +- The EC identity graph retains the source, UID and lifecycle metadata, including known provider expiry and the acquisition writer. Later auctions reconstruct fixed Lockr ID5 provenance only for a usable Lockr-written record. KV does not store arbitrary provenance, provider `ext` or additional UIDs. - Publisher SDK storage remains the SDK's responsibility. Capture does not add an EID cookie. **Consent Validation**: diff --git a/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md b/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md index 718cd88d5..9fa1f1c10 100644 --- a/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md +++ b/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md @@ -66,7 +66,7 @@ cargo test --package trusted-server-cli --target "$HOST_TARGET" **Files:** `crates/trusted-server-core/src/{auction/endpoints.rs,auction/formats.rs,auction/types.rs,openrtb.rs,ec/prebid_eids.rs,ec/finalize.rs,ec/admin.rs,ec/eids.rs,constants.rs,cookies/template_cache_policy.rs,publisher.rs}`, `crates/trusted-server-adapter-fastly/src/{app.rs,main.rs}`, `crates/trusted-server-js/lib/src/integrations/prebid/index.ts` and its existing tests; update operator docs at `docs/guide/integrations/prebid.md` and `docs/guide/configuration.md` for changed transport. Limit other adapters to compile/behavior parity; they currently have no EC KV graph. - Red: registered body EIDs enrich an existing root even on no-bid or handled auction-provider error; unknown sources remain current-request-only; no-token/consent-denied/missing/failed/tombstone roots do not write or recover. Prove one combined browser finalization attempt with `sharedId`, using the auction snapshot rather than a second unconditional KV read. Directly reuse the EID-to-update normalization in `ec/prebid_eids.rs`, not a fake cookie. Stage normalized updates in `EcContext` or a typed response effect before dispatch so `Err` paths retain them through the adapter's error conversion; inspect `app.rs` response-finalization behavior rather than assuming an error response keeps extensions. -- Green: keep normal EC finalization timing and publisher-only orphan recovery, including no seeding from the retired cookie. Extend `Eid` with optional string `inserter`/`matcher` and an OpenRTB-compatible integer `mm`, omitted only when absent. Update both the manual parser in `parse_client_auction_eids` and `merge_auction_eids`, not only serde derives. Body-supplied provenance wins over missing KV provenance; preserve empty `matcher` and supplied zero-valued `mm`. Keep existing UID merge precedence and retain UID `ext` for the current auction. Update affected struct constructors and serialization tests. KV-derived EIDs remain registry-`atype` only. Preserve source/UID caps and consent gates. Filter the matching known-expired body UID before merging. +- Green: keep normal EC finalization timing and publisher-only orphan recovery, including no seeding from the retired cookie. Extend `Eid` with optional string `inserter`/`matcher` and an OpenRTB-compatible integer `mm`, omitted only when absent. Update both the manual parser in `parse_client_auction_eids` and `merge_auction_eids`, not only serde derives. Explicit body-supplied provenance wins over reconstructed KV provenance; preserve empty `matcher` and supplied zero-valued `mm`. Keep existing UID merge precedence and retain UID `ext` for the current auction. Update affected struct constructors and serialization tests. KV-derived EIDs retain registry `atype` and reconstruct fixed Lockr ID5 provenance only from the persisted `lockr` writer. Keep the mapping in `integrations/lockr.rs`, not a source-only attribution table. Different browser UIDs suppress reconstruction for a mixed-origin EID without changing UID merging. Preserve source/UID caps and consent gates. Filter the matching known-expired body UID before merging. - Only after body tests pass: remove `/auction` cookie fallback, server cookie ingest, JS writer, header emission and old cookie-dependent diagnostics. If retiring an admin route, keep it locally denied with its existing authentication/no-store behavior; do not remove its route and forward admin credentials to publisher fallback. Apply that protection to the Axum and Spin routes as well as Fastly. Keep `x-ts-eids` and `x-ts-eids-truncated` on the inbound strip list; keep EC/consent headers. Keep one transition JS release that expires the old host-only `Path=/` cookie, with a named removal condition: cached bundles that write it have aged beyond the existing one-day max-age and deployment cache lifetime. New servers ignore old bundles' writes. No unconditional server `Set-Cookie` cleanup. Agree an admin diagnostic replacement/removal and identify external header consumers before this deletion. - Check: filtered `cargo test-fastly` auction/finalize/admin/publisher tests, JS Prebid tests, `rg -n 'ts-eids|x-ts-eids'` review of remaining intentional compatibility references. Verify before/after request cookies and response cache privacy through Fastly tests. @@ -82,7 +82,7 @@ cargo test --package trusted-server-cli --target "$HOST_TARGET" **Files:** `crates/trusted-server-core/src/integrations/lockr.rs`, shared acquisition consent handling in `crates/trusted-server-core/src/{consent/mod.rs,consent/types.rs,ec/consent.rs,ec/identity.rs}`, its tests and sanitized fixtures under its existing test module or a dedicated `crates/trusted-server-core/tests/fixtures/lockr/` directory; `trusted-server.example.toml` and `docs/guide/integrations/lockr.md` for confirmed configuration and operator behavior. Register capture only, not a resolver. Add default-true `capture_identity` kill switch without removing the ownership descriptor. Keep SDK, settings and unlisted paths on the existing proxy path; inspect exact POST `page-view`, `generate-tokens`, `refresh-tokens`, `sync-no-hem-ids`, `revoke-consent` paths. Reuse the already buffered POST bytes for request consent without changing forwarding. -- Fixture gate before enabling each mapping: sanitized SDK `tokenMappings` and `tokenSourceMappings` **and** response examples prove token shape, key/source/type, expiry unit and consent contract. Start with ID5, using `id5id`, `id5-sync.com` and `atype=1`, after its fixture and consent gates pass; decode URL-encoded JSON `advertising_token.universal_uid`, associate `ids..eid` when present, reject mismatched source/oversized UID, convert `identity_expires` milliseconds to seconds and reject expired issue. Do not infer LiveRamp, UID2, EUID, First ID, Yahoo, Criteo or Epsilon token decoding from ID5; leave each unmapped until its own fixture passes. Panorama source/type remains disabled until Lockr confirms it. No provider provenance or `ext` is stored in KV. +- Fixture gate before enabling each mapping: sanitized SDK `tokenMappings` and `tokenSourceMappings` **and** response examples prove token shape, key/source/type, expiry unit and consent contract. Start with ID5, using `id5id`, `id5-sync.com` and `atype=1`, after its fixture and consent gates pass; decode URL-encoded JSON `advertising_token.universal_uid`, associate `ids..eid` when present, reject mismatched source/oversized UID, convert `identity_expires` milliseconds to seconds and reject expired issue. Do not infer LiveRamp, UID2, EUID, First ID, Yahoo, Criteo or Epsilon token decoding from ID5; leave each unmapped until its own fixture passes. Panorama source/type remains disabled until Lockr confirms it. No arbitrary provider provenance or `ext` is stored in KV. Reconstruct fixed `inserter="Lockr-For-Publishers"`, `matcher=""`, `mm=2` for usable Lockr-written ID5 records without a new field or mutation; other writers and sources remain unattributed. - Red: exact route and method selection, aimTokens-only and associated EID, unknown key, source mismatch, non-2xx/malformed/no-token no invalidation, missing/extended expiry, repeat equal observation no write, original/decoded overflow with unchanged proxy response. Consent tests include populated opt-out, malformed non-empty strings, empty optional strings, cookie/body conflict, GPC and otherwise permissive/non-regulated or proxy-mode request policy. Map Lockr's buffered `gppString`, `consentString` and `ccpaString` into named shared consent inputs. Reuse shared decoders and request jurisdiction/expiration policy. Require explicit parse success for populated signals; the existing decoder's absent decoded value or a permissive jurisdiction must not turn malformed body consent into permission. Empty fields impose no positive permission, and body consent can restrict but never override denied request consent. Keep this acquisition check separate from approved explicit withdrawal handling. Never log UID, query-bearing URL, consent string or token; audit shared consent display and error logs as well as Lockr logs. Remove the current full-target-URL logging/error attachment in `lockr.rs` on captured paths, and audit shared error formatting. - Withdrawal is gated separately: only after Lockr confirms 2xx application-success semantics **and** whether revoke is global EC withdrawal or per-source invalidation. Test revoke despite denied acquisition consent, finalization precedence, cookie expiry and existing-key-only tombstone if global scope is approved. Until then, leave revoke capture disabled and let ordinary request consent withdrawal continue; never infer withdrawal from empty tokens or a network error. Clarify ordering if overlapping SDK calls can return older tokens after newer ones; absent a provider sequence, CAS order is observation order only. - Check: filtered `cargo test-fastly` Lockr parser/proxy/post-send tests with synthetic responses; no live IDs in fixtures. Then run approved publisher test-route acceptance: consent, unchanged original response, delayed KV readback of UID/source/expiry/writer/revision without exporting values, later bidder payload source presence, same-UID extension, known-expired omission including matching body UID, negative consent and approved revoke. Record statuses, counts and booleans only. Capture stays off for a source until its fixture, consent, ownership and compatibility gates pass. @@ -113,3 +113,11 @@ cargo test --package trusted-server-cli --target "$HOST_TARGET" - Ownership/config, restrictive consent, missing/failed/withdrawn/full-ID isolation, CAS/no-op/ABA/expiry, browser one-finalization, expiry bypass and withdrawal tests passed. Independent bounded source review found no additional verified privacy/concurrency defect. Not verified: real provider acceptance, sanitized provider-approved mappings/fixtures, consent/revoke scope approval, overlapping response ordering, deployment propagation/drain, remote CI or external diagnostic consumers. Synthetic fixtures are not provider approval. Keep both Lockr capture switches explicitly false until their independent gates pass. Native LiveRamp, refresh/invalidation scheduling, durable retry and cross-browser linking remain deferred. + +## Writer-based provenance follow-up verification + +- Adopted the fixed Lockr ID5 provenance values for issuance and refresh. No KV schema, acquisition policy or rollout switch changed. +- Added failing regressions before implementation. Native `cargo test -p trusted-server-core --target x86_64-unknown-linux-gnu lockr_provenance` now passes all three tests: stored source/writer eligibility, persisted KV-to-bidder payload with no mutation, and mixed-origin browser UID handling. The existing body-provenance regression also passes with explicit fields overriding reconstructed values. +- `cargo test-fastly`: 219 adapter tests and 2,933 core tests passed, with six core tests ignored; shared-crate and doc tests also passed. +- `cargo clippy-fastly`, `cargo fmt --all -- --check`, scoped Prettier checks for the updated specs/plan/guide, and `git diff --check` passed. +- Other adapter suites, the CLI/browser-fixture suites, remote CI and live provider acceptance were not rerun for this follow-up. Capture remains gated as above. diff --git a/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md b/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md index 70f0016d3..9210ab07c 100644 --- a/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md +++ b/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md @@ -122,6 +122,8 @@ retiring the cookie path. including when current request consent no longer allows identity use. 7. Retain consent, size, UID, root and tombstone safety requirements from the foundation. A failed capture must not break the SDK call. +8. Reconstruct the fixed ID5 OpenRTB provenance on later auctions only for + usable records whose persisted acquisition writer is `lockr`. Out of scope: server-to-server calls to lockr, identifier intake, cross-browser linking, changing Prebid provider discovery, direct ID5 or LiveRamp proxies, @@ -251,10 +253,30 @@ response bytes. Other adapters follow the foundation's supported timing and KV capability rules. Auctions resolve usable registered records from KV, deriving `atype` from the -source registry. Set the registry type to the confirmed mapping. Do not stamp -`Lockr-For-Publishers` onto every KV ID for a source: another input may have -populated it. Provider `ext`, extra UIDs and OpenRTB provenance remain absent -from later KV-only auctions, as defined by the foundation. +source registry. Set the registry type to the confirmed mapping. Carry the +persisted record's `writer` through resolution. For `source = "id5-sync.com"` +and `writer = "lockr"`, reconstruct the fixed SDK provenance: + +```json +{ + "inserter": "Lockr-For-Publishers", + "matcher": "", + "mm": 2 +} +``` + +This delivery adopts these values for Lockr ID5 issuance and refresh. The mapping +belongs to the Lockr integration and adds no KV fields. Do not infer it from +`identity_owner`, source alone, or current capture configuration. Browser, +legacy, other-writer and unknown-origin records remain without reconstructed +Lockr provenance; other sources are not covered by this ID5 mapping. Disabling +new capture does not remove provenance from already stored usable Lockr records. + +Explicit browser provenance retains current-request merge precedence, including +an empty `matcher` and zero-valued `mm`. If browser input adds a different UID +for the same source, omit reconstructed provenance for that merged EID rather +than attribute all its UIDs to Lockr. Keep existing UID merging unchanged. +Provider `ext` and extra UIDs are still not stored in KV. ## 5. Data flow and freshness @@ -266,7 +288,7 @@ flowchart TD D --> E[Original response sent without an EID cookie] E --> F[Shared service applies authoritative updates after send] F --> G[EC identity graph stores IDs and lifecycle metadata] - G --> H[Later auctions read visible unexpired IDs] + G --> H[Later auctions read visible unexpired IDs and reconstruct writer-based provenance] ``` | Event | Persistence | Auction availability | @@ -348,6 +370,13 @@ prove that this consumer reaches those paths correctly: scope and does not duplicate request-finalization tombstones. - Owned/registered sources persist; unregistered or differently owned sources do not; browser input cannot impersonate Lockr authority. +- Persisted Lockr ID5 writer reconstructs `inserter`, empty `matcher` and `mm=2` + in later bidder-facing payloads without a KV mutation. Configured ownership + alone, browser/legacy/unknown writers and other sources do not acquire these + fields. Existing expiry, withdrawal and bidstream gates still apply. +- Matching body UIDs retain reconstructed provenance unless explicit body fields + override it. Different body UIDs suppress reconstruction for the merged EID, + without dropping UIDs or explicit body provenance. - Capture effects execute after Fastly send through the foundation, not a separate Lockr adapter implementation; persisted snapshot handoff is retained. - No newly emitted TS EID cookie or diagnostic EID header; provider browser @@ -359,7 +388,8 @@ Live acceptance on an approved publisher test route: 2. Confirm original response bytes and no TS EID cookie from capture. 3. Allow for post-send work and KV visibility. Inspect the session's admin EC record for the registered source and lifecycle metadata, without exporting IDs. -4. Trigger a later auction and inspect bidder-facing payload evidence. Do not +4. Trigger a later auction and inspect bidder-facing payload evidence, including + the stored Lockr ID5 record's `inserter`, empty `matcher` and `mm=2`. Do not require the browser body to contain a server-added ID or rely on `x-ts-eids`. 5. Reload with no new tokens: unchanged UID and expiry do not cause a write. 6. Trigger SDK refresh: confirm changed UID or expiry persists after visibility. @@ -400,7 +430,8 @@ rollback contract. No `ts-eids-s` is introduced at any stage. infer provider issuance order from edge wall clocks. Known expiry persistence is now a proposed foundation decision rather than an -unimplemented Lockr-local open question. OpenRTB provenance storage remains -outside this delivery. Active Lockr resolution from `ltid`, direct provider +unimplemented Lockr-local open question. Arbitrary OpenRTB provenance storage +remains outside this delivery; the fixed Lockr ID5 mapping is reconstructed from +the persisted writer instead. Active Lockr resolution from `ltid`, direct provider capture and client-side consumers require separate follow-up specs; no current S2S contract is assumed. diff --git a/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md b/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md index a1eafd9c5..f122a7070 100644 --- a/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md +++ b/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md @@ -242,8 +242,12 @@ that legacy IDs are still valid; tightening unknown-expiry policy is a separate operator decision. The record extension does not store OpenRTB provenance, UID `ext`, or extra UIDs. -Writer identity is storage metadata, not proof that a provider requires a -particular `inserter` or `matcher` value in the bidstream. +Writer identity identifies the acquisition path. An integration can use that +persisted writer and source to reconstruct fixed provenance defined by its spec; +do not infer provenance from the current configured owner or source alone. +The first mapping is Lockr ID5, with `inserter = "Lockr-For-Publishers"`, +`matcher = ""` and `mm = 2`. Other sources and browser, legacy or unknown writers +remain without reconstructed provenance. No additional storage fields are needed. ### 6.2 Mutation classes @@ -285,9 +289,17 @@ becoming an authoritative replacement in KV. Body-supplied OpenRTB provenance remains current-request metadata. Add optional `inserter`, `matcher` and `mm` to `Eid` parsing and merging; an empty `matcher` -is a supplied value. Later KV-derived EIDs use source registry `atype` and do -not synthesize provider provenance. The first delivery preserves existing caps -and consent gates across all auction paths. +is a supplied value. Later KV-derived EIDs use source registry `atype` and the +persisted acquisition writer. For usable Lockr-written `id5-sync.com` records, +reconstruct the fixed provenance in section 6.1 through Lockr's mapping, even +when new capture is disabled. Do not label all records for an owned source. + +Explicit body fields take precedence over reconstructed fields, including empty +`matcher` and zero-valued `mm`. If browser input adds a different UID for the +same source, suppress reconstructed provenance for the resulting mixed-origin +EID because its top-level fields apply to every UID. Preserve explicit body +provenance and the existing UID merge rules. The first delivery preserves +existing caps and consent gates across all auction paths. ## 7. Consent and execution timing @@ -447,6 +459,10 @@ Foundation tests must cover: skips writes; unrelated sources survive CAS conflicts. - Known expiry boundaries across publisher, `/auction` and SPA auction paths, including an expired UID resubmitted in the auction body. +- Writer-aware Lockr ID5 provenance through KV reads and bidder-facing auctions, + including browser/legacy/unknown writers, configured-owner-only records, + unrelated sources, matching versus different body UIDs and explicit body + field precedence. Reconstruction must not write KV or bypass usability gates. - Revision-conditional invalidation and stale resolver-result rejection, with same-UID/new-expiry and concurrent-withdrawal cases. - Missing versus failed roots, no-create enrichment, snapshot identity @@ -470,4 +486,5 @@ Resolve before implementation activation: Provider expiry, writer and revision storage are proposed foundation decisions, not already implemented capabilities. LiveRamp protocol approval, identifier intake, refresh response semantics and country-specific TTL remain for its -future spec. OpenRTB provenance storage remains outside this first delivery. +future spec. Arbitrary OpenRTB provenance storage remains outside this first +delivery; fixed Lockr ID5 provenance is reconstructed from the persisted writer. From 16b6d09a5056ae190b03b52a5573f9600271667d Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 12 Oct 2026 11:52:10 -0500 Subject: [PATCH 6/7] Add authenticated identity capture handoff Restore bounded absent-source availability and post-send retry through a default-off publisher/full-EC-bound ts-eids-s handoff. Use dedicated rotating signing keys, fixed deadlines and fill-absent-only CAS. Preserve pending expiry and writer metadata without granting cookies authority over stored records. Keep browser-written ts-eids retired and document the existing KV snapshot consistency and live rollout gates. --- .../trusted-server-adapter-fastly/src/app.rs | 3 +- .../src/auction/endpoints.rs | 6 +- crates/trusted-server-core/src/config.rs | 338 +++++++- crates/trusted-server-core/src/constants.rs | 2 + crates/trusted-server-core/src/cookies.rs | 86 +- .../src/cookies/template_cache_policy.rs | 6 +- crates/trusted-server-core/src/ec/finalize.rs | 24 +- crates/trusted-server-core/src/ec/identity.rs | 103 ++- .../src/ec/identity_handoff.rs | 367 ++++++++ .../src/ec/identity_handoff_codec.rs | 808 ++++++++++++++++++ .../src/ec/identity_handoff_tests.rs | 786 +++++++++++++++++ crates/trusted-server-core/src/ec/kv.rs | 106 +++ crates/trusted-server-core/src/ec/mod.rs | 26 +- .../src/integrations/prebid.rs | 86 +- .../src/integrations/registry.rs | 49 +- crates/trusted-server-core/src/settings.rs | 250 ++++++ docs/guide/configuration.md | 62 +- docs/guide/integrations/lockr.md | 72 +- ...rver-side-identity-foundation-and-lockr.md | 33 +- ...026-10-08-lockr-identity-capture-design.md | 233 +++-- ...-server-side-identity-foundation-design.md | 212 ++++- trusted-server.example.toml | 17 + 22 files changed, 3521 insertions(+), 154 deletions(-) create mode 100644 crates/trusted-server-core/src/ec/identity_handoff.rs create mode 100644 crates/trusted-server-core/src/ec/identity_handoff_codec.rs create mode 100644 crates/trusted-server-core/src/ec/identity_handoff_tests.rs diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 5e5fd7728..28d784553 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -418,7 +418,7 @@ fn build_ec_request_state( None }); - let (ec_context, setup_error) = + let (mut ec_context, setup_error) = match EcContext::read_from_request_with_geo(settings, req, services, geo_info.as_ref()) { Ok(mut context) => { context.set_device_signals(device_signals); @@ -446,6 +446,7 @@ fn build_ec_request_state( None }; let kv_graph = if is_real_browser { kv_graph } else { None }; + ec_context.set_identity_handoff_supported(finalize_kv_graph.is_some()); EcRequestState { ec_context, diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index 8aff7f3c2..a3d88278e 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -453,7 +453,11 @@ pub(crate) fn resolve_auction_eids( } let resolved = resolve_partner_ids(registry, entry, now); - Some(to_eids(&resolved)) + let mut eids = to_eids(&resolved); + eids.extend(crate::ec::identity_handoff::resolve_eids( + ec_context, snapshot, registry, now, + )); + Some(eids) } /// Merges browser and KV EIDs without letting browser input revive an unusable UID. diff --git a/crates/trusted-server-core/src/config.rs b/crates/trusted-server-core/src/config.rs index 9b72f4e7d..1572b2b09 100644 --- a/crates/trusted-server-core/src/config.rs +++ b/crates/trusted-server-core/src/config.rs @@ -8,6 +8,7 @@ use std::borrow::Cow; +use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD}; use edgezero_core::app_config::{SecretField, SecretKind, SecretPathSegment}; use error_stack::Report; use serde::{Deserialize, Deserializer, Serialize, Serializer}; @@ -145,6 +146,16 @@ impl edgezero_core::app_config::AppConfigMeta for TrustedServerAppConfig { vec![ field(vec![object("publisher"), object("proxy_secret")], false), field(vec![object("ec"), object("passphrase")], false), + field( + vec![ + object("ec"), + optional_object("identity_handoff"), + optional_object("keys"), + SecretPathSegment::ArrayEach, + object("secret"), + ], + false, + ), field( vec![ object("ec"), @@ -245,7 +256,9 @@ impl edgezero_core::app_config::AppConfigMeta for TrustedServerAppConfig { /// Returns [`TrustedServerError`] when non-secret configuration or a secret key /// reference is invalid. pub fn validate_settings_for_deploy(settings: &Settings) -> Result<(), Report> { + validate_identity_handoff_structure(settings)?; validate_secret_key_references(settings)?; + validate_identity_handoff_uses_dedicated_keys(settings)?; validate_non_secret_deploy_placeholders(settings)?; validate_js_asset_proxy_config(settings)?; @@ -269,6 +282,9 @@ pub fn validate_settings_for_deploy(settings: &Settings) -> Result<(), Report Result<(), Report> { + validate_identity_handoff_structure(settings)?; + validate_identity_handoff_runtime_keys(settings)?; + validate_identity_handoff_uses_dedicated_keys(settings)?; settings.reject_placeholder_secrets()?; validate_js_asset_proxy_config(settings)?; settings.validate_admin_handler_passwords()?; @@ -387,6 +403,73 @@ fn validate_non_secret_deploy_placeholders( })) } +fn validate_identity_handoff_structure( + settings: &Settings, +) -> Result<(), Report> { + settings.ec.identity_handoff.validate().map_err(|error| { + Report::new(TrustedServerError::Configuration { + message: format!("ec.identity_handoff configuration is invalid: {error}"), + }) + }) +} + +fn validate_identity_handoff_runtime_keys( + settings: &Settings, +) -> Result<(), Report> { + for (index, key) in settings.ec.identity_handoff.keys.iter().enumerate() { + let decoded = URL_SAFE_NO_PAD.decode(key.secret.expose()).map_err(|_| { + invalid_identity_handoff_secret(index, &key.id, "must use URL_SAFE_NO_PAD base64") + })?; + if decoded.len() != 32 { + return Err(invalid_identity_handoff_secret( + index, + &key.id, + "must decode to exactly 32 bytes", + )); + } + if decoded.as_slice() == settings.ec.passphrase.expose().as_bytes() { + return Err(invalid_identity_handoff_secret( + index, + &key.id, + "must not reuse ec.passphrase key material", + )); + } + } + Ok(()) +} + +fn validate_identity_handoff_uses_dedicated_keys( + settings: &Settings, +) -> Result<(), Report> { + if let Some((index, key)) = settings + .ec + .identity_handoff + .keys + .iter() + .enumerate() + .find(|(_, key)| key.secret.expose() == settings.ec.passphrase.expose()) + { + return Err(invalid_identity_handoff_secret( + index, + &key.id, + "must not reuse ec.passphrase", + )); + } + Ok(()) +} + +fn invalid_identity_handoff_secret( + index: usize, + key_id: &str, + requirement: &str, +) -> Report { + Report::new(TrustedServerError::Configuration { + message: format!( + "secret at `ec.identity_handoff.keys[{index}].secret` for key `{key_id}` {requirement}" + ), + }) +} + fn validate_secret_key_references(settings: &Settings) -> Result<(), Report> { validate_secret_key_reference( "publisher.proxy_secret", @@ -394,6 +477,20 @@ fn validate_secret_key_references(settings: &Settings) -> Result<(), Report IdentityHandoffKey { + IdentityHandoffKey { + id: id.to_owned(), + secret: Redacted::new(secret.to_owned()), + } + } + #[test] fn settings_collections_produce_stable_envelope_hashes_across_parses() { let mut source = serde_json::to_value(valid_settings()).expect("should serialize settings"); @@ -828,6 +935,7 @@ formats = [{ width = 300, height = 250 }] vec![ ("publisher.proxy_secret".to_owned(), false), ("ec.passphrase".to_owned(), false), + ("ec.identity_handoff.keys[*].secret".to_owned(), false,), ("ec.partners[*].api_token".to_owned(), true), ("ec.partners[*].ts_pull_token".to_owned(), true), ("handlers[*].password".to_owned(), false), @@ -861,7 +969,7 @@ formats = [{ width = 300, height = 250 }] } #[test] - fn partner_secret_metadata_makes_the_defaulted_array_optional() { + fn defaulted_array_secret_metadata_uses_optional_container_segments() { let fields = TrustedServerAppConfig::secret_fields(); for field in fields.iter().filter(|field| { @@ -875,6 +983,19 @@ formats = [{ width = 300, height = 250 }] SecretPathSegment::OptionalField(name) if name == "partners" )); } + + let handoff = fields + .iter() + .find(|field| field.dotted_path() == "ec.identity_handoff.keys[*].secret") + .expect("should expose identity handoff key secret metadata"); + assert!(matches!( + &handoff.path[1], + SecretPathSegment::OptionalField(name) if name == "identity_handoff" + )); + assert!(matches!( + &handoff.path[2], + SecretPathSegment::OptionalField(name) if name == "keys" + )); } #[test] @@ -1083,6 +1204,140 @@ gam_network_id = "99999" ); } + #[test] + fn identity_handoff_disabled_empty_config_validates_at_deploy_and_runtime() { + let settings = valid_settings(); + + validate_settings_for_deploy(&settings) + .expect("disabled empty handoff should pass deploy validation"); + validate_settings_for_runtime(&settings) + .expect("disabled empty handoff should pass runtime validation"); + } + + #[test] + fn identity_handoff_deploy_validation_accepts_unresolved_rotation_key_references() { + let mut settings = valid_settings(); + settings.ec.identity_handoff = IdentityHandoffConfig { + enabled: true, + active_key_id: "current".to_owned(), + keys: vec![ + handoff_key("previous", "identity_handoff_previous"), + handoff_key("current", "identity_handoff_current"), + ], + }; + + validate_settings_for_deploy(&settings) + .expect("deploy validation should treat handoff secrets as key references"); + } + + #[test] + fn identity_handoff_deploy_validation_rejects_invalid_or_reused_key_references() { + for secret in [" ", " padded_reference ", "same_key_as_ec_passphrase"] { + let mut settings = valid_settings(); + if secret == "same_key_as_ec_passphrase" { + settings.ec.passphrase = Redacted::new(secret.to_owned()); + } + settings.ec.identity_handoff.keys = vec![handoff_key("key-1", secret)]; + + let error = validate_settings_for_deploy(&settings) + .expect_err("deploy validation should reject an unusable handoff key reference"); + assert!( + error + .to_string() + .contains("ec.identity_handoff.keys[0].secret"), + "error should name the handoff secret path without its value: {error:?}" + ); + if !secret.trim().is_empty() { + assert!( + !error.to_string().contains(secret.trim()), + "error should not expose the handoff secret or reference: {error:?}" + ); + } + } + } + + #[test] + fn identity_handoff_structure_fails_the_same_deploy_and_runtime_boundaries() { + let mut settings = valid_settings(); + settings.ec.identity_handoff = IdentityHandoffConfig { + enabled: true, + active_key_id: "not-configured".to_owned(), + keys: vec![handoff_key("configured", "identity_handoff_key")], + }; + + for error in [ + validate_settings_for_deploy(&settings) + .expect_err("deploy should reject a missing active key"), + validate_settings_for_runtime(&settings) + .expect_err("runtime should reject a missing active key"), + ] { + assert!( + error.to_string().contains("active_key_id"), + "both boundaries should report the active key invariant: {error:?}" + ); + } + } + + #[test] + fn identity_handoff_runtime_validation_requires_canonical_32_byte_keys() { + let valid = URL_SAFE_NO_PAD.encode([7_u8; 32]); + let invalid_cases = [ + ("not-base64url".to_owned(), "URL_SAFE_NO_PAD"), + (URL_SAFE_NO_PAD.encode([7_u8; 31]), "exactly 32 bytes"), + (format!("{valid}="), "URL_SAFE_NO_PAD"), + ]; + + for (secret, expected) in invalid_cases { + let mut settings = valid_settings(); + settings.ec.identity_handoff.keys = vec![handoff_key("key-1", &secret)]; + + let error = validate_settings_for_runtime(&settings) + .expect_err("runtime should reject invalid handoff key material"); + let message = error.to_string(); + assert!( + message.contains(expected), + "runtime error should explain the key shape: {error:?}" + ); + assert!( + !message.contains(&secret), + "runtime error should not expose handoff key material: {error:?}" + ); + } + } + + #[test] + fn identity_handoff_runtime_rejects_encoded_ec_passphrase_key_material() { + let mut settings = valid_settings(); + let ec_key = "synthetic-ec-key-with-32-bytes!!"; + assert_eq!(ec_key.len(), 32); + settings.ec.passphrase = Redacted::new(ec_key.to_owned()); + settings.ec.identity_handoff.keys = + vec![handoff_key("dedicated", &URL_SAFE_NO_PAD.encode(ec_key))]; + let error = validate_settings_for_runtime(&settings) + .expect_err("should reject reused decoded EC key material"); + assert!(error.to_string().contains("must not reuse ec.passphrase")); + assert!(!error.to_string().contains(ec_key)); + } + + #[test] + fn identity_handoff_runtime_validation_accepts_rotation_and_active_key_changes() { + let mut settings = valid_settings(); + settings.ec.identity_handoff = IdentityHandoffConfig { + enabled: true, + active_key_id: "current".to_owned(), + keys: vec![ + handoff_key("previous", &URL_SAFE_NO_PAD.encode([3_u8; 32])), + handoff_key("current", &URL_SAFE_NO_PAD.encode([7_u8; 32])), + ], + }; + + validate_settings_for_runtime(&settings) + .expect("runtime should accept current and previous rotation keys"); + settings.ec.identity_handoff.active_key_id = "previous".to_owned(); + validate_settings_for_runtime(&settings) + .expect("runtime should accept switching issuance to a retained key"); + } + #[test] fn app_config_new_rejects_invalid_non_secret_settings() { let mut settings = valid_settings(); @@ -1677,6 +1932,85 @@ password = "production-admin-password-32-bytes" ); } + #[test] + fn startup_blob_resolves_provided_disabled_identity_handoff_keys() { + let mut settings = valid_settings(); + settings.publisher.proxy_secret = Redacted::new("handoff-proxy-secret".to_owned()); + settings.ec.passphrase = Redacted::new("handoff-ec-passphrase".to_owned()); + for (index, handler) in settings.handlers.iter_mut().enumerate() { + handler.password = Redacted::new(format!("handoff-handler-password-{index}")); + } + settings.ec.identity_handoff = IdentityHandoffConfig { + enabled: false, + active_key_id: "next".to_owned(), + keys: vec![handoff_key("next", "identity-handoff-next")], + }; + + validate_settings_for_deploy(&settings) + .expect("deploy should accept a disabled provisioned key reference"); + + let resolved_key = URL_SAFE_NO_PAD.encode([11_u8; 32]); + let secret_store = |handoff_secret: &str| { + let mut secrets = HashMap::new(); + for (key, value) in [ + ( + "handoff-proxy-secret".to_owned(), + "resolved-proxy-secret-not-placeholder".to_owned(), + ), + ( + "handoff-ec-passphrase".to_owned(), + "resolved-ec-passphrase-32-bytes-valid".to_owned(), + ), + ( + "identity-handoff-next".to_owned(), + handoff_secret.to_owned(), + ), + ] { + secrets.insert(key, value.into_bytes()); + } + for index in 0..settings.handlers.len() { + secrets.insert( + format!("handoff-handler-password-{index}"), + format!("resolved-handler-password-{index}-32-bytes").into_bytes(), + ); + } + HashMapSecretStore::new(secrets) + }; + let envelope = BlobEnvelope::new( + serde_json::to_value(&settings).expect("should serialize settings"), + "2026-10-10T00:00:00Z".to_owned(), + ); + let envelope_json = serde_json::to_string(&envelope).expect("should serialize envelope"); + + let started = crate::config_payload::settings_from_config_blob( + &envelope_json, + &secret_store(&resolved_key), + &StoreName::from("identity-handoff-test-secrets"), + ) + .expect("startup should resolve and validate the disabled provisioned key"); + + assert!(!started.ec.identity_handoff.enabled); + assert_eq!(started.ec.identity_handoff.active_key_id, "next"); + assert_eq!( + started.ec.identity_handoff.keys[0].secret.expose(), + &resolved_key, + "startup should replace the key reference with resolved key material" + ); + + let malformed = "not-valid-key-material"; + let error = crate::config_payload::settings_from_config_blob( + &envelope_json, + &secret_store(malformed), + &StoreName::from("identity-handoff-test-secrets"), + ) + .expect_err("startup should reject malformed resolved handoff key material"); + assert!(error.to_string().contains("URL_SAFE_NO_PAD")); + assert!( + !error.to_string().contains(malformed), + "startup error should not expose resolved handoff key material: {error:?}" + ); + } + #[test] fn startup_blob_validation_keeps_disabled_owner_descriptor_and_resolves_partner_token() { let (mut settings, source) = lockr_identity_owner_settings(); diff --git a/crates/trusted-server-core/src/constants.rs b/crates/trusted-server-core/src/constants.rs index 68ccff808..6422bdff8 100644 --- a/crates/trusted-server-core/src/constants.rs +++ b/crates/trusted-server-core/src/constants.rs @@ -1,6 +1,8 @@ use http::header::HeaderName; pub const COOKIE_TS_EC: &str = "ts-ec"; +/// Authenticated, short-lived handoff for server-captured identity observations. +pub const COOKIE_TS_EIDS_SERVER: &str = "ts-eids-s"; /// Short-lived signed proof that the current EC row has every pull-partner UID. pub const COOKIE_TS_EC_PULL_COMPLETE: &str = "ts-ec-pull-complete"; /// Retired EID cookie name, retained only for transition regression fixtures. diff --git a/crates/trusted-server-core/src/cookies.rs b/crates/trusted-server-core/src/cookies.rs index 3fe6719ef..9f95bdc7e 100644 --- a/crates/trusted-server-core/src/cookies.rs +++ b/crates/trusted-server-core/src/cookies.rs @@ -8,8 +8,8 @@ pub(crate) mod template_cache_policy; use cookie::{Cookie, CookieJar}; use edgezero_core::body::Body as EdgeBody; use error_stack::{Report, ResultExt as _}; -use http::Request; use http::header; +use http::{HeaderValue, Request}; use crate::constants::{COOKIE_EUCONSENT_V2, COOKIE_GPP, COOKIE_GPP_SID, COOKIE_US_PRIVACY}; use crate::error::TrustedServerError; @@ -110,34 +110,52 @@ pub fn strip_cookies(cookie_header: &str, cookie_names: &[&str]) -> String { /// When `strip_consent` is `true`, cookies listed in [`CONSENT_COOKIE_NAMES`] /// are removed before forwarding. If stripping leaves no cookies or yields an /// invalid header value, the stripped header is omitted. Non-UTF-8 cookie -/// headers are forwarded unchanged. Existing `Cookie` headers on `to` are -/// preserved; source values are appended after them. +/// headers are forwarded unchanged unless they contain a capture handoff. +/// The authenticated `ts-eids-s` handoff is never forwarded to auction providers. +/// Existing `Cookie` headers on `to` are preserved; source values are appended +/// after them. pub fn forward_cookie_header( from: &Request, to: &mut Request, strip_consent: bool, ) { for cookie_value in from.headers().get_all(header::COOKIE) { - if !strip_consent { - to.headers_mut() - .append(header::COOKIE, cookie_value.clone()); - continue; + if let Some(value) = auction_cookie_header(cookie_value, strip_consent) { + to.headers_mut().append(header::COOKIE, value); } + } +} - match cookie_value.to_str() { - Ok(s) => { - let stripped = strip_cookies(s, CONSENT_COOKIE_NAMES); - if !stripped.is_empty() - && let Ok(value) = http::HeaderValue::from_str(&stripped) - { - to.headers_mut().append(header::COOKIE, value); - } +/// Removes capture replay proof, and optionally consent, from provider cookies. +/// +/// Opaque headers retain legacy handling only when they cannot carry a handoff. +pub(crate) fn auction_cookie_header( + cookie_value: &HeaderValue, + strip_consent: bool, +) -> Option { + match cookie_value.to_str() { + Ok(value) => { + let mut names = vec![crate::constants::COOKIE_TS_EIDS_SERVER]; + if strip_consent { + names.extend_from_slice(CONSENT_COOKIE_NAMES); } - Err(_) => { - // Non-UTF-8 Cookie header — forward as-is. - to.headers_mut() - .append(header::COOKIE, cookie_value.clone()); + let stripped = strip_cookies(value, &names); + if stripped.is_empty() { + return None; } + HeaderValue::from_str(&stripped).ok() + } + Err(_) => { + // Never forward replay proof in an otherwise unparseable header. + let has_handoff = cookie_value + .as_bytes() + .split(|byte| *byte == b';') + .any(|part| { + part.split(|byte| *byte == b'=') + .next() + .is_some_and(|name| name.trim_ascii() == b"ts-eids-s") + }); + (!has_handoff).then(|| cookie_value.clone()) } } } @@ -444,6 +462,36 @@ mod tests { assert_eq!(stripped, ""); } + #[test] + fn authenticated_handoff_is_not_forwarded_to_auction_providers() { + for strip_consent in [false, true] { + let from = build_request(Some( + "session=abc; ts-eids-s=signed-replay-proof; __gpp=signal", + )); + let mut to = build_request(None); + forward_cookie_header(&from, &mut to, strip_consent); + let cookies = to.headers()[header::COOKIE] + .to_str() + .expect("should read cookies"); + assert!(!cookies.contains("ts-eids-s")); + assert!(cookies.contains("session=abc")); + assert_eq!(cookies.contains("__gpp"), !strip_consent); + } + for opaque in [ + b"session=\xff; ts-eids-s=signed-replay-proof".as_slice(), + b"session=\xff; ts-eids-s \t =signed-replay-proof".as_slice(), + ] { + let mut from = build_request(None); + from.headers_mut().insert( + header::COOKIE, + HeaderValue::from_bytes(opaque).expect("should create opaque header"), + ); + let mut to = build_request(None); + forward_cookie_header(&from, &mut to, false); + assert!(to.headers().get(header::COOKIE).is_none()); + } + } + #[test] fn test_strip_cookies_with_complex_values() { // Cookie values can contain '=' characters. diff --git a/crates/trusted-server-core/src/cookies/template_cache_policy.rs b/crates/trusted-server-core/src/cookies/template_cache_policy.rs index 71ec389ee..3e28f9f29 100644 --- a/crates/trusted-server-core/src/cookies/template_cache_policy.rs +++ b/crates/trusted-server-core/src/cookies/template_cache_policy.rs @@ -2,7 +2,7 @@ use std::collections::{HashMap, HashSet}; -use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EC}; +use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EC, COOKIE_TS_EIDS_SERVER}; use crate::platform::TemplateCookieValue; /// Whether the request can share a template and its explicit cookie dimensions. @@ -32,7 +32,7 @@ pub(crate) fn validate_cookie_names( return Err(format!("{field} contains invalid cookie name `{name}`")); } if field == "template_cache_key_cookies" - && [COOKIE_TS_EC, COOKIE_SHAREDID].contains(&name.as_str()) + && [COOKIE_TS_EC, COOKIE_SHAREDID, COOKIE_TS_EIDS_SERVER].contains(&name.as_str()) { return Err(format!( "{field} must not key on Trusted Server identity cookie `{name}`" @@ -232,7 +232,7 @@ mod tests { #[test] fn template_cookie_policy_rejects_identity_keys_but_allows_bypass() { - for name in [COOKIE_TS_EC, COOKIE_SHAREDID] { + for name in [COOKIE_TS_EC, COOKIE_SHAREDID, COOKIE_TS_EIDS_SERVER] { let error = validate_cookie_names(&names(&[name]), &[]) .expect_err("should reject identity cookie keys"); assert!( diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index 4d735cc7a..1e210f02e 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -75,6 +75,7 @@ pub fn ec_finalize_response( consent_withdrawn, response, ); + super::identity_handoff::finalize_response(settings, registry, ec_context, response); return; } @@ -82,11 +83,12 @@ pub fn ec_finalize_response( if ec_context.ec_was_present() && !ec_context.ec_generated() && consent_allows_ec { if let (Some(graph), Some(ec_id)) = (kv, ec_context.ec_value().map(str::to_owned)) { let source = ec_context.eid_sync_source(); - let collected = source + let mut collected = source .map(|_| { collect_browser_eid_updates(body_updates.clone(), sharedid_cookie, registry) }) .unwrap_or_default(); + protect_pending_capture_updates(&mut collected, ec_context, response); let had_eid_updates = !collected.is_empty(); let updates = gate_eid_updates_by_consent(collected, ec_context.consent()); // `sync_eid_cookie_updates` early-returns without refreshing the @@ -121,8 +123,9 @@ pub fn ec_finalize_response( reconcile_pull_sync_marker(settings, registry, ec_context, response); - // Ordinary returning-user page views no longer refresh the browser - // cookie, emit the EC header, or update KV TTL. + // Capture handoff reconciliation is separate from browser EID ingestion. + super::identity_handoff::finalize_response(settings, registry, ec_context, response); + // Ordinary returning-user page views do not refresh the EC cookie. return; } @@ -149,6 +152,21 @@ pub fn ec_finalize_response( } reconcile_pull_sync_marker(settings, registry, ec_context, response); + super::identity_handoff::finalize_response(settings, registry, ec_context, response); +} + +fn protect_pending_capture_updates( + updates: &mut Vec, + context: &EcContext, + response: &Response, +) { + let capture = response + .extensions() + .get::(); + updates.retain(|update| { + !super::identity_handoff::protects_source(context, &update.partner_id) + && !capture.is_some_and(|effects| effects.protects_source(&update.partner_id)) + }); } fn sync_eid_cookie_updates( diff --git a/crates/trusted-server-core/src/ec/identity.rs b/crates/trusted-server-core/src/ec/identity.rs index 056c17b7d..c1711d927 100644 --- a/crates/trusted-server-core/src/ec/identity.rs +++ b/crates/trusted-server-core/src/ec/identity.rs @@ -8,6 +8,7 @@ use std::collections::BTreeMap; use crate::consent::{RawConsentSignals, identity_capture_consent_allowed}; use crate::settings::Settings; +use super::identity_handoff_codec::{HANDOFF_TTL_SECS, HandoffEntry}; use super::kv::KvIdentityGraph; use super::kv_types::MAX_UID_LENGTH; use super::partner::normalize_partner_source_domain; @@ -61,6 +62,8 @@ pub(crate) struct ManagedPartnerIdUpdate { pub(crate) source: String, pub(crate) uid: String, pub(crate) expires_at: Option, + /// Matching authenticated pending expiry, used only when filling an absent source. + pub(crate) pending_expires_at: Option, } /// Request-scoped effects sealed by the framework's proxy dispatch boundary. @@ -73,6 +76,7 @@ pub struct IdentityEffects { ec_id: String, snapshot: EcKvSnapshot, action: IdentityAction, + replay: Vec, } #[derive(Clone)] @@ -82,6 +86,61 @@ enum IdentityAction { } impl IdentityEffects { + pub(crate) fn handoff_entries(&self, context: &EcContext, now: u64) -> Vec { + if context.ec_value() != Some(self.ec_id.as_str()) { + return Vec::new(); + } + let IdentityAction::Enrich(updates) = &self.action else { + return Vec::new(); + }; + updates + .iter() + .map(|update| { + let expiry = update.expires_at.or(update.pending_expires_at); + let mut entry = HandoffEntry { + module: self.module.to_owned(), + source: update.source.clone(), + uid: update.uid.clone(), + expires_at: expiry, + deadline: expiry.map_or(now.saturating_add(HANDOFF_TTL_SECS), |expiry| { + expiry.min(now.saturating_add(HANDOFF_TTL_SECS)) + }), + }; + if let Some(pending) = context.identity_handoff.matching_capture( + context, + self.module, + &update.source, + &update.uid, + now, + ) && pending.expires_at == expiry + { + entry.deadline = entry.deadline.min(pending.deadline); + } + entry + }) + .collect() + } + + pub(crate) fn handoff_replay(context: &EcContext, replay: Vec) -> Self { + Self { + module: "handoff", + ec_id: context.ec_value().unwrap_or_default().to_owned(), + snapshot: context.kv_snapshot().clone(), + action: IdentityAction::Enrich(Vec::new()), + replay, + } + } + + pub(crate) fn add_handoff_replay(&mut self, replay: Vec) { + if matches!(self.action, IdentityAction::Enrich(_)) { + self.replay = replay; + } + } + + pub(crate) fn protects_source(&self, source: &str) -> bool { + matches!(&self.action, IdentityAction::Enrich(updates) if updates.iter().any(|update| update.source == source)) + } + /// Whether this effect requests withdrawal for the current full EC identity. #[must_use] pub fn withdraws(&self, context: &EcContext) -> bool { @@ -121,6 +180,11 @@ impl IdentityEffects { ), IdentityAction::Enrich(_) => return, }; + let result = if !self.replay.is_empty() && context.ec_allowed() { + graph.upsert_handoff_ids_from_snapshot(&self.ec_id, partners, &self.replay, result) + } else { + result + }; context.set_kv_snapshot(result); } } @@ -174,18 +238,10 @@ pub(crate) fn stage_capture( ec_id: ec_id.to_owned(), snapshot: context.kv_snapshot().clone(), action: IdentityAction::Withdraw, + replay: Vec::new(), }); } - let signals = RawConsentSignals { - raw_tc_string: observation.consent.tcf, - raw_gpp_string: observation.consent.gpp, - raw_us_privacy: observation.consent.usp, - ..Default::default() - }; - if observation.consent.malformed - || !context.ec_allowed() - || !identity_capture_consent_allowed(context.consent(), &signals, &settings.consent) - { + if !observation_consent_allowed(&observation.consent, context, settings) { return None; } let now = checked_current_timestamp(); @@ -218,12 +274,20 @@ pub(crate) fn stage_capture( if !updates.contains_key(&source) && updates.len() >= MAX_CAPTURE_SOURCES { continue; } + let pending_expires_at = now + .and_then(|now| { + context + .identity_handoff + .matching_capture(context, module, &source, &uid, now) + }) + .and_then(|entry| entry.expires_at); updates.insert( source.clone(), ManagedPartnerIdUpdate { source, uid, expires_at, + pending_expires_at, }, ); } @@ -235,9 +299,26 @@ pub(crate) fn stage_capture( ec_id: ec_id.to_owned(), snapshot: context.kv_snapshot().clone(), action: IdentityAction::Enrich(updates.into_values().collect()), + replay: Vec::new(), }) } +pub(crate) fn observation_consent_allowed( + signals: &IdentityConsentSignals, + context: &EcContext, + settings: &Settings, +) -> bool { + let raw = RawConsentSignals { + raw_tc_string: signals.tcf.clone(), + raw_gpp_string: signals.gpp.clone(), + raw_us_privacy: signals.usp.clone(), + ..Default::default() + }; + !signals.malformed + && context.ec_allowed() + && identity_capture_consent_allowed(context.consent(), &raw, &settings.consent) +} + #[cfg(test)] mod tests { use std::cell::{Cell, RefCell}; @@ -788,7 +869,9 @@ mod tests { source: SOURCE.to_owned(), uid: "expired-uid".to_owned(), expires_at: Some(1), + pending_expires_at: None, }]), + replay: Vec::new(), }; effect.apply(&graph, &partners, &mut request); assert_eq!( diff --git a/crates/trusted-server-core/src/ec/identity_handoff.rs b/crates/trusted-server-core/src/ec/identity_handoff.rs new file mode 100644 index 000000000..77659f55e --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity_handoff.rs @@ -0,0 +1,367 @@ +//! Shared request policy and response reconciliation for authenticated capture handoffs. + +use core::fmt; +use std::collections::BTreeMap; + +use edgezero_core::body::Body as EdgeBody; +use http::{HeaderValue, Response, header}; + +use crate::consent::{RawConsentSignals, identity_capture_consent_allowed}; +use crate::consent_config::ConsentConfig; +use crate::constants::COOKIE_TS_EIDS_SERVER; +use crate::integrations::IntegrationRegistry; +use crate::openrtb::Eid; +use crate::response_privacy::enforce_set_cookie_cache_privacy; +use crate::settings::Settings; + +use super::eids::{ResolvedPartnerId, to_eids}; +use super::identity::IdentityEffects; +use super::identity_handoff_codec::{self as codec, HANDOFF_TTL_SECS, HandoffEntry}; +use super::partner::normalize_partner_source_domain; +use super::registry::PartnerRegistry; +use super::{EcContext, EcKvSnapshot, checked_current_timestamp}; + +/// Authenticated incoming observations, never raw cookie data in diagnostics. +#[derive(Clone, Default)] +pub(crate) struct HandoffState { + present: bool, + ec_id: Option, + entries: Vec, + consent_config: Option, + pub(crate) supported: bool, + pub(crate) proxy_replay_allowed: bool, + pub(crate) proxy_replay_denied: bool, +} + +impl fmt::Debug for HandoffState { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + formatter + .debug_struct("HandoffState") + .field("present", &self.present) + .field("entries", &self.entries.len()) + .field("supported", &self.supported) + .finish() + } +} + +impl HandoffState { + pub(crate) fn from_cookie( + settings: &Settings, + ec_id: Option<&str>, + cookie: Option<&str>, + ) -> Self { + let mut state = Self { + present: cookie.is_some(), + ..Self::default() + }; + let Some(value) = cookie else { + return state; + }; + let Some(ec_id) = ec_id else { + return state; + }; + let Some(now) = checked_current_timestamp() else { + return state; + }; + let Some(entries) = codec::decode( + &settings.ec.identity_handoff, + &settings.publisher.domain, + ec_id, + value, + now, + ) else { + return state; + }; + state.entries = entries + .into_iter() + .filter(|entry| source_allowed(settings, entry)) + .collect(); + state.ec_id = Some(ec_id.to_owned()); + state.consent_config = Some(settings.consent.clone()); + state + } + + pub(crate) fn matching_capture( + &self, + context: &EcContext, + module: &str, + source: &str, + uid: &str, + now: u64, + ) -> Option<&HandoffEntry> { + if !self.allowed(context) { + return None; + } + self.entries.iter().find(|entry| { + entry.module == module && entry.source == source && entry.uid == uid && live(entry, now) + }) + } + + fn allowed(&self, context: &EcContext) -> bool { + self.supported + && !self.proxy_replay_denied + && context.ec_was_present() + && !context.ec_generated() + && context.ec_value() == self.ec_id.as_deref() + && self.consent_config.as_ref().is_some_and(|config| { + identity_capture_consent_allowed( + context.consent(), + &RawConsentSignals::default(), + config, + ) + }) + } +} + +fn source_allowed(settings: &Settings, entry: &HandoffEntry) -> bool { + settings.ec.partners.iter().any(|partner| { + normalize_partner_source_domain(&partner.source_domain) + .ok() + .as_deref() + == Some(entry.source.as_str()) + && partner.identity_owner.as_deref() == Some(entry.module.as_str()) + }) && IntegrationRegistry::capture_claim_enabled(settings, &entry.module, &entry.source) +} + +fn live(entry: &HandoffEntry, now: u64) -> bool { + now < entry.deadline && entry.expires_at.is_none_or(|expiry| now < expiry) +} + +/// Resolves handoff IDs only for absent sources in the current live root. +pub(crate) fn resolve_eids( + context: &EcContext, + snapshot: &EcKvSnapshot, + registry: &PartnerRegistry, + now: Option, +) -> Vec { + if !context.identity_handoff.allowed(context) { + return Vec::new(); + } + let Some(now) = now else { + return Vec::new(); + }; + let Some(ec_id) = context.ec_value() else { + return Vec::new(); + }; + let Some(root) = snapshot.entry_for(ec_id).filter(|root| root.consent.ok) else { + return Vec::new(); + }; + let ids = context + .identity_handoff + .entries + .iter() + .filter_map(|entry| { + let partner = registry.get(&entry.source)?; + (live(entry, now) + && partner.bidstream_enabled + && partner.identity_owner.as_deref() == Some(entry.module.as_str()) + && !root.ids.contains_key(&entry.source)) + .then(|| ResolvedPartnerId { + source_domain: entry.source.clone(), + uid: entry.uid.clone(), + writer: Some(entry.module.clone()), + openrtb_atype: partner.openrtb_atype, + }) + }) + .collect::>(); + to_eids(&ids) +} + +/// Prevents ordinary browser ingestion from stripping a trusted pending record's metadata. +pub(crate) fn protects_source(context: &EcContext, source: &str) -> bool { + let Some(now) = checked_current_timestamp() else { + return false; + }; + context.identity_handoff.allowed(context) + && context + .identity_handoff + .entries + .iter() + .any(|entry| entry.source == source && live(entry, now)) +} + +/// Sets a new capture handoff or reconciles an existing one without opening KV. +/// +/// A `NotRead` snapshot may stage replay, but only the post-send mutation can +/// authorize persistence against an existing live row. Auction reads separately +/// require an already loaded live root. +pub(crate) fn finalize_response( + settings: &Settings, + partners: &PartnerRegistry, + context: &mut EcContext, + response: &mut Response, +) { + if !context.identity_handoff.supported { + return; + } + let withdrawal = response + .extensions() + .get::() + .is_some_and(|effects| effects.withdraws(context)) + || super::consent::ec_consent_withdrawn(context.consent()); + let Some(now) = checked_current_timestamp() else { + clear_if_present(settings, context, response); + return; + }; + if withdrawal + || !settings.ec.identity_handoff.enabled + || !context.ec_allowed() + || context.ec_generated() + { + clear_if_present(settings, context, response); + return; + } + let Some(ec_id) = context.ec_value().map(str::to_owned) else { + clear_if_present(settings, context, response); + return; + }; + let new_entries = response + .extensions() + .get::() + .map(|effects| effects.handoff_entries(context, now)) + .unwrap_or_default(); + let mut pending = BTreeMap::new(); + let mut replay = Vec::new(); + if context.identity_handoff.allowed(context) { + for entry in &context.identity_handoff.entries { + if !live(entry, now) || !source_allowed(settings, entry) { + continue; + } + let partner_owned = partners.get(&entry.source).is_some_and(|partner| { + partner.identity_owner.as_deref() == Some(entry.module.as_str()) + }); + if !partner_owned { + continue; + } + match context.kv_snapshot() { + EcKvSnapshot::Present { .. } => { + let Some(root) = context + .kv_snapshot() + .entry_for(&ec_id) + .filter(|root| root.consent.ok) + else { + continue; + }; + // Both acknowledgement and conflict retire the handoff. An + // expired record is still an existing record, never a hole. + if root.ids.contains_key(&entry.source) { + continue; + } + } + EcKvSnapshot::Missing { .. } => continue, + EcKvSnapshot::Failed { .. } => { + // A failed lookup cannot authorize replay, but need not + // destroy the browser's bounded chance on a later request. + pending.insert(entry.source.clone(), entry.clone()); + continue; + } + EcKvSnapshot::NotRead => {} + } + pending.insert(entry.source.clone(), entry.clone()); + if context.eid_sync_source().is_some() || context.identity_handoff.proxy_replay_allowed + { + replay.push(entry.clone()); + } + } + } + let capture_sources = new_entries + .iter() + .map(|entry| entry.source.clone()) + .collect::>(); + replay.retain(|entry| !capture_sources.contains(&entry.source)); + for entry in new_entries { + if !live(&entry, now) || !source_allowed(settings, &entry) { + continue; + } + match context.kv_snapshot() { + EcKvSnapshot::Present { .. } => { + let Some(root) = context + .kv_snapshot() + .entry_for(&ec_id) + .filter(|root| root.consent.ok) + else { + continue; + }; + if root.ids.contains_key(&entry.source) { + continue; + } + } + EcKvSnapshot::Missing { .. } | EcKvSnapshot::Failed { .. } => continue, + EcKvSnapshot::NotRead => {} + } + pending.insert(entry.source.clone(), entry); + } + if !replay.is_empty() { + if let Some(effects) = response.extensions_mut().get_mut::() { + effects.add_handoff_replay(replay); + } else { + response + .extensions_mut() + .insert(IdentityEffects::handoff_replay(context, replay)); + } + } + let entries = pending.into_values().collect::>(); + if entries.is_empty() { + clear_if_present(settings, context, response); + return; + } + if let Some(value) = codec::encode( + &settings.ec.identity_handoff, + &settings.publisher.domain, + &ec_id, + entries, + now, + ) && let Some(retained) = codec::decode( + &settings.ec.identity_handoff, + &settings.publisher.domain, + &ec_id, + &value, + now, + ) { + let max_age = retained + .iter() + .map(|entry| entry.deadline) + .max() + .unwrap_or(now) + .saturating_sub(now) + .min(HANDOFF_TTL_SECS); + append_cookie(settings, response, &value, max_age); + } else { + clear_if_present(settings, context, response); + } +} + +fn clear_if_present( + settings: &Settings, + context: &mut EcContext, + response: &mut Response, +) { + if context.identity_handoff.present { + append_cookie(settings, response, "", 0); + } + context.identity_handoff.entries.clear(); +} + +fn append_cookie( + settings: &Settings, + response: &mut Response, + value: &str, + max_age: u64, +) { + let cookie = format!( + "{COOKIE_TS_EIDS_SERVER}={value}; Domain={}; Path=/; Secure; HttpOnly; SameSite=Lax; Max-Age={max_age}", + settings.publisher.ec_cookie_domain() + ); + if let Ok(value) = HeaderValue::from_str(&cookie) { + response.headers_mut().append(header::SET_COOKIE, value); + enforce_set_cookie_cache_privacy(response); + response.headers_mut().insert( + header::CACHE_CONTROL, + HeaderValue::from_static("private, no-store"), + ); + } +} + +#[cfg(test)] +#[path = "identity_handoff_tests.rs"] +mod tests; diff --git a/crates/trusted-server-core/src/ec/identity_handoff_codec.rs b/crates/trusted-server-core/src/ec/identity_handoff_codec.rs new file mode 100644 index 000000000..c1990d3c3 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity_handoff_codec.rs @@ -0,0 +1,808 @@ +//! Bounded authenticated codec for the short-lived server identity handoff. + +use std::collections::{BTreeMap, BTreeSet}; + +use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD}; +use hmac::{Hmac, Mac as _}; +use serde::{Deserialize, Serialize}; +use sha2::Sha256; + +use crate::settings::{IdentityHandoffConfig, IdentityHandoffKey}; + +use super::is_valid_ec_id; +use super::kv_types::MAX_UID_LENGTH; +use super::partner::normalize_partner_source_domain; + +/// Maximum size of the complete encoded cookie value. +pub(crate) const MAX_HANDOFF_BYTES: usize = 2048; +/// Maximum remaining lifetime of an entry accepted by the codec. +pub(crate) const HANDOFF_TTL_SECS: u64 = 300; + +const TOKEN_VERSION: &str = "v1"; +const SIGNING_DOMAIN: &[u8] = b"trusted-server/ts-eids-s/hmac-sha256\0"; +const MAX_HANDOFF_SOURCES: usize = 64; +const MAX_KEY_ID_BYTES: usize = 32; +const MAX_MODULE_ID_BYTES: usize = 64; + +type HmacSha256 = Hmac; + +/// One authenticated server-capture observation. +/// +/// This type deliberately omits `Debug`: UIDs must not leak through routine +/// formatting of handoff state. +#[derive(Clone, Serialize, Deserialize, PartialEq)] +#[serde(deny_unknown_fields)] +pub(crate) struct HandoffEntry { + #[serde(rename = "m")] + pub(crate) module: String, + #[serde(rename = "s")] + pub(crate) source: String, + #[serde(rename = "u")] + pub(crate) uid: String, + #[serde(rename = "x", skip_serializing_if = "Option::is_none")] + pub(crate) expires_at: Option, + #[serde(rename = "d")] + pub(crate) deadline: u64, +} + +#[derive(Serialize, Deserialize)] +#[serde(deny_unknown_fields)] +struct HandoffEnvelope { + #[serde(rename = "p")] + publisher: String, + #[serde(rename = "e")] + ec_id: String, + #[serde(rename = "d")] + deadline: u64, + #[serde(rename = "i")] + entries: Vec, +} + +/// Encodes entries with the configured active key. +/// +/// Invalid entries fail closed. When multiple valid entries exceed either the +/// source-count or encoded-size bound, complete entries are removed by +/// `(deadline, source)` order, earliest first. +pub(crate) fn encode( + config: &IdentityHandoffConfig, + publisher: &str, + ec_id: &str, + entries: Vec, + now: u64, +) -> Option { + if !config.enabled || publisher.is_empty() || !is_valid_ec_id(ec_id) { + return None; + } + + let key_id = config.active_key_id.as_str(); + let key = configured_key(config, key_id)?; + let secret = decode_secret(key)?; + + let max_deadline = now.saturating_add(HANDOFF_TTL_SECS); + let mut by_source = BTreeMap::new(); + for mut entry in entries { + if !valid_module_id(&entry.module) + || entry.uid.trim().is_empty() + || entry.uid.len() > MAX_UID_LENGTH + || entry.deadline > max_deadline + || entry + .expires_at + .is_some_and(|expires_at| entry.deadline > expires_at) + { + return None; + } + + entry.source = normalize_partner_source_domain(&entry.source).ok()?; + if by_source.insert(entry.source.clone(), entry).is_some() { + return None; + } + } + + // Never issue already-expired observations. This also lets callers rewrite + // a merged handoff without extending or reissuing stale entries. + by_source.retain(|_, entry| entry_is_live(entry, now)); + if by_source.is_empty() { + return None; + } + + // Reject an individually unencodable entry without letting it suppress + // smaller entries in the same handoff. The exact signed value is measured, + // so JSON escaping, publisher binding, key ID, and signature all count. + let mut individually_oversized = Vec::new(); + for (source, entry) in &by_source { + let value = encode_envelope_value(publisher, ec_id, key_id, &secret, vec![entry.clone()])?; + if value.len() > MAX_HANDOFF_BYTES { + individually_oversized.push(source.clone()); + } + } + for source in individually_oversized { + by_source.remove(&source); + } + if by_source.is_empty() { + return None; + } + + while by_source.len() > MAX_HANDOFF_SOURCES { + remove_earliest(&mut by_source)?; + } + + loop { + let entries = by_source.values().cloned().collect(); + let value = encode_envelope_value(publisher, ec_id, key_id, &secret, entries)?; + if value.len() <= MAX_HANDOFF_BYTES { + return Some(value); + } + remove_earliest(&mut by_source)?; + } +} + +/// Authenticates and decodes a handoff for the exact publisher and full EC ID. +/// +/// An authenticated envelope whose entries have all expired returns +/// `Some(Vec::new())`, allowing the caller to clear it. Structural, signature, +/// key, and binding failures return `None`. +pub(crate) fn decode( + config: &IdentityHandoffConfig, + publisher: &str, + ec_id: &str, + value: &str, + now: u64, +) -> Option> { + if !config.enabled || publisher.is_empty() || value.len() > MAX_HANDOFF_BYTES { + return None; + } + + let mut segments = value.split('.'); + let version = segments.next()?; + let key_id = segments.next()?; + let payload_b64 = segments.next()?; + let signature_b64 = segments.next()?; + if segments.next().is_some() || version != TOKEN_VERSION || !valid_key_id(key_id) { + return None; + } + + let key = configured_key(config, key_id)?; + let secret = decode_secret(key)?; + let signature = URL_SAFE_NO_PAD.decode(signature_b64.as_bytes()).ok()?; + let mut mac = new_mac(&secret)?; + update_mac(&mut mac, version, key_id, payload_b64); + mac.verify_slice(&signature).ok()?; + + let payload = URL_SAFE_NO_PAD.decode(payload_b64.as_bytes()).ok()?; + let envelope: HandoffEnvelope = serde_json::from_slice(&payload).ok()?; + if envelope.publisher != publisher + || envelope.ec_id != ec_id + || !is_valid_ec_id(&envelope.ec_id) + || envelope.entries.is_empty() + || envelope.entries.len() > MAX_HANDOFF_SOURCES + { + return None; + } + + let max_future_deadline = now.saturating_add(HANDOFF_TTL_SECS); + let mut seen_sources = BTreeSet::new(); + for entry in &envelope.entries { + let normalized_source = normalize_partner_source_domain(&entry.source).ok()?; + if normalized_source != entry.source + || !valid_module_id(&entry.module) + || entry.uid.trim().is_empty() + || entry.uid.len() > MAX_UID_LENGTH + || entry.deadline > max_future_deadline + || entry + .expires_at + .is_some_and(|expires_at| entry.deadline > expires_at) + || !seen_sources.insert(entry.source.as_str()) + { + return None; + } + } + + let latest_deadline = envelope.entries.iter().map(|entry| entry.deadline).max()?; + if envelope.deadline != latest_deadline || envelope.deadline > max_future_deadline { + return None; + } + + Some( + envelope + .entries + .into_iter() + .filter(|entry| entry_is_live(entry, now)) + .collect(), + ) +} + +fn configured_key<'a>( + config: &'a IdentityHandoffConfig, + key_id: &str, +) -> Option<&'a IdentityHandoffKey> { + if !valid_key_id(key_id) { + return None; + } + + let mut matches = config.keys.iter().filter(|key| key.id == key_id); + let key = matches.next()?; + if matches.next().is_some() { + return None; + } + Some(key) +} + +fn decode_secret(key: &IdentityHandoffKey) -> Option<[u8; 32]> { + if !valid_key_id(&key.id) { + return None; + } + URL_SAFE_NO_PAD + .decode(key.secret.expose().as_bytes()) + .ok()? + .try_into() + .ok() +} + +fn valid_key_id(value: &str) -> bool { + valid_safe_id(value, MAX_KEY_ID_BYTES) +} + +fn valid_module_id(value: &str) -> bool { + valid_safe_id(value, MAX_MODULE_ID_BYTES) +} + +fn valid_safe_id(value: &str, max_bytes: usize) -> bool { + !value.is_empty() + && value.len() <= max_bytes + && value + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'-' | b'_')) +} + +fn entry_is_live(entry: &HandoffEntry, now: u64) -> bool { + entry.deadline > now && entry.expires_at.is_none_or(|expires_at| expires_at > now) +} + +fn remove_earliest(entries: &mut BTreeMap) -> Option<()> { + let source = entries + .values() + .min_by(|left, right| { + left.deadline + .cmp(&right.deadline) + .then_with(|| left.source.cmp(&right.source)) + })? + .source + .clone(); + entries.remove(&source).map(|_| ()) +} + +fn encode_envelope_value( + publisher: &str, + ec_id: &str, + key_id: &str, + secret: &[u8; 32], + entries: Vec, +) -> Option { + let deadline = entries.iter().map(|entry| entry.deadline).max()?; + let envelope = HandoffEnvelope { + publisher: publisher.to_owned(), + ec_id: ec_id.to_owned(), + deadline, + entries, + }; + let payload = serde_json::to_vec(&envelope).ok()?; + sign_value(key_id, secret, &payload) +} + +fn sign_value(key_id: &str, secret: &[u8; 32], payload: &[u8]) -> Option { + let payload_b64 = URL_SAFE_NO_PAD.encode(payload); + let mut mac = new_mac(secret)?; + update_mac(&mut mac, TOKEN_VERSION, key_id, &payload_b64); + let signature_b64 = URL_SAFE_NO_PAD.encode(mac.finalize().into_bytes()); + Some(format!( + "{TOKEN_VERSION}.{key_id}.{payload_b64}.{signature_b64}" + )) +} + +fn new_mac(secret: &[u8; 32]) -> Option { + ::new_from_slice(secret).ok() +} + +fn update_mac(mac: &mut HmacSha256, version: &str, key_id: &str, payload_b64: &str) { + mac.update(SIGNING_DOMAIN); + mac.update(version.as_bytes()); + mac.update(b"\0"); + mac.update(key_id.as_bytes()); + mac.update(b"\0"); + mac.update(payload_b64.as_bytes()); +} + +#[cfg(test)] +mod tests { + use super::*; + use crate::redacted::Redacted; + + const NOW: u64 = 1_800_000_000; + const PUBLISHER: &str = "publisher.example"; + const SOURCE: &str = "id.example"; + + fn ec_id(seed: char) -> String { + format!("{}.Ab123Z", seed.to_string().repeat(64)) + } + + fn key(id: &str, byte: u8) -> IdentityHandoffKey { + IdentityHandoffKey { + id: id.to_owned(), + secret: Redacted::new(URL_SAFE_NO_PAD.encode([byte; 32])), + } + } + + fn config() -> IdentityHandoffConfig { + IdentityHandoffConfig { + enabled: true, + active_key_id: "current".to_owned(), + keys: vec![key("current", 7), key("previous", 9)], + } + } + + fn entry(source: &str, uid: &str, deadline: u64) -> HandoffEntry { + HandoffEntry { + module: "lockr".to_owned(), + source: source.to_owned(), + uid: uid.to_owned(), + expires_at: None, + deadline, + } + } + + fn signed_envelope( + config: &IdentityHandoffConfig, + envelope: &HandoffEnvelope, + key_id: &str, + ) -> String { + let key = configured_key(config, key_id).expect("test key should be configured"); + let secret = decode_secret(key).expect("test key should decode"); + let payload = serde_json::to_vec(envelope).expect("test envelope should serialize"); + sign_value(key_id, &secret, &payload).expect("test envelope should sign") + } + + fn resign_payload(config: &IdentityHandoffConfig, payload: &serde_json::Value) -> String { + let key = configured_key(config, "current").expect("should configure test key"); + let secret = decode_secret(key).expect("should decode test key"); + let payload = serde_json::to_vec(payload).expect("should serialize test payload"); + sign_value("current", &secret, &payload).expect("should sign test payload") + } + + fn token_payload(value: &str) -> serde_json::Value { + let payload = value.split('.').nth(2).expect("token should have payload"); + let decoded = URL_SAFE_NO_PAD + .decode(payload.as_bytes()) + .expect("payload should decode"); + serde_json::from_slice(&decoded).expect("payload should be JSON") + } + + #[test] + fn round_trip_normalizes_sources_and_preserves_entry_deadlines() { + let config = config(); + let ec_id = ec_id('a'); + let entries = vec![HandoffEntry { + module: "lockr_capture".to_owned(), + source: "ID.Example.".to_owned(), + uid: "opaque-user-id".to_owned(), + expires_at: Some(NOW + 240), + deadline: NOW + 180, + }]; + + let value = + encode(&config, PUBLISHER, &ec_id, entries, NOW).expect("valid handoff should encode"); + let decoded = + decode(&config, PUBLISHER, &ec_id, &value, NOW).expect("valid handoff should decode"); + + assert_eq!(decoded.len(), 1); + assert_eq!(decoded[0].source, SOURCE); + assert_eq!(decoded[0].module, "lockr_capture"); + assert_eq!(decoded[0].uid, "opaque-user-id"); + assert_eq!(decoded[0].expires_at, Some(NOW + 240)); + assert_eq!(decoded[0].deadline, NOW + 180); + assert!(value.len() <= MAX_HANDOFF_BYTES); + } + + #[test] + fn disabled_codec_neither_encodes_nor_decodes() { + let mut config = config(); + let ec_id = ec_id('a'); + let value = encode( + &config, + PUBLISHER, + &ec_id, + vec![entry(SOURCE, "uid", NOW + 100)], + NOW, + ) + .expect("enabled codec should encode"); + + config.enabled = false; + assert!( + encode( + &config, + PUBLISHER, + &ec_id, + vec![entry(SOURCE, "uid", NOW + 100)], + NOW, + ) + .is_none() + ); + assert!(decode(&config, PUBLISHER, &ec_id, &value, NOW).is_none()); + } + + #[test] + fn decode_accepts_rotated_non_active_key() { + let mut old_config = config(); + old_config.active_key_id = "previous".to_owned(); + let ec_id = ec_id('a'); + let value = encode( + &old_config, + PUBLISHER, + &ec_id, + vec![entry(SOURCE, "uid", NOW + 100)], + NOW, + ) + .expect("old active key should encode"); + + let new_config = config(); + assert!(decode(&new_config, PUBLISHER, &ec_id, &value, NOW).is_some()); + } + + #[test] + fn invalid_or_ambiguous_key_material_fails_closed() { + let ec_id = ec_id('a'); + for secret in [ + URL_SAFE_NO_PAD.encode([1_u8; 31]), + URL_SAFE_NO_PAD.encode([1_u8; 33]), + format!("{}=", URL_SAFE_NO_PAD.encode([1_u8; 32])), + "not base64!".to_owned(), + ] { + let config = IdentityHandoffConfig { + enabled: true, + active_key_id: "current".to_owned(), + keys: vec![IdentityHandoffKey { + id: "current".to_owned(), + secret: Redacted::new(secret), + }], + }; + assert!( + encode( + &config, + PUBLISHER, + &ec_id, + vec![entry(SOURCE, "uid", NOW + 100)], + NOW, + ) + .is_none() + ); + } + + let mut duplicate = config(); + duplicate.keys.push(key("current", 11)); + assert!( + encode( + &duplicate, + PUBLISHER, + &ec_id, + vec![entry(SOURCE, "uid", NOW + 100)], + NOW, + ) + .is_none() + ); + + let mut unsafe_id = config(); + unsafe_id.active_key_id = "bad.key".to_owned(); + unsafe_id.keys.push(key("bad.key", 4)); + assert!( + encode( + &unsafe_id, + PUBLISHER, + &ec_id, + vec![entry(SOURCE, "uid", NOW + 100)], + NOW, + ) + .is_none() + ); + } + + #[test] + fn signature_and_exact_bindings_are_required() { + let config = config(); + let bound_ec_id = ec_id('a'); + let value = encode( + &config, + PUBLISHER, + &bound_ec_id, + vec![entry(SOURCE, "uid", NOW + 100)], + NOW, + ) + .expect("valid handoff should encode"); + + assert!(decode(&config, "other.example", &bound_ec_id, &value, NOW).is_none()); + assert!( + decode(&config, PUBLISHER, &ec_id('b'), &value, NOW).is_none(), + "the full EC ID must match" + ); + + let mut tampered = value.into_bytes(); + let index = tampered + .iter() + .rposition(|byte| *byte == b'.') + .expect("token should have a signature separator") + + 1; + tampered[index] = if tampered[index] == b'A' { b'B' } else { b'A' }; + let tampered = String::from_utf8(tampered).expect("test token remains ASCII"); + assert!(decode(&config, PUBLISHER, &bound_ec_id, &tampered, NOW).is_none()); + } + + #[test] + fn rejects_oversized_value_before_decoding() { + let config = config(); + assert!( + decode( + &config, + PUBLISHER, + &ec_id('a'), + &"A".repeat(MAX_HANDOFF_BYTES + 1), + NOW, + ) + .is_none() + ); + } + + #[test] + fn rejects_unknown_envelope_and_entry_fields() { + let config = config(); + let ec_id = ec_id('a'); + let value = encode( + &config, + PUBLISHER, + &ec_id, + vec![entry(SOURCE, "uid", NOW + 100)], + NOW, + ) + .expect("valid handoff should encode"); + + let mut envelope_unknown = token_payload(&value); + envelope_unknown + .as_object_mut() + .expect("envelope should be an object") + .insert("unknown".to_owned(), serde_json::json!(true)); + let value = resign_payload(&config, &envelope_unknown); + assert!(decode(&config, PUBLISHER, &ec_id, &value, NOW).is_none()); + + let valid = encode( + &config, + PUBLISHER, + &ec_id, + vec![entry(SOURCE, "uid", NOW + 100)], + NOW, + ) + .expect("valid handoff should encode"); + let mut entry_unknown = token_payload(&valid); + entry_unknown["i"][0]["unknown"] = serde_json::json!(true); + let value = resign_payload(&config, &entry_unknown); + assert!(decode(&config, PUBLISHER, &ec_id, &value, NOW).is_none()); + } + + #[test] + fn rejects_invalid_entries_and_duplicate_normalized_sources() { + let config = config(); + let ec_id = ec_id('a'); + let cases = vec![ + entry(SOURCE, " ", NOW + 100), + entry(SOURCE, &"u".repeat(MAX_UID_LENGTH + 1), NOW + 100), + HandoffEntry { + module: "bad.module".to_owned(), + ..entry(SOURCE, "uid", NOW + 100) + }, + entry("not_a_domain", "uid", NOW + 100), + entry(SOURCE, "uid", NOW + HANDOFF_TTL_SECS + 1), + HandoffEntry { + expires_at: Some(NOW + 50), + ..entry(SOURCE, "uid", NOW + 100) + }, + ]; + for invalid in cases { + assert!( + encode(&config, PUBLISHER, &ec_id, vec![invalid], NOW).is_none(), + "invalid entry should fail closed" + ); + } + + assert!( + encode( + &config, + PUBLISHER, + &ec_id, + vec![ + entry("ID.Example", "one", NOW + 100), + entry("id.example.", "two", NOW + 200), + ], + NOW, + ) + .is_none(), + "duplicates are checked after normalization" + ); + } + + #[test] + fn decode_filters_each_expiry_and_returns_some_when_all_expired() { + let config = config(); + let ec_id = ec_id('a'); + let value = encode( + &config, + PUBLISHER, + &ec_id, + vec![ + entry("early.example", "early", NOW + 10), + HandoffEntry { + expires_at: Some(NOW + 20), + ..entry("provider.example", "provider", NOW + 20) + }, + entry("live.example", "live", NOW + 100), + ], + NOW, + ) + .expect("valid handoff should encode"); + + let decoded = decode(&config, PUBLISHER, &ec_id, &value, NOW + 20) + .expect("authenticated mixed-expiry handoff should decode"); + assert_eq!(decoded.len(), 1); + assert_eq!(decoded[0].source, "live.example"); + + let decoded = decode(&config, PUBLISHER, &ec_id, &value, NOW + 100) + .expect("authenticated expired handoff should support cleanup"); + assert!(decoded.is_empty()); + } + + #[test] + fn decode_rejects_wrong_envelope_deadline_and_excess_future_deadline() { + let config = config(); + let ec_id = ec_id('a'); + let envelope = HandoffEnvelope { + publisher: PUBLISHER.to_owned(), + ec_id: ec_id.clone(), + deadline: NOW + 99, + entries: vec![entry(SOURCE, "uid", NOW + 100)], + }; + let value = signed_envelope(&config, &envelope, "current"); + assert!(decode(&config, PUBLISHER, &ec_id, &value, NOW).is_none()); + + let envelope = HandoffEnvelope { + deadline: NOW + HANDOFF_TTL_SECS + 1, + entries: vec![entry(SOURCE, "uid", NOW + HANDOFF_TTL_SECS + 1)], + ..envelope + }; + let value = signed_envelope(&config, &envelope, "current"); + assert!(decode(&config, PUBLISHER, &ec_id, &value, NOW).is_none()); + } + + #[test] + fn decode_rejects_noncanonical_or_duplicate_sources_even_when_signed() { + let config = config(); + let ec_id = ec_id('a'); + let envelope = HandoffEnvelope { + publisher: PUBLISHER.to_owned(), + ec_id: ec_id.clone(), + deadline: NOW + 100, + entries: vec![entry("ID.Example", "uid", NOW + 100)], + }; + let value = signed_envelope(&config, &envelope, "current"); + assert!(decode(&config, PUBLISHER, &ec_id, &value, NOW).is_none()); + + let envelope = HandoffEnvelope { + entries: vec![ + entry(SOURCE, "one", NOW + 100), + entry(SOURCE, "two", NOW + 100), + ], + ..envelope + }; + let value = signed_envelope(&config, &envelope, "current"); + assert!(decode(&config, PUBLISHER, &ec_id, &value, NOW).is_none()); + } + + #[test] + fn individually_oversized_entry_does_not_suppress_smaller_entry() { + let config = config(); + let ec_id = ec_id('a'); + let value = encode( + &config, + PUBLISHER, + &ec_id, + vec![ + entry("small.example", "small", NOW + 100), + entry("oversized.example", &"\0".repeat(MAX_UID_LENGTH), NOW + 200), + ], + NOW, + ) + .expect("smaller entry should still encode"); + + let decoded = + decode(&config, PUBLISHER, &ec_id, &value, NOW).expect("bounded handoff should decode"); + assert_eq!(decoded.len(), 1); + assert_eq!(decoded[0].source, "small.example"); + } + + #[test] + fn deterministic_truncation_removes_earliest_deadlines_first() { + let config = config(); + let ec_id = ec_id('a'); + let entries: Vec<_> = (0..10) + .map(|index| { + entry( + &format!("id-{index}.example"), + &format!("{index}-{}", "u".repeat(500)), + NOW + 100 + index, + ) + }) + .collect(); + let mut reversed = entries.clone(); + reversed.reverse(); + + let value = + encode(&config, PUBLISHER, &ec_id, entries, NOW).expect("bounded subset should encode"); + let reversed_value = encode(&config, PUBLISHER, &ec_id, reversed, NOW) + .expect("reordered bounded subset should encode"); + assert_eq!(value, reversed_value, "encoding should be deterministic"); + assert!(value.len() <= MAX_HANDOFF_BYTES); + + let decoded = decode(&config, PUBLISHER, &ec_id, &value, NOW) + .expect("truncated handoff should decode"); + assert!(decoded.len() < 10); + assert!(decoded.iter().all(|entry| entry.deadline > NOW + 100)); + assert!( + decoded.iter().any(|entry| entry.deadline == NOW + 109), + "latest entry should survive" + ); + } + + #[test] + fn source_count_truncation_keeps_the_latest_entries() { + let config = config(); + let ec_id = ec_id('a'); + let entries: Vec<_> = (0..=MAX_HANDOFF_SOURCES) + .map(|index| { + entry( + &format!("id-{index}.example"), + "u", + NOW + 100 + index as u64, + ) + }) + .collect(); + + let value = encode(&config, PUBLISHER, &ec_id, entries, NOW) + .expect("bounded source subset should encode"); + let decoded = decode(&config, PUBLISHER, &ec_id, &value, NOW) + .expect("bounded source subset should decode"); + + assert!(decoded.len() <= MAX_HANDOFF_SOURCES); + assert!(decoded.iter().all(|entry| entry.deadline > NOW + 100)); + assert!( + decoded + .iter() + .any(|entry| entry.deadline == NOW + 100 + MAX_HANDOFF_SOURCES as u64), + "latest entry should survive" + ); + } + + #[test] + fn decode_rejects_more_than_sixty_four_sources_even_when_signed() { + let config = config(); + let ec_id = ec_id('a'); + let entries: Vec<_> = (0..=MAX_HANDOFF_SOURCES) + .map(|index| entry(&format!("id-{index}.example"), "u", NOW + 100)) + .collect(); + let envelope = HandoffEnvelope { + publisher: PUBLISHER.to_owned(), + ec_id: ec_id.clone(), + deadline: NOW + 100, + entries, + }; + let value = signed_envelope(&config, &envelope, "current"); + + // The normal encoded-size cap usually rejects this fixture first. Its + // source-count check is still exercised if compact schema changes ever + // make the same signed fixture fit. + assert!(decode(&config, PUBLISHER, &ec_id, &value, NOW).is_none()); + } +} diff --git a/crates/trusted-server-core/src/ec/identity_handoff_tests.rs b/crates/trusted-server-core/src/ec/identity_handoff_tests.rs new file mode 100644 index 000000000..da5d9d4f2 --- /dev/null +++ b/crates/trusted-server-core/src/ec/identity_handoff_tests.rs @@ -0,0 +1,786 @@ +use super::*; +use std::sync::Arc; + +use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD}; +use serde_json::json; + +use crate::auction::endpoints::resolve_auction_eids; +use crate::consent::jurisdiction::Jurisdiction; +use crate::ec::EidSyncSource; +use crate::ec::finalize::ec_finalize_response; +use crate::ec::identity::{ + IdentityConsentSignals, IdentityObservation, IdentityOutcome, send_then_apply_identity, + stage_capture, +}; +use crate::ec::kv::{KvIdentityGraph, PartnerIdUpdate}; +use crate::ec::kv_types::{KvEntry, KvPartnerId}; +use crate::integrations::ProxyDispatchInput; +use crate::platform::test_support::{ + StubHttpClient, build_services_with_http_client, noop_services, +}; +use crate::redacted::Redacted; +use crate::settings::{IdentityHandoffConfig, IdentityHandoffKey}; +use crate::test_support::tests::create_test_settings; + +const SOURCE: &str = "id5-sync.com"; +const UID: &str = "ID5*synthetic-handoff"; +const PATH: &str = "/integrations/lockr/api/publisher/app/v2/identityLockr/page-view"; + +fn id() -> String { + format!("{}.ABC123", "a".repeat(64)) +} +fn now() -> u64 { + checked_current_timestamp().expect("should read test clock") +} +fn settings() -> Settings { + let mut settings = create_test_settings(); + settings.ec.identity_handoff = IdentityHandoffConfig { + enabled: true, + active_key_id: "current".to_owned(), + keys: vec![IdentityHandoffKey { + id: "current".to_owned(), + secret: Redacted::new(URL_SAFE_NO_PAD.encode([7u8; 32])), + }], + }; + settings.ec.partners = vec![serde_json::from_value(json!({ + "name":"Synthetic capture source", "source_domain":SOURCE, "identity_owner":"lockr", "openrtb_atype":1, "bidstream_enabled":true + })).expect("should parse source policy")]; + settings + .integrations + .insert_config( + "lockr", + &json!({"app_id":"synthetic", "capture_identity":true}), + ) + .expect("should configure capture"); + settings +} +fn partners(settings: &Settings) -> PartnerRegistry { + PartnerRegistry::from_config(&settings.ec.partners).expect("should build registry") +} +fn graph() -> KvIdentityGraph { + let graph = KvIdentityGraph::in_memory("handoff-tests"); + let mut root = KvEntry::minimal("other.example.com", "other-id", now()); + root.consent.ok = true; + graph + .create(&id(), &root) + .expect("should seed consenting root"); + graph +} +fn entry() -> HandoffEntry { + HandoffEntry { + module: "lockr".to_owned(), + source: SOURCE.to_owned(), + uid: UID.to_owned(), + expires_at: Some(now() + 600), + deadline: now() + 300, + } +} +fn cookie(settings: &Settings, entries: Vec) -> String { + codec::encode( + &settings.ec.identity_handoff, + &settings.publisher.domain, + &id(), + entries, + now(), + ) + .expect("should encode fixture") +} +fn context(settings: &Settings, value: Option<&str>) -> EcContext { + let mut cookies = format!("ts-ec={}", id()); + if let Some(value) = value { + cookies.push_str(&format!("; ts-eids-s={value}")); + } + let request = http::Request::builder() + .uri("https://publisher.example/auction") + .header(header::COOKIE, cookies) + .body(EdgeBody::empty()) + .expect("should build request"); + let mut context = EcContext::read_from_request(settings, &request, &noop_services()) + .expect("should read request"); + context.consent_mut().jurisdiction = Jurisdiction::NonRegulated; + context.set_identity_handoff_supported(true); + context +} +fn response() -> Response { + Response::builder() + .status(200) + .header(header::CACHE_CONTROL, "public, max-age=60") + .header(header::CONTENT_ENCODING, "gzip") + .body(EdgeBody::from("original bytes")) + .expect("should build response") +} +fn handoff_header(response: &Response) -> Option<&str> { + response + .headers() + .get_all(header::SET_COOKIE) + .iter() + .filter_map(|value| value.to_str().ok()) + .find(|value| value.starts_with("ts-eids-s=")) +} +fn captured_effect( + settings: &Settings, + context: &EcContext, + partners: &PartnerRegistry, +) -> IdentityEffects { + stage_capture( + "lockr", + &[SOURCE], + IdentityObservation { + outcomes: vec![IdentityOutcome::Issued { + source: SOURCE.to_owned(), + uid: UID.to_owned(), + expires_at: Some(now() + 600), + }], + consent: IdentityConsentSignals::default(), + }, + context, + partners, + settings, + ) + .expect("should seal acquisition") +} +fn send( + response: &mut Response, + context: &mut EcContext, + graph: &KvIdentityGraph, + partners: &PartnerRegistry, +) { + let effects = response.extensions_mut().remove::(); + assert!(send_then_apply_identity( + effects, + context, + partners, + || {}, + || Some(graph.clone()) + )); +} + +#[test] +fn capture_sets_private_handoff_before_storage_and_preserves_bytes() { + let settings = settings(); + let partners = partners(&settings); + let graph = graph(); + let mut context = context(&settings, None); + let mut response = response(); + response + .extensions_mut() + .insert(captured_effect(&settings, &context, &partners)); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + let header = handoff_header(&response).expect("should set capture handoff"); + for attribute in [ + "HttpOnly", + "Secure", + "SameSite=Lax", + "Path=/", + "Max-Age=300", + ] { + assert!( + header.contains(attribute), + "should use bounded secure cookie" + ); + } + let value = header + .strip_prefix("ts-eids-s=") + .expect("should get value") + .split(';') + .next() + .expect("should get token"); + let decoded = codec::decode( + &settings.ec.identity_handoff, + &settings.publisher.domain, + &id(), + value, + now(), + ) + .expect("should authenticate cookie"); + assert_eq!(decoded.len(), 1); + assert_eq!(decoded[0].uid, UID); + assert!( + !graph + .get(&id()) + .expect("should read graph") + .expect("should find root") + .0 + .ids + .contains_key(SOURCE) + ); + assert!( + response.headers()[header::CACHE_CONTROL] + .to_str() + .expect("should read cache policy") + .contains("no-store") + ); + assert_eq!(response.headers()[header::CONTENT_ENCODING], "gzip"); + assert!(matches!(response.body(),EdgeBody::Once(bytes) if bytes.as_ref()==b"original bytes")); + let effects = response.extensions_mut().remove::(); + assert!(send_then_apply_identity( + effects, + &mut context, + &partners, + || { + assert!( + !graph + .get(&id()) + .expect("should read graph at send") + .expect("should find root") + .0 + .ids + .contains_key(SOURCE) + ); + }, + || Some(graph.clone()) + )); + let stored = graph + .get(&id()) + .expect("should read persisted acquisition") + .expect("should find root") + .0; + assert_eq!(stored.ids[SOURCE].writer.as_deref(), Some("lockr")); + assert!(stored.ids[SOURCE].expires_at.is_some()); + assert_eq!(stored.ids["other.example.com"].uid, "other-id"); +} + +#[test] +fn pending_capture_delivers_before_visibility_then_replays_and_acknowledges() { + let settings = settings(); + let partners = partners(&settings); + let graph = graph(); + let original_entry = entry(); + let value = cookie(&settings, vec![original_entry.clone()]); + let mut context = context(&settings, Some(&value)); + context.set_kv_snapshot(graph.load_snapshot(&id())); + context.set_eid_sync_source(EidSyncSource::Auction); + let eids = resolve_auction_eids( + context.kv_snapshot(), + Some(&partners), + &context, + Some(now()), + ) + .expect("should resolve handoff auction"); + assert_eq!(eids.len(), 1); + assert_eq!(eids[0].uids[0].id, UID); + assert_eq!(eids[0].inserter.as_deref(), Some("Lockr-For-Publishers")); + assert_eq!(eids[0].matcher.as_deref(), Some("")); + assert_eq!(eids[0].mm, Some(2)); + // The matching browser value must not be persisted as unknown-expiry browser state. + context.stage_browser_eid_updates(vec![PartnerIdUpdate::new(SOURCE, UID)]); + let mut response = response(); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + assert!( + !graph + .get(&id()) + .expect("should read before replay") + .expect("should find root") + .0 + .ids + .contains_key(SOURCE) + ); + send(&mut response, &mut context, &graph, &partners); + let stored = graph + .get(&id()) + .expect("should read replay") + .expect("should find root") + .0; + assert_eq!(stored.ids[SOURCE].uid, UID); + assert_eq!(stored.ids[SOURCE].writer.as_deref(), Some("lockr")); + assert_eq!(stored.ids[SOURCE].expires_at, original_entry.expires_at); + let mut acknowledged = self::context(&settings, Some(&value)); + acknowledged.set_kv_snapshot(graph.load_snapshot(&id())); + let mut response = self::response(); + ec_finalize_response( + &settings, + &mut acknowledged, + Some(&graph), + &partners, + None, + &mut response, + ); + assert!( + handoff_header(&response) + .expect("should clear acknowledged cookie") + .contains("Max-Age=0") + ); + assert!(response.extensions().get::().is_none()); +} + +#[test] +fn matching_capture_retains_pending_deadline_and_missing_expiry_without_overriding_kv() { + let settings = settings(); + let partners = partners(&settings); + let mut original_entry = entry(); + let expiry = now() + 500; + original_entry.expires_at = Some(expiry); + original_entry.deadline = now() + 200; + let value = cookie(&settings, vec![original_entry]); + for reported_expiry in [Some(expiry), None] { + let graph = graph(); + let mut context = context(&settings, Some(&value)); + let effects = stage_capture( + "lockr", + &[SOURCE], + IdentityObservation { + outcomes: vec![IdentityOutcome::Issued { + source: SOURCE.to_owned(), + uid: UID.to_owned(), + expires_at: reported_expiry, + }], + consent: IdentityConsentSignals::default(), + }, + &context, + &partners, + &settings, + ) + .expect("should seal matching capture"); + let mut response = response(); + response.extensions_mut().insert(effects); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + let returned = handoff_header(&response) + .expect("should retain pending observation") + .strip_prefix("ts-eids-s=") + .expect("should get value") + .split(';') + .next() + .expect("should get token"); + assert_eq!( + returned, value, + "identical capture or missing expiry must not renew pending metadata" + ); + if reported_expiry.is_none() { + // A newer stored expiry appears after seal. The cookie fallback is + // fill-absent-only metadata, never authority over this record. + let mut fresh_context = self::context(&settings, None); + captured_effect(&settings, &fresh_context, &partners).apply( + &graph, + &partners, + &mut fresh_context, + ); + let before = graph.get(&id()).expect("should read newer expiry"); + send(&mut response, &mut context, &graph, &partners); + assert_eq!( + graph.get(&id()).expect("should retain newer expiry"), + before + ); + } else { + send(&mut response, &mut context, &graph, &partners); + assert_eq!( + graph + .get(&id()) + .expect("should read capture") + .expect("should find root") + .0 + .ids[SOURCE] + .expires_at, + Some(expiry) + ); + } + } + // Missing provider expiry also preserves known pending expiry when the + // post-send read still finds the source absent. + let graph = graph(); + let mut context = context(&settings, Some(&value)); + let effects = stage_capture( + "lockr", + &[SOURCE], + IdentityObservation { + outcomes: vec![IdentityOutcome::Issued { + source: SOURCE.to_owned(), + uid: UID.to_owned(), + expires_at: None, + }], + consent: IdentityConsentSignals::default(), + }, + &context, + &partners, + &settings, + ) + .expect("should seal capture without expiry"); + effects.apply(&graph, &partners, &mut context); + assert_eq!( + graph + .get(&id()) + .expect("should read filled source") + .expect("should find root") + .0 + .ids[SOURCE] + .expires_at, + Some(expiry) + ); +} + +#[test] +fn existing_records_including_expired_and_same_uid_refresh_block_replay() { + let settings = settings(); + let partners = partners(&settings); + for (uid, expiry) in [ + ("newer-uid", Some(now() + 900)), + (UID, Some(now() + 900)), + (UID, Some(1)), + ] { + let graph = KvIdentityGraph::in_memory("handoff-existing-records"); + let mut root = KvEntry::minimal(SOURCE, uid, now()); + root.consent.ok = true; + let record = root.ids.get_mut(SOURCE).expect("should get fixture record"); + record.expires_at = expiry; + record.writer = Some("lockr".to_owned()); + record.revision = Some("newer-revision".to_owned()); + graph + .create(&id(), &root) + .expect("should seed existing record"); + let before = graph.get(&id()).expect("should read before"); + let value = cookie(&settings, vec![entry()]); + let mut context = context(&settings, Some(&value)); + context.set_kv_snapshot(graph.load_snapshot(&id())); + context.set_eid_sync_source(EidSyncSource::Auction); + assert!(resolve_eids(&context, context.kv_snapshot(), &partners, Some(now())).is_empty()); + let mut response = response(); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + assert!( + handoff_header(&response) + .expect("should retire conflicting handoff") + .contains("Max-Age=0") + ); + send(&mut response, &mut context, &graph, &partners); + assert_eq!(graph.get(&id()).expect("should read after"), before); + } +} + +#[test] +fn replay_cas_cannot_revive_withdrawal_or_replace_a_concurrent_capture() { + let settings = settings(); + let partners = partners(&settings); + for withdrawal in [true, false] { + let graph = graph(); + let value = cookie(&settings, vec![entry()]); + let mut context = context(&settings, Some(&value)); + context.set_kv_snapshot(graph.load_snapshot(&id())); + context.set_eid_sync_source(EidSyncSource::Auction); + let mut response = response(); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + if withdrawal { + graph.tombstone_existing_from_snapshot(&id(), graph.load_snapshot(&id())); + } else { + graph + .upsert_partner_id(&id(), SOURCE, "concurrent-newer-uid") + .expect("should write concurrent fixture"); + } + let before = graph.get(&id()).expect("should read concurrent state"); + send(&mut response, &mut context, &graph, &partners); + assert_eq!(graph.get(&id()).expect("should read after replay"), before); + } +} + +#[test] +fn observed_stale_snapshot_can_deliver_but_replay_rechecks_authoritative_generation() { + let settings = settings(); + let partners = partners(&settings); + let graph = graph(); + let value = cookie(&settings, vec![entry()]); + let mut context = context(&settings, Some(&value)); + context.set_kv_snapshot(graph.load_snapshot(&id())); + context.set_eid_sync_source(EidSyncSource::Auction); + // Model an older live point read after another request has withdrawn. + graph.tombstone_existing_from_snapshot(&id(), graph.load_snapshot(&id())); + assert_eq!( + resolve_eids(&context, context.kv_snapshot(), &partners, Some(now())).len(), + 1, + "delivery retains the existing snapshot consistency limit" + ); + let before = graph + .get(&id()) + .expect("should read authoritative tombstone"); + let mut response = response(); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + send(&mut response, &mut context, &graph, &partners); + assert_eq!( + graph.get(&id()).expect("should read after replay CAS"), + before, + "replay must not revive the authoritative tombstone" + ); +} + +#[test] +fn deadline_and_provider_expiry_are_rechecked_after_send() { + let settings = settings(); + let partners = partners(&settings); + let graph = graph(); + let before = graph.get(&id()).expect("should read before"); + for expired_deadline in [true, false] { + let mut update = entry(); + if expired_deadline { + update.deadline = 1; + } else { + update.expires_at = Some(1); + } + let mut context = context(&settings, None); + context.set_kv_snapshot(graph.load_snapshot(&id())); + IdentityEffects::handoff_replay(&context, vec![update]).apply( + &graph, + &partners, + &mut context, + ); + assert_eq!( + graph.get(&id()).expect("should read after expired replay"), + before + ); + } +} + +#[test] +fn missing_failed_or_withdrawn_roots_and_unknown_clock_do_not_deliver() { + let settings = settings(); + let partners = partners(&settings); + let value = cookie(&settings, vec![entry()]); + let mut context = context(&settings, Some(&value)); + let mut withdrawn = KvEntry::tombstone(now()); + withdrawn + .ids + .insert(SOURCE.to_owned(), KvPartnerId::default()); + for snapshot in [ + EcKvSnapshot::NotRead, + EcKvSnapshot::Missing { ec_id: id() }, + EcKvSnapshot::Failed { ec_id: id() }, + EcKvSnapshot::Present { + ec_id: id(), + entry: Box::new(withdrawn), + generation: Some(1), + }, + ] { + context.set_kv_snapshot(snapshot); + assert!(resolve_eids(&context, context.kv_snapshot(), &partners, Some(now())).is_empty()); + } + context.set_kv_snapshot(graph().load_snapshot(&id())); + assert!(resolve_eids(&context, context.kv_snapshot(), &partners, None).is_empty()); +} + +#[test] +fn tampering_rotation_disabled_capture_and_request_denial_never_authorize_handoff() { + let settings = settings(); + let partners = partners(&settings); + let mut value = cookie(&settings, vec![entry()]); + value.push('x'); + let mut context = context(&settings, Some(&value)); + context.set_kv_snapshot(graph().load_snapshot(&id())); + assert!(resolve_eids(&context, context.kv_snapshot(), &partners, Some(now())).is_empty()); + let value = cookie(&settings, vec![entry()]); + let mut disabled = settings.clone(); + disabled + .integrations + .insert_config( + "lockr", + &json!({"app_id":"synthetic","capture_identity":false}), + ) + .expect("should disable acquisition"); + let mut context = self::context(&disabled, Some(&value)); + context.set_kv_snapshot(graph().load_snapshot(&id())); + assert!(resolve_eids(&context, context.kv_snapshot(), &partners, Some(now())).is_empty()); + let mut context = self::context(&settings, Some(&value)); + context.set_kv_snapshot(graph().load_snapshot(&id())); + context.consent_mut().gpc = true; + assert!(resolve_eids(&context, context.kv_snapshot(), &partners, Some(now())).is_empty()); + let mut context = self::context(&settings, Some(&value)); + context.replace_with_generated(format!("{}.XYZ789", "a".repeat(64)), EcKvSnapshot::NotRead); + assert!(resolve_eids(&context, context.kv_snapshot(), &partners, Some(now())).is_empty()); + assert!(!format!("{context:?}").contains(UID)); +} + +#[test] +fn no_token_proxy_request_repairs_pending_capture_after_send_but_body_optout_clears_it() { + let settings = settings(); + let partners = partners(&settings); + let value = cookie(&settings, vec![entry()]); + for denied in [false, true] { + let graph = graph(); + let mut context = context(&settings, Some(&value)); + let stub = Arc::new(StubHttpClient::new()); + stub.push_response( + 200, + serde_json::to_vec(&json!({"success":true,"aimTokens":[]})) + .expect("should encode no-token response"), + ); + let services = build_services_with_http_client( + Arc::clone(&stub) as Arc + ); + let registry = + IntegrationRegistry::new(&settings).expect("should create integration registry"); + let body = if denied { + json!({"ccpaString":"1YYN"}) + } else { + json!({}) + }; + let request = http::Request::builder() + .method(http::Method::POST) + .uri(PATH) + .body(EdgeBody::from( + serde_json::to_vec(&body).expect("should serialize body"), + )) + .expect("should build proxy request"); + let mut response = futures::executor::block_on(registry.handle_proxy(ProxyDispatchInput { + method: &http::Method::POST, + path: PATH, + settings: &settings, + kv: Some(&graph), + ec_context: &mut context, + services: &services, + req: request, + })) + .expect("should select proxy") + .expect("should proxy upstream"); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + assert!( + matches!(context.kv_snapshot(), EcKvSnapshot::NotRead), + "no-token proxy must not add a pre-send KV lookup" + ); + if denied { + assert!( + handoff_header(&response) + .expect("should clear denied handoff") + .contains("Max-Age=0") + ); + } else { + let header = handoff_header(&response).expect("should retain pending handoff"); + let returned_value = header + .strip_prefix("ts-eids-s=") + .expect("should get cookie value") + .split(';') + .next() + .expect("should get token"); + assert_eq!( + returned_value, value, + "NoChange must retain the original authenticated deadline" + ); + } + send(&mut response, &mut context, &graph, &partners); + assert_eq!( + graph + .get(&id()) + .expect("should read repair") + .expect("should find root") + .0 + .ids + .contains_key(SOURCE), + !denied + ); + } +} + +#[test] +fn oversized_replacement_expires_existing_cookie_when_nothing_fits() { + let settings = settings(); + let partners = partners(&settings); + let graph = graph(); + let value = cookie(&settings, vec![entry()]); + let mut context = context(&settings, Some(&value)); + let effects = stage_capture( + "lockr", + &[SOURCE], + IdentityObservation { + outcomes: vec![IdentityOutcome::Issued { + source: SOURCE.to_owned(), + uid: "\u{1}".repeat(512), + expires_at: None, + }], + consent: IdentityConsentSignals::default(), + }, + &context, + &partners, + &settings, + ) + .expect("should seal bounded but JSON-expanding UID"); + let mut response = response(); + response.extensions_mut().insert(effects); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + assert!( + handoff_header(&response) + .expect("should retire oversized replacement") + .contains("Max-Age=0") + ); +} + +#[test] +fn capture_and_handoff_stay_off_on_unsupported_adapters_and_unrelated_requests_do_not_clean_up() { + let settings = settings(); + let partners = partners(&settings); + let graph = graph(); + let mut context = context(&settings, None); + context.set_identity_handoff_supported(false); + let mut response = response(); + response + .extensions_mut() + .insert(captured_effect(&settings, &context, &partners)); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + assert!(handoff_header(&response).is_none()); + let mut context = self::context(&settings, None); + let mut response = self::response(); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + assert!(handoff_header(&response).is_none()); +} diff --git a/crates/trusted-server-core/src/ec/kv.rs b/crates/trusted-server-core/src/ec/kv.rs index f47ae74e2..97de1768f 100644 --- a/crates/trusted-server-core/src/ec/kv.rs +++ b/crates/trusted-server-core/src/ec/kv.rs @@ -905,6 +905,10 @@ impl KvIdentityGraph { let expiry = update.expires_at.or_else(|| { if same_uid { previous.and_then(|record| record.expires_at) + } else if previous.is_none() { + // Preserve known pending metadata only in an absent + // slot. A cookie never changes any stored expiry. + update.pending_expires_at } else { None } @@ -966,6 +970,108 @@ impl KvIdentityGraph { } } + /// Replays authenticated capture only into absent sources on an existing live root. + /// + /// Rechecks deadline, expiry and ownership after every conflict. Existing + /// records, including expired records, never become replacement slots. + pub(crate) fn upsert_handoff_ids_from_snapshot( + &self, + ec_id: &str, + partners: &super::registry::PartnerRegistry, + updates: &[super::identity_handoff_codec::HandoffEntry], + snapshot: EcKvSnapshot, + ) -> EcKvSnapshot { + if updates.is_empty() { + return snapshot; + } + let mut current = match snapshot { + EcKvSnapshot::NotRead => self.load_snapshot(ec_id), + EcKvSnapshot::Present { + ec_id: ref id, + generation: None, + .. + } if id == ec_id => self.load_snapshot(ec_id), + state if state.belongs_to(ec_id) => state, + _ => { + return EcKvSnapshot::Failed { + ec_id: ec_id.to_owned(), + }; + } + }; + for _ in 0..MAX_CAS_RETRIES { + let EcKvSnapshot::Present { + ec_id: ref id, + ref entry, + generation: Some(generation), + } = current + else { + return current; + }; + if id != ec_id || !entry.consent.ok { + return current; + } + let Some(now) = checked_current_timestamp() else { + return current; + }; + let mut candidate = entry.as_ref().clone(); + let mut changed = false; + for update in updates { + if now >= update.deadline + || update.expires_at.is_some_and(|expiry| now >= expiry) + || partners + .get(&update.source) + .and_then(|partner| partner.identity_owner.as_deref()) + != Some(update.module.as_str()) + || candidate.ids.contains_key(&update.source) + { + continue; + } + candidate.ids.insert( + update.source.clone(), + super::kv_types::KvPartnerId { + uid: update.uid.clone(), + expires_at: update.expires_at, + writer: Some(update.module.clone()), + revision: Some(uuid::Uuid::new_v4().to_string()), + }, + ); + changed = true; + } + if !changed { + return current; + } + let Ok((body, metadata)) = Self::serialize_entry(&candidate, self.store_name()) else { + return EcKvSnapshot::Failed { + ec_id: ec_id.to_owned(), + }; + }; + match self.write_entry( + ec_id, + &body, + &metadata, + ENTRY_TTL, + EcKvWriteMode::IfGenerationMatch(generation), + ) { + Ok(EcKvWriteOutcome::Written) => { + return EcKvSnapshot::Present { + ec_id: ec_id.to_owned(), + entry: Box::new(candidate), + generation: None, + }; + } + Ok(EcKvWriteOutcome::PreconditionFailed) => current = self.load_snapshot(ec_id), + Err(_) => { + return EcKvSnapshot::Failed { + ec_id: ec_id.to_owned(), + }; + } + } + } + EcKvSnapshot::Failed { + ec_id: ec_id.to_owned(), + } + } + /// Returns `proven` instead of a read outcome that cannot disprove it. /// /// Partner-ID enrichment is best effort. When the caller already held proof diff --git a/crates/trusted-server-core/src/ec/mod.rs b/crates/trusted-server-core/src/ec/mod.rs index 44b827fd3..abb9dc489 100644 --- a/crates/trusted-server-core/src/ec/mod.rs +++ b/crates/trusted-server-core/src/ec/mod.rs @@ -41,6 +41,8 @@ pub mod finalize; pub mod generation; pub mod identify; pub mod identity; +pub(crate) mod identity_handoff; +pub(crate) mod identity_handoff_codec; pub mod kv; pub mod kv_backend; pub mod kv_types; @@ -74,7 +76,7 @@ use error_stack::Report; use http::Request; use crate::consent::{self as consent_mod, ConsentContext, ConsentPipelineInput}; -use crate::constants::{COOKIE_TS_EC, COOKIE_TS_EC_PULL_COMPLETE}; +use crate::constants::{COOKIE_TS_EC, COOKIE_TS_EC_PULL_COMPLETE, COOKIE_TS_EIDS_SERVER}; use crate::cookies::handle_request_cookies; use crate::ec::cookies::ec_id_has_only_allowed_chars; use crate::error::TrustedServerError; @@ -278,6 +280,8 @@ pub struct EcContext { eid_sync_source: Option, /// Normalized browser body updates retained even when auction dispatch fails. browser_eid_updates: Vec, + /// Authenticated capture observations, bound to the incoming full EC ID. + pub(crate) identity_handoff: identity_handoff::HandoffState, } impl EcContext { @@ -346,7 +350,17 @@ impl EcContext { consent.jurisdiction, ); + let identity_handoff = identity_handoff::HandoffState::from_cookie( + settings, + ec_value.as_deref(), + parsed + .jar + .as_ref() + .and_then(|jar| jar.get(COOKIE_TS_EIDS_SERVER)) + .map(cookie::Cookie::value), + ); Ok(Self { + identity_handoff, ec_value, cookie_ec_value: parsed.cookie_ec, ec_was_present, @@ -527,6 +541,13 @@ impl EcContext { self.geo_info.as_ref() } + /// Allows authenticated handoff work on an adapter with supported KV/post-send execution. + /// + /// Adapters must leave this disabled for bot-gated or unsupported requests. + pub fn set_identity_handoff_supported(&mut self, supported: bool) { + self.identity_handoff.supported = supported; + } + /// Returns the request-scoped identity-graph snapshot. #[must_use] pub fn kv_snapshot(&self) -> &EcKvSnapshot { @@ -646,6 +667,7 @@ impl EcContext { #[must_use] pub fn new_for_test(ec_value: Option, consent: ConsentContext) -> Self { Self { + identity_handoff: identity_handoff::HandoffState::default(), browser_eid_updates: Vec::new(), ec_was_present: ec_value.is_some(), cookie_ec_value: ec_value.clone(), @@ -671,6 +693,7 @@ impl EcContext { client_ip: Option, ) -> Self { Self { + identity_handoff: identity_handoff::HandoffState::default(), browser_eid_updates: Vec::new(), ec_was_present: ec_value.is_some(), cookie_ec_value: ec_value.clone(), @@ -699,6 +722,7 @@ impl EcContext { consent: ConsentContext, ) -> Self { Self { + identity_handoff: identity_handoff::HandoffState::default(), browser_eid_updates: Vec::new(), ec_value, cookie_ec_value, diff --git a/crates/trusted-server-core/src/integrations/prebid.rs b/crates/trusted-server-core/src/integrations/prebid.rs index 6cf87aec2..924f59483 100644 --- a/crates/trusted-server-core/src/integrations/prebid.rs +++ b/crates/trusted-server-core/src/integrations/prebid.rs @@ -36,7 +36,7 @@ use crate::auction::types::{AuctionContext, AuctionRequest, MediaType}; use crate::auction::types::{AuctionResponse, Bid as AuctionBid}; use crate::cache_policy::{CacheControlPolicy, EdgeCacheHeader}; use crate::consent_config::ConsentForwardingMode; -use crate::cookies::{CONSENT_COOKIE_NAMES, strip_cookies}; +use crate::cookies::auction_cookie_header; use crate::error::TrustedServerError; #[cfg(test)] use crate::http_util::RequestInfo; @@ -2088,24 +2088,10 @@ fn apply_prebid_header_values( let Some(cookie_value) = cookie else { return; }; - if !consent_forwarding.strips_consent_cookies() { - to.headers_mut() - .insert(header::COOKIE, cookie_value.clone()); - return; - } - match cookie_value.to_str() { - Ok(value) => { - let stripped = strip_cookies(value, CONSENT_COOKIE_NAMES); - if !stripped.is_empty() - && let Ok(cookie_header) = HeaderValue::from_str(&stripped) - { - to.headers_mut().insert(header::COOKIE, cookie_header); - } - } - Err(_) => { - to.headers_mut() - .insert(header::COOKIE, cookie_value.clone()); - } + if let Some(value) = + auction_cookie_header(cookie_value, consent_forwarding.strips_consent_cookies()) + { + to.headers_mut().insert(header::COOKIE, value); } } @@ -9740,6 +9726,68 @@ set = { networkId = 42 } ); } + #[test] + fn planned_transport_never_forwards_authenticated_handoff_proof() { + let plan = + crate::auction::plan::AuctionPlan::compile(crate::auction::plan::AuctionPlanConfig { + timeout_ms: 321, + providers: std::collections::BTreeMap::new(), + bidders: std::collections::BTreeMap::new(), + mediator: None, + request_signing: None, + }) + .expect("should compile empty plan"); + for mode in [ + ConsentForwardingMode::Both, + ConsentForwardingMode::CookiesOnly, + ConsentForwardingMode::OpenrtbOnly, + ] { + for cookie in [ + HeaderValue::from_static("session=fictional; ts-eids-s=signed-proof; __gpp=signal"), + HeaderValue::from_bytes(b"session=\xff; ts-eids-s \t =signed-proof") + .expect("should build opaque header"), + ] { + let inbound = http::Request::builder() + .uri("https://publisher.example/auction") + .header(header::COOKIE, cookie.clone()) + .body(EdgeBody::empty()) + .expect("should build inbound request"); + let routed = crate::auction::routing::route_auction( + canonical_parity_auction_request(), + &inbound, + &plan, + None, + ); + let mut outbound = http::Request::builder() + .uri("https://pbs.example.test/openrtb2/auction") + .body(EdgeBody::empty()) + .expect("should build outbound request"); + apply_prebid_transport_headers( + routed.prebid_transport_headers(), + &mut outbound, + mode, + None, + ); + if cookie.to_str().is_err() { + assert!( + !outbound.headers().contains_key(header::COOKIE), + "should drop opaque headers carrying replay proof" + ); + } else { + let forwarded = outbound.headers()[header::COOKIE] + .to_str() + .expect("should read forwarded cookies"); + assert!( + !forwarded.contains("ts-eids-s"), + "should never disclose the shared handoff to PBS" + ); + assert!(forwarded.contains("session=fictional")); + assert_eq!(forwarded.contains("__gpp"), !mode.strips_consent_cookies()); + } + } + } + } + #[test] fn planned_transport_preserves_malformed_cookie_bytes_and_attested_xff() { let inbound = http::Request::builder() diff --git a/crates/trusted-server-core/src/integrations/registry.rs b/crates/trusted-server-core/src/integrations/registry.rs index f0c1bd643..1c853e049 100644 --- a/crates/trusted-server-core/src/integrations/registry.rs +++ b/crates/trusted-server-core/src/integrations/registry.rs @@ -990,6 +990,23 @@ impl IntegrationRegistry { Self::check_identity_owners(settings, &captures) } + /// Checks a replay writer against its current registered capture descriptor. + /// + /// Construct only the selected module, using the same builder as startup. + pub(crate) fn capture_claim_enabled(settings: &Settings, module: &str, source: &str) -> bool { + crate::integrations::builders() + .iter() + .find(|builder| builder.id == module) + .and_then(|builder| (builder.build)(settings).ok().flatten()) + .and_then(|registration| registration.identity_capture) + .is_some_and(|capture| { + capture.enabled() + && capture.sources().iter().any(|claim| { + normalize_partner_source_domain(claim).ok().as_deref() == Some(source) + }) + }) + } + fn check_identity_owners( settings: &Settings, captures: &BTreeMap<&'static str, Arc>, @@ -1158,17 +1175,29 @@ impl IntegrationRegistry { status: response.status(), decoded_body: &decoded, }) - && let Ok(partners) = PartnerRegistry::from_config(&settings.ec.partners) - && let Some(effects) = stage_capture( - integration_id, - capture.sources(), - observation, - ec_context, - &partners, - settings, - ) { - response.extensions_mut().insert(effects); + // An admitted no-token response can independently replay an + // incoming handoff, but body privacy signals still restrict it. + ec_context.identity_handoff.proxy_replay_allowed = + crate::ec::identity::observation_consent_allowed( + &observation.consent, + ec_context, + settings, + ); + ec_context.identity_handoff.proxy_replay_denied = + !ec_context.identity_handoff.proxy_replay_allowed; + if let Ok(partners) = PartnerRegistry::from_config(&settings.ec.partners) + && let Some(effects) = stage_capture( + integration_id, + capture.sources(), + observation, + ec_context, + &partners, + settings, + ) + { + response.extensions_mut().insert(effects); + } } } Some(Ok(response)) diff --git a/crates/trusted-server-core/src/settings.rs b/crates/trusted-server-core/src/settings.rs index 0063bea4b..acda0c0b5 100644 --- a/crates/trusted-server-core/src/settings.rs +++ b/crates/trusted-server-core/src/settings.rs @@ -539,6 +539,98 @@ impl EcPartner { } } +/// One signing key for the authenticated identity capture handoff. +#[derive(Debug, Clone, Deserialize, Serialize)] +#[serde(deny_unknown_fields)] +pub struct IdentityHandoffKey { + /// Stable key identifier carried in authenticated handoff envelopes. + pub id: String, + /// URL-safe unpadded base64 key material, resolved from the secret store at runtime. + pub secret: Redacted, +} + +/// Configuration for the optional authenticated identity capture handoff. +#[derive(Debug, Default, Clone, Deserialize, Serialize, Validate)] +#[serde(deny_unknown_fields)] +#[validate(schema(function = validate_identity_handoff_config))] +pub struct IdentityHandoffConfig { + /// Whether handoff issuance, auction delivery, and replay are enabled. + #[serde(default)] + pub enabled: bool, + /// Identifier of the key used to issue new handoff envelopes. + #[serde(default)] + pub active_key_id: String, + /// Verification keys retained for issuance and rotation overlap. + #[serde(default)] + pub keys: Vec, +} + +impl IdentityHandoffConfig { + fn is_default(&self) -> bool { + !self.enabled && self.active_key_id.is_empty() && self.keys.is_empty() + } +} + +fn validate_identity_handoff_config(config: &IdentityHandoffConfig) -> Result<(), ValidationError> { + let validation_error = |message: String| { + let mut error = ValidationError::new("invalid_identity_handoff_config"); + error.message = Some(message.into()); + error + }; + + if config.keys.len() > 4 { + return Err(validation_error( + "ec.identity_handoff.keys must contain at most 4 keys".to_owned(), + )); + } + + let mut key_ids = HashSet::new(); + for key in &config.keys { + if key.id.is_empty() + || key.id.len() > 32 + || !key + .id + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-')) + { + return Err(validation_error( + "ec.identity_handoff key ids must be 1 to 32 ASCII alphanumeric, '_' or '-' characters" + .to_owned(), + )); + } + if !key_ids.insert(key.id.as_str()) { + return Err(validation_error(format!( + "ec.identity_handoff key id `{}` is configured more than once", + key.id + ))); + } + } + + if !config.active_key_id.is_empty() + && (config.active_key_id.len() > 32 + || !config + .active_key_id + .bytes() + .all(|byte| byte.is_ascii_alphanumeric() || matches!(byte, b'_' | b'-'))) + { + return Err(validation_error( + "ec.identity_handoff.active_key_id must be 1 to 32 ASCII alphanumeric, '_' or '-' characters when set" + .to_owned(), + )); + } + + if config.enabled + && (config.active_key_id.is_empty() || !key_ids.contains(config.active_key_id.as_str())) + { + return Err(validation_error( + "enabled ec.identity_handoff requires active_key_id to name a configured key" + .to_owned(), + )); + } + + Ok(()) +} + /// Edge Cookie (EC) configuration. /// /// Mapped from the `[ec]` TOML section. Controls EC identity generation, @@ -572,6 +664,11 @@ pub struct Ec { #[serde(default = "Ec::default_cluster_recheck_secs")] pub cluster_recheck_secs: u64, + /// Optional authenticated handoff for validated server-side identity capture. + #[serde(default, skip_serializing_if = "IdentityHandoffConfig::is_default")] + #[validate(nested)] + pub identity_handoff: IdentityHandoffConfig, + /// Partners (SSPs, DSPs, identity vendors) for EC identity sync. #[serde(default, deserialize_with = "vec_from_seq_or_map")] #[validate(nested)] @@ -3957,6 +4054,7 @@ mod tests { fn settings_debug_output_redacts_every_secret_field() { const CANARY_PROXY_SECRET: &str = "CANARY-PROXY-SECRET-0123456789"; const CANARY_EC_PASSPHRASE: &str = "CANARY-EC-PASSPHRASE-0123456789"; + const CANARY_IDENTITY_HANDOFF_KEY: &str = "CANARY-IDENTITY-HANDOFF-KEY-0123456789"; const CANARY_HANDLER_USERNAME: &str = "CANARY-HANDLER-USERNAME-0123456789"; const CANARY_HANDLER_PASSWORD: &str = "CANARY-HANDLER-PASSWORD-0123456789"; const CANARY_EC_PARTNER_API_TOKEN: &str = "CANARY-EC-PARTNER-API-TOKEN-0123456789"; @@ -3974,6 +4072,14 @@ mod tests { settings.publisher.proxy_secret = Redacted::new(CANARY_PROXY_SECRET.to_string()); settings.ec.passphrase = Redacted::new(CANARY_EC_PASSPHRASE.to_string()); + settings.ec.identity_handoff = IdentityHandoffConfig { + enabled: true, + active_key_id: "canary-key".to_owned(), + keys: vec![IdentityHandoffKey { + id: "canary-key".to_owned(), + secret: Redacted::new(CANARY_IDENTITY_HANDOFF_KEY.to_owned()), + }], + }; settings.handlers = vec![Handler { path: "^/secure".to_string(), @@ -4057,6 +4163,10 @@ mod tests { let canaries = [ ("publisher.proxy_secret", CANARY_PROXY_SECRET), ("ec.passphrase", CANARY_EC_PASSPHRASE), + ( + "ec.identity_handoff.keys[].secret", + CANARY_IDENTITY_HANDOFF_KEY, + ), ("handlers[].username", CANARY_HANDLER_USERNAME), ("handlers[].password", CANARY_HANDLER_PASSWORD), ("ec.partners[].api_token", CANARY_EC_PARTNER_API_TOKEN), @@ -5645,6 +5755,146 @@ source_domain = "partner.example.com" ); } + #[test] + fn identity_handoff_defaults_disabled_without_keys() { + let config = IdentityHandoffConfig::default(); + + assert!(!config.enabled, "handoff should default off"); + assert!( + config.active_key_id.is_empty(), + "handoff should not select a key by default" + ); + assert!(config.keys.is_empty(), "handoff should default to no keys"); + config + .validate() + .expect("disabled empty handoff config should validate"); + } + + #[test] + fn identity_handoff_default_is_omitted_but_provisioned_disabled_keys_serialize() { + let mut settings = Settings::from_toml(&crate_test_settings_str()) + .expect("should parse settings without identity handoff configuration"); + let value = serde_json::to_value(&settings).expect("should serialize default settings"); + assert!( + value["ec"].get("identity_handoff").is_none(), + "default handoff should remain absent from pushed config" + ); + + settings.ec.identity_handoff = IdentityHandoffConfig { + enabled: false, + active_key_id: "next".to_owned(), + keys: vec![IdentityHandoffKey { + id: "next".to_owned(), + secret: Redacted::new("next-secret-reference".to_owned()), + }], + }; + let value = serde_json::to_value(&settings) + .expect("should serialize provisioned disabled handoff settings"); + assert_eq!(value["ec"]["identity_handoff"]["enabled"], false); + assert_eq!( + value["ec"]["identity_handoff"]["keys"][0]["secret"], + "next-secret-reference" + ); + } + + #[test] + fn identity_handoff_structure_accepts_rotation_keys() { + let config = IdentityHandoffConfig { + enabled: true, + active_key_id: "current_key-2".to_owned(), + keys: vec![ + IdentityHandoffKey { + id: "previous-key".to_owned(), + secret: Redacted::new("previous-secret-reference".to_owned()), + }, + IdentityHandoffKey { + id: "current_key-2".to_owned(), + secret: Redacted::new("current-secret-reference".to_owned()), + }, + ], + }; + + config + .validate() + .expect("rotation config should retain an old verification key"); + } + + #[test] + fn identity_handoff_structure_rejects_unsafe_duplicate_and_missing_active_ids() { + let cases = [ + IdentityHandoffConfig { + enabled: false, + active_key_id: String::new(), + keys: vec![IdentityHandoffKey { + id: String::new(), + secret: Redacted::new("secret-reference".to_owned()), + }], + }, + IdentityHandoffConfig { + enabled: false, + active_key_id: String::new(), + keys: vec![IdentityHandoffKey { + id: "x".repeat(33), + secret: Redacted::new("secret-reference".to_owned()), + }], + }, + IdentityHandoffConfig { + enabled: false, + active_key_id: String::new(), + keys: vec![IdentityHandoffKey { + id: "unsafe key".to_owned(), + secret: Redacted::new("secret-reference".to_owned()), + }], + }, + IdentityHandoffConfig { + enabled: false, + active_key_id: String::new(), + keys: vec![ + IdentityHandoffKey { + id: "duplicate".to_owned(), + secret: Redacted::new("first-secret-reference".to_owned()), + }, + IdentityHandoffKey { + id: "duplicate".to_owned(), + secret: Redacted::new("second-secret-reference".to_owned()), + }, + ], + }, + IdentityHandoffConfig { + enabled: true, + active_key_id: "missing".to_owned(), + keys: vec![IdentityHandoffKey { + id: "configured".to_owned(), + secret: Redacted::new("secret-reference".to_owned()), + }], + }, + ]; + + for config in cases { + config + .validate() + .expect_err("invalid handoff key selection should fail validation"); + } + } + + #[test] + fn identity_handoff_structure_rejects_more_than_four_keys() { + let config = IdentityHandoffConfig { + enabled: false, + active_key_id: String::new(), + keys: (0..5) + .map(|index| IdentityHandoffKey { + id: format!("key-{index}"), + secret: Redacted::new(format!("secret-reference-{index}")), + }) + .collect(), + }; + + config + .validate() + .expect_err("handoff should cap the rotation key ring at four keys"); + } + #[test] fn validate_passphrase_rejects_under_32_characters() { let passphrase = Redacted::new("a".repeat(31)); diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 6d4e645b3..553437974 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -56,6 +56,7 @@ publisher, trusted-client-IP, EC, handler, Tinybird, DataDome, and S3 fields: - `publisher.proxy_secret` - `trusted_client_ip.shared_secret`, when trusted client-IP forwarding is configured - `ec.passphrase` +- `ec.identity_handoff.keys[*].secret`, for each provisioned handoff signing key - `ec.partners[*].api_token`, when inbound identify or batch sync is used - `ec.partners[*].ts_pull_token`, when pull sync is enabled - `handlers[*].password` @@ -99,6 +100,7 @@ serialization, runtime, and secret axes for every exceptional field: | `DataDomeProtectionTestBypassConfig.credential_secret_name` | canonical | canonical | serialized | active | store resolved | | `DataDomeProtectionTestBypassConfig.credential_secret_store` | deprecated | canonical | skipped | normalized away | none | | `Ec.passphrase` | canonical | canonical | serialized | active | store resolved | +| `IdentityHandoffKey.secret` | canonical | canonical | serialized | active when provided | store resolved | | `EcPartner.api_token` | canonical | canonical | serialized | active | store resolved | | `EcPartner.ts_pull_token` | canonical | canonical | serialized | active | store resolved | | `Handler.password` | canonical | canonical | serialized | active | store resolved | @@ -646,6 +648,7 @@ be at least 32 bytes. Keep it stable to preserve EC identifier continuity. | `pull_sync_concurrency` | Integer | No | Maximum concurrent pull-sync requests per organic response | | `cluster_trust_threshold` | Integer | No | Cluster size threshold for identity trust decisions | | `cluster_recheck_secs` | Integer | No | Legacy compatibility setting; cluster rechecks no longer use timestamps | +| `identity_handoff` | Table | No | Authenticated short-lived capture handoff, disabled by default | | `partners` | Array | No | Static partner registry entries | ::: tip Partner keying @@ -672,7 +675,9 @@ writes. Unowned legacy pull remains fill-missing-only even for expired records. Drain metadata-unaware readers and writers before enabling managed capture. `/auction` body EIDs enrich existing consenting rows conservatively; the retired `ts-eids` cookie and EID diagnostic response headers are no longer identity -transport. Inbound diagnostic headers remain stripped. See the +transport. The optional `ts-eids-s` cookie is an authenticated server-capture +handoff only. It does not restore `ts-eids` or collect browser IDs. Inbound +diagnostic headers remain stripped. See the [Prebid transport migration](./integrations/prebid.md) and [Lockr capture gates](./integrations/lockr.md). @@ -691,6 +696,61 @@ bidstream_enabled = true # ts_pull_token = "partner_ts_pull_token" # required when pull sync is enabled ``` +### `[ec.identity_handoff]` + +The authenticated `ts-eids-s` capture handoff is an explicit publisher opt-in. +It defaults off with no keys. Keep it off until capture, consent, source +ownership, deployment compatibility, synthetic checks, and live acceptance have +all passed for the publisher route. + +```toml +[ec.identity_handoff] +enabled = false +active_key_id = "handoff-2026-10" + +[[ec.identity_handoff.keys]] +id = "handoff-2026-10" +secret = "identity_handoff_2026_10" +``` + +`secret` is a key name in `trusted_server_secrets`. Its resolved value must be +URL-safe unpadded base64 for exactly 32 random bytes. Use a dedicated value, not +`ec.passphrase` or another application credential. Key IDs must be unique, +non-empty ASCII alphanumeric, `_`, or `-`, with at most 32 characters. Configure +no more than four keys. When `enabled = true`, `active_key_id` must name one of +them. + +Generate a value locally, then write it directly to the mapped secret store: + +```bash +openssl rand 32 | openssl base64 -A | tr '+/' '-_' | tr -d '=' +``` + +Trusted Server resolves every provided key at startup, even while handoff is +disabled, and fails startup if a value is missing, malformed, or does not decode +to exactly 32 bytes. This lets operators provision and check keys before opting +in. Store changes are startup-scoped. + +For rotation, add the new key and secret while retaining the old key. Push and +deploy that verification set first. After every instance can read both keys, +change `active_key_id` to the new key and push again. Remove the old key only +after old invocations have drained and at least the five-minute handoff lifetime +has elapsed since all instances began issuing with the new key. The active key +signs new envelopes; retained keys verify envelopes created before the switch. + +The cookie value has a fixed five-minute maximum lifetime and a 2,048-byte +encoded-value limit. Provider expiry may shorten the lifetime. The handoff is +eligible only for an absent registered source on an existing live, consenting +EC root. Current consent, source ownership, bidstream policy, the integration's +capture switch, and the shared handoff switch all still apply. Existing, +expired, invalidated, or conflicting stored records win. + +Persistence retry is best-effort. A later qualifying request may stage one +bounded fill-absent attempt after the response is sent on Fastly. There is no +durable queue, recursive retry, root creation, existing-source refresh, or +first-auction guarantee. Keep rollout and live acceptance gated as described in +the [Lockr integration guide](./integrations/lockr.md). + **Environment Override**: ```bash diff --git a/docs/guide/integrations/lockr.md b/docs/guide/integrations/lockr.md index ca796228b..e2805e968 100644 --- a/docs/guide/integrations/lockr.md +++ b/docs/guide/integrations/lockr.md @@ -1,7 +1,7 @@ # Lockr Integration **Category**: Identity -**Status**: Production +**Status**: Proxy production; identity capture and handoff rollout gated **Type**: Identity Management & Privacy Vault ## Overview @@ -27,9 +27,12 @@ flowchart TD A[Browser loads first-party Lockr SDK] --> B[SDK calls first-party API proxy] B --> C[Lockr returns provider tokens] C --> D[Capture validates ID5, consent and source ownership] - D --> E[Original response sent to browser] + D --> E[Original response sent with optional authenticated handoff] E --> F[Eligible EC record updated after send on Fastly] - F --> G[Later auctions read usable IDs from KV] + E --> G[Later live-root request validates handoff] + G --> H[Absent source reaches auction and stages bounded retry] + F --> I[Later auctions read usable IDs from KV] + H --> I ``` ## Configuration @@ -37,6 +40,15 @@ flowchart TD Add Lockr configuration to `trusted-server.toml`: ```toml +[ec.identity_handoff] +# Separate opt-in. Leave off until the handoff rollout and live gates pass. +enabled = false +active_key_id = "handoff-2026-10" + +[[ec.identity_handoff.keys]] +id = "handoff-2026-10" +secret = "identity_handoff_2026_10" + [integrations.lockr] enabled = true app_id = "example-app-id" @@ -66,7 +78,11 @@ identity_owner = "lockr" | `rewrite_sdk` | boolean | No | Rewrite matching SDK URLs to the first-party route (default: `true`) | | `origin_override` | URL | No | Origin header override for local or approved domains | -`rewrite_sdk_host` is deprecated and ignored. An absent Lockr configuration does not register the integration. +`rewrite_sdk_host` is deprecated and ignored. An absent Lockr configuration +does not register the integration. `[ec.identity_handoff]` is shared EC +configuration, not a Lockr option. Its keys come from +`trusted_server_secrets`; see the [configuration reference](../configuration) +for provisioning and rotation. ### Environment Variables @@ -139,7 +155,7 @@ The Lockr integration is implemented in [crates/trusted-server-core/src/integrat - The only supported capture mapping is Lockr `id5id` to registered `id5-sync.com` with `atype=1`. - The EC identity graph retains the source, UID and lifecycle metadata, including known provider expiry and the acquisition writer. Later auctions reconstruct fixed Lockr ID5 provenance only for a usable Lockr-written record. KV does not store arbitrary provenance, provider `ext` or additional UIDs. -- Publisher SDK storage remains the SDK's responsibility. Capture does not add an EID cookie. +- Publisher SDK storage remains the SDK's responsibility. Capture never restores the browser-written `ts-eids` cookie. The separately enabled shared handoff may add authenticated `ts-eids-s`. **Consent Validation**: @@ -151,13 +167,55 @@ The Lockr integration is implemented in [crates/trusted-server-core/src/integrat Capture inspects only exact POST `page-view`, `generate-tokens`, `refresh-tokens` and `sync-no-hem-ids` routes under `/publisher/app/v2/identityLockr/`. It decodes URL-encoded `advertising_token.universal_uid` from `aimTokens`, verifies an associated `ids.id5id.eid` when present, and converts `identity_expires` milliseconds to Unix seconds. Unknown provider keys, mismatched source or UID, expired tokens and malformed responses do not change stored identity. SDK, settings and unlisted API requests stay on the original proxy path. The proxy still strips publisher `Cookie` and `Authorization` headers. -Original and decoded capture responses have separate 64 KiB limits. Inspection failure leaves upstream response bytes and headers intact. With capture enabled, Fastly sends the SDK response before shared capture persistence against the existing consenting EC record. Missing, unreadable or tombstoned roots cannot be created or revived by acquisition. There is no native resolver, extra EID cookie, first-auction guarantee or durable retry queue. +Original and decoded capture responses have separate 64 KiB limits. Inspection failure leaves upstream response bytes and headers intact. With capture enabled, Fastly sends the SDK response before shared capture persistence against the existing consenting EC record. Missing, unreadable or tombstoned roots cannot be created or revived by acquisition. There is no native resolver, first-auction guarantee or durable retry queue. + +### Authenticated capture handoff + +The optional shared handoff can add `ts-eids-s` to a validated capture response. +It defaults off independently of Lockr capture. Enabling it requires both +`ec.identity_handoff.enabled = true` and `integrations.lockr.capture_identity = +true`, plus an active dedicated signing key. Do not reuse `ec.passphrase`. Keep +the handoff off until key management, source ownership, consent behavior, +metadata-compatible deployment, synthetic verification, and live acceptance +have passed for the publisher route. + +`ts-eids-s` has a maximum five-minute lifetime and a 2,048-byte encoded-value +budget. It binds the publisher, full EC ID, capture writer, source, UID, expiry, +and original handoff deadline in an authenticated envelope. `Secure`, +`HttpOnly`, and `SameSite=Lax` cookie flags do not replace signature checking. +Provider expiry may shorten the deadline, and reads or retries never extend it. + +A later request can use an eligible ID5 handoff only when the registered source +is absent from an existing live, consenting EC root. Current Lockr capture, +source ownership, consent, expiry, and bidstream gates are checked again. Any +existing record wins, including an expired, invalidated, or conflicting record. +The handoff never creates or recovers a root, replaces a UID, extends stored +expiry, or accelerates refresh of an existing source. + +Eligibility uses the existing request-local KV snapshot. A stale live read can +still miss a newer record or withdrawal at another POP; the handoff does not add +strong reads or instant cross-POP revocation. Current request consent still +applies. Replay uses generation-checked writes and cannot overwrite that newer +record or revive its tombstone. + +The same later request may stage one bounded fill-absent persistence attempt. +Fastly runs it after sending the response. This retry is best-effort and needs a +qualifying request before the original deadline. It has no durable queue or +recursive provider call. A successful after-send write cannot clear the cookie +already sent; a subsequent request reconciles and clears acknowledged entries. + +This feature does not restore the retired browser-written `ts-eids` cookie. +`ts-eids-s` contains validated server capture only. It does not ingest late +browser EIDs or restore the old diagnostic headers. Cookie-bearing responses +are private/no-store, and auction-provider Cookie forwarding strips the handoff. +Only Fastly's admitted EC/KV requests support issuance and post-send replay; +other adapters keep the feature inactive. `capture_identity` defaults to `true` but must be explicitly set to `false` during rollout until sanitized provider fixtures, consent requirements, source ownership, metadata-aware binary drain and an approved publisher test route have been verified. Disabling capture keeps the registered ID5 source claim and already-stored metadata. Do not roll back to a metadata-unaware binary after managed writes without an approved compatibility procedure. Reject conflicting legacy push or pull writers before assigning `identity_owner = "lockr"`. `capture_global_withdrawal` remains `false` until Lockr confirms that a 2xx `revoke-consent` response with `"success": true` means global withdrawal for this EC identity. Only then may operators opt in: the normal finalizer expires `ts-ec` and the existing-key-only tombstone prevents later enrichment. Empty token arrays, network failures and other responses never imply withdrawal. Overlapping SDK calls have no provider sequence contract; CAS follows observation order. -Use synthetic or sanitized source/count/status evidence for an approved test route. Verify unchanged proxy response, delayed KV read-back of UID/source/expiry/writer/revision without exporting values, later auction source presence, same-UID expiry extension, expired omission and confirmed opt-out/revoke. No live IDs, consent strings, credentials or query URLs belong in test records. +Use synthetic or sanitized source/count/status evidence for an approved test route. Verify unchanged proxy response, delayed KV read-back of UID/source/expiry/writer/revision without exporting values, later auction source presence, same-UID expiry extension, expired omission and confirmed opt-out/revoke. For handoff acceptance, also hold back KV visibility, verify absent-source auction delivery only after cookie installation, inject a failed post-send write, observe later best-effort repair and acknowledgement cleanup, then test expiry, consent denial, root read failure, EC rotation, ownership changes and key rotation. These are rollout gates, not completed acceptance. No live IDs, consent strings, credentials or query URLs belong in test records. ## Best Practices diff --git a/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md b/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md index 9fa1f1c10..8e4ece62b 100644 --- a/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md +++ b/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md @@ -1,10 +1,10 @@ # Server-side identity foundation and Lockr capture implementation plan -**Status:** Milestones 1–5 implemented in the draft change. Local verification is recorded below. Provider activation and deployment compatibility remain gated; this is not live acceptance. +**Status:** Milestones 1–6 implemented in the draft change. Local verification is recorded below, with the default-off capture handoff checked separately. Provider activation and deployment compatibility remain gated; this is not live acceptance. **Specs:** [Foundation](../specs/2026-10-09-server-side-identity-foundation-design.md), [Lockr capture](../specs/2026-10-08-lockr-identity-capture-design.md), [KV snapshot and recovery](../specs/2026-07-10-kv-eid-request-snapshot-ec-recovery-design.md). The [existing LiveRamp browser design](../specs/2026-08-21-liveramp-integration-design.md) does not approve a native resolver. -**Goal:** Persist registered browser auction EIDs and trusted Lockr proxy observations on an existing, consenting EC row, retain expiry and writer metadata through every mutation, and expose only usable records to later auctions. Keep the response intact and send it before capture writes on Fastly. No extra EID cookie, root creation by acquisition, or new provider fetch loop. +**Goal:** Persist registered browser auction EIDs and trusted Lockr proxy observations on an existing, consenting EC row, retain expiry and writer metadata through every mutation, and expose only usable records to later auctions. Keep the response intact and send it before capture writes on Fastly. Keep browser-written `ts-eids` retired. The optional authenticated `ts-eids-s` handoff carries bounded pending capture only. No root creation by acquisition or new provider fetch loop. ```mermaid flowchart TD @@ -14,7 +14,8 @@ flowchart TD B --> E[Bounded capture and Fastly after-send effects] D --> F[Lockr capture consumer] E --> F - F --> G[Approved fixture and deployment gates] + F --> H[Default-off authenticated capture handoff] + H --> G[Approved fixture and deployment gates] ``` ## Boundaries and starting code (pre-implementation baseline) @@ -87,13 +88,25 @@ cargo test --package trusted-server-cli --target "$HOST_TARGET" - Withdrawal is gated separately: only after Lockr confirms 2xx application-success semantics **and** whether revoke is global EC withdrawal or per-source invalidation. Test revoke despite denied acquisition consent, finalization precedence, cookie expiry and existing-key-only tombstone if global scope is approved. Until then, leave revoke capture disabled and let ordinary request consent withdrawal continue; never infer withdrawal from empty tokens or a network error. Clarify ordering if overlapping SDK calls can return older tokens after newer ones; absent a provider sequence, CAS order is observation order only. - Check: filtered `cargo test-fastly` Lockr parser/proxy/post-send tests with synthetic responses; no live IDs in fixtures. Then run approved publisher test-route acceptance: consent, unchanged original response, delayed KV readback of UID/source/expiry/writer/revision without exporting values, later bidder payload source presence, same-UID extension, known-expired omission including matching body UID, negative consent and approved revoke. Record statuses, counts and booleans only. Capture stays off for a source until its fixture, consent, ownership and compatibility gates pass. +### 6. Restore bounded capture availability and retry without cookie authority over stored records + +**Files:** `ec/identity_handoff_codec.rs`, `ec/identity_handoff.rs`, `ec/identity_handoff_tests.rs`, shared context/effects/KV/finalization, auction resolution, integration dispatch, Fastly request admission, settings/app-config validation, cookie forwarding/cache policy and operator docs. + +- Implemented default-off `[ec.identity_handoff]`, dedicated secret-store signing keys, safe active/rotation IDs and deploy/runtime validation. Runtime keys decode to exactly 32 bytes and cannot reuse EC passphrase material. Disabled provisioned keys still resolve and validate. +- Implemented a publisher/full-EC-bound, versioned HMAC-SHA256 envelope. The complete signed value is bounded to 2,048 bytes. Entries retain absolute deadlines of at most five minutes, shortened by provider expiry. Oversized entries are dropped whole; multi-source eviction removes earliest deadlines first. Response Max-Age derives from retained signed entries, not pre-truncation candidates. +- Shared finalization issues capture handoff before send without an added KV read, revalidates incoming entries, retires acknowledgements/conflicts and clears empty or rejected cookies only when one was present. Cookie responses are private/no-store, preserve upstream body/encoding and exclude the handoff from auction-provider Cookie forwarding and template cache keys. +- All shared auction callers can deliver an eligible absent-source entry from a loaded, consenting full-ID-bound snapshot. Any observed stored record wins, including expired records. Current consent, capture kill switch, source claims/ownership, clock and expiry still apply. Eligibility retains existing KV read consistency; no strong-read or instant cross-POP withdrawal guarantee is added. +- Browser ingestion cannot strip authenticated pending capture expiry/writer metadata. Matching captures retain the original pending deadline and known expiry when the provider omits it. That pending expiry only fills an absent source and cannot change any stored expiry. A new direct capture supersedes pending replay for its source; existing-source refresh remains direct capture only. +- At most one eligible replay batch is staged for a browser persistence route or an observed permitted Lockr proxy request, including `NoChange`. Body denial clears the handoff and cannot borrow cookie permission. Fastly enables the feature only with its admitted EC/KV finalization graph; other adapters remain unsupported. +- The existing shared executor sends first, then proves the root and applies fill-absent-only CAS before legacy pull work. Conflicts recheck source absence and withdrawal. Replay never creates a root, replaces UIDs/metadata or extends the original deadline. Successful persistence is acknowledged on a later response. No provider refetch, durable queue or unconditional first-auction guarantee. + ## Release and rollback 1. Ship optional schema-v1 metadata-aware readers and **every** metadata-preserving writer first, with managed acquisition off. Verify mixed-record fixtures. Obtain deployment propagation and old-invocation drain evidence; old UID-only readers ignore expiry and old writers erase metadata. Confirm operator procedure before the first managed write. 2. Treat browser writer/revision output as metadata activation too; complete the compatibility gate before enabling any path that produces lifecycle metadata. Activate body persistence in a coordinated server release; prove body/no-cookie behavior before the transition JS writer removal. Check diagnostic consumers before header retirement. Roll back the JS transition only with a server that still understands body submission; old cached JS may write the ignored cookie for its remaining lifetime. 3. Deploy synthetic capture framework inactive and verify Fastly send ordering. Although the Lockr setting defaults to true, explicitly deploy `capture_identity = false` until approval and compatibility gates pass; ownership validation must still work. On an approved route, add one confirmed Lockr source with owner and `atype` after resolving conflicting legacy writes or pull configuration. Enable capture, measure source-only counts, KV readback and later auction visibility. Canary first; do not claim first-auction guarantees or automatic retry. 4. Safe immediate mitigation after activation is `capture_identity = false` or removing ownership through an **approved migration**; disabling capture alone does not erase stored expiry. Rolling back to a UID-only binary is unsafe until managed writes stop and expiry-aware read/metadata compatibility is explicitly resolved. Do not automatically transfer owner on module failure. DSR deletion and global consent withdrawal remain authoritative over acquired records. -5. At intermediate checkpoints, run only suites affected by the completed milestone. At the final stable implementation checkpoint, run `cargo fmt --all -- --check`, `cargo test-fastly`, `cargo test-fastly-reuse`, `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin`, `./scripts/test-cli.sh`, JS tests/build/format, integration parity and affected target-matched clippy aliases per `AGENTS.md`. Reuse unchanged passing results rather than repeat this matrix at every checkpoint. Confirm CI and approved live-route evidence separately. Core tests are not a substitute for an observed post-send KV write and later auction. +5. At intermediate checkpoints, run only suites affected by the completed milestone. At the final stable implementation checkpoint, run `cargo fmt --all -- --check`, `cargo test-fastly`, `cargo test-fastly-reuse`, `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin`, `./scripts/test-cli.sh`, JS tests/build/format, integration parity and affected target-matched clippy aliases per `AGENTS.md`. Reuse unchanged passing results rather than repeat this matrix at every checkpoint. Confirm CI and approved live-route evidence separately. Core tests are not a substitute for an observed deployed post-send KV write and later auction. Handoff activation additionally requires dedicated publisher key provisioning/rotation and approved delayed-visibility, failure-repair, cleanup and withdrawal checks. Keep `ec.identity_handoff.enabled = false` until those gates pass. ## Decisions and remaining approval gates @@ -121,3 +134,15 @@ Not verified: real provider acceptance, sanitized provider-approved mappings/fix - `cargo test-fastly`: 219 adapter tests and 2,933 core tests passed, with six core tests ignored; shared-crate and doc tests also passed. - `cargo clippy-fastly`, `cargo fmt --all -- --check`, scoped Prettier checks for the updated specs/plan/guide, and `git diff --check` passed. - Other adapter suites, the CLI/browser-fixture suites, remote CI and live provider acceptance were not rerun for this follow-up. Capture remains gated as above. + +## Authenticated capture handoff verification (2026-10-12) + +- Native `cargo test -p trusted-server-core --target x86_64-unknown-linux-gnu --lib`: 2,981 passed. The handoff/config/codec subset has 40 tests. It covers signed bounds/bindings/rotation, secret resolution and dedicated key material, response bytes/cache policy, absent-source delivery, metadata protection, replay/acknowledgement, retained expired records, same-UID changed-expiry conflicts, concurrent capture/withdrawal, deadline rechecks and no-token proxy repair/body denial. +- `cargo test-fastly`: 219 adapter tests and 2,975 core tests passed, with six native-only core tests ignored; shared-crate and doc tests passed. `cargo test-fastly-reuse`: 219 passed. An initial Fastly run found that the newly added example prose was contiguous with an uncommentable TOML block. Added a blank-line separator and reran the entire core/Fastly suite successfully. +- `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin`, `./scripts/test-cli.sh`, integration parity and native build-digest tests passed. CLI verification includes all four required ignored browser-fixture filters. Other adapters remain unsupported for handoff issuance/replay. +- All target-matched Clippy aliases passed, including native build-digest lint. Fastly/Axum/Cloudflare builds passed. Full JS Vitest, JS formatting and bundle build passed, with 48 files and 1,186 tests. Full docs and non-docs Markdown formatting, Rust formatting and whitespace checks passed. +- Executed safety proof: no handoff becomes auction input without valid authentication, current policy and an observed live consenting root. Proxy and shared-finalizer tests inspect original response bytes, cookie policy and auction provenance. Missing/failed/withdrawn/full-ID mismatch, disabled capture, tampering and unknown clock do not deliver. +- Executed safety proof: replay cannot replace any stored source or revive a concurrent withdrawal. Tests read back stored state after CAS conflicts and prove send precedes mutation. Provider expiry/writer survive replay, expired deadlines do not write, and `NoChange` retains the original signed deadline. Matching capture retains pending expiry/deadline; omitted provider expiry fills only an absent source and cannot override a newer stored expiry. Existing expiry changes win even when the UID matches. A planned PBS transport regression proves that all consent-forwarding modes strip the handoff, including opaque headers with whitespace around its name. +- Executed consistency limit: a simulated stale live snapshot can deliver its pending handoff despite an authoritative withdrawal elsewhere. The subsequent CAS rejects replay and preserves the tombstone. This is the existing snapshot-read consistency limit, not a strong-read or instant cross-POP withdrawal guarantee; deployed propagation/frequency remains unverified. + +Not verified: remote CI, live provider acceptance, actual cross-POP visibility/withdrawal timing, operational key provisioning/rotation, provider-approved fixture sizes, consent/revoke scope, overlapping SDK response ordering or metadata-compatible deployment drain. Keep handoff default-off and Lockr capture explicitly disabled until the applicable rollout gates pass. diff --git a/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md b/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md index 9210ab07c..2aaeb2edb 100644 --- a/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md +++ b/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md @@ -2,7 +2,9 @@ Status: ID5-only capture implemented in the draft change, awaiting code review. Provider fixtures, consent approval, revocation semantics and live acceptance -remain gated; rollout configuration explicitly disables capture. Depends on the +remain gated; rollout configuration explicitly disables capture. The default-off +`ts-eids-s` handoff is implemented in the draft change, with separate local +verification in the implementation plan. Depends on the [server-side identity foundation](2026-10-09-server-side-identity-foundation-design.md). See the [implementation plan and verification](../plans/2026-10-09-server-side-identity-foundation-and-lockr.md). @@ -35,12 +37,18 @@ common identity outcomes to the shared service, which persists them under the visitor's existing EC ID after sending the response on Fastly. The lockr SDK still runs in the browser. Capture does not depend on Prebid -configuration or the TS JavaScript EID collector. There is no `ts-eids-s` or -replacement EID cookie. `ts-ec` remains the browser's identity pointer. - -Delayed availability is accepted: a later auction sees an ID only after the -deferred write completes and its KV read can see the record. The first auction -may miss it. There is no cookie fallback or durable capture retry queue. +configuration or the TS JavaScript EID collector. `ts-ec` remains the browser's +identity pointer. Reintroduce `ts-eids-s` as an optional, short-lived authenticated +capture handoff through the shared identity service, not a long-lived identity +store or restoration of the browser-written `ts-eids` cookie. + +For a source absent from a live, consenting EC row, a request carrying the handoff +can use the captured ID before its post-send write becomes visible and can stage +a bounded persistence retry. An auction that starts before the capture response +installs the cookie can still miss it. Existing-source refresh still waits for +the direct capture update; stale cookie replay must not replace stored state. +Without handoff opt-in, the implemented KV-only behavior remains unchanged. +There is no durable capture retry queue. ## 2. Observed lockr flow @@ -112,8 +120,9 @@ retiring the cookie path. a TS browser capture module. 2. Implement the foundation's optional capture capability only. Lockr has no active server-side resolver in this delivery. -3. Return original proxy responses intact, with no added EID cookie and no - Fastly KV enrichment write before send. +3. Preserve original proxy body bytes and encoding/length headers. With shared + handoff enabled, finalization may add `ts-eids-s` and cache privacy headers. + No Fastly capture or handoff-replay KV write runs before send. 4. Submit authoritative `Issued` outcomes for owned, registered sources; subsequent SDK token responses use the same path. 5. Preserve known provider expiry through the foundation's lifecycle record. @@ -124,6 +133,9 @@ retiring the cookie path. foundation. A failed capture must not break the SDK call. 8. Reconstruct the fixed ID5 OpenRTB provenance on later auctions only for usable records whose persisted acquisition writer is `lockr`. +9. Offer immediate absent-source delivery and bounded retry through the + foundation's authenticated capture handoff, without weakening live-root, + consent, ownership, expiry or stale-replay rules. Out of scope: server-to-server calls to lockr, identifier intake, cross-browser linking, changing Prebid provider discovery, direct ID5 or LiveRamp proxies, @@ -136,8 +148,9 @@ OpenRTB provenance/provider `ext` storage, and client-side delivery of KV IDs. Register `LockrIntegration` through `.with_identity_capture()` alongside its existing proxy and attribute rewriter. Its identity descriptor claims the confirmed source mappings in section 4.2. It registers no resolver capability. -`capture_identity = false` disables observation while preserving the descriptor -needed to validate configured source ownership. +`capture_identity = false` disables observation and Lockr handoff delivery/retry +while preserving the descriptor needed to validate configured source ownership. +Usable records already persisted in KV retain their existing delivery rules. The capture paths are exact POST paths relative to `/integrations/lockr/api`: @@ -230,9 +243,9 @@ provider-specific opt-out must not be widened into a global withdrawal. This outcome does not pass through token-consent eligibility: revocation must still work after consent becomes denied. The shared finalizer expires `ts-ec` -and the service applies the existing-key-only tombstone path. Normal request -withdrawal takes precedence and suppresses duplicate work. Lockr never clears -or recreates a root independently. +and any capture handoff cookie, and the service applies the existing-key-only +tombstone path. Normal request withdrawal takes precedence and suppresses +duplicate work. Lockr never clears or recreates a root independently. ### 4.5 Shared persistence and auction delivery @@ -278,6 +291,66 @@ for the same source, omit reconstructed provenance for that merged EID rather than attribute all its UIDs to Lockr. Keep existing UID merging unchanged. Provider `ext` and extra UIDs are still not stored in KV. +### 4.6 Short-lived capture handoff + +Use the foundation's section 8.2 contract for `ts-eids-s`. The shared service, +not `LockrIntegration`, authenticates the envelope, sets the cookie, resolves +eligible auction IDs, stages retry and reconciles acknowledgement. Start with +ID5 only; no unconfirmed mapping becomes eligible through the cookie. + +Each entry binds the publisher, full EC ID, acquisition module `lockr`, registered +source, UID, optional provider expiry and fixed handoff deadline. The foundation +implements a five-minute lifetime and a 2,048-byte cap for the complete authenticated +value. Provider expiry can shorten that lifetime. Cookie reads, retries, merging +and no-token responses do not extend it. Keep `Secure; HttpOnly; SameSite=Lax`; +base64 and `HttpOnly` alone do not authenticate replay. + +Create entries only from acquisitions that pass the existing capture consent, +source-claim and ownership gates. Normal deferred capture persistence still runs +if cookie creation fails. A `NotRead` snapshot at capture does not force an added +pre-send lookup, but it also does not prove root eligibility for later delivery. + +On later requests, validate the handoff against current policy and the existing +full-ID-bound snapshot. An absent source on a live, consenting root may use the +handoff UID for a bidstream-enabled auction and stage fill-absent-only persistence +after send. Recheck absence, consent, provider expiry and the original handoff +deadline at mutation time. A matching persisted UID, writer and effective expiry +acknowledges the entry. A conflicting existing record wins and +retires the handoff; expired or invalidated records also block replay. No handoff +replaces a stored UID, extends its expiry, changes its writer or creates a root. +This first version does not accelerate refresh of an existing source. + +Snapshot eligibility retains existing KV read consistency. A stale live snapshot +that lacks the source can still deliver the handoff before it observes a newer +record or withdrawal elsewhere. Replay uses generation-checked writes and cannot +replace that newer record or revive its tombstone. Current request consent always +applies; the handoff does not promise instant cross-POP withdrawal propagation. + +Authenticated, eligible ID5 handoff entries use the fixed Lockr provenance from +section 4.5. Do not infer acquisition authority from an unsigned cookie or current +configured ownership. Browser mixed-origin and explicit-field precedence remain +unchanged. Provider `ext` is not added to the cookie. + +A later request carrying an existing valid entry may retry a missed first-issue +write even if its Lockr response contains no new tokens. Retry is shared, bounded +and conditional; it is not another provider call. Remove acknowledged, conflicting, +expired or unauthorized entries on a subsequent response, clearing the cookie +when none remain. After-send success cannot alter a cookie already sent. +Withdrawal or EC rotation invalidates the handoff. No cookie permits delivery +from a missing, unreadable or withdrawn root. + +The original standalone draft proposed an unsigned source/UID array lasting up +to 30 days, with +cookie-first precedence and generic cookie ingestion. Do not restore it as-is. +It omitted expiry needed by its own merge rules and offered no authenticated +full-EC binding or stale-replay contract. The revised handoff restores useful +absent-source availability and retry while keeping lifecycle policy in EC. + +The existing `ts-eids` fallback was largely redundant with normal `/auction` +body submission and did not merge missing sources into a valid body. Restoring +its late-browser-ID ingestion benefit is a separate decision, not part of this +capture handoff. + ## 5. Data flow and freshness ```mermaid @@ -285,36 +358,44 @@ flowchart TD A[Lockr SDK calls TS first-party proxy] --> B[Lockr API returns tokens] B --> C[Lockr capture module normalizes IDs and expiry] C --> D[Shared identity service validates consent and ownership] - D --> E[Original response sent without an EID cookie] + D --> E[Original body sent with optional authenticated handoff cookie] E --> F[Shared service applies authoritative updates after send] F --> G[EC identity graph stores IDs and lifecycle metadata] - G --> H[Later auctions read visible unexpired IDs and reconstruct writer-based provenance] + E --> H[Later request validates handoff against live EC root] + H --> I[Absent-source auction delivery and bounded after-send retry] + I --> G + G --> J[Later requests use KV state and clear acknowledged handoff] ``` -| Event | Persistence | Auction availability | -| ------------------------------- | ------------------------------------------------- | ------------------------------------------------------------------------------ | -| First issue | Authoritative update after send | Once a later KV read sees the record | -| SDK refresh | Replacement or expiry update after send | Once new state is visible | -| Provider expiry | Stored record becomes unusable; no timer required | Omitted when request time reaches known expiry | -| Explicit global revoke | EC cookie expired; existing-key-only tombstone | Later requests stop using the identity; dispatched auctions cannot be recalled | -| Returning visitor, no new token | No capture mutation | Existing usable KV state only | +| Event | Persistence | Auction availability | +| ---------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | +| First issue, source absent | Authoritative update after send; eligible handoff can stage a later fill-absent retry | After cookie installation on a qualifying live-root request, or once KV exposes the record | +| SDK refresh, source already stored | Authoritative replacement or expiry update after send; replay cannot replace existing state | Existing usable KV record until the new state is visible | +| Provider or handoff expiry | No read-triggered cleanup write | Expired handoff is rejected; known-expired stored IDs remain omitted | +| Explicit global revoke | EC and handoff cookies expired; existing-key-only tombstone | Later requests stop using the identity; dispatched auctions cannot be recalled | +| Returning visitor, no new token | No new capture effect; valid pending handoff can independently stage replay | Existing usable KV state, or an eligible absent-source handoff before its original deadline | Inspection can retain 64 KiB of original bytes and 64 KiB of decoded JSON, plus -bounded stream/decoder overhead. There are no additional TS EID cookie bytes -on later requests. - -There is no cookie-based repair of a failed capture write. A later token-bearing -SDK response, browser-body submission or separately authorized partner path can -attempt persistence again under the foundation's writer rules. A returning -`page-view` with no token does not repair a missed capture. Expiry-aware omission -also does not initiate a Lockr fetch. These limits and delayed KV visibility are -accepted for this draft. +bounded stream/decoder overhead. While pending, the handoff adds up to 2,048 +cookie-value bytes plus cookie-name/separator overhead to same-origin requests. +Cleanup requires a later request that observes acknowledgement or ineligibility; +otherwise the fixed deadline ends the handoff. + +Cookie repair is best-effort, limited to absent sources and dependent on another +qualifying request within that deadline. Once the handoff expires, a returning +`page-view` with no token cannot repair a missed capture. Later token-bearing +responses, browser-body submissions or separately authorized partner paths keep +their existing mutation rules. Expiry-aware omission does not initiate a Lockr +fetch. Without shared handoff opt-in, availability still waits for KV visibility. ## 6. Configuration Add `capture_identity` under `[integrations.lockr]`, default `true`, as the -publisher capture kill switch. It does not disable browser-body ingestion, -expiry-aware reads or normal EC consent withdrawal. +publisher capture kill switch. It also disables Lockr handoff delivery and retry, +but does not disable browser-body ingestion, expiry-aware KV reads or normal EC +consent withdrawal. Shared handoff is separately opt-in and defaults off; its +publisher key/configuration contract must be approved before implementation. +Capture alone must not silently enable a new cookie. Capturing a source requires its foundation source policy, for example: @@ -327,12 +408,12 @@ bidstream_enabled = true identity_owner = "lockr" ``` -`identity_owner` is a proposed foundation field, not an existing runtime option. +`identity_owner` is implemented by the foundation and validated at deployment and runtime. Check the publisher registry and disable conflicting legacy authoritative paths before enabling the owner. Missing registration or another selected owner means -Lockr does not persist the source. There is no cookie to forward an unregistered -captured ID. Valid browser-body EIDs can still reach their current auction under -the foundation's existing forwarding rules. +Lockr does not persist the source or issue an eligible handoff entry. The handoff +cannot forward an unregistered captured ID. Valid browser-body EIDs can still +reach their current auction under the foundation's existing forwarding rules. ## 7. Error handling @@ -345,9 +426,15 @@ Use the foundation's redaction, bounds and root safety policy: - Denied token consent, missing EC, missing registration or owner mismatch: no enrichment; the SDK call still succeeds independently. - Missing, unreadable or withdrawn root: no creation or resurrection. -- Post-send persistence failure: redacted log; no immediate retry, durable - queue or change to the already-sent response. -- No tokens: `NoChange`, not invalidation or consent withdrawal. +- Post-send persistence failure: redacted log; an already-issued handoff may + support a later bounded retry until its original deadline. No immediate retry, + durable queue or change to the already-sent response. +- Invalid signature, expired handoff, EC/publisher mismatch, disabled capture or + changed ownership: no handoff delivery or retry; no unsigned fallback. +- Cookie creation or size failure: preserve direct deferred capture and the + upstream body; do not turn cookie failure into an SDK error. +- No tokens: `NoChange`, not invalidation or withdrawal. An independently valid + pending handoff can still be reconciled and replayed. ## 8. Verification @@ -362,8 +449,9 @@ prove that this consumer reaches those paths correctly: same-UID expiry extension; known expiry survives the persisted record. - Original and decoded cap overflow, including compressed expansion, skips Lockr effects while retaining response bytes and encoding headers. -- No-token response produces no token mutation; non-2xx/malformed token - response cannot invalidate existing records. +- No-token observation produces no new direct capture mutation; incoming handoff + replay is independently validated. Non-2xx/malformed token responses cannot + invalidate existing records. - Populated body opt-outs, malformed signals, empty optional fields and cookie consent conflicts exercise shared consent policy without modifying forwarding. - Explicit revoke works with denied consent, chooses the confirmed withdrawal @@ -379,24 +467,43 @@ prove that this consumer reaches those paths correctly: without dropping UIDs or explicit body provenance. - Capture effects execute after Fastly send through the foundation, not a separate Lockr adapter implementation; persisted snapshot handoff is retained. -- No newly emitted TS EID cookie or diagnostic EID header; provider browser - stores and existing publisher credential stripping remain unchanged. +- Capture handoff defaults off; enabling it emits only the bounded authenticated + `ts-eids-s`, never the retired `ts-eids` or diagnostic EID headers. Provider + browser stores and existing publisher credential stripping remain unchanged. +- With capture KV visibility held back, an installed valid handoff reaches a + later live-root auction for an absent registered source. An earlier auction, + root-read failure or existing stored source receives no handoff guarantee. +- No-token requests can repair pending absent-source capture after send; changed + UID/expiry/writer, expired stored records, CAS conflicts and withdrawal cannot + be overwritten by replay. Acknowledgement cleanup waits for a later response. +- Tampered/oversized cookies, expiry, wrong full EC, owner/capture changes and + key rotation exercise foundation rejection and cleanup without SDK failure. Live acceptance on an approved publisher test route: 1. Fresh profile, grant consent and wait for `page-view`. -2. Confirm original response bytes and no TS EID cookie from capture. +2. Confirm unchanged upstream body/encoding and, with handoff explicitly enabled, + authenticated `ts-eids-s` plus private/no-store response handling. Confirm + `ts-eids` and diagnostic EID headers remain absent. 3. Allow for post-send work and KV visibility. Inspect the session's admin EC record for the registered source and lifecycle metadata, without exporting IDs. 4. Trigger a later auction and inspect bidder-facing payload evidence, including the stored Lockr ID5 record's `inserter`, empty `matcher` and `mm=2`. Do not require the browser body to contain a server-added ID or rely on `x-ts-eids`. -5. Reload with no new tokens: unchanged UID and expiry do not cause a write. +5. Observe a later request clearing an acknowledged handoff. Reload with no new + tokens: unchanged UID and expiry do not cause a capture write. Separately + inject a failed first-issue write and confirm an eligible pending handoff + repairs the absent source after a no-token request, without extending its + deadline or overwriting an existing record. 6. Trigger SDK refresh: confirm changed UID or expiry persists after visibility. 7. Exercise known expiry in a controlled fixture and confirm omission, including a stale matching UID resubmitted in a browser auction body. -8. Trigger confirmed global revoke: EC cookie expires, tombstone exists and - later enrichment cannot revive it. +8. Trigger confirmed global revoke: EC and handoff cookies expire, tombstone + exists and later enrichment or handoff replay cannot revive it. +9. On an approved controlled fixture, delay first-issue KV visibility and prove + handoff delivery for an absent source after cookie installation. Verify + rejection after deadline, root-read failure and EC rotation. Measure later + auction coverage; do not infer an unconditional next-auction guarantee. No live identifiers, placement settings or credentials belong in fixtures or committed validation notes. Keep sanitized status/source/count evidence only. @@ -404,19 +511,26 @@ committed validation notes. Keep sanitized status/source/count evidence only. ## 9. Rollout 1. Deliver the foundation, including metadata-preserving writers/readers, - source ownership, browser-body migration and EID cookie/header retirement. + source ownership, browser-body migration and `ts-eids` cookie/header retirement. 2. Confirm Lockr path, provider token shapes, consent and revoke contracts using sanitized fixtures; leave unconfirmed source mappings disabled. 3. Register the capture consumer and its `capture_identity` kill switch. 4. Configure the affected registered sources with `identity_owner = "lockr"` and their confirmed `atype`, after resolving conflicting authoritative paths. 5. Complete the foundation's deployment compatibility gate before activation. -6. Enable capture on an approved test route and run live acceptance, including - visibility delay, expiry extension and withdrawal. +6. Provision the shared handoff's dedicated publisher signing keys through + app-config secret handling. Keep its opt-in off until provider-approved fixture + sizes, rotation and deployed auction/replay/cache-privacy checks pass. +7. Enable capture and separately opt into handoff on an approved test route. + Run live acceptance, including delayed visibility, failed-write repair, + acknowledgement cleanup, expiry extension and withdrawal. Disabling capture stops new observations; it does not make already stored expiry metadata safe for an older UID-only binary. Follow the foundation's -rollback contract. No `ts-eids-s` is introduced at any stage. +rollback contract. Disabling capture also stops Lockr handoff delivery and retry; +turning handoff off alone leaves direct capture and usable KV delivery intact. +Local synthetic tests cover the handoff's codec, policy and proxy/send/replay paths. +They do not prove provider approval or deployed availability. ## 10. Open questions and future work @@ -427,10 +541,15 @@ rollback contract. No `ts-eids-s` is introduced at any stage. 3. Confirm successful revoke response semantics and whether withdrawal is global or source-specific before enabling that outcome. 4. Confirm token ordering when SDK requests overlap. The foundation does not - infer provider issuance order from edge wall clocks. - -Known expiry persistence is now a proposed foundation decision rather than an -unimplemented Lockr-local open question. Arbitrary OpenRTB provenance storage + infer provider issuance order from edge wall clocks. Handoff signatures also + do not establish order; first-version replay never replaces an existing source. +5. Verify publisher-scoped signing-key provisioning and rotation on deployment. + The approved implementation uses a five-minute lifetime and 2,048-byte + complete signed-value cap. Validate provider-approved fixture sizes including + binding, expiry and signature overhead before enabling it. Immediate existing-source refresh through a + cookie requires a later revision-bound design, not cookie-first precedence. + +Known expiry persistence is implemented by the foundation. Arbitrary OpenRTB provenance storage remains outside this delivery; the fixed Lockr ID5 mapping is reconstructed from the persisted writer instead. Active Lockr resolution from `ltid`, direct provider capture and client-side consumers require separate follow-up specs; no current diff --git a/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md b/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md index f122a7070..17f79f413 100644 --- a/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md +++ b/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md @@ -3,6 +3,9 @@ Status: Architecture implemented in the draft change, awaiting code review. See the [implementation plan and verification](../plans/2026-10-09-server-side-identity-foundation-and-lockr.md). Provider activation and metadata-compatible deployment remain unverified. +The default-off `ts-eids-s` capture handoff in section 8.2 is implemented in the +draft change. Its local checks are recorded separately in the implementation +plan; deployed acceptance remains gated. First consumer: [Lockr proxy-side identity capture](2026-10-08-lockr-identity-capture-design.md). @@ -70,7 +73,9 @@ approved provider spec. Invariants: -1. No additional EID cookie. `ts-ec` remains the pointer to an existing EC row. +1. `ts-ec` remains the pointer to an existing EC row. An optional, short-lived + `ts-eids-s` cookie can carry authenticated capture observations across + requests; it is not a replacement identity root or a long-lived EID store. 2. Integration capture and `/auction` enrichment never create or recover a root. Existing eligible publisher-navigation creation and recovery remain. 3. Missing and unreadable rows are distinct. Neither authorizes enrichment. @@ -165,9 +170,12 @@ If the provider needs consent values from a POST body, the integration extracts them from its existing forwarding buffer and supplies normalized signals. It must not consume the body twice or let a second parser modify forwarding. -Only framework-associated upstream responses can produce authoritative capture -updates. An arbitrary browser body or inbound diagnostic header cannot select -this trust level or impersonate a module. +Only framework-associated upstream responses can originate authoritative capture +updates. The shared service can seal a validated observation into the capture +handoff defined in section 8.2. Later requests must authenticate and revalidate +that handoff before staging its restricted retry. An arbitrary browser body, +unsigned cookie or inbound diagnostic header cannot select this trust level or +impersonate a module. ### 5.2 Resolution and refresh @@ -275,6 +283,14 @@ They do not promise chronological provider freshness across overlapping SDK requests. A provider requiring stronger ordering must supply an ordering contract in its integration spec, not rely on edge wall clocks. +Authenticated handoff replay is a separate, fill-absent-only mutation class. +It retains the capture writer and known expiry but cannot replace an existing +source or alter its lifecycle metadata. Recheck source absence, root consent, +provider expiry and the original handoff deadline at mutation time; a conflict +that introduces the source makes the replay a no-op. +Do not send replay through browser enrichment, which discards lifecycle metadata, +or unrestricted authoritative replacement, which could replay an older token. + ### 6.3 Auction reads Resolve only live, consented, bidstream-enabled records whose known expiry has @@ -301,6 +317,12 @@ EID because its top-level fields apply to every UID. Preserve explicit body provenance and the existing UID merge rules. The first delivery preserves existing caps and consent gates across all auction paths. +When the optional handoff is enabled, resolve eligible captured IDs under +section 8.2 before the existing browser merge. Handoff entries require a live, +consenting root and an absent stored source. Stored records retain precedence, +including expired or invalidated records that block replay. Handoff delivery +does not relax current-request browser consent gates. + ## 7. Consent and execution timing The shared service evaluates global EC consent and each module's additional @@ -313,7 +335,7 @@ root consent through conditional-write semantics. flowchart TD A[Request has existing EC and consent context] --> B[Integration captures identity outcomes] B --> C[Shared service stages permitted effects] - C --> D[Normal response finalization handles withdrawal] + C --> D[Finalization handles withdrawal and optional capture handoff cookie] D --> E[Response sent to browser] E --> F[Shared service applies captured updates] F --> G[Persisted snapshot passed to later identity work] @@ -334,15 +356,21 @@ forward, including changed generations and failed states. Capture routes can have `NotRead` snapshots. Their first required lookup occurs after send. A reusable snapshot avoids that lookup, but unusable generations, stale misses and CAS conflicts can require refresh reads. No zero-read claim is -made. No-token responses queue no token mutation. +made. No-token observations queue no new direct capture mutation. A valid +incoming handoff can independently stage restricted replay under section 8.2. Best-effort after-send work adds execution time, not Fastly response latency. There is no durable queue, guaranteed retry or first-auction availability. +The optional handoff allows another qualifying request to attempt persistence +within a fixed deadline, without holding the capture response for a KV write. +Replayed capture persistence also runs after send on Fastly, before legacy pull. Other adapters use the shared service only where EC KV support exists. Document before-send behavior where no post-send execution hook exists; do not claim persistence on unsupported adapters. -## 8. Browser-body migration and EID cookie retirement +## 8. Browser-body migration and capture handoff + +### 8.1 Retire the redundant browser EID cookie Use validated `/auction` body EIDs both for current-request forwarding and for registered-source browser enrichment. Expose the existing EID-to-update helper @@ -352,14 +380,22 @@ finalization instead of performing a second unconditional lookup. Remove the `ts-eids` writer and server readers only after body ingestion is verified. Final behavior: -- `/auction` gets current-request client EIDs from the body, with no EID-cookie - fallback. Unknown sources remain usable only in that current auction. -- Publisher and SPA page-bid auctions read registered IDs from KV. +- `/auction` gets current-request client EIDs from the body, with no `ts-eids` + fallback. Unknown browser sources remain usable only in that current auction. +- Publisher and SPA page-bid auctions read registered IDs from KV. All auction + paths may also use eligible `ts-eids-s` observations under section 8.2. - The separate `sharedId` cookie retains its existing ingestion behavior; combine browser inputs into one finalization mutation attempt. - Orphan recovery no longer seeds IDs from a `ts-eids` cookie. - Late browser IDs need another auction submission for persistence. No new - page-exit beacon or cookie fallback is introduced. + page-exit beacon or browser EID cookie fallback is introduced. `ts-eids-s` + contains validated server capture only; it does not collect late browser IDs. + +Normal `/auction` callers already submit available browser EIDs in the body. +The old `ts-eids` reader used body-or-cookie fallback, not missing-source merge. +Its current-auction role was therefore largely redundant. It could also ingest +IDs discovered after request construction on a later navigation. Restoring that +behavior remains a separate benefit decision; this handoff does not restore it. Retire `x-ts-eids` and `x-ts-eids-truncated` emission, checking external diagnostic tooling first. Keep both names on the inbound internal-header strip list. @@ -371,6 +407,131 @@ bundles still write it. Remove the cleanup shim once the transition artifacts have aged out. Do not add unconditional server cookie cleanup that changes cache privacy on every response. +### 8.2 Short-lived authenticated capture handoff + +Reintroduce `ts-eids-s` as an optional transient handoff, not the original +unsigned source/UID array. It has two material benefits for a newly captured +source: auction delivery before capture persistence becomes visible, and a +later persistence attempt after a failed post-send write. It does not duplicate +browser-body intake or replace KV as the identity graph. + +The shared identity service owns encoding, validation, auction resolution, +retry staging and cleanup. Integrations provide observations and approved +source/writer mappings, not private cookie readers or writers. + +#### Envelope and lifetime + +- Use a versioned authenticated envelope with a key identifier, publisher + binding, full EC ID, absolute handoff deadline and a bounded source-keyed + list. Each entry carries the acquisition module, UID, its original absolute + deadline and optional provider expiry in Unix seconds. Do not include raw + consent strings or credentials. +- Protect the complete envelope with a publisher-scoped signing key and a + reviewed encoding and verification scheme. Base64 is encoding, not integrity + protection; `HttpOnly` does not authenticate an incoming cookie. Never fall + back to unsigned acceptance when a key or signature is unavailable. +- Use `Path=/; Secure; HttpOnly; SameSite=Lax` and the configured EC cookie + domain. Default entry lifetime is five minutes, never longer than its known + provider expiry. Set the envelope deadline to the latest retained entry + deadline and browser `Max-Age` to the remaining envelope lifetime. Filter each + entry by its own authenticated deadline; `Max-Age` alone cannot establish expiry. +- Cap the complete encoded value, including authentication overhead, at 2,048 + bytes. Retain the existing UID and source caps. Reject an entry that cannot + fit; never emit a partial envelope. Any multi-source truncation is + deterministic and removes the earliest-expiring entries first. +- Each entry must retain its original absolute deadline when cookies merge or + entries are removed. Reading, retrying, clearing another entry or receiving + `NoChange` must not extend that deadline. An actually new validated capture + can get a new deadline. A matching UID and effective expiry retain the pending + deadline; omission of expiry retains the same UID's known pending expiry. + That pending expiry may only fill an absent source, never alter a stored record. + Enforce both entry and envelope deadlines. + +The shared finalizer can set the cookie from already validated capture without +adding a pre-send KV lookup or write. A `NotRead` snapshot remains permitted at +issuance; the cookie itself does not prove that a live root exists. Handoff +creation failure leaves the original response and normal deferred capture +persistence intact. Responses carrying the cookie remain private and uncacheable. +Upstream body bytes and encoding/length headers remain unchanged, but the response +is no longer header-identical because it adds `Set-Cookie` and cache privacy. + +#### Eligibility, precedence and replay + +On a later request, authenticate within the encoded size cap and revalidate the +full EC ID, publisher, clock, entry expiry, current acquisition consent, module +claims, enabled capture and configured ownership. Use the normal request-local +KV snapshot. Missing, unreadable or withdrawn roots authorize neither handoff +auction delivery nor handoff persistence. No cookie creates or recovers a root. +An unusable clock cannot authorize a time-bounded handoff. + +Eligibility reflects the row observed in this request, not a strongly consistent +or globally current read. A stale live snapshot that still lacks the source can +deliver a handoff while another POP has newer stored state or a withdrawal. +This retains the foundation's existing KV read-consistency limit. Current request +consent still gates delivery; post-send replay checks the authoritative generation +and cannot overwrite the newer record or revive its tombstone. No cross-POP +instant-withdrawal guarantee is made. + +For each registered source: + +| Stored source state | Handoff action | +| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | +| Absent on a live, consenting root | Allow the unexpired captured UID for a bidstream-enabled source and stage a fill-absent-only retry. | +| Expired or invalidated record | Retire the handoff without delivery or retry for that source; the retained record is not an empty slot. | +| Usable record with the same UID, writer and effective expiry | Acknowledge the handoff and remove it; use the stored record. | +| Any other existing record | Use the stored record if usable and retire the conflicting handoff; do not replace the UID, extend expiry or change its writer. | + +Source/UID equality alone cannot acknowledge a handoff with different expiry or +writer metadata. When capture omitted expiry, a stored same-UID known expiry is +retained rather than erased. A matching browser UID does not turn handoff +metadata into browser authority. Provenance reconstruction uses only the +validated acquisition module and its approved source mapping; mixed-origin +browser merges follow section 6.3. + +This conservative first version accelerates absent-source capture only. A refresh +of an existing source still depends on the direct authoritative capture write. +It does not make an older cookie beat a newer record observed in the snapshot. Supporting immediate +replacement or expiry extension from replay requires a separate revision-bound +contract; edge timestamps and signatures do not prove provider ordering. + +The shared service owns retry. A new direct capture observation supersedes any +incoming handoff for the same source; do not replay the older entry alongside it. +Combine remaining eligible handoff entries into at most one replay batch per +qualifying request, using bounded CAS and preserving unrelated sources. On +conflict, refresh and recheck source absence; do not blindly retry replacement. Fastly applies the replay after send, in the shared capture executor +before pull sync. No immediate recursive retry, private integration dispatcher, +provider refetch or durable queue is introduced. + +Remove acknowledged, conflicting, expired or no-longer-authorized entries on +responses that already have a validated handoff to reconcile. Clear the cookie +when none remain. Withdrawal, EC rotation or publisher/owner mismatch invalidate +the affected handoff. Do not emit unconditional cleanup cookies on unrelated +responses. A post-send success cannot clear the cookie in an already-sent response; +cleanup waits for a later request that observes the persisted result. A failed +write can retain the handoff only until its original deadline. + +A later SDK response with no new tokens may therefore repair a previously missed +capture by carrying an existing valid handoff. No new token means no new direct +capture effect; it does not mean no independently validated replay. Retry remains +best-effort and requires another qualifying request before expiry. + +The capture response must install the cookie before a later request can carry it. +An auction starting earlier can still miss the ID. The handoff does not guarantee +the first auction, immediate refresh of an existing source, delivery during a +root-read failure, or repair after its deadline. + +Shared handoff issuance, delivery and retries default off behind explicit +publisher opt-in. The integration capture kill switch also stops handoff delivery +and retry for that module, without making stored KV records unusable. Confirm the +publisher key provisioning and rotation before activation. The implemented +interface is `[ec.identity_handoff]` with `enabled`, `active_key_id` and at most +four `keys`. Each key has a safe identifier and a dedicated app-config secret-store +reference; runtime material must be URL-safe unpadded base64 of 32 random bytes. +Issuance uses the active key and verification accepts retained rotation keys. +The versioned envelope uses domain-separated HMAC-SHA256; the EC passphrase +cannot be reused. Only Fastly requests admitted to its EC/KV finalization path +enable handoff work. Unsupported adapters neither issue nor replay it. + ## 9. Bounded transport and failure policy Capture response inspection has independent 64 KiB limits for original body @@ -439,11 +600,12 @@ Delivery stages: mutation plumbing with managed acquisition inactive. 2. Add common outcomes, source ownership and capture registration. Keep authoritative and browser mutation classes distinct. -3. Add browser-body ingestion, verify it, then retire EID cookie/header paths. +3. Add browser-body ingestion, verify it, then retire `ts-eids` cookie/header paths. 4. Add bounded capture handling and shared Fastly effect execution, using synthetic integration fixtures before enabling Lockr. -5. Implement and enable the separate Lockr consumer after ownership and - provider validation checks pass. +5. Implement the separate Lockr consumer, then the optional authenticated + handoff through the shared service. Enable each only after its ownership, + provider, key-management and safety gates pass. 6. Implement active resolution and refresh only with its later provider spec. ## 12. Verification and open decisions @@ -474,6 +636,20 @@ Foundation tests must cover: - Fastly send-before-capture-write ordering and resulting snapshot handoff to pull sync; no-token and failed-update paths; supported adapter behavior. - Redacted errors and no private data in logs, including provider query URLs. +- Handoff authentication, publisher/full-ID binding, key rotation, malformed and + oversized values, independent entry deadlines and no deadline extension on + replay, merge, cleanup or `NoChange`. +- Handoff auction eligibility across publisher, `/auction` and SPA paths before + capture KV visibility; no delivery for missing/failed/withdrawn roots, denied + consent, expired entries, disabled capture or changed ownership. +- Absent-source-only replay, exact acknowledgement, different-UID and same-UID + changed-expiry conflicts, retained expired records, duplicate requests and CAS + races with acquisition or withdrawal. No stale replacement or metadata loss. +- Send-before-replay-write ordering, combined replay bounds, acknowledgement + cleanup on a later response, no-token request repair and failure retention only + until the original deadline. Cookie failure must not break direct capture. +- `Set-Cookie` cache privacy, unchanged upstream bytes/encoding, deterministic + complete-envelope size limits and no cleanup cookies on unrelated responses. Resolve before implementation activation: @@ -482,9 +658,13 @@ Resolve before implementation activation: 2. Confirm capture ordering expectations when an SDK issues overlapping token requests; no provider sequence is assumed by this foundation. 3. Identify external consumers of the diagnostic EID headers before retirement. +4. Provision dedicated publisher signing keys and verify rotation operationally. + The implemented defaults are five minutes and 2,048 encoded bytes, including + binding, metadata and signature overhead. Validate provider-approved ID5 + fixtures and cross-POP visibility/withdrawal behavior on the approved route. + Synthetic local tests do not establish live acceptance or strong KV reads. -Provider expiry, writer and revision storage are proposed foundation decisions, -not already implemented capabilities. LiveRamp protocol approval, identifier +Provider expiry, writer and revision storage are implemented foundation decisions. LiveRamp protocol approval, identifier intake, refresh response semantics and country-specific TTL remain for its future spec. Arbitrary OpenRTB provenance storage remains outside this first delivery; fixed Lockr ID5 provenance is reconstructed from the persisted writer. diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 5eea58fdb..85a57a272 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -85,6 +85,18 @@ pull_sync_concurrency = 3 # cluster_trust_threshold = 10 # entries with cluster_size <= this are individual users # cluster_recheck_secs = 3600 # re-evaluate cluster_size after this many seconds +# Optional authenticated server-capture handoff. It defaults off. Keep it off +# until capture, consent, ownership, key rotation, deployment compatibility, and +# live acceptance gates pass. The resolved secret must be URL_SAFE_NO_PAD base64 +# for exactly 32 random bytes and must not reuse ec.passphrase. +# [ec.identity_handoff] +# enabled = false +# active_key_id = "handoff-2026-10" +# +# [[ec.identity_handoff.keys]] +# id = "handoff-2026-10" +# secret = "identity_handoff_2026_10" # Key in trusted_server_secrets + # Example partner configuration. Provision referenced keys in # trusted_server_secrets before validating/pushing. # [[ec.partners]] @@ -589,6 +601,11 @@ geo_query_parameters = false # capture_identity = false # require provider approval and old-binary drain # capture_global_withdrawal = false # require confirmed global revoke semantics +# The separate [ec.identity_handoff] opt-in stays off until its synthetic and +# live acceptance gates pass. It is absent-source-only, expires within five +# minutes, and provides only best-effort after-send retry. It does not restore +# the retired browser-written ts-eids cookie. + # DataDome bot protection. Proxies tags.js + signal-collection API first-party. # Kept active but disabled so `ts audit` can flip `enabled` when detected. [integrations.datadome] From 77dc8c74d9c0789be4e541c96903bbe2afde5da5 Mon Sep 17 00:00:00 2001 From: Christian Date: Mon, 12 Oct 2026 23:42:32 -0500 Subject: [PATCH 7/7] Restore browser EIDs and expired capture fallback Keep browser cookie writers, readers and EID diagnostics alongside body ingestion; defer their retirement to a separate PR. Preserve consent, metadata and pending authenticated capture protections. Allow valid authenticated handoff IDs over known-expired stored records for auctions without replacing stored state or enabling replay writes. Verified native and adapter tests, all target Clippy gates, JS tests and build, and formatting checks. --- crates/trusted-server-adapter-axum/src/app.rs | 11 +- .../tests/routes.rs | 27 +- .../src/app.rs | 12 +- .../tests/routes.rs | 19 +- .../trusted-server-adapter-fastly/src/app.rs | 8 +- crates/trusted-server-adapter-spin/src/app.rs | 19 +- .../tests/routes.rs | 12 +- .../src/auction/endpoints.rs | 83 +++- .../src/auction/formats.rs | 78 +++- crates/trusted-server-core/src/constants.rs | 7 +- crates/trusted-server-core/src/ec/admin.rs | 246 ++++++++++-- crates/trusted-server-core/src/ec/eids.rs | 46 +++ crates/trusted-server-core/src/ec/finalize.rs | 163 +++++++- .../src/ec/identity_handoff.rs | 38 +- .../src/ec/identity_handoff_tests.rs | 373 +++++++++++++++++- crates/trusted-server-core/src/ec/mod.rs | 43 +- .../trusted-server-core/src/ec/prebid_eids.rs | 300 +++++++++++++- crates/trusted-server-core/src/publisher.rs | 50 ++- .../lib/src/integrations/prebid/index.ts | 118 +++++- .../test/integrations/prebid/index.test.ts | 83 +++- .../test/prebid-artifact-integration.test.mjs | 8 +- docs/guide/configuration.md | 30 +- docs/guide/integrations/lockr.md | 38 +- docs/guide/integrations/prebid.md | 52 ++- ...rver-side-identity-foundation-and-lockr.md | 44 ++- ...026-10-08-lockr-identity-capture-design.md | 112 ++++-- ...-server-side-identity-foundation-design.md | 129 +++--- trusted-server.example.toml | 6 +- 28 files changed, 1859 insertions(+), 296 deletions(-) diff --git a/crates/trusted-server-adapter-axum/src/app.rs b/crates/trusted-server-adapter-axum/src/app.rs index fea635bcc..caba714d6 100644 --- a/crates/trusted-server-adapter-axum/src/app.rs +++ b/crates/trusted-server-adapter-axum/src/app.rs @@ -16,8 +16,9 @@ use trusted_server_core::auction::{ use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::ec::EcContext; use trusted_server_core::ec::admin::{ - admin_ec_lookup_not_supported, admin_eids_lookup_retired, deny_admin_diagnostic_fallback, + admin_ec_lookup_not_supported, deny_admin_diagnostic_fallback, handle_admin_eids_lookup, }; +use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; use trusted_server_core::proxy::{ @@ -494,9 +495,11 @@ fn named_route_handler( // dev server has no store to read. Ok(admin_ec_lookup_not_supported()) } - // Keep the authenticated route local. Removing it would - // risk forwarding admin credentials to publisher fallback. - NamedRouteHandler::AdminEidsLookup => Ok(admin_eids_lookup_retired()), + NamedRouteHandler::AdminEidsLookup => { + let partner_registry = + PartnerRegistry::from_config(&state.settings.ec.partners)?; + handle_admin_eids_lookup(&partner_registry, &req) + } NamedRouteHandler::LegacyAdminDenied => Ok(legacy_admin_alias_denied()), NamedRouteHandler::Auction => { // Build the geo-aware EC context so the auction consent diff --git a/crates/trusted-server-adapter-axum/tests/routes.rs b/crates/trusted-server-adapter-axum/tests/routes.rs index 0711574e0..9771264f3 100644 --- a/crates/trusted-server-adapter-axum/tests/routes.rs +++ b/crates/trusted-server-adapter-axum/tests/routes.rs @@ -412,7 +412,7 @@ async fn admin_ec_route_without_credentials_returns_401() { } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn retired_admin_eids_route_still_requires_authentication() { +async fn admin_eids_route_still_requires_authentication() { let req = Request::builder() .method("GET") .uri("/_ts/admin/eids") @@ -430,8 +430,8 @@ async fn retired_admin_eids_route_still_requires_authentication() { } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn authenticated_admin_eids_route_returns_local_retirement_denial() { - // Retired routes must not forward admin credentials to publisher fallback. +async fn authenticated_admin_eids_route_returns_diagnostic_locally() { + // The diagnostic is served locally, never via publisher fallback. let mut svc = make_service(); let req = Request::builder() .method("GET") @@ -447,10 +447,13 @@ async fn authenticated_admin_eids_route_returns_local_retirement_denial() { .call(req) .await .expect("should respond"); + assert_eq!(resp.status().as_u16(), 200); assert_eq!( - resp.status().as_u16(), - 410, - "/_ts/admin/eids should deny locally after authentication" + resp.headers() + .get("content-type") + .and_then(|v| v.to_str().ok()) + .map(|v| v.split(';').next().unwrap_or(v)), + Some("application/json") ); assert_eq!( resp.headers() @@ -464,6 +467,18 @@ async fn authenticated_admin_eids_route_returns_local_retirement_denial() { .and_then(|v| v.to_str().ok()), Some("nosniff") ); + let body = axum::body::to_bytes(resp.into_body(), usize::MAX) + .await + .expect("should read admin EIDs response body"); + let body = std::str::from_utf8(&body).expect("should read UTF-8 admin EIDs response"); + assert!( + body.contains("\"cookie_present\":true"), + "should inspect the request cookie locally" + ); + assert!( + body.contains("ts-eids"), + "should include the cookie diagnostic" + ); } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] diff --git a/crates/trusted-server-adapter-cloudflare/src/app.rs b/crates/trusted-server-adapter-cloudflare/src/app.rs index a0907b24b..5860a8f4e 100644 --- a/crates/trusted-server-adapter-cloudflare/src/app.rs +++ b/crates/trusted-server-adapter-cloudflare/src/app.rs @@ -19,9 +19,10 @@ use trusted_server_core::config_payload::CONFIG_BLOB_KEY; use trusted_server_core::config_payload::{DEFAULT_SECRET_STORE_ID, settings_from_config_blob}; use trusted_server_core::ec::EcContext; use trusted_server_core::ec::admin::{ - admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, admin_eids_lookup_retired, - deny_admin_diagnostic_fallback, + admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, + deny_admin_diagnostic_fallback, handle_admin_eids_lookup, }; +use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; use trusted_server_core::platform::RuntimeServices; @@ -642,11 +643,12 @@ fn build_router(state: &Arc) -> RouterService { .get("/_ts/admin/ec/{id}", |_ctx: RequestContext| async { Ok::(admin_ec_lookup_not_supported()) }) - // Keep the retired diagnostic local and behind existing authentication. + // Keep the authenticated diagnostic local; do not route it to publisher fallback. .get( "/_ts/admin/eids", - make_handler(Arc::clone(&state), |_s, _services, _req| async move { - Ok(admin_eids_lookup_retired()) + make_handler(Arc::clone(&state), |s, _services, req| async move { + let partner_registry = PartnerRegistry::from_config(&s.settings.ec.partners)?; + handle_admin_eids_lookup(&partner_registry, &req) }), ) .post( diff --git a/crates/trusted-server-adapter-cloudflare/tests/routes.rs b/crates/trusted-server-adapter-cloudflare/tests/routes.rs index 8ae9e6c4c..c26168693 100644 --- a/crates/trusted-server-adapter-cloudflare/tests/routes.rs +++ b/crates/trusted-server-adapter-cloudflare/tests/routes.rs @@ -370,8 +370,17 @@ async fn admin_ec_route_without_credentials_returns_401() { } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn authenticated_admin_eids_route_returns_local_retirement() { - // Retired diagnostics must not forward authenticated requests to origin. +async fn authenticated_admin_eids_route_returns_diagnostic_locally() { + // The diagnostic is local and must never forward credentials to origin. + let unauthenticated = request_builder() + .method("GET") + .uri("/_ts/admin/eids") + .body(edgezero_core::body::Body::empty()) + .expect("should build unauthenticated admin EIDs request"); + let unauthorized = route(test_router(), unauthenticated).await; + assert_eq!(unauthorized.status().as_u16(), 401); + assert!(unauthorized.headers().contains_key("www-authenticate")); + let req = request_builder() .method("GET") .uri("/_ts/admin/eids") @@ -380,11 +389,7 @@ async fn authenticated_admin_eids_route_returns_local_retirement() { .expect("should build request"); let resp = route(test_router(), req).await; - assert_eq!( - resp.status().as_u16(), - 410, - "should deny the retired EID diagnostic locally" - ); + assert_eq!(resp.status().as_u16(), 200); assert_eq!(resp.headers()["cache-control"], "no-store"); assert_eq!(resp.headers()["x-content-type-options"], "nosniff"); } diff --git a/crates/trusted-server-adapter-fastly/src/app.rs b/crates/trusted-server-adapter-fastly/src/app.rs index 28d784553..2b336b37e 100644 --- a/crates/trusted-server-adapter-fastly/src/app.rs +++ b/crates/trusted-server-adapter-fastly/src/app.rs @@ -24,7 +24,7 @@ //! | POST | `/_ts/admin/keys/deactivate` | [`handle_deactivate_key`] | //! | GET | `/_ts/admin/ec` | [`handle_admin_ec_lookup`] | //! | GET | `/_ts/admin/ec/{id}` | [`handle_admin_ec_lookup`] | -//! | GET | `/_ts/admin/eids` | [`admin_eids_lookup_retired`] | +//! | GET | `/_ts/admin/eids` | [`handle_admin_eids_lookup`] | //! | POST | `/_ts/api/v1/batch-sync` | [`handle_batch_sync`] | //! | GET | `/_ts/api/v1/identify` | [`handle_identify`] | //! | GET | `/_ts/set-tester` | [`handle_set_tester`] | @@ -109,7 +109,7 @@ use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::config_payload::DEFAULT_SECRET_STORE_ID; use trusted_server_core::constants::COOKIE_SHAREDID; use trusted_server_core::ec::admin::{ - admin_eids_lookup_retired, deny_admin_diagnostic_fallback, handle_admin_ec_lookup, + deny_admin_diagnostic_fallback, handle_admin_ec_lookup, handle_admin_eids_lookup, }; use trusted_server_core::ec::batch_sync::handle_batch_sync; use trusted_server_core::ec::consent::ec_consent_withdrawn; @@ -575,7 +575,7 @@ async fn execute_named( let kv = crate::maybe_identity_graph(&state.settings); handle_admin_ec_lookup(kv.as_ref(), ®istry, &req) } - NamedRouteHandler::AdminEidsLookup => Ok(admin_eids_lookup_retired()), + NamedRouteHandler::AdminEidsLookup => handle_admin_eids_lookup(®istry, &req), _ => unreachable!("admin diagnostics should use early dispatch"), }) .unwrap_or_else(|error| http_error(&error)); @@ -2547,7 +2547,7 @@ mod tests { let response = route(&router, request); - assert_eq!(response.status(), StatusCode::GONE); + assert_eq!(response.status(), StatusCode::OK); assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); assert!( response diff --git a/crates/trusted-server-adapter-spin/src/app.rs b/crates/trusted-server-adapter-spin/src/app.rs index c8400bb16..75f35f1bf 100644 --- a/crates/trusted-server-adapter-spin/src/app.rs +++ b/crates/trusted-server-adapter-spin/src/app.rs @@ -21,9 +21,10 @@ use trusted_server_core::cache_policy::EdgeCacheHeader; use trusted_server_core::config_payload::settings_from_config_blob; use trusted_server_core::ec::EcContext; use trusted_server_core::ec::admin::{ - admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, admin_eids_lookup_retired, - deny_admin_diagnostic_fallback, + admin_ec_lookup_not_supported as core_admin_ec_lookup_not_supported, + deny_admin_diagnostic_fallback, handle_admin_eids_lookup, }; +use trusted_server_core::ec::registry::PartnerRegistry; use trusted_server_core::error::{IntoHttpResponse as _, TrustedServerError}; use trusted_server_core::http_util::sanitize_forwarded_headers; use trusted_server_core::integrations::{IntegrationRegistry, ProxyDispatchInput}; @@ -644,9 +645,17 @@ fn build_router(state: &Arc) -> RouterService { Ok::(admin_ec_lookup_not_supported()) }; - // Retain authenticated local denial, never publisher fallback. - let admin_eids_handler = - |_ctx: RequestContext| async { Ok::(admin_eids_lookup_retired()) }; + // Keep the authenticated diagnostic local; do not route it to publisher fallback. + let s = Arc::clone(&state); + let admin_eids_handler = move |ctx: RequestContext| { + let s = Arc::clone(&s); + async move { + let req = ctx.into_request(); + let result = PartnerRegistry::from_config(&s.settings.ec.partners) + .and_then(|registry| handle_admin_eids_lookup(®istry, &req)); + Ok::(result.unwrap_or_else(|error| http_error(&error))) + } + }; // /auction let s = Arc::clone(&state); diff --git a/crates/trusted-server-adapter-spin/tests/routes.rs b/crates/trusted-server-adapter-spin/tests/routes.rs index be9add93c..0031ddbd2 100644 --- a/crates/trusted-server-adapter-spin/tests/routes.rs +++ b/crates/trusted-server-adapter-spin/tests/routes.rs @@ -168,7 +168,7 @@ async fn admin_ec_route_without_credentials_returns_401() { } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn retired_admin_eids_route_still_requires_authentication() { +async fn admin_eids_route_still_requires_authentication() { let req = request_builder() .method("GET") .uri("/_ts/admin/eids") @@ -180,8 +180,8 @@ async fn retired_admin_eids_route_still_requires_authentication() { } #[tokio::test(flavor = "multi_thread", worker_threads = 2)] -async fn authenticated_admin_eids_route_returns_local_retirement_denial() { - // Retired routes must not forward admin credentials to publisher fallback. +async fn authenticated_admin_eids_route_returns_diagnostic_locally() { + // The diagnostic is local and must not forward admin credentials to fallback. let req = request_builder() .method("GET") .uri("/_ts/admin/eids") @@ -191,11 +191,7 @@ async fn authenticated_admin_eids_route_returns_local_retirement_denial() { .expect("should build request"); let resp = route(test_router(), req).await; - assert_eq!( - resp.status().as_u16(), - 410, - "/_ts/admin/eids should deny locally after authentication" - ); + assert_eq!(resp.status().as_u16(), 200); assert_eq!( resp.headers() .get("cache-control") diff --git a/crates/trusted-server-core/src/auction/endpoints.rs b/crates/trusted-server-core/src/auction/endpoints.rs index a3d88278e..81896d49f 100644 --- a/crates/trusted-server-core/src/auction/endpoints.rs +++ b/crates/trusted-server-core/src/auction/endpoints.rs @@ -10,13 +10,15 @@ use serde_json::Value as JsonValue; use crate::auction::formats::AdRequest; use crate::auction::orchestrator::OrchestrationResult; use crate::consent::{consent_allows_server_side_auction, gate_eids_by_consent}; +use crate::constants::COOKIE_TS_EIDS; +use crate::cookies::extract_cookie_value; use crate::ec::EcContext; use crate::ec::EcKvSnapshot; use crate::ec::eids::{resolve_partner_ids, to_eids}; use crate::ec::kv::KvIdentityGraph; use crate::ec::kv_types::MAX_UID_LENGTH; use crate::ec::partner::normalize_partner_source_domain; -use crate::ec::prebid_eids::collect_prebid_eid_updates_from_eids; +use crate::ec::prebid_eids::{collect_prebid_eid_updates_from_eids, parse_prebid_eids_cookie}; use crate::ec::registry::PartnerRegistry; use crate::error::TrustedServerError; use crate::openrtb::{Eid, Uid}; @@ -183,7 +185,10 @@ pub async fn handle_auction( // Keep normalized body updates on the adapter-owned context, not the // response, so handled provider errors cannot lose persistence work. let client_eids = if ec_id.is_some() { - parse_client_auction_eids(body.eids.as_ref()) + resolve_client_auction_eids( + body.eids.as_ref(), + extract_cookie_value(&http_req, COOKIE_TS_EIDS).as_deref(), + ) } else { None }; @@ -301,7 +306,7 @@ pub async fn handle_auction( }; // Hand the loaded row to the request context so response finalization — // which runs on an EC context the adapter owns, after this handler returns - // — ingests body/sharedId updates from this read instead of paying + // — ingests cookie/body/sharedId updates from this read instead of paying // for a second lookup. if !matches!(auction_kv_snapshot, EcKvSnapshot::NotRead) { ec_context.set_kv_snapshot(auction_kv_snapshot.clone()); @@ -490,6 +495,23 @@ pub(crate) fn resolve_and_merge_auction_eids( merge_auction_eids(client_eids, resolved) } +/// Prefers the current body EID set, falling back to the browser-written cookie. +pub(crate) fn resolve_client_auction_eids( + raw: Option<&JsonValue>, + cookie_value: Option<&str>, +) -> Option> { + parse_client_auction_eids(raw).or_else(|| { + let cookie = cookie_value?; + match parse_prebid_eids_cookie(cookie) { + Ok(eids) if !eids.is_empty() => Some(eids), + _ => { + log::trace!("Auction EIDs: empty or invalid ts-eids cookie; dropping"); + None + } + } + }) +} + fn parse_client_auction_eids(raw: Option<&JsonValue>) -> Option> { let Some(JsonValue::Array(entries)) = raw else { return None; @@ -677,7 +699,6 @@ mod tests { use crate::auction::types::{AuctionRequest, AuctionResponse}; use crate::consent::jurisdiction::Jurisdiction; use crate::consent::types::ConsentContext; - use crate::constants::COOKIE_TS_EIDS; use crate::ec::kv_backend::test_support::InMemoryEcKv; use crate::ec::kv_backend::{ EcKvLookup, EcKvStore, EcKvWrite, EcKvWriteMode, EcKvWriteOutcome, @@ -979,7 +1000,7 @@ mod tests { }}); let cookie = BASE64.encode( serde_json::to_vec(&json!([ - {"source":"id5-sync.com", "uids":[{"id":"retired-cookie-id"}]} + {"source":"id5-sync.com", "uids":[{"id":"cookie-id"}]} ])) .expect("should encode legacy cookie"), ); @@ -1012,7 +1033,7 @@ mod tests { ); assert_eq!( writes.lock().expect("should lock writes").len(), - usize::from(case == "live"), + usize::from(matches!(case, "live" | "cookie-only")), "{case}" ); // A Missing point-read snapshot is revalidated once by the @@ -1028,10 +1049,15 @@ mod tests { "{case}: auction must not create or recover roots" ); let stored = graph.get(&ec_id).expect("should inspect row"); - if case == "live" { + if matches!(case, "live" | "cookie-only") { assert_eq!( - stored.expect("live root remains").0.ids["id5-sync.com"].uid, - "body-id" + stored.expect("should retain live root").0.ids["id5-sync.com"].uid, + if case == "live" { + "body-id" + } else { + "cookie-id" + }, + "body or cookie fallback must enrich only existing live roots" ); } else if let Some((entry, _)) = stored { assert!(entry.ids.is_empty(), "{case}: no browser enrichment"); @@ -1621,6 +1647,7 @@ mod tests { (vec!["expired-uid"], false, vec![]), (vec!["expired-uid"], true, vec![]), (vec!["different-uid"], false, vec!["different-uid"]), + (vec!["different-uid"], true, vec!["different-uid"]), ( vec!["expired-uid", "different-uid"], false, @@ -2010,6 +2037,44 @@ mod tests { assert!(parse_client_auction_eids(None).is_none()); } + #[test] + fn restored_cookie_fallback_keeps_body_first_semantics() { + let cookie = BASE64.encode( + serde_json::to_vec(&json!([ + {"source":"ids.example.com", "uids":[{"id":"cookie-id"}]}, + {"source":"extra.example.com", "uids":[{"id":"cookie-only"}]} + ])) + .expect("should encode cookie"), + ); + let body = json!([{ "source":"ids.example.com", "uids":[{"id":"body-id"}] }]); + let from_body = resolve_client_auction_eids(Some(&body), Some(&cookie)) + .expect("should prefer body EIDs"); + assert_eq!( + from_body.len(), + 1, + "valid body must not merge missing cookie sources" + ); + assert_eq!( + from_body[0].uids[0].id, "body-id", + "body must win over cookie" + ); + let fallback = + resolve_client_auction_eids(None, Some(&cookie)).expect("should fall back to cookie"); + assert_eq!( + fallback.len(), + 2, + "cookie-only request must retain its full EID set" + ); + assert_eq!( + fallback[0].uids[0].id, "cookie-id", + "should retain cookie UID" + ); + assert!( + resolve_client_auction_eids(None, Some("invalid-cookie")).is_none(), + "malformed cookie must not become auction input" + ); + } + #[test] fn parse_client_auction_eids_ignores_malformed_entries() { let raw = json!([ diff --git a/crates/trusted-server-core/src/auction/formats.rs b/crates/trusted-server-core/src/auction/formats.rs index c9bb9e8f2..f24472ad2 100644 --- a/crates/trusted-server-core/src/auction/formats.rs +++ b/crates/trusted-server-core/src/auction/formats.rs @@ -15,8 +15,9 @@ use uuid::Uuid; use crate::auction::context::ContextValue; use crate::consent::ConsentContext; -use crate::constants::HEADER_X_TS_EC_CONSENT; +use crate::constants::{HEADER_X_TS_EC_CONSENT, HEADER_X_TS_EIDS, HEADER_X_TS_EIDS_TRUNCATED}; use crate::creative; +use crate::ec::eids::encode_eids_header; use crate::error::TrustedServerError; use crate::geo::GeoInfo; use crate::openrtb::{ @@ -532,6 +533,20 @@ pub(crate) fn convert_to_openrtb_response_with_report( .insert(HEADER_X_TS_EC_CONSENT, HeaderValue::from_static("ok")); } + if let Some(eids) = &auction_request.user.eids { + let (encoded, truncated) = encode_eids_header(eids)?; + let value = + HeaderValue::from_str(&encoded).change_context(TrustedServerError::Auction { + message: "Failed to encode EIDs header value".to_owned(), + })?; + response.headers_mut().insert(HEADER_X_TS_EIDS, value); + if truncated { + response + .headers_mut() + .insert(HEADER_X_TS_EIDS_TRUNCATED, HeaderValue::from_static("true")); + } + } + Ok(OpenRtbResponseConversion { response, delivery }) } @@ -545,7 +560,6 @@ mod tests { use crate::auction::types::{ ApsRendererV1, ApsTagType, AuctionResponse, Bid, BidRenderer, BidStatus, }; - use crate::constants::{HEADER_X_TS_EIDS, HEADER_X_TS_EIDS_TRUNCATED}; use crate::openrtb::{Eid, Uid}; use crate::platform::test_support::noop_services; use crate::test_support::tests::create_test_settings; @@ -796,7 +810,7 @@ mod tests { } #[test] - fn response_keeps_eid_diagnostics_retired_when_eids_present() { + fn response_includes_eid_headers_when_eids_present() { let mut request = make_auction_request(); request.user.eids = Some(vec![Eid { inserter: None, @@ -816,10 +830,19 @@ mod tests { let response = convert_to_openrtb_response(&result, &settings, &request, true) .expect("should build response"); - assert!( - response.headers().get(&HEADER_X_TS_EIDS).is_none(), - "EID diagnostic headers stay retired even when EIDs are present" - ); + let encoded = response + .headers() + .get(&HEADER_X_TS_EIDS) + .expect("should include x-ts-eids when EIDs are present"); + let decoded = base64::Engine::decode( + &base64::engine::general_purpose::STANDARD, + encoded.as_bytes(), + ) + .expect("should decode EID diagnostic header"); + let payload: serde_json::Value = + serde_json::from_slice(&decoded).expect("should parse diagnostic EID JSON"); + assert_eq!(payload[0]["source"], "ssp.com"); + assert_eq!(payload[0]["uids"][0]["id"], "uid-1"); assert_eq!( response .headers() @@ -833,10 +856,49 @@ mod tests { .headers() .get(&HEADER_X_TS_EIDS_TRUNCATED) .is_none(), - "should never emit the retired truncation header" + "should omit the truncation header when all EIDs fit" ); } + #[test] + fn response_marks_truncated_eid_diagnostic_header() { + let mut request = make_auction_request(); + request.user.eids = Some( + (0..64) + .map(|index| Eid { + source: format!("{index}-{}", "s".repeat(120)), + uids: vec![Uid { + id: format!("{}-{index}", "u".repeat(100)), + atype: Some(1), + ext: None, + }], + ..Eid::default() + }) + .collect(), + ); + + let response = + convert_to_openrtb_response(&make_empty_result(), &make_settings(), &request, true) + .expect("should build response with truncated EID diagnostics"); + + assert_eq!( + response.headers().get(&HEADER_X_TS_EIDS_TRUNCATED), + Some(&HeaderValue::from_static("true")), + ); + let encoded = response + .headers() + .get(&HEADER_X_TS_EIDS) + .expect("should retain a bounded EID header"); + let decoded = base64::Engine::decode( + &base64::engine::general_purpose::STANDARD, + encoded.as_bytes(), + ) + .expect("should decode truncated EID header"); + let payload: serde_json::Value = + serde_json::from_slice(&decoded).expect("should parse truncated EID payload"); + assert!(payload.as_array().unwrap().len() < 64); + } + #[test] fn response_sets_consent_header_even_without_eids() { let request = make_auction_request(); diff --git a/crates/trusted-server-core/src/constants.rs b/crates/trusted-server-core/src/constants.rs index 6422bdff8..94b553d7d 100644 --- a/crates/trusted-server-core/src/constants.rs +++ b/crates/trusted-server-core/src/constants.rs @@ -5,18 +5,17 @@ pub const COOKIE_TS_EC: &str = "ts-ec"; pub const COOKIE_TS_EIDS_SERVER: &str = "ts-eids-s"; /// Short-lived signed proof that the current EC row has every pull-partner UID. pub const COOKIE_TS_EC_PULL_COMPLETE: &str = "ts-ec-pull-complete"; -/// Retired EID cookie name, retained only for transition regression fixtures. -/// New servers neither read nor write it. +/// Browser-written EID cookie used for auction fallback and EC ingestion. pub const COOKIE_TS_EIDS: &str = "ts-eids"; pub const COOKIE_TS_TESTER: &str = "ts-tester"; pub const COOKIE_SHAREDID: &str = "sharedId"; pub const HEADER_X_PUB_USER_ID: HeaderName = HeaderName::from_static("x-pub-user-id"); pub const HEADER_X_TS_EC: HeaderName = HeaderName::from_static("x-ts-ec"); -/// Retired diagnostic header. Keep inbound stripping and withdrawal clearing. +/// Diagnostic header carrying consent-gated EIDs; keep inbound stripping and withdrawal clearing. pub const HEADER_X_TS_EIDS: HeaderName = HeaderName::from_static("x-ts-eids"); pub const HEADER_X_TS_EC_CONSENT: HeaderName = HeaderName::from_static("x-ts-ec-consent"); -/// Retired diagnostic header. Never emitted by current servers. +/// Diagnostic header indicating that `x-ts-eids` was truncated. pub const HEADER_X_TS_EIDS_TRUNCATED: HeaderName = HeaderName::from_static("x-ts-eids-truncated"); pub const HEADER_X_TS_ERROR: HeaderName = HeaderName::from_static("x-ts-error"); pub const HEADER_X_TS_JS_ASSET_PROXY: HeaderName = HeaderName::from_static("x-ts-js-asset-proxy"); diff --git a/crates/trusted-server-core/src/ec/admin.rs b/crates/trusted-server-core/src/ec/admin.rs index aad4d4ed6..ab37c56b4 100644 --- a/crates/trusted-server-core/src/ec/admin.rs +++ b/crates/trusted-server-core/src/ec/admin.rs @@ -15,6 +15,7 @@ //! internal detail (raw consent strings, partner UIDs, parse errors). use std::borrow::Cow; +use std::collections::BTreeMap; use http::{HeaderValue, Method, Request, Response, StatusCode, header}; use serde::Serialize; @@ -23,6 +24,8 @@ use serde_json::Value as JsonValue; use edgezero_core::body::Body as EdgeBody; use error_stack::{Report, ResultExt as _}; +use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EIDS}; +use crate::cookies::extract_cookie_value; use crate::error::TrustedServerError; use crate::openrtb::Eid; @@ -32,6 +35,9 @@ use super::kv::KvIdentityGraph; use super::kv_backend::EcKvLookup; use super::kv_types::{KvEntry, KvMetadata}; use super::log_id; +use super::prebid_eids::{ + analyze_prebid_eids_cookie, collect_sharedid_update, dedupe_partner_updates, is_valid_eid_uid, +}; use super::registry::PartnerRegistry; /// Route prefix shared by the cookie-based and explicit-ID lookup routes. @@ -506,14 +512,135 @@ fn build_auction_view(registry: &PartnerRegistry, entry: &KvEntry) -> AuctionEid AuctionEidsView { eids, skipped } } -/// Locally denies the retired EID-cookie diagnostic after adapter authentication. -/// Never fall through to a publisher origin with admin credentials. -#[must_use] -pub fn admin_eids_lookup_retired() -> Response { - json_error( - StatusCode::GONE, - "EID cookie diagnostic retired; use /_ts/admin/ec", - ) +/// Admin EIDs echo payload. +#[derive(Debug, Serialize)] +struct AdminEidsResponse { + /// Whether a `ts-eids` cookie was present on the request. + cookie_present: bool, + /// EIDs parsed from the cookie. Absent when missing or malformed. + #[serde(skip_serializing_if = "Option::is_none")] + eids: Option>, + /// Parse failure detail when the cookie could not be decoded. + #[serde(skip_serializing_if = "Option::is_none")] + parse_error: Option, + /// Whether a `sharedId` cookie was present. + sharedid_present: bool, + /// Number of partners configured in the registry. + partners_configured: usize, + /// Candidate writes inferred from cookies; this is diagnostic-only and does + /// not model consent gating, request-body EIDs, or existing KV values. + ingest: IngestPreview, +} + +#[derive(Debug, Serialize)] +struct IngestPreview { + matched: Vec, + unmatched: Vec, +} + +#[derive(Debug, Serialize)] +struct MatchedPartnerId { + source_domain: String, + uid: String, +} + +#[derive(Debug, Serialize)] +struct DroppedEidSource { + source: String, + reason: DroppedEidReason, +} + +#[derive(Debug, Serialize)] +#[serde(rename_all = "snake_case")] +enum DroppedEidReason { + NoPartner, + NoValidUid, +} + +/// Handles `GET /_ts/admin/eids` without KV access. +/// +/// The authenticated diagnostic echoes the parsed client EIDs and previews +/// cookie-derived updates. Missing and malformed cookies are represented in +/// the JSON response, not as HTTP errors. +/// +/// # Errors +/// +/// Returns a configuration error only if JSON serialization fails. +pub fn handle_admin_eids_lookup( + registry: &PartnerRegistry, + req: &Request, +) -> Result, Report> { + let eids_cookie = extract_cookie_value(req, COOKIE_TS_EIDS); + let sharedid_cookie = extract_cookie_value(req, COOKIE_SHAREDID); + let (eids, parse_error, diagnostic_sources, mut updates) = match &eids_cookie { + None => (None, None, Vec::new(), Vec::new()), + Some(value) => match analyze_prebid_eids_cookie(value, registry) { + Ok(analysis) => ( + Some(analysis.eids), + None, + analysis.diagnostic_sources, + analysis.updates, + ), + Err(error) => ( + None, + Some(format!("failed to parse ts-eids cookie: {error}")), + Vec::new(), + Vec::new(), + ), + }, + }; + + if let Some(value) = &sharedid_cookie + && let Some(update) = collect_sharedid_update(value, registry) + { + updates.push(update); + } + let matched = dedupe_partner_updates(updates) + .into_iter() + .map(|update| MatchedPartnerId { + source_domain: update.partner_id, + uid: update.uid, + }) + .collect(); + + let mut source_has_valid_uid = BTreeMap::new(); + for diagnostic_source in diagnostic_sources { + let has_valid_uid = diagnostic_source + .uids + .iter() + .any(|uid| is_valid_eid_uid(uid)); + source_has_valid_uid + .entry(diagnostic_source.source) + .and_modify(|valid| *valid |= has_valid_uid) + .or_insert(has_valid_uid); + } + let unmatched = source_has_valid_uid + .into_iter() + .filter_map(|(source, has_valid_uid)| { + let reason = if registry.find_by_source_domain(&source).is_none() { + DroppedEidReason::NoPartner + } else if !has_valid_uid { + DroppedEidReason::NoValidUid + } else { + return None; + }; + Some(DroppedEidSource { source, reason }) + }) + .collect(); + + let payload = AdminEidsResponse { + cookie_present: eids_cookie.is_some(), + eids, + parse_error, + sharedid_present: sharedid_cookie.is_some(), + partners_configured: registry.len(), + ingest: IngestPreview { matched, unmatched }, + }; + let body = + serde_json::to_string(&payload).change_context(TrustedServerError::Configuration { + message: "failed to serialize admin EIDs response".to_owned(), + })?; + Ok(json_response(StatusCode::OK, body)) } fn json_error(status: StatusCode, message: &str) -> Response { @@ -535,6 +662,8 @@ fn json_response(status: StatusCode, body: String) -> Response { mod tests { use std::time::Duration; + use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64}; + use crate::ec::kv::TombstoneOutcome; use super::*; @@ -1200,20 +1329,93 @@ mod tests { assert!(result.is_err(), "should propagate KV read failures"); } + fn eids_cookie_for(entries: &serde_json::Value) -> String { + BASE64.encode(entries.to_string()) + } + #[test] - fn retired_eids_diagnostic_denies_locally_without_echoing_cookies() { - for cookie in ["ts-eids=secret; sharedId=secret", "ts-eids=!!!", ""] { - let _req = get_request_with_cookie("/_ts/admin/eids", cookie); - let response = admin_eids_lookup_retired(); - assert_eq!(response.status(), StatusCode::GONE); - assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); - assert_eq!( - response.headers()[header::X_CONTENT_TYPE_OPTIONS], - "nosniff" - ); - let payload = response_json(response); - assert!(payload.get("eids").is_none()); - assert!(!payload.to_string().contains("secret")); - } + fn eids_lookup_without_cookies_returns_empty_payload() { + let response = handle_admin_eids_lookup(&test_registry(), &get_request("/_ts/admin/eids")) + .expect("should handle EIDs diagnostic"); + assert_eq!(response.status(), StatusCode::OK); + assert_eq!(response.headers()[header::CACHE_CONTROL], "no-store"); + assert_eq!( + response.headers()[header::X_CONTENT_TYPE_OPTIONS], + "nosniff" + ); + let json = response_json(response); + assert_eq!(json["cookie_present"], false); + assert_eq!(json["sharedid_present"], false); + assert_eq!(json["partners_configured"], 2); + assert!(json["ingest"]["matched"].as_array().unwrap().is_empty()); + } + + #[test] + fn eids_lookup_parses_cookie_and_previews_ingestion() { + let cookie = eids_cookie_for(&serde_json::json!([ + {"source":"bidstream.example", "uids":[{"id":"uid-configured", "atype":1}]}, + {"source":"unknown.example", "uids":[{"id":"uid-unknown", "atype":1}]} + ])); + let req = get_request_with_cookie("/_ts/admin/eids", &format!("ts-eids={cookie}")); + let response = handle_admin_eids_lookup(&test_registry(), &req) + .expect("should handle EIDs diagnostic"); + let json = response_json(response); + assert_eq!(json["cookie_present"], true); + assert_eq!(json["eids"].as_array().unwrap().len(), 2); + let matched = json["ingest"]["matched"].as_array().unwrap(); + assert_eq!(matched.len(), 1); + assert_eq!(matched[0]["source_domain"], "bidstream.example"); + assert_eq!(matched[0]["uid"], "uid-configured"); + let unmatched = json["ingest"]["unmatched"].as_array().unwrap(); + assert_eq!(unmatched.len(), 1); + assert_eq!(unmatched[0]["source"], "unknown.example"); + assert_eq!(unmatched[0]["reason"], "no_partner"); + } + + #[test] + fn eids_lookup_reports_configured_source_without_valid_uid() { + let cookie = eids_cookie_for(&serde_json::json!([{ + "source":"bidstream.example", + "uids":[{"id":""}, {"id":" "}, {"id":"x".repeat(513)}] + }])); + let req = get_request_with_cookie("/_ts/admin/eids", &format!("ts-eids={cookie}")); + let response = handle_admin_eids_lookup(&test_registry(), &req) + .expect("should handle EIDs diagnostic"); + let json = response_json(response); + let unmatched = json["ingest"]["unmatched"].as_array().unwrap(); + assert_eq!(unmatched.len(), 1); + assert_eq!(unmatched[0]["reason"], "no_valid_uid"); + } + + #[test] + fn eids_lookup_reports_parse_error_without_failing_request() { + let req = get_request_with_cookie("/_ts/admin/eids", "ts-eids=!!!not-base64!!!"); + let response = handle_admin_eids_lookup(&test_registry(), &req) + .expect("malformed cookie should be diagnostic data"); + assert_eq!(response.status(), StatusCode::OK); + let json = response_json(response); + assert_eq!(json["cookie_present"], true); + assert!(json["parse_error"].as_str().unwrap().contains("ts-eids")); + assert!(json.get("eids").is_none()); + assert!(json["ingest"]["matched"].as_array().unwrap().is_empty()); + } + + #[test] + fn eids_lookup_includes_sharedid_match() { + let registry = PartnerRegistry::from_config(&[ + make_test_partner("bidstream.example", true), + make_test_partner("sharedid.org", true), + ]) + .expect("should build sharedId registry"); + let req = get_request_with_cookie("/_ts/admin/eids", "sharedId=shared-uid-123"); + let response = + handle_admin_eids_lookup(®istry, &req).expect("should handle EIDs diagnostic"); + let json = response_json(response); + assert_eq!(json["cookie_present"], false); + assert_eq!(json["sharedid_present"], true); + let matched = json["ingest"]["matched"].as_array().unwrap(); + assert_eq!(matched.len(), 1); + assert_eq!(matched[0]["source_domain"], "sharedid.org"); + assert_eq!(matched[0]["uid"], "shared-uid-123"); } } diff --git a/crates/trusted-server-core/src/ec/eids.rs b/crates/trusted-server-core/src/ec/eids.rs index a1a380200..bcf7a374b 100644 --- a/crates/trusted-server-core/src/ec/eids.rs +++ b/crates/trusted-server-core/src/ec/eids.rs @@ -3,11 +3,18 @@ //! Used by both `/_ts/api/v1/identify` and `/auction` to resolve source-domain //! keyed IDs from KV entries and convert them to `OpenRTB` EID structures. +use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64}; +use error_stack::{Report, ResultExt as _}; + +use crate::error::TrustedServerError; use crate::openrtb::{Eid, Uid}; use super::kv_types::KvEntry; use super::registry::PartnerRegistry; +/// Maximum base64-encoded size of the diagnostic EID response header. +pub const MAX_EIDS_HEADER_BYTES: usize = 4096; + /// A source-domain keyed ID resolved from a KV entry against the partner registry. /// /// Only includes usable records from live entries and bidstream-enabled partners. @@ -84,6 +91,45 @@ pub fn to_eids(resolved: &[ResolvedPartnerId]) -> Vec { .collect() } +/// Encodes the final auction EIDs for diagnostics, trimming tail sources to fit. +/// +/// Returns the encoded value and whether truncation was necessary. This header +/// is diagnostic output, never an identity acquisition input. +/// +/// # Errors +/// +/// Returns an error if JSON serialization fails. +pub fn encode_eids_header(eids: &[Eid]) -> Result<(String, bool), Report> { + let try_encode = |size: usize| -> Result> { + let json = serde_json::to_vec(&eids[..size]).change_context( + TrustedServerError::Configuration { + message: "Failed to serialize eids header payload".to_owned(), + }, + )?; + Ok(BASE64.encode(json)) + }; + let encoded = try_encode(eids.len())?; + if encoded.len() <= MAX_EIDS_HEADER_BYTES { + return Ok((encoded, false)); + } + let mut lo = 0; + let mut hi = eids.len(); + while lo + 1 < hi { + let mid = lo + (hi - lo) / 2; + if try_encode(mid)?.len() <= MAX_EIDS_HEADER_BYTES { + lo = mid; + } else { + hi = mid; + } + } + if lo == 0 && !eids.is_empty() { + log::warn!( + "No EIDs fit the diagnostic header byte limit; emitting an empty truncated header" + ); + } + Ok((try_encode(lo)?, true)) +} + #[cfg(test)] mod tests { use super::*; diff --git a/crates/trusted-server-core/src/ec/finalize.rs b/crates/trusted-server-core/src/ec/finalize.rs index 1e210f02e..e7ab719a6 100644 --- a/crates/trusted-server-core/src/ec/finalize.rs +++ b/crates/trusted-server-core/src/ec/finalize.rs @@ -42,7 +42,7 @@ const EC_RESPONSE_HEADERS: &[&str] = &[ /// immediately and the EC identity-graph KV tombstone is the authoritative /// revocation marker. There is no separate consent KV store to clean up. /// -/// Staged auction body updates and sharedId are combined into one browser mutation. +/// Browser cookie, staged auction body and sharedId updates are combined once. pub fn ec_finalize_response( settings: &Settings, ec_context: &mut EcContext, @@ -85,7 +85,12 @@ pub fn ec_finalize_response( let source = ec_context.eid_sync_source(); let mut collected = source .map(|_| { - collect_browser_eid_updates(body_updates.clone(), sharedid_cookie, registry) + collect_browser_eid_updates( + body_updates.clone(), + ec_context.prebid_eids_cookie(), + sharedid_cookie, + registry, + ) }) .unwrap_or_default(); protect_pending_capture_updates(&mut collected, ec_context, response); @@ -140,7 +145,12 @@ pub fn ec_finalize_response( }; let updates = gate_eid_updates_by_consent( - collect_browser_eid_updates(body_updates, sharedid_cookie, registry), + collect_browser_eid_updates( + body_updates, + ec_context.prebid_eids_cookie(), + sharedid_cookie, + registry, + ), ec_context.consent(), ); sync_eid_cookie_updates(graph, ec_context, &ec_id, &updates, EidSyncSource::NewEc); @@ -1366,6 +1376,151 @@ mod tests { ); } + #[test] + fn finalize_restored_cookie_combines_with_body_and_sharedid_once() { + for source in [ + EidSyncSource::Navigation, + EidSyncSource::PageBids, + EidSyncSource::Auction, + ] { + let settings = create_test_settings(); + let ec_id = sample_ec_id("cook01"); + let graph = KvIdentityGraph::in_memory("restored-cookie"); + let root = KvEntry::new( + &granting_consent(), + None, + current_timestamp(), + &settings.publisher.domain, + ); + graph.create(&ec_id, &root).expect("should seed live root"); + let value = base64::Engine::encode( + &base64::engine::general_purpose::STANDARD, + serde_json::to_vec(&serde_json::json!([ + {"source":"id5-sync.com", "uids":[{"id":"cookie-id5"}]}, + {"source":"ids.example.com", "uids":[{"id":"cookie-only"}]}, + {"source":"sharedid.org", "uids":[{"id":"cookie-shared"}]} + ])) + .expect("should encode browser cookie"), + ); + let request = http::Request::builder() + .header("cookie", format!("ts-ec={ec_id}; ts-eids={value}")) + .body(EdgeBody::empty()) + .expect("should build request"); + let mut context = EcContext::read_from_request( + &settings, + &request, + &crate::platform::test_support::noop_services(), + ) + .expect("should read browser cookie"); + *context.consent_mut() = granting_consent(); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + context.set_eid_sync_source(source); + context.stage_browser_eid_updates(vec![ + PartnerIdUpdate::new("id5-sync.com", "body-id5"), + PartnerIdUpdate::new("sharedid.org", "body-shared"), + ]); + let registry = PartnerRegistry::from_config(&[ + make_partner("id5-sync.com"), + make_partner("ids.example.com"), + make_partner("sharedid.org"), + ]) + .expect("should build registry"); + let mut response = empty_response(); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + ®istry, + Some("direct-shared"), + &mut response, + ); + let stored = graph + .get(&ec_id) + .expect("should read combined write") + .expect("should retain root") + .0; + assert_eq!( + stored.ids["id5-sync.com"].uid, "body-id5", + "body must override browser cookie" + ); + assert_eq!( + stored.ids["ids.example.com"].uid, "cookie-only", + "cookie-only source must still persist" + ); + assert_eq!( + stored.ids["sharedid.org"].uid, "direct-shared", + "direct sharedId must retain final precedence" + ); + } + } + + #[test] + fn restored_cookie_cannot_replace_managed_or_expired_metadata() { + for cookie_uid in ["stored-id", "different-browser-id"] { + let settings = create_test_settings(); + let ec_id = sample_ec_id("cook02"); + let graph = KvIdentityGraph::in_memory("cookie-metadata"); + let mut root = KvEntry::minimal("ids.example.com", "stored-id", current_timestamp()); + root.consent.ok = true; + let record = root + .ids + .get_mut("ids.example.com") + .expect("should get managed record"); + record.expires_at = Some(1); + record.writer = Some("lockr".to_owned()); + record.revision = Some("stored-revision".to_owned()); + let original_record = record.clone(); + graph + .create(&ec_id, &root) + .expect("should seed managed record"); + let value = base64::Engine::encode(&base64::engine::general_purpose::STANDARD, + serde_json::to_vec(&serde_json::json!([ + {"source":"ids.example.com", "uids":[{"id":cookie_uid}], "writer":"forged", "expires_at":9999999999_u64}, + {"source":"browser.example.com", "uids":[{"id":"new-browser-id"}]} + ])).expect("should encode browser cookie")); + let request = http::Request::builder() + .header("cookie", format!("ts-ec={ec_id}; ts-eids={value}")) + .body(EdgeBody::empty()) + .expect("should build request"); + let mut context = EcContext::read_from_request( + &settings, + &request, + &crate::platform::test_support::noop_services(), + ) + .expect("should read cookie"); + *context.consent_mut() = granting_consent(); + context.set_kv_snapshot(graph.load_snapshot(&ec_id)); + context.set_eid_sync_source(EidSyncSource::Navigation); + let registry = PartnerRegistry::from_config(&[ + make_partner("ids.example.com"), + make_partner("browser.example.com"), + ]) + .expect("should build registry"); + let mut response = empty_response(); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + ®istry, + None, + &mut response, + ); + let stored = graph + .get(&ec_id) + .expect("should read store") + .expect("should retain root") + .0; + assert_eq!( + stored.ids["ids.example.com"], original_record, + "browser cookie must preserve managed lifecycle metadata" + ); + assert_eq!( + stored.ids["browser.example.com"].uid, "new-browser-id", + "independent absent source may still fill" + ); + } + } + #[test] fn finalize_navigation_routes_persist_returning_user_eid_updates() { for (source, suffix, cookie_id) in [ @@ -1600,7 +1755,7 @@ mod tests { #[test] fn finalize_persists_every_configured_partner_from_staged_body_eids() { // Regression test for #1184: persistence must use the full auction body, - // without depending on the retired size-capped EID cookie. + // without depending on the size-capped EID cookie. let settings = create_test_settings(); let ec_id = sample_ec_id("cleids1"); let graph = KvIdentityGraph::in_memory("test_store"); diff --git a/crates/trusted-server-core/src/ec/identity_handoff.rs b/crates/trusted-server-core/src/ec/identity_handoff.rs index 77659f55e..2bb71e090 100644 --- a/crates/trusted-server-core/src/ec/identity_handoff.rs +++ b/crates/trusted-server-core/src/ec/identity_handoff.rs @@ -17,6 +17,7 @@ use crate::settings::Settings; use super::eids::{ResolvedPartnerId, to_eids}; use super::identity::IdentityEffects; use super::identity_handoff_codec::{self as codec, HANDOFF_TTL_SECS, HandoffEntry}; +use super::kv_types::KvPartnerId; use super::partner::normalize_partner_source_domain; use super::registry::PartnerRegistry; use super::{EcContext, EcKvSnapshot, checked_current_timestamp}; @@ -127,7 +128,22 @@ fn live(entry: &HandoffEntry, now: u64) -> bool { now < entry.deadline && entry.expires_at.is_none_or(|expiry| now < expiry) } -/// Resolves handoff IDs only for absent sources in the current live root. +/// Allows auction fallback for absent or known-expired IDs, not empty records. +/// +/// A matching UID needs a refreshed known expiry; omission cannot revive it. +fn stored_source_allows_handoff( + record: Option<&KvPartnerId>, + entry: &HandoffEntry, + now: u64, +) -> bool { + record.is_none_or(|record| { + !record.uid.is_empty() + && record.expires_at.is_some_and(|expiry| now >= expiry) + && (record.uid != entry.uid || entry.expires_at.is_some_and(|expiry| now < expiry)) + }) +} + +/// Resolves handoff IDs for absent or expired sources in the current live root. pub(crate) fn resolve_eids( context: &EcContext, snapshot: &EcKvSnapshot, @@ -155,7 +171,7 @@ pub(crate) fn resolve_eids( (live(entry, now) && partner.bidstream_enabled && partner.identity_owner.as_deref() == Some(entry.module.as_str()) - && !root.ids.contains_key(&entry.source)) + && stored_source_allows_handoff(root.ids.get(&entry.source), entry, now)) .then(|| ResolvedPartnerId { source_domain: entry.source.clone(), uid: entry.uid.clone(), @@ -233,7 +249,7 @@ pub(crate) fn finalize_response( if !partner_owned { continue; } - match context.kv_snapshot() { + let source_absent = match context.kv_snapshot() { EcKvSnapshot::Present { .. } => { let Some(root) = context .kv_snapshot() @@ -242,11 +258,11 @@ pub(crate) fn finalize_response( else { continue; }; - // Both acknowledgement and conflict retire the handoff. An - // expired record is still an existing record, never a hole. - if root.ids.contains_key(&entry.source) { + if !stored_source_allows_handoff(root.ids.get(&entry.source), entry, now) { continue; } + // An expired record permits auction fallback, not persistence replay. + !root.ids.contains_key(&entry.source) } EcKvSnapshot::Missing { .. } => continue, EcKvSnapshot::Failed { .. } => { @@ -255,10 +271,12 @@ pub(crate) fn finalize_response( pending.insert(entry.source.clone(), entry.clone()); continue; } - EcKvSnapshot::NotRead => {} - } + EcKvSnapshot::NotRead => true, + }; pending.insert(entry.source.clone(), entry.clone()); - if context.eid_sync_source().is_some() || context.identity_handoff.proxy_replay_allowed + if source_absent + && (context.eid_sync_source().is_some() + || context.identity_handoff.proxy_replay_allowed) { replay.push(entry.clone()); } @@ -282,7 +300,7 @@ pub(crate) fn finalize_response( else { continue; }; - if root.ids.contains_key(&entry.source) { + if !stored_source_allows_handoff(root.ids.get(&entry.source), &entry, now) { continue; } } diff --git a/crates/trusted-server-core/src/ec/identity_handoff_tests.rs b/crates/trusted-server-core/src/ec/identity_handoff_tests.rs index da5d9d4f2..a569a1d1a 100644 --- a/crates/trusted-server-core/src/ec/identity_handoff_tests.rs +++ b/crates/trusted-server-core/src/ec/identity_handoff_tests.rs @@ -4,7 +4,7 @@ use std::sync::Arc; use base64::{Engine as _, engine::general_purpose::URL_SAFE_NO_PAD}; use serde_json::json; -use crate::auction::endpoints::resolve_auction_eids; +use crate::auction::endpoints::{resolve_and_merge_auction_eids, resolve_auction_eids}; use crate::consent::jurisdiction::Jurisdiction; use crate::ec::EidSyncSource; use crate::ec::finalize::ec_finalize_response; @@ -317,6 +317,71 @@ fn pending_capture_delivers_before_visibility_then_replays_and_acknowledges() { assert!(response.extensions().get::().is_none()); } +#[test] +fn restored_browser_cookie_cannot_strip_authenticated_pending_capture_metadata() { + let settings = settings(); + let partners = partners(&settings); + let graph = graph(); + let pending = entry(); + let signed = cookie(&settings, vec![pending.clone()]); + let browser = base64::engine::general_purpose::STANDARD.encode( + serde_json::to_vec(&json!([{ "source": SOURCE, "uids": [{ "id": UID }] }])) + .expect("should encode browser cookie"), + ); + let request = http::Request::builder() + .uri("https://publisher.example/") + .header( + header::COOKIE, + format!("ts-ec={}; ts-eids-s={signed}; ts-eids={browser}", id()), + ) + .body(EdgeBody::empty()) + .expect("should build request with both cookies"); + let mut context = EcContext::read_from_request(&settings, &request, &noop_services()) + .expect("should read both cookies"); + context.consent_mut().jurisdiction = Jurisdiction::NonRegulated; + context.set_identity_handoff_supported(true); + context.set_eid_sync_source(EidSyncSource::Navigation); + context.set_kv_snapshot(graph.load_snapshot(&id())); + let mut response = response(); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + assert!( + !graph + .get(&id()) + .expect("should inspect before send") + .expect("should retain root") + .0 + .ids + .contains_key(SOURCE), + "matching unsigned cookie must not persist an unknown-expiry browser record before trusted replay" + ); + send(&mut response, &mut context, &graph, &partners); + let stored = graph + .get(&id()) + .expect("should inspect after replay") + .expect("should retain root") + .0; + assert_eq!( + stored.ids[SOURCE].uid, UID, + "should persist captured UID after send" + ); + assert_eq!( + stored.ids[SOURCE].writer.as_deref(), + Some("lockr"), + "should retain authenticated writer" + ); + assert_eq!( + stored.ids[SOURCE].expires_at, pending.expires_at, + "should retain authenticated provider expiry" + ); +} + #[test] fn matching_capture_retains_pending_deadline_and_missing_expiry_without_overriding_kv() { let settings = settings(); @@ -429,13 +494,315 @@ fn matching_capture_retains_pending_deadline_and_missing_expiry_without_overridi } #[test] -fn existing_records_including_expired_and_same_uid_refresh_block_replay() { +fn expired_record_uses_live_handoff_without_replay_or_deadline_renewal() { + let settings = settings(); + let partners = partners(&settings); + for stored_uid in [UID, "expired-different-uid"] { + let graph = graph_with_record(stored_uid, Some(1)); + let before = graph.get(&id()).expect("should read expired record"); + let value = cookie(&settings, vec![entry()]); + let mut context = context(&settings, Some(&value)); + context.set_kv_snapshot(graph.load_snapshot(&id())); + context.set_eid_sync_source(EidSyncSource::Auction); + let client_eids = Some(to_eids(&[ResolvedPartnerId { + source_domain: SOURCE.to_owned(), + uid: stored_uid.to_owned(), + writer: None, + openrtb_atype: 1, + }])); + let eids = resolve_and_merge_auction_eids( + client_eids, + context.kv_snapshot(), + Some(&partners), + &context, + ) + .expect("should supply authenticated capture instead of expired browser input"); + assert_eq!(eids.len(), 1, "should emit only the valid capture source"); + assert_eq!(eids[0].uids.len(), 1, "should omit expired browser input"); + assert_eq!(eids[0].uids[0].id, UID, "should use the live captured UID"); + assert_eq!( + eids[0].inserter.as_deref(), + Some("Lockr-For-Publishers"), + "should preserve authenticated capture provenance" + ); + context.stage_browser_eid_updates(vec![PartnerIdUpdate::new(SOURCE, UID)]); + let mut response = response(); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + let returned = handoff_header(&response) + .expect("should retain live fallback over an expired record") + .strip_prefix("ts-eids-s=") + .expect("should get cookie value") + .split(';') + .next() + .expect("should get token"); + assert_eq!( + returned, value, + "should retain the original signed deadline" + ); + assert!( + response.extensions().get::().is_none(), + "an expired source must not stage persistence replay" + ); + send(&mut response, &mut context, &graph, &partners); + assert_eq!( + graph.get(&id()).expect("should read after send"), + before, + "auction fallback must not replace stored UID, expiry, writer or revision" + ); + } +} + +#[test] +fn capture_issues_handoff_over_expired_record_then_acknowledges_direct_write() { + let settings = settings(); + let partners = partners(&settings); + for stored_uid in [UID, "expired-different-uid"] { + let graph = graph_with_record(stored_uid, Some(1)); + let before = graph.get(&id()).expect("should read expired record"); + let mut context = context(&settings, None); + context.set_kv_snapshot(graph.load_snapshot(&id())); + let mut response = response(); + response + .extensions_mut() + .insert(captured_effect(&settings, &context, &partners)); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + let value = handoff_header(&response) + .expect("should issue capture fallback over expired stored state") + .strip_prefix("ts-eids-s=") + .expect("should get cookie value") + .split(';') + .next() + .expect("should get token") + .to_owned(); + let mut later = self::context(&settings, Some(&value)); + later.set_kv_snapshot(graph.load_snapshot(&id())); + let eids = resolve_auction_eids(later.kv_snapshot(), Some(&partners), &later, Some(now())) + .expect("should resolve before capture persistence"); + assert_eq!( + eids.len(), + 1, + "should deliver capture before write visibility" + ); + assert_eq!(eids[0].uids[0].id, UID, "should deliver the captured UID"); + assert_eq!( + graph.get(&id()).expect("should read before send"), + before, + "should not persist capture before send" + ); + send(&mut response, &mut context, &graph, &partners); + later.set_kv_snapshot(graph.load_snapshot(&id())); + assert!( + resolve_eids(&later, later.kv_snapshot(), &partners, Some(now())).is_empty(), + "usable stored capture must take precedence over the handoff" + ); + let stored = graph + .get(&id()) + .expect("should read direct capture") + .expect("should find root") + .0; + assert_eq!(stored.ids[SOURCE].uid, UID, "should persist direct capture"); + assert!( + stored.ids[SOURCE].is_usable(Some(now())), + "direct capture should persist the refreshed expiry" + ); + let mut acknowledged = self::response(); + ec_finalize_response( + &settings, + &mut later, + Some(&graph), + &partners, + None, + &mut acknowledged, + ); + assert!( + handoff_header(&acknowledged) + .expect("should clear persisted capture") + .contains("Max-Age=0"), + "should clear the acknowledged handoff" + ); + } +} + +#[test] +fn expired_source_fallback_checks_expiry_boundary_and_current_policy() { + let settings = settings(); + let partners = partners(&settings); + let timestamp = now(); + let graph = graph_with_record("expired-different-uid", Some(timestamp)); + for provider_expiry in [None, Some(timestamp + 30)] { + let mut pending = entry(); + pending.expires_at = provider_expiry; + pending.deadline = timestamp + 30; + let value = cookie(&settings, vec![pending]); + let mut context = context(&settings, Some(&value)); + context.set_kv_snapshot(graph.load_snapshot(&id())); + assert_eq!( + resolve_eids(&context, context.kv_snapshot(), &partners, Some(timestamp)).len(), + 1, + "stored expiry is exclusive, including at the exact boundary" + ); + assert!( + resolve_eids( + &context, + context.kv_snapshot(), + &partners, + Some(timestamp + 30) + ) + .is_empty(), + "provider expiry and handoff deadline remain exclusive" + ); + assert!( + resolve_eids(&context, context.kv_snapshot(), &partners, None).is_empty(), + "an unknown clock must not authorize fallback" + ); + context.consent_mut().gpc = true; + assert!( + resolve_eids(&context, context.kv_snapshot(), &partners, Some(timestamp)).is_empty(), + "current request denial must block expired-source fallback" + ); + context.consent_mut().gpc = false; + let mut blocked = settings.clone(); + blocked.ec.partners[0].bidstream_enabled = false; + assert!( + resolve_eids( + &context, + context.kv_snapshot(), + &self::partners(&blocked), + Some(timestamp) + ) + .is_empty(), + "bidstream-disabled sources must not use fallback" + ); + blocked + .integrations + .insert_config( + "lockr", + &json!({"app_id":"synthetic", "capture_identity":false}), + ) + .expect("should disable capture"); + let mut disabled = self::context(&blocked, Some(&value)); + disabled.set_kv_snapshot(graph.load_snapshot(&id())); + assert!( + resolve_eids( + &disabled, + disabled.kv_snapshot(), + &partners, + Some(timestamp) + ) + .is_empty(), + "disabled capture must not authorize fallback" + ); + } +} + +#[test] +fn same_uid_without_refreshed_expiry_cannot_bypass_known_expiry() { + let settings = settings(); + let partners = partners(&settings); + let graph = graph_with_record(UID, Some(1)); + let before = graph.get(&id()).expect("should read expired record"); + let mut pending = entry(); + pending.expires_at = None; + let value = cookie(&settings, vec![pending]); + let mut context = context(&settings, Some(&value)); + context.set_kv_snapshot(graph.load_snapshot(&id())); + assert!( + resolve_eids(&context, context.kv_snapshot(), &partners, Some(now())).is_empty(), + "missing expiry must not revive a known-expired matching UID" + ); + let mut response = response(); + ec_finalize_response( + &settings, + &mut context, + Some(&graph), + &partners, + None, + &mut response, + ); + assert!( + handoff_header(&response) + .expect("should clear unproven refresh") + .contains("Max-Age=0"), + "same-UID fallback needs a known refreshed expiry" + ); + let mut fresh = self::context(&settings, None); + fresh.set_kv_snapshot(graph.load_snapshot(&id())); + let effects = stage_capture( + "lockr", + &[SOURCE], + IdentityObservation { + outcomes: vec![IdentityOutcome::Issued { + source: SOURCE.to_owned(), + uid: UID.to_owned(), + expires_at: None, + }], + consent: IdentityConsentSignals::default(), + }, + &fresh, + &partners, + &settings, + ) + .expect("should seal capture without expiry"); + let mut response = self::response(); + response.extensions_mut().insert(effects); + ec_finalize_response( + &settings, + &mut fresh, + Some(&graph), + &partners, + None, + &mut response, + ); + assert!( + handoff_header(&response).is_none(), + "should not issue an unproven same-UID refresh" + ); + send(&mut response, &mut fresh, &graph, &partners); + assert_eq!( + graph.get(&id()).expect("should read after capture"), + before, + "missing expiry must retain the expired stored record" + ); +} + +fn graph_with_record(uid: &str, expires_at: Option) -> KvIdentityGraph { + let graph = KvIdentityGraph::in_memory("handoff-existing-records"); + let mut root = KvEntry::minimal(SOURCE, uid, now()); + root.consent.ok = true; + let record = root.ids.get_mut(SOURCE).expect("should get fixture record"); + record.expires_at = expires_at; + record.writer = Some("lockr".to_owned()); + record.revision = Some("stored-revision".to_owned()); + graph + .create(&id(), &root) + .expect("should seed existing record"); + graph +} + +#[test] +fn usable_or_empty_records_block_handoff_and_all_existing_records_block_replay() { let settings = settings(); let partners = partners(&settings); for (uid, expiry) in [ ("newer-uid", Some(now() + 900)), (UID, Some(now() + 900)), - (UID, Some(1)), + (UID, None), + ("", Some(1)), + ("", None), ] { let graph = KvIdentityGraph::in_memory("handoff-existing-records"); let mut root = KvEntry::minimal(SOURCE, uid, now()); diff --git a/crates/trusted-server-core/src/ec/mod.rs b/crates/trusted-server-core/src/ec/mod.rs index abb9dc489..f551de717 100644 --- a/crates/trusted-server-core/src/ec/mod.rs +++ b/crates/trusted-server-core/src/ec/mod.rs @@ -76,12 +76,15 @@ use error_stack::Report; use http::Request; use crate::consent::{self as consent_mod, ConsentContext, ConsentPipelineInput}; -use crate::constants::{COOKIE_TS_EC, COOKIE_TS_EC_PULL_COMPLETE, COOKIE_TS_EIDS_SERVER}; +use crate::constants::{ + COOKIE_TS_EC, COOKIE_TS_EC_PULL_COMPLETE, COOKIE_TS_EIDS, COOKIE_TS_EIDS_SERVER, +}; use crate::cookies::handle_request_cookies; use crate::ec::cookies::ec_id_has_only_allowed_chars; use crate::error::TrustedServerError; use crate::geo::GeoInfo; use crate::platform::RuntimeServices; +use crate::redacted::Redacted; use crate::settings::Settings; use device::DeviceSignals; @@ -278,6 +281,8 @@ pub struct EcContext { pull_sync_marker: PullSyncMarkerState, /// Allowed returning-user EID persistence source, assigned only after request filters pass. eid_sync_source: Option, + /// Browser-written EID cookie retained for response-time ingestion, never logging. + prebid_eids_cookie: Option>, /// Normalized browser body updates retained even when auction dispatch fails. browser_eid_updates: Vec, /// Authenticated capture observations, bound to the incoming full EC ID. @@ -350,6 +355,11 @@ impl EcContext { consent.jurisdiction, ); + let prebid_eids_cookie = parsed + .jar + .as_ref() + .and_then(|jar| jar.get(COOKIE_TS_EIDS)) + .map(|cookie| Redacted::new(cookie.value().to_owned())); let identity_handoff = identity_handoff::HandoffState::from_cookie( settings, ec_value.as_deref(), @@ -361,6 +371,7 @@ impl EcContext { ); Ok(Self { identity_handoff, + prebid_eids_cookie, ec_value, cookie_ec_value: parsed.cookie_ec, ec_was_present, @@ -569,6 +580,13 @@ impl EcContext { self.eid_sync_source = Some(source); } + /// Returns the browser-written cookie for decoding and consent-gated use. + pub(crate) fn prebid_eids_cookie(&self) -> Option<&str> { + self.prebid_eids_cookie + .as_ref() + .map(|cookie| cookie.expose().as_str()) + } + /// Stages registered browser body updates before dispatch. No root is created. pub(crate) fn stage_browser_eid_updates(&mut self, updates: Vec) { self.browser_eid_updates = updates; @@ -667,6 +685,7 @@ impl EcContext { #[must_use] pub fn new_for_test(ec_value: Option, consent: ConsentContext) -> Self { Self { + prebid_eids_cookie: None, identity_handoff: identity_handoff::HandoffState::default(), browser_eid_updates: Vec::new(), ec_was_present: ec_value.is_some(), @@ -693,6 +712,7 @@ impl EcContext { client_ip: Option, ) -> Self { Self { + prebid_eids_cookie: None, identity_handoff: identity_handoff::HandoffState::default(), browser_eid_updates: Vec::new(), ec_was_present: ec_value.is_some(), @@ -722,6 +742,7 @@ impl EcContext { consent: ConsentContext, ) -> Self { Self { + prebid_eids_cookie: None, identity_handoff: identity_handoff::HandoffState::default(), browser_eid_updates: Vec::new(), ec_value, @@ -990,6 +1011,26 @@ mod tests { assert!(!ec.ec_generated(), "should not mark as generated"); } + #[test] + fn read_from_request_retains_browser_eid_cookie_without_debug_disclosure() { + let settings = create_test_settings(); + let ec_id = valid_ec_id("a", "Eids01"); + let value = "synthetic-encoded-cookie-content"; + let cookies = format!("ts-ec={ec_id}; ts-eids={value}"); + let req = create_test_request(&[("cookie", &cookies)]); + let context = EcContext::read_from_request(&settings, &req, &noop_services()) + .expect("should read browser cookie"); + assert_eq!( + context.prebid_eids_cookie(), + Some(value), + "should retain request-local cookie for ingestion" + ); + assert!( + !format!("{context:?}").contains(value), + "should redact browser EID cookie from debug output" + ); + } + #[test] fn read_from_request_with_cookie_ec() { let settings = create_test_settings(); diff --git a/crates/trusted-server-core/src/ec/prebid_eids.rs b/crates/trusted-server-core/src/ec/prebid_eids.rs index 8fe71c882..d01fc18e5 100644 --- a/crates/trusted-server-core/src/ec/prebid_eids.rs +++ b/crates/trusted-server-core/src/ec/prebid_eids.rs @@ -1,7 +1,12 @@ //! Browser EID normalization for existing-root enrichment. //! -//! Auction bodies supply current-request EIDs. Only registered sources produce -//! UID-only storage updates; provenance and lifecycle metadata are never stored. +//! Auction bodies and the browser-written `ts-eids` cookie supply current-request +//! EIDs. Only registered sources produce UID-only storage updates; browser input +//! cannot confer acquisition authority or lifecycle metadata. + +use base64::{Engine as _, engine::general_purpose::STANDARD as BASE64}; +use serde::Deserialize; +use serde_json::Value as JsonValue; use crate::openrtb::{Eid, Uid}; @@ -9,6 +14,182 @@ use super::kv::PartnerIdUpdate; use super::kv_types::MAX_UID_LENGTH; use super::registry::PartnerRegistry; +/// Maximum raw `ts-eids` cookie size accepted before base64 decode. +const MAX_EIDS_COOKIE_BYTES: usize = 8 * 1024; + +#[derive(Deserialize)] +struct LegacyCookieEid { + source: String, + id: String, + atype: i32, +} + +#[derive(Deserialize)] +struct StructuredCookieEid { + source: String, + #[serde(default)] + uids: Vec, + #[serde(default)] + inserter: Option, + #[serde(default)] + matcher: Option, + #[serde(default)] + mm: Option, +} + +#[derive(Deserialize)] +struct StructuredCookieUid { + id: String, + #[serde(default)] + atype: Option, + #[serde(default)] + ext: Option, +} + +enum DecodedCookieEids { + Legacy(Vec), + Structured(Vec), +} + +pub(crate) struct DiagnosticEidSource { + pub(crate) source: String, + pub(crate) uids: Vec, +} + +pub(crate) struct PrebidEidAnalysis { + pub(crate) eids: Vec, + pub(crate) diagnostic_sources: Vec, + pub(crate) updates: Vec, +} + +/// Parses a browser `ts-eids` cookie into current-request [`Eid`] entries. +/// +/// Accepts structured `OpenRTB` entries and legacy flattened entries. This is +/// browser input, not an authenticated server-capture handoff. +/// +/// # Errors +/// +/// Returns an error for oversized input, invalid base64, or unsupported JSON. +pub fn parse_prebid_eids_cookie(cookie_value: &str) -> Result, String> { + decode_prebid_eids_cookie(cookie_value).map(DecodedCookieEids::into_openrtb) +} + +fn decode_prebid_eids_cookie(cookie_value: &str) -> Result { + if cookie_value.len() > MAX_EIDS_COOKIE_BYTES { + return Err(format!( + "ts-eids cookie too large ({} bytes)", + cookie_value.len() + )); + } + let bytes = BASE64 + .decode(cookie_value) + .map_err(|error| format!("base64 decode failed: {error}"))?; + if let Ok(entries) = serde_json::from_slice::>(&bytes) { + return Ok(DecodedCookieEids::Legacy(entries)); + } + let entries = serde_json::from_slice::>(&bytes) + .map_err(|error| format!("JSON parse failed: {error}"))?; + Ok(DecodedCookieEids::Structured(entries)) +} + +impl DecodedCookieEids { + fn diagnostic_sources(&self) -> Vec { + match self { + Self::Legacy(entries) => entries + .iter() + .filter(|entry| !entry.source.is_empty()) + .map(|entry| DiagnosticEidSource { + source: entry.source.clone(), + uids: vec![entry.id.clone()], + }) + .collect(), + Self::Structured(entries) => entries + .iter() + .filter(|entry| !entry.source.is_empty()) + .map(|entry| DiagnosticEidSource { + source: entry.source.clone(), + uids: entry.uids.iter().map(|uid| uid.id.clone()).collect(), + }) + .collect(), + } + } + + fn into_openrtb(self) -> Vec { + match self { + Self::Legacy(entries) => entries + .into_iter() + .filter(|entry| !entry.source.is_empty() && !entry.id.is_empty()) + .map(|entry| Eid { + source: entry.source, + uids: vec![Uid { + id: entry.id, + atype: (entry.atype >= 0).then_some(entry.atype), + ext: None, + }], + ..Eid::default() + }) + .collect(), + Self::Structured(entries) => entries + .into_iter() + .filter_map(|entry| { + if entry.source.is_empty() { + return None; + } + let uids = entry + .uids + .into_iter() + .filter_map(|uid| { + if uid.id.is_empty() { + return None; + } + Some(Uid { + id: uid.id, + atype: uid.atype.filter(|atype| *atype >= 0), + ext: uid.ext.filter(JsonValue::is_object), + }) + }) + .collect::>(); + (!uids.is_empty()).then_some(Eid { + source: entry.source, + uids, + inserter: entry.inserter, + matcher: entry.matcher, + mm: entry.mm, + }) + }) + .collect(), + } + } +} + +pub(crate) fn analyze_prebid_eids_cookie( + cookie_value: &str, + registry: &PartnerRegistry, +) -> Result { + let decoded = decode_prebid_eids_cookie(cookie_value)?; + let diagnostic_sources = decoded.diagnostic_sources(); + let eids = decoded.into_openrtb(); + let updates = collect_prebid_eid_updates_from_eids(&eids, registry); + Ok(PrebidEidAnalysis { + eids, + diagnostic_sources, + updates, + }) +} + +fn collect_prebid_eid_updates( + cookie_value: &str, + registry: &PartnerRegistry, +) -> Vec { + match parse_prebid_eids_cookie(cookie_value) { + Ok(eids) => collect_prebid_eid_updates_from_eids(&eids, registry), + Err(_) => { + log::trace!("Prebid EIDs: failed to decode ts-eids cookie; dropping"); + Vec::new() + } + } +} + /// Collects one valid UID per registered source without KV I/O. pub(crate) fn collect_prebid_eid_updates_from_eids( eids: &[Eid], @@ -25,19 +206,25 @@ pub(crate) fn collect_prebid_eid_updates_from_eids( dedupe_partner_updates(updates) } -/// Combines staged body updates with the separate sharedId cookie once. -/// The direct sharedId cookie retains its legacy precedence over body input. +/// Combines cookie, staged body and sharedId updates into one browser mutation. +/// +/// Body updates override `ts-eids`; direct sharedId retains last precedence. pub(crate) fn collect_browser_eid_updates( - mut updates: Vec, + updates: Vec, + eids_cookie: Option<&str>, sharedid_cookie: Option<&str>, registry: &PartnerRegistry, ) -> Vec { + let mut combined = eids_cookie + .map(|cookie| collect_prebid_eid_updates(cookie, registry)) + .unwrap_or_default(); + combined.extend(updates); if let Some(cookie) = sharedid_cookie && let Some(update) = collect_sharedid_update(cookie, registry) { - updates.push(update); + combined.push(update); } - dedupe_partner_updates(updates) + dedupe_partner_updates(combined) } pub(crate) fn dedupe_partner_updates(updates: Vec) -> Vec { @@ -96,6 +283,103 @@ mod tests { PartnerRegistry::from_config(&partners).expect("should build registry") } + fn encoded(value: &JsonValue) -> String { + base64::Engine::encode( + &BASE64, + serde_json::to_vec(value).expect("should encode fixture"), + ) + } + + #[test] + fn cookie_parser_retains_structured_and_legacy_browser_eids() { + let legacy = parse_prebid_eids_cookie(&encoded(&json!([ + {"source":"ids.example.com", "id":"legacy", "atype":571187} + ]))) + .expect("should parse legacy cookie"); + assert_eq!( + legacy[0].uids[0].atype, + Some(571187), + "should retain vendor-specific atype" + ); + let structured = parse_prebid_eids_cookie(&encoded(&json!([ + {"source":"ids.example.com", "inserter":"browser", "matcher":"", "mm":0, + "uids":[{"id":"opaque-envelope", "atype":1, "ext":{"linkType":1}}]} + ]))) + .expect("should parse structured cookie"); + assert_eq!( + structured[0].uids[0].id, "opaque-envelope", + "should preserve opaque IDs" + ); + assert_eq!( + structured[0].uids[0].ext, + Some(json!({"linkType":1})), + "should preserve current-request provider metadata" + ); + assert_eq!( + structured[0].matcher.as_deref(), + Some(""), + "should retain supplied empty matcher" + ); + assert_eq!( + structured[0].mm, + Some(0), + "should retain supplied zero match method" + ); + } + + #[test] + fn cookie_parser_is_bounded_and_sanitizes_uid_metadata() { + for invalid in [ + "not-base64".to_owned(), + encoded(&json!({"not":"array"})), + "x".repeat(MAX_EIDS_COOKIE_BYTES + 1), + ] { + assert!( + parse_prebid_eids_cookie(&invalid).is_err(), + "should reject malformed or oversized cookie" + ); + } + let eids = parse_prebid_eids_cookie(&encoded(&json!([ + {"source":"ids.example.com", "uids":[{"id":"", "atype":1}, {"id":"valid", "atype":-1, "ext":[]}], "writer":"lockr", "revision":"forged"} + ]))).expect("should parse tolerant browser shape"); + assert_eq!(eids[0].uids.len(), 1, "should omit empty UID"); + assert_eq!(eids[0].uids[0].atype, None, "should omit negative atype"); + assert_eq!(eids[0].uids[0].ext, None, "should omit non-object ext"); + } + + #[test] + fn cookie_body_and_sharedid_updates_preserve_legacy_precedence() { + let value = encoded(&json!([ + {"source":"id5-sync.com", "uids":[{"id":"cookie-id5"}]}, + {"source":"sharedid.org", "uids":[{"id":"cookie-shared"}]}, + {"source":"unknown.example.com", "uids":[{"id":"unregistered"}]} + ])); + assert_eq!( + collect_browser_eid_updates( + vec![ + PartnerIdUpdate::new("id5-sync.com", "body-id5"), + PartnerIdUpdate::new("sharedid.org", "body-shared") + ], + Some(&value), + Some("direct-shared"), + ®istry() + ), + vec![ + PartnerIdUpdate::new("id5-sync.com", "body-id5"), + PartnerIdUpdate::new("sharedid.org", "direct-shared") + ], + "body should beat ts-eids, and sharedId should beat both" + ); + assert_eq!( + collect_browser_eid_updates(Vec::new(), Some(&value), None, ®istry()), + vec![ + PartnerIdUpdate::new("id5-sync.com", "cookie-id5"), + PartnerIdUpdate::new("sharedid.org", "cookie-shared") + ], + "cookie-only routes should still ingest registered sources" + ); + } + #[test] fn body_updates_are_registered_uid_only_and_skip_invalid_candidates() { let eids: Vec = serde_json::from_value(json!([ @@ -117,7 +401,7 @@ mod tests { PartnerIdUpdate::new("id5-sync.com", "id5"), ]; assert_eq!( - collect_browser_eid_updates(updates, Some(" cookie-shared "), ®istry()), + collect_browser_eid_updates(updates, None, Some(" cookie-shared "), ®istry()), vec![ PartnerIdUpdate::new("id5-sync.com", "id5"), PartnerIdUpdate::new("sharedid.org", "cookie-shared") diff --git a/crates/trusted-server-core/src/publisher.rs b/crates/trusted-server-core/src/publisher.rs index f82184b4e..939bd3959 100644 --- a/crates/trusted-server-core/src/publisher.rs +++ b/crates/trusted-server-core/src/publisher.rs @@ -27,6 +27,7 @@ use std::time::{Duration, Instant, SystemTime}; use brotli::Decompressor; use brotli::enc::BrotliEncoderParams; use brotli::enc::writer::CompressorWriter; +use cookie::CookieJar; use edgezero_core::body::Body as EdgeBody; use error_stack::{Report, ResultExt}; use flate2::read::ZlibDecoder; @@ -35,7 +36,7 @@ use futures::StreamExt as _; use http::{HeaderValue, Method, Request, Response, StatusCode, Uri, header}; use sha2::Digest as _; -use crate::auction::endpoints::resolve_and_merge_auction_eids; +use crate::auction::endpoints::{resolve_and_merge_auction_eids, resolve_client_auction_eids}; use crate::auction::formats::sanitize_publisher_page_url; use crate::auction::orchestrator::{ AuctionOrchestrator, DispatchAuctionOutcome, DispatchedAuction, ERROR_TYPE_ALL, @@ -53,7 +54,7 @@ use crate::cache_policy::{ CachePolicy, EdgeCacheHeader, cache_control_headers_are_private_or_no_store, }; use crate::consent::{consent_allows_server_side_auction, gate_eids_by_consent}; -use crate::constants::{COOKIE_SHAREDID, HEADER_X_COMPRESS_HINT}; +use crate::constants::{COOKIE_SHAREDID, COOKIE_TS_EIDS, HEADER_X_COMPRESS_HINT}; use crate::cookies::handle_request_cookies; use crate::cookies::template_cache_policy::{TemplateCookieDecision, evaluate_cookie_policy}; use crate::creative_opportunities::{ @@ -4809,7 +4810,7 @@ pub async fn handle_publisher_request( let mut origin_request = Some(req); let eid_cookie_may_need_persistence = cookie_jar .as_ref() - .is_some_and(|jar| jar.get(COOKIE_SHAREDID).is_some()); + .is_some_and(|jar| jar.get(COOKIE_TS_EIDS).is_some() || jar.get(COOKIE_SHAREDID).is_some()); // This decision also gates the concurrent origin send below. A marker skip // cannot currently delay the origin behind an auction because marker // validation requires a non-empty registry, and every such auction sets @@ -4929,6 +4930,7 @@ pub async fn handle_publisher_request( apply_auction_eids_and_device( &mut auction_request, &AuctionEidTargeting { + cookie_jar: cookie_jar.as_ref(), kv_snapshot: ec_context.kv_snapshot(), partner_registry: auction.registry, ec_context, @@ -5506,6 +5508,7 @@ pub(crate) struct MatchedSlotsContext<'a> { /// Borrowed inputs for [`apply_auction_eids_and_device`], bundled to keep the /// helper within the project's 7-argument cap. struct AuctionEidTargeting<'a> { + cookie_jar: Option<&'a CookieJar>, kv_snapshot: &'a crate::ec::EcKvSnapshot, partner_registry: Option<&'a PartnerRegistry>, ec_context: &'a EcContext, @@ -5525,8 +5528,20 @@ fn apply_auction_eids_and_device( auction_request: &mut AuctionRequest, targeting: &AuctionEidTargeting<'_>, ) { + let client_eids = + if targeting.ec_context.ec_allowed() && targeting.ec_context.ec_value().is_some() { + resolve_client_auction_eids( + None, + targeting + .cookie_jar + .and_then(|jar| jar.get(COOKIE_TS_EIDS)) + .map(cookie::Cookie::value), + ) + } else { + None + }; let merged_eids = resolve_and_merge_auction_eids( - None, + client_eids, targeting.kv_snapshot, targeting.partner_registry, targeting.ec_context, @@ -6861,8 +6876,8 @@ fn normalize_page_bids_path(raw: &str) -> String { /// /// `kv` enriches the bid request with server-side EIDs from the EC identity /// graph. Only the Fastly adapter has a KV identity store, so Axum, Cloudflare, -/// and Spin pass `None` and cannot resolve server-side EIDs. Retired EID cookies -/// are ignored on every adapter. +/// and Spin pass `None`; consented browser cookie EIDs remain available on +/// every adapter through the same expiry-aware merge. /// /// # Errors /// @@ -6982,6 +6997,7 @@ pub async fn handle_page_bids( .map(str::to_owned); let consent_context = ec_context.consent().clone(); let geo = ec_context.geo_info().cloned(); + let cookie_jar = handle_request_cookies(&req)?; // Same fail-closed jurisdiction-aware gate the publisher navigation path // uses — relies on the adapter's geo-aware EC context. @@ -7055,7 +7071,7 @@ pub async fn handle_page_bids( }; // Hand the loaded row to the request context so response // finalization — which runs on an EC context the adapter owns, - // after this handler returns — ingests sharedId updates + // after this handler returns — ingests ts-eids/sharedId updates // from this read instead of paying for a second lookup. if !matches!(page_bids_kv_snapshot, crate::ec::EcKvSnapshot::NotRead) { ec_context.set_kv_snapshot(page_bids_kv_snapshot.clone()); @@ -7071,6 +7087,7 @@ pub async fn handle_page_bids( apply_auction_eids_and_device( &mut auction_request, &AuctionEidTargeting { + cookie_jar: cookie_jar.as_ref(), kv_snapshot: &page_bids_kv_snapshot, partner_registry: auction.registry, ec_context, @@ -7963,7 +7980,7 @@ mod tests { } #[test] - fn publisher_and_page_bids_ignore_retired_cookie_ids() { + fn publisher_and_page_bids_filter_matching_expired_cookie_ids() { let ec_id = format!("{}.ABC123", "a".repeat(64)); let context = EcContext::new_for_test( Some(ec_id.clone()), @@ -8017,6 +8034,7 @@ mod tests { apply_auction_eids_and_device( &mut request, &AuctionEidTargeting { + cookie_jar: Some(&jar), kv_snapshot: &snapshot, partner_registry: Some(®istry), ec_context: &context, @@ -8033,9 +8051,14 @@ mod tests { .flat_map(|eid| &eid.uids) .map(|uid| uid.id.as_str()) .collect(); - assert!( - uids.is_empty(), - "publisher and SPA must not consume retired EID cookies" + let expected = if uid == "expired-uid" { + Vec::new() + } else { + vec![uid] + }; + assert_eq!( + uids, expected, + "publisher and SPA should forward valid cookie IDs without reviving expired UIDs" ); assert_eq!( snapshot.entry_for(&ec_id), @@ -8071,10 +8094,11 @@ mod tests { } #[tokio::test] - async fn valid_marker_ignores_retired_eid_cookie() { + async fn valid_marker_does_not_skip_browser_eid_cookie_snapshot_lookup() { assert_eq!( run_marker_lookup_probe(MarkerProbe::Valid, true, false).await, - 0 + 1, + "browser cookie enrichment still requires the shared EC snapshot" ); } diff --git a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts index 020d3909f..3032ab668 100644 --- a/crates/trusted-server-js/lib/src/integrations/prebid/index.ts +++ b/crates/trusted-server-js/lib/src/integrations/prebid/index.ts @@ -2001,7 +2001,6 @@ function installApsBidResponseRegistry(): void { } export function installPrebidNpm(config?: Partial): typeof pbjs { - expireRetiredPrebidEidsCookie(); // The prebid integration requires the external Prebid.js bundle // (integrations.prebid.external_bundle_url). When it failed to load (network // error, SRI mismatch) window.pbjs is still the head-injected stub with no @@ -2372,7 +2371,11 @@ export function installPrebidNpm(config?: Partial): typeof pbjs buildRequests(validBidRequests: TrustedServerBidRequest[]): TrustedServerRequest { log.debug('[tsjs-prebid] buildRequests', { count: validBidRequests.length }); const requestScopedBidRequests = [...validBidRequests]; + const hasUserIdApi = typeof pbjs.getUserIdsAsEids === 'function'; const auctionEids = collectAuctionEids(); + if (hasUserIdApi && !auctionEids) { + clearPrebidEidsCookie(); + } const payload = buildAdRequest(validBidRequests, { eids: auctionEids }); return { method: 'POST', @@ -2549,6 +2552,7 @@ export function installPrebidNpm(config?: Partial): typeof pbjs // and persist them as a cookie for backend sync. const originalBidsBack = opts.bidsBackHandler; opts.bidsBackHandler = function (...args: unknown[]) { + syncPrebidEidsCookie(); const registrationId = isSyntheticRefresh ? undefined : registerPendingPublisherBids(publisherAdUnitCodes, args[0], firstImpressionTokens); @@ -2907,14 +2911,114 @@ export function installUserIdModules(): void { } } +// --------------------------------------------------------------------------- +// Prebid EID cookie sync +// --------------------------------------------------------------------------- + +/** Maximum cookie payload size in bytes (leave room for other cookies). */ +const MAX_EID_COOKIE_BYTES = 3072; + +/** Cookie name for persisted Prebid EIDs. */ +const EID_COOKIE_NAME = 'ts-eids'; + +/** Cookie max-age in seconds (1 day). */ +const EID_COOKIE_MAX_AGE = 86400; + +/** Clears any previously persisted Prebid EIDs cookie. */ +function clearPrebidEidsCookie(): void { + document.cookie = `${EID_COOKIE_NAME}=; Path=/; Secure; SameSite=Lax; Max-Age=0`; +} + +/** + * Trims an EID payload so its base64-encoded cookie fits `MAX_EID_COOKIE_BYTES`, + * dropping UIDs then whole sources from the tail. `/auction` requests still + * forward the untrimmed set in the request body (see `buildAdRequest`); this + * cap only bounds what the `ts-eids` cookie carries for routes without a body + * (e.g. `GET /_ts/page-bids`) and for backend ingestion at response finalize. + */ +function fitAuctionEidsToCookie(eids: AuctionEid[]): AuctionEid[] | undefined { + const payload = eids.map((eid) => ({ source: eid.source, uids: [...eid.uids] })); + const droppedSources = new Set(); + const trimmedSources = new Set(); + + while (payload.length > 0) { + const encoded = btoa(JSON.stringify(payload)); + if (encoded.length <= MAX_EID_COOKIE_BYTES) { + logTrimmedEidSources(droppedSources, trimmedSources); + return payload; + } + + const last = payload[payload.length - 1]; + if (last && last.uids.length > 1) { + last.uids = last.uids.slice(0, last.uids.length - 1); + trimmedSources.add(last.source); + continue; + } + + const dropped = payload.pop(); + if (dropped) { + droppedSources.add(dropped.source); + // A source trimmed and then dropped is reported once, as dropped. + trimmedSources.delete(dropped.source); + } + } + + logTrimmedEidSources(droppedSources, trimmedSources); + return undefined; +} + +/** + * Logs which EID sources `fitAuctionEidsToCookie` had to drop or trim UIDs + * from, if any. Debug level: `/auction` sends the untrimmed set in the request + * body, so a trimmed cookie is expected steady state rather than a warning. + */ +function logTrimmedEidSources(droppedSources: Set, trimmedSources: Set): void { + if (droppedSources.size === 0 && trimmedSources.size === 0) { + return; + } + const parts: string[] = []; + if (droppedSources.size > 0) { + parts.push(`dropped sources: ${[...droppedSources].join(', ')}`); + } + if (trimmedSources.size > 0) { + parts.push(`trimmed uids from sources: ${[...trimmedSources].join(', ')}`); + } + log.debug( + `[tsjs-prebid] ts-eids cookie exceeded ${MAX_EID_COOKIE_BYTES} bytes; ${parts.join('; ')}` + ); +} + /** - * Transition-only cleanup for the retired host-only Path=/ EID cookie. - * Remove once bundles that write it have aged past both their deployment cache - * lifetime and the old cookie's one-day max-age. New servers ignore those writes. + * Collects EIDs from Prebid's User ID Module and writes them as a + * base64-encoded OpenRTB-style JSON cookie (`ts-eids`) for backend ingestion + * and auction fallback on later requests. */ -function expireRetiredPrebidEidsCookie(): void { - if (typeof document !== 'undefined') { - document.cookie = 'ts-eids=; Path=/; Secure; SameSite=Lax; Max-Age=0'; +function syncPrebidEidsCookie(): void { + try { + if (typeof pbjs.getUserIdsAsEids !== 'function') { + // Without Prebid EIDs to forward, stale auction fallback IDs must not persist. + clearPrebidEidsCookie(); + return; + } + + const eids = collectAuctionEids(); + if (!eids) { + clearPrebidEidsCookie(); + return; + } + + const payload = fitAuctionEidsToCookie(eids); + if (!payload) { + clearPrebidEidsCookie(); + return; + } + + const encoded = btoa(JSON.stringify(payload)); + document.cookie = `${EID_COOKIE_NAME}=${encoded}; Path=/; Secure; SameSite=Lax; Max-Age=${EID_COOKIE_MAX_AGE}`; + + log.debug(`[tsjs-prebid] synced ${payload.length} EID sources to cookie`); + } catch (err) { + log.warn('[tsjs-prebid] failed to sync EIDs cookie', err); } } diff --git a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts index 1351206c1..81e72aaa2 100644 --- a/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts +++ b/crates/trusted-server-js/lib/test/integrations/prebid/index.test.ts @@ -2543,9 +2543,9 @@ describe('prebid/installPrebidNpm', () => { expect(logged.some((value) => JSON.stringify(value).includes(sentinel))).toBe(false); }); - it('installation expires the retired ts-eids cookie when EIDs are absent', () => { - document.cookie = 'ts-eids=stale-value'; + it('buildRequests clears a stale ts-eids cookie when current Prebid EIDs are absent', () => { const spec = getAdapterSpec(); + document.cookie = 'ts-eids=stale-value'; mockGetUserIdsAsEids.mockReturnValue([]); spec.buildRequests([ @@ -3196,7 +3196,7 @@ describe('prebid/installPrebidNpm', () => { expect(hasTsBidder).toBe(true); }); - it('never writes a ts-eids cookie after bidsBackHandler', () => { + it('syncs a structured ts-eids cookie after bidsBackHandler', () => { mockRequestBids.mockImplementation((opts?: { bidsBackHandler?: () => void }) => { opts?.bidsBackHandler?.(); }); @@ -3215,10 +3215,20 @@ describe('prebid/installPrebidNpm', () => { adUnits: [{ bids: [{ bidder: 'appnexus', params: {} }] }], } as unknown as RequestBidsArg); - expect(document.cookie).not.toMatch(/(?:^|; )ts-eids=/); + const cookieValue = document.cookie.match(/(?:^|; )ts-eids=([^;]+)/)?.[1]; + expect(cookieValue).toBeDefined(); + expect(JSON.parse(atob(cookieValue!))).toEqual([ + { + source: 'sharedid.org', + uids: [ + { id: 'shared_123', atype: 3 }, + { id: 'shared_456', ext: { provider: 'example' } }, + ], + }, + ]); }); - it('does not persist an opaque LiveRamp envelope in a cookie', () => { + it('preserves an opaque LiveRamp envelope in the ts-eids cookie', () => { mockRequestBids.mockImplementation((opts?: { bidsBackHandler?: () => void }) => { opts?.bidsBackHandler?.(); }); @@ -3234,10 +3244,69 @@ describe('prebid/installPrebidNpm', () => { adUnits: [{ bids: [{ bidder: 'appnexus', params: {} }] }], } as unknown as RequestBidsArg); - expect(document.cookie).not.toMatch(/(?:^|; )ts-eids=/); + const cookieValue = document.cookie.match(/(?:^|; )ts-eids=([^;]+)/)?.[1]; + expect(cookieValue).toBeDefined(); + expect(JSON.parse(atob(cookieValue!))).toEqual([ + { + source: 'liveramp.com', + uids: [{ id: 'opaque-test-envelope', atype: 3 }], + }, + ]); + }); + + it('trims an oversized ts-eids cookie without trimming /auction body EIDs', () => { + const debugSpy = vi.spyOn(log, 'debug').mockImplementation(() => {}); + mockRequestBids.mockImplementation((opts?: { bidsBackHandler?: () => void }) => { + opts?.bidsBackHandler?.(); + }); + const longUid = 'x'.repeat(400); + const inputEids = [ + { source: 'id5-sync.com', uids: [{ id: longUid, atype: 1 }] }, + { source: 'liveramp.com', uids: [{ id: longUid, atype: 3 }] }, + { source: 'criteo.com', uids: [{ id: longUid, atype: 1 }] }, + { source: 'uidapi.com', uids: [{ id: longUid, atype: 3 }] }, + { source: 'pubcid.org', uids: [{ id: longUid, atype: 1 }] }, + { source: 'sharedid.org', uids: [{ id: longUid, atype: 1 }] }, + { source: 'dropped.example', uids: [{ id: longUid, atype: 1 }] }, + ]; + mockGetUserIdsAsEids.mockReturnValue(inputEids); + const pbjs = installPrebidNpm(); + const spec = mockRegisterBidAdapter.mock.calls[0][2] as TestAdapterSpec; + + const request = spec.buildRequests([ + { + adUnitCode: 'slot', + bidder: 'trustedServer', + mediaTypes: { banner: { sizes: [[300, 250]] } }, + params: {}, + }, + ]); + expect(JSON.parse(request.data).eids).toEqual(inputEids); + + pbjs.requestBids({ + adUnits: [{ bids: [{ bidder: 'appnexus', params: {} }] }], + } as unknown as RequestBidsArg); + + const cookieValue = document.cookie.match(/(?:^|; )ts-eids=([^;]+)/)?.[1]; + expect(cookieValue).toBeDefined(); + expect(cookieValue!.length).toBeLessThanOrEqual(3072); + const cookieEids = JSON.parse(atob(cookieValue!)) as Array<{ source: string }>; + expect(cookieEids.map((eid) => eid.source)).toEqual([ + 'id5-sync.com', + 'liveramp.com', + 'criteo.com', + 'uidapi.com', + 'pubcid.org', + ]); + const loggedMessage = debugSpy.mock.calls + .map((call) => call.at(-1)) + .find( + (arg): arg is string => typeof arg === 'string' && arg.includes('ts-eids cookie exceeded') + ); + expect(loggedMessage).toContain('dropped.example'); }); - it('expires the retired cookie on installation before any auction', () => { + it('clears ts-eids cookie after bidsBackHandler when no current EIDs remain', () => { document.cookie = `ts-eids=${btoa(JSON.stringify([{ source: 'sharedid.org', uids: [{ id: 'stale' }] }]))}`; mockRequestBids.mockImplementation((opts?: { bidsBackHandler?: () => void }) => { opts?.bidsBackHandler?.(); diff --git a/crates/trusted-server-js/lib/test/prebid-artifact-integration.test.mjs b/crates/trusted-server-js/lib/test/prebid-artifact-integration.test.mjs index 24db647d3..1b28fc710 100644 --- a/crates/trusted-server-js/lib/test/prebid-artifact-integration.test.mjs +++ b/crates/trusted-server-js/lib/test/prebid-artifact-integration.test.mjs @@ -307,10 +307,10 @@ describe('tsjs-prebid production artifacts', () => { expect(analyticsArtifact.bundleCode.length).toBeGreaterThan(200_000); // A value-import of Prebid or a private rendering helper would multiply // the shim size. The bound sits just above the normal compact shim output, - // which is roughly 40 KB: tight enough that material growth has to be - // noticed and re-justified here, and far enough below a multiplication - // that one still fails loudly. The bundle beside it is 200 KB and up. - expect(shimCode.length).toBeLessThan(41_000); + // now roughly 41 KB with the restored bounded EID cookie writer. Allow + // 42 KB so that material growth still needs justification and an accidental + // Prebid import fails loudly. The bundle beside it is 200 KB and up. + expect(shimCode.length).toBeLessThan(42_000); expect(shimCode).toContain('markWinningBidAsUsed'); }); diff --git a/docs/guide/configuration.md b/docs/guide/configuration.md index 553437974..39689be77 100644 --- a/docs/guide/configuration.md +++ b/docs/guide/configuration.md @@ -673,12 +673,14 @@ Provider expiry, writer and opaque revision are optional schema-v1 KV fields. Known-expired IDs are omitted from auctions and `identify` without cleanup writes. Unowned legacy pull remains fill-missing-only even for expired records. Drain metadata-unaware readers and writers before enabling managed capture. -`/auction` body EIDs enrich existing consenting rows conservatively; the retired -`ts-eids` cookie and EID diagnostic response headers are no longer identity -transport. The optional `ts-eids-s` cookie is an authenticated server-capture -handoff only. It does not restore `ts-eids` or collect browser IDs. Inbound -diagnostic headers remain stripped. See the -[Prebid transport migration](./integrations/prebid.md) and +`/auction` body EIDs and the browser-written `ts-eids` cookie enrich existing +consenting rows conservatively. Body EIDs take precedence for auction forwarding; +publisher and SPA auctions retain cookie fallback. EID diagnostic response headers +and authenticated `/_ts/admin/eids` remain available. Their retirement is deferred +to a separate PR; inbound diagnostic headers remain stripped. The optional +`ts-eids-s` cookie is a separate authenticated server-capture handoff only and does +not collect browser IDs. See the +[Prebid identity transport](./integrations/prebid.md) and [Lockr capture gates](./integrations/lockr.md). **Example**: @@ -740,15 +742,19 @@ signs new envelopes; retained keys verify envelopes created before the switch. The cookie value has a fixed five-minute maximum lifetime and a 2,048-byte encoded-value limit. Provider expiry may shorten the lifetime. The handoff is -eligible only for an absent registered source on an existing live, consenting -EC root. Current consent, source ownership, bidstream policy, the integration's -capture switch, and the shared handoff switch all still apply. Existing, -expired, invalidated, or conflicting stored records win. +eligible for an absent or known-expired registered source on an existing live, +consenting EC root. Current consent, source ownership, bidstream policy, the +integration's capture switch, and the shared handoff switch all still apply. +Usable stored IDs win. A valid capture can supply an auction ID over an expired +record without changing KV or renewing its handoff deadline. A matching +known-expired UID requires a known refreshed provider expiry. Empty or invalidated +records still block fallback. Persistence retry is best-effort. A later qualifying request may stage one bounded fill-absent attempt after the response is sent on Fastly. There is no -durable queue, recursive retry, root creation, existing-source refresh, or -first-auction guarantee. Keep rollout and live acceptance gated as described in +durable queue, recursive retry, root creation, existing-source persistence +refresh, or first-auction guarantee. Expired-source auction fallback never stages +a replacement write; persistence replay remains absent-source-only. Keep rollout and live acceptance gated as described in the [Lockr integration guide](./integrations/lockr.md). **Environment Override**: diff --git a/docs/guide/integrations/lockr.md b/docs/guide/integrations/lockr.md index e2805e968..b80e35bd5 100644 --- a/docs/guide/integrations/lockr.md +++ b/docs/guide/integrations/lockr.md @@ -30,9 +30,10 @@ flowchart TD D --> E[Original response sent with optional authenticated handoff] E --> F[Eligible EC record updated after send on Fastly] E --> G[Later live-root request validates handoff] - G --> H[Absent source reaches auction and stages bounded retry] + G --> H[Absent or expired source gets valid capture for auction] + G --> J[Only absent sources stage bounded retry] F --> I[Later auctions read usable IDs from KV] - H --> I + J --> I ``` ## Configuration @@ -185,12 +186,16 @@ and original handoff deadline in an authenticated envelope. `Secure`, `HttpOnly`, and `SameSite=Lax` cookie flags do not replace signature checking. Provider expiry may shorten the deadline, and reads or retries never extend it. -A later request can use an eligible ID5 handoff only when the registered source -is absent from an existing live, consenting EC root. Current Lockr capture, -source ownership, consent, expiry, and bidstream gates are checked again. Any -existing record wins, including an expired, invalidated, or conflicting record. -The handoff never creates or recovers a root, replaces a UID, extends stored -expiry, or accelerates refresh of an existing source. +A later request can use an eligible ID5 handoff when the registered source is +absent or known-expired in an existing live, consenting EC root. Current Lockr +capture, source ownership, consent, expiry, and bidstream gates are checked again. +A usable stored record always wins. An expired record permits auction fallback +from a valid capture, including the same UID with a known refreshed expiry or a +different captured UID. Missing expiry cannot revive a matching known-expired UID. +Empty or invalidated records still block fallback. The cookie keeps +its original deadline and never changes the stored UID, expiry, writer or revision. +Persistence replay remains absent-source-only; existing-source updates require +direct capture. The handoff never creates or recovers a root. Eligibility uses the existing request-local KV snapshot. A stale live read can still miss a newer record or withdrawal at another POP; the handoff does not add @@ -198,16 +203,19 @@ strong reads or instant cross-POP revocation. Current request consent still applies. Replay uses generation-checked writes and cannot overwrite that newer record or revive its tombstone. -The same later request may stage one bounded fill-absent persistence attempt. +If the source is absent, the same later request may stage one bounded fill-absent +persistence attempt. An expired stored record allows auction fallback only, not +persistence replay. Fastly runs it after sending the response. This retry is best-effort and needs a qualifying request before the original deadline. It has no durable queue or recursive provider call. A successful after-send write cannot clear the cookie already sent; a subsequent request reconciles and clears acknowledged entries. -This feature does not restore the retired browser-written `ts-eids` cookie. -`ts-eids-s` contains validated server capture only. It does not ingest late -browser EIDs or restore the old diagnostic headers. Cookie-bearing responses -are private/no-store, and auction-provider Cookie forwarding strips the handoff. +The browser-written `ts-eids` cookie and auction EID diagnostics remain separate +legacy paths; their retirement is deferred to another PR. `ts-eids-s` contains +validated server capture only and does not ingest browser EIDs. Handoff-bearing +responses are private/no-store, and auction-provider Cookie forwarding strips +`ts-eids-s` while retaining legacy cookie handling. Only Fastly's admitted EC/KV requests support issuance and post-send replay; other adapters keep the feature inactive. @@ -215,7 +223,7 @@ other adapters keep the feature inactive. `capture_global_withdrawal` remains `false` until Lockr confirms that a 2xx `revoke-consent` response with `"success": true` means global withdrawal for this EC identity. Only then may operators opt in: the normal finalizer expires `ts-ec` and the existing-key-only tombstone prevents later enrichment. Empty token arrays, network failures and other responses never imply withdrawal. Overlapping SDK calls have no provider sequence contract; CAS follows observation order. -Use synthetic or sanitized source/count/status evidence for an approved test route. Verify unchanged proxy response, delayed KV read-back of UID/source/expiry/writer/revision without exporting values, later auction source presence, same-UID expiry extension, expired omission and confirmed opt-out/revoke. For handoff acceptance, also hold back KV visibility, verify absent-source auction delivery only after cookie installation, inject a failed post-send write, observe later best-effort repair and acknowledgement cleanup, then test expiry, consent denial, root read failure, EC rotation, ownership changes and key rotation. These are rollout gates, not completed acceptance. No live IDs, consent strings, credentials or query URLs belong in test records. +Use synthetic or sanitized source/count/status evidence for an approved test route. Verify unchanged proxy response, delayed KV read-back of UID/source/expiry/writer/revision without exporting values, later auction source presence, same-UID expiry extension, expired omission and confirmed opt-out/revoke. For handoff acceptance, also hold back KV visibility, verify absent-source and expired-source auction delivery only after cookie installation, confirm expired-source fallback leaves KV metadata and the handoff deadline unchanged, inject a failed post-send write, observe later best-effort repair and acknowledgement cleanup, then test expiry, consent denial, root read failure, EC rotation, ownership changes and key rotation. These are rollout gates, not completed acceptance. No live IDs, consent strings, credentials or query URLs belong in test records. ## Best Practices @@ -268,7 +276,7 @@ On an approved synthetic test route, first confirm the SDK request and original ### Response timing -The SDK call still waits for the upstream Lockr API response. Fastly runs eligible KV capture updates after sending that response; later auctions see a UID only after the write is visible. The SDK route uses `cache_ttl_seconds` for its own response. No separate Lockr fetch loop or refresh timeout is configured by capture. +The SDK call still waits for the upstream Lockr API response. Fastly runs eligible KV capture updates after sending that response; later auctions can use a valid installed handoff for an absent or expired source before that write is visible. Usable stored IDs remain preferred. The SDK route uses `cache_ttl_seconds` for its own response. No separate Lockr fetch loop or refresh timeout is configured by capture. ### Optimization Tips diff --git a/docs/guide/integrations/prebid.md b/docs/guide/integrations/prebid.md index ebdd4179d..65e1db0f4 100644 --- a/docs/guide/integrations/prebid.md +++ b/docs/guide/integrations/prebid.md @@ -934,11 +934,15 @@ not passed by this implementation. ## Identity forwarding Prebid.js sends current EIDs from `pbjs.getUserIdsAsEids()` in the `/auction` -body. Trusted Server merges them with usable registered IDs from the current -EC/KV row, applies consent gating, and forwards `user.ext.eids` to Prebid Server. +body. When no usable body EIDs are supplied, Trusted Server falls back to the +browser-written `ts-eids` cookie. It merges the selected client EIDs with usable +registered IDs from the current EC/KV row, applies consent gating, and forwards +`user.ext.eids` to Prebid Server. A valid body takes precedence as a complete set; +missing body sources are not merged from the cookie. Registered body sources are staged before provider dispatch. Response finalization -combines them with the separate `sharedId` cookie in one conservative mutation, +combines `ts-eids`, staged body updates and the separate `sharedId` cookie in one +conservative mutation (body beats `ts-eids`; direct `sharedId` wins last), using the auction's KV snapshot. No bid and handled provider-error responses still finalize these updates. Browser input fills absent sources only. It cannot replace stored IDs, forge lifecycle metadata, create a root, recover a missing root, or @@ -946,36 +950,42 @@ revive a withdrawal tombstone. Unknown sources remain current-request-only. This persistence requires EC KV support. Fastly supports it; adapters without an EC identity graph can forward permitted body EIDs but do not persist them. -Publisher and SPA page-bid auctions use KV IDs only. An ID arriving late needs -another `/auction` body submission; there is no page-exit beacon or cookie repair. +Publisher and SPA page-bid auctions also read permitted `ts-eids` cookie EIDs +through the same expiry-aware merge. IDs discovered after request construction +can reach a later navigation through that cookie. There is no page-exit beacon. ### Identity flow ```mermaid flowchart TD - A[Prebid User ID modules resolve EIDs] --> B[POST auction with current body EIDs] - B --> C[Stage registered UID-only updates on EC context] - C --> D[Read current EC row and merge usable KV EIDs] + A[Prebid resolves current EIDs] --> B[POST auction with untrimmed body EIDs] + A --> C[Bid callback writes bounded ts-eids cookie] + B --> D[Merge body or cookie EIDs with usable KV IDs] + C -->|Later request without body EIDs| D D --> E[Consent-gated OpenRTB request to Prebid Server] E --> F[Bid, no-bid, or handled error response] - F --> G[Finalize body plus sharedId on existing consenting root] + F --> G[Finalize cookie plus body plus sharedId on consenting root] G --> H[Send response] G --> I[Later auctions read stored IDs from KV] ``` -### Cookie and diagnostic retirement +### Browser cookie and diagnostics -New servers ignore `ts-eids`, including writes by cached older bundles. No TS EID -cookie replaces it. The transition JS bundle expires the old host-only `Path=/` -cookie once installed. Remove that cleanup only after writing bundles have aged -past both the old cookie's one-day max-age and the deployment cache lifetime. -The server does not emit unconditional cleanup `Set-Cookie` headers. +The Prebid bid callback writes a one-day, host-only `ts-eids` cookie with +`Path=/; Secure; SameSite=Lax`. Its base64 payload is capped at 3,072 bytes, +trimming UIDs and then sources from the tail deterministically. The auction body +remains untrimmed. Empty or unavailable IDs clear stale cookie state; installation +alone does not expire it. This unsigned cookie is browser input, not authenticated +capture authority, and cannot strip stored or pending capture lifecycle metadata. -`x-ts-eids` and `x-ts-eids-truncated` are no longer emitted and remain stripped -from inbound requests. Check external diagnostic consumers before rollout. -`/_ts/admin/eids` remains authenticated and returns a local `410 Gone` with -`Cache-Control: no-store`, never publisher fallback. Use `/_ts/admin/ec` to inspect -persisted state on adapters with EC KV support. EC and consent headers remain. +Auction responses emit base64 `x-ts-eids` diagnostics capped at 4,096 bytes; +`x-ts-eids-truncated: true` appears only when tail sources were removed. Both +names remain stripped from inbound requests. `/_ts/admin/eids` remains an +authenticated local cookie diagnostic with `Cache-Control: no-store` and +`X-Content-Type-Options: nosniff`, never publisher fallback. Use `/_ts/admin/ec` +to inspect persisted lifecycle state on adapters with EC KV support. EC and +consent headers remain independent. Cookie and diagnostic retirement is deferred +to a separate PR. Deploy metadata-preserving readers and writers everywhere and drain old invocations before activating body writer/revision metadata. Old binaries can erase metadata @@ -991,7 +1001,7 @@ on a new binary is not deployment-drain evidence. - Supplied `inserter`, `matcher`, and integer `mm` remain current-request provenance, including empty `matcher` and zero `mm`; they are not stored in KV - UID `ext` remains current-request-only; existing UID merge precedence is unchanged -- Known-expired KV IDs are omitted, including an identical body UID for that source +- Known-expired KV IDs are omitted, including an identical body or cookie UID for that source - Consent gating is applied to the merged set before forwarding ### What reaches Prebid Server diff --git a/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md b/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md index 8e4ece62b..33e8b5c07 100644 --- a/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md +++ b/docs/superpowers/plans/2026-10-09-server-side-identity-foundation-and-lockr.md @@ -4,13 +4,13 @@ **Specs:** [Foundation](../specs/2026-10-09-server-side-identity-foundation-design.md), [Lockr capture](../specs/2026-10-08-lockr-identity-capture-design.md), [KV snapshot and recovery](../specs/2026-07-10-kv-eid-request-snapshot-ec-recovery-design.md). The [existing LiveRamp browser design](../specs/2026-08-21-liveramp-integration-design.md) does not approve a native resolver. -**Goal:** Persist registered browser auction EIDs and trusted Lockr proxy observations on an existing, consenting EC row, retain expiry and writer metadata through every mutation, and expose only usable records to later auctions. Keep the response intact and send it before capture writes on Fastly. Keep browser-written `ts-eids` retired. The optional authenticated `ts-eids-s` handoff carries bounded pending capture only. No root creation by acquisition or new provider fetch loop. +**Goal:** Persist registered browser auction EIDs and trusted Lockr proxy observations on an existing, consenting EC row, retain expiry and writer metadata through every mutation, and expose only usable records to later auctions. Keep the response intact and send it before capture writes on Fastly. Retain browser-written `ts-eids` and EID diagnostics; retire them in a separate PR. The optional authenticated `ts-eids-s` handoff carries bounded pending capture only. No root creation by acquisition or new provider fetch loop. ```mermaid flowchart TD A[Record compatibility and expiry-aware readers] --> B[Ownership and mutation policy] B --> C[Auction body ingestion] - C --> D[Retire EID cookie and diagnostic headers] + C --> D[Preserve browser cookie and EID diagnostics] B --> E[Bounded capture and Fastly after-send effects] D --> F[Lockr capture consumer] E --> F @@ -62,13 +62,13 @@ cargo test --package trusted-server-cli --target "$HOST_TARGET" - Green: descriptor claims and runtime registry are one source of truth; validate source policy against claims in the shared deploy/runtime validation functions. Shared service validates trusted origin, registered source, owner, UID length/source count, consent and full EC-ID-bound snapshot. Implement the capture-used `Issued`, `NoChange` and approved withdrawal actions; reserve the remaining outcome contracts without an unused active resolver. Test expiry again at mutation time, because a token can expire between capture and post-send persistence. Browser update cannot set authority or lifecycle metadata. Keep `ec/kv.rs` browser one-CAS/one-conflict-read/no-retry policy distinct from authoritative bounded CAS/re-merge. Authoritative replacement preserves unrelated sources and never creates a missing, failed or tombstoned root. Same UID with changed expiry is a write; unchanged UID and lifecycle metadata is not. A stale result tied to a prior revision must be rejected when revision-conditional mutation becomes necessary; do not expose an unused resolver callback merely to test an imagined provider. - Check: filtered `cargo test-fastly` config, registry, push/pull and KV tests; host CLI validation tests. Inspect log assertions for redaction. -### 3. Persist browser body once, then retire cookie/header transport +### 3. Persist browser body once while preserving cookie/header transport **Files:** `crates/trusted-server-core/src/{auction/endpoints.rs,auction/formats.rs,auction/types.rs,openrtb.rs,ec/prebid_eids.rs,ec/finalize.rs,ec/admin.rs,ec/eids.rs,constants.rs,cookies/template_cache_policy.rs,publisher.rs}`, `crates/trusted-server-adapter-fastly/src/{app.rs,main.rs}`, `crates/trusted-server-js/lib/src/integrations/prebid/index.ts` and its existing tests; update operator docs at `docs/guide/integrations/prebid.md` and `docs/guide/configuration.md` for changed transport. Limit other adapters to compile/behavior parity; they currently have no EC KV graph. - Red: registered body EIDs enrich an existing root even on no-bid or handled auction-provider error; unknown sources remain current-request-only; no-token/consent-denied/missing/failed/tombstone roots do not write or recover. Prove one combined browser finalization attempt with `sharedId`, using the auction snapshot rather than a second unconditional KV read. Directly reuse the EID-to-update normalization in `ec/prebid_eids.rs`, not a fake cookie. Stage normalized updates in `EcContext` or a typed response effect before dispatch so `Err` paths retain them through the adapter's error conversion; inspect `app.rs` response-finalization behavior rather than assuming an error response keeps extensions. -- Green: keep normal EC finalization timing and publisher-only orphan recovery, including no seeding from the retired cookie. Extend `Eid` with optional string `inserter`/`matcher` and an OpenRTB-compatible integer `mm`, omitted only when absent. Update both the manual parser in `parse_client_auction_eids` and `merge_auction_eids`, not only serde derives. Explicit body-supplied provenance wins over reconstructed KV provenance; preserve empty `matcher` and supplied zero-valued `mm`. Keep existing UID merge precedence and retain UID `ext` for the current auction. Update affected struct constructors and serialization tests. KV-derived EIDs retain registry `atype` and reconstruct fixed Lockr ID5 provenance only from the persisted `lockr` writer. Keep the mapping in `integrations/lockr.rs`, not a source-only attribution table. Different browser UIDs suppress reconstruction for a mixed-origin EID without changing UID merging. Preserve source/UID caps and consent gates. Filter the matching known-expired body UID before merging. -- Only after body tests pass: remove `/auction` cookie fallback, server cookie ingest, JS writer, header emission and old cookie-dependent diagnostics. If retiring an admin route, keep it locally denied with its existing authentication/no-store behavior; do not remove its route and forward admin credentials to publisher fallback. Apply that protection to the Axum and Spin routes as well as Fastly. Keep `x-ts-eids` and `x-ts-eids-truncated` on the inbound strip list; keep EC/consent headers. Keep one transition JS release that expires the old host-only `Path=/` cookie, with a named removal condition: cached bundles that write it have aged beyond the existing one-day max-age and deployment cache lifetime. New servers ignore old bundles' writes. No unconditional server `Set-Cookie` cleanup. Agree an admin diagnostic replacement/removal and identify external header consumers before this deletion. +- Green: keep normal EC finalization timing and publisher-only orphan recovery, including conservative cookie ingestion. Extend `Eid` with optional string `inserter`/`matcher` and an OpenRTB-compatible integer `mm`, omitted only when absent. Update both the manual parser in `parse_client_auction_eids` and `merge_auction_eids`, not only serde derives. Explicit body-supplied provenance wins over reconstructed KV provenance; preserve empty `matcher` and supplied zero-valued `mm`. Keep existing UID merge precedence and retain UID `ext` for the current auction. Update affected struct constructors and serialization tests. KV-derived EIDs retain registry `atype` and reconstruct fixed Lockr ID5 provenance only from the persisted `lockr` writer. Keep the mapping in `integrations/lockr.rs`, not a source-only attribution table. Different browser UIDs suppress reconstruction for a mixed-origin EID without changing UID merging. Preserve source/UID caps and consent gates. Filter the matching known-expired body UID before merging. +- Retain `/auction` body-or-cookie fallback, publisher/SPA readers, browser cookie ingestion, JS writer, EID header emission and authenticated local admin diagnostics. Store the raw cookie privately/redacted in `EcContext`; combine cookie → staged body → direct `sharedId` into one snapshot/CAS mutation without adapter argument threading. Preserve expiry, metadata, ownership, consent and pending-capture protection. Restore the one-day 3,072-byte cookie cap, deterministic tail UID/source trimming and stale clearing, without install-time cleanup or trimming the body. Restore bounded 4,096-byte EID response headers and conditional truncation marker. Keep inbound stripping, EC/consent headers and authenticated admin namespace/fallback guards. Retirement, diagnostic consumer inventory and transition cleanup belong in a separate PR. - Check: filtered `cargo test-fastly` auction/finalize/admin/publisher tests, JS Prebid tests, `rg -n 'ts-eids|x-ts-eids'` review of remaining intentional compatibility references. Verify before/after request cookies and response cache privacy through Fastly tests. ### 4. Add bounded shared capture and one Fastly post-send executor @@ -95,15 +95,15 @@ cargo test --package trusted-server-cli --target "$HOST_TARGET" - Implemented default-off `[ec.identity_handoff]`, dedicated secret-store signing keys, safe active/rotation IDs and deploy/runtime validation. Runtime keys decode to exactly 32 bytes and cannot reuse EC passphrase material. Disabled provisioned keys still resolve and validate. - Implemented a publisher/full-EC-bound, versioned HMAC-SHA256 envelope. The complete signed value is bounded to 2,048 bytes. Entries retain absolute deadlines of at most five minutes, shortened by provider expiry. Oversized entries are dropped whole; multi-source eviction removes earliest deadlines first. Response Max-Age derives from retained signed entries, not pre-truncation candidates. - Shared finalization issues capture handoff before send without an added KV read, revalidates incoming entries, retires acknowledgements/conflicts and clears empty or rejected cookies only when one was present. Cookie responses are private/no-store, preserve upstream body/encoding and exclude the handoff from auction-provider Cookie forwarding and template cache keys. -- All shared auction callers can deliver an eligible absent-source entry from a loaded, consenting full-ID-bound snapshot. Any observed stored record wins, including expired records. Current consent, capture kill switch, source claims/ownership, clock and expiry still apply. Eligibility retains existing KV read consistency; no strong-read or instant cross-POP withdrawal guarantee is added. -- Browser ingestion cannot strip authenticated pending capture expiry/writer metadata. Matching captures retain the original pending deadline and known expiry when the provider omits it. That pending expiry only fills an absent source and cannot change any stored expiry. A new direct capture supersedes pending replay for its source; existing-source refresh remains direct capture only. +- All shared auction callers can deliver an eligible entry for an absent or known-expired source from a loaded, consenting full-ID-bound snapshot. Usable stored IDs win; empty records block delivery. A valid same-UID refresh or different captured UID can supply the auction over an expired record without changing KV metadata or the original handoff deadline. Issuance and reconciliation retain this fallback; replay remains absent-source-only. Current consent, capture kill switch, source claims/ownership, clock and expiry still apply. Eligibility retains existing KV read consistency; no strong-read or instant cross-POP withdrawal guarantee is added. +- Browser ingestion cannot strip authenticated pending capture expiry/writer metadata. Matching captures retain the original pending deadline and known expiry when the provider omits it. That pending expiry only fills an absent source and cannot change any stored expiry. A new direct capture supersedes pending replay for its source; existing-source persistence refresh remains direct capture only. - At most one eligible replay batch is staged for a browser persistence route or an observed permitted Lockr proxy request, including `NoChange`. Body denial clears the handoff and cannot borrow cookie permission. Fastly enables the feature only with its admitted EC/KV finalization graph; other adapters remain unsupported. - The existing shared executor sends first, then proves the root and applies fill-absent-only CAS before legacy pull work. Conflicts recheck source absence and withdrawal. Replay never creates a root, replaces UIDs/metadata or extends the original deadline. Successful persistence is acknowledged on a later response. No provider refetch, durable queue or unconditional first-auction guarantee. ## Release and rollback 1. Ship optional schema-v1 metadata-aware readers and **every** metadata-preserving writer first, with managed acquisition off. Verify mixed-record fixtures. Obtain deployment propagation and old-invocation drain evidence; old UID-only readers ignore expiry and old writers erase metadata. Confirm operator procedure before the first managed write. -2. Treat browser writer/revision output as metadata activation too; complete the compatibility gate before enabling any path that produces lifecycle metadata. Activate body persistence in a coordinated server release; prove body/no-cookie behavior before the transition JS writer removal. Check diagnostic consumers before header retirement. Roll back the JS transition only with a server that still understands body submission; old cached JS may write the ignored cookie for its remaining lifetime. +2. Treat browser writer/revision output as metadata activation too; complete the compatibility gate before enabling any path that produces lifecycle metadata. Activate body persistence in a coordinated server release while retaining browser cookie and diagnostic compatibility. Verify body/no-cookie persistence, body-over-cookie precedence and lifecycle-safe cookie ingestion. Defer transport retirement, consumer inventory and any cleanup transition to a separate PR. 3. Deploy synthetic capture framework inactive and verify Fastly send ordering. Although the Lockr setting defaults to true, explicitly deploy `capture_identity = false` until approval and compatibility gates pass; ownership validation must still work. On an approved route, add one confirmed Lockr source with owner and `atype` after resolving conflicting legacy writes or pull configuration. Enable capture, measure source-only counts, KV readback and later auction visibility. Canary first; do not claim first-auction guarantees or automatic retry. 4. Safe immediate mitigation after activation is `capture_identity = false` or removing ownership through an **approved migration**; disabling capture alone does not erase stored expiry. Rolling back to a UID-only binary is unsafe until managed writes stop and expiry-aware read/metadata compatibility is explicitly resolved. Do not automatically transfer owner on module failure. DSR deletion and global consent withdrawal remain authoritative over acquired records. 5. At intermediate checkpoints, run only suites affected by the completed milestone. At the final stable implementation checkpoint, run `cargo fmt --all -- --check`, `cargo test-fastly`, `cargo test-fastly-reuse`, `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin`, `./scripts/test-cli.sh`, JS tests/build/format, integration parity and affected target-matched clippy aliases per `AGENTS.md`. Reuse unchanged passing results rather than repeat this matrix at every checkpoint. Confirm CI and approved live-route evidence separately. Core tests are not a substitute for an observed deployed post-send KV write and later auction. Handoff activation additionally requires dedicated publisher key provisioning/rotation and approved delayed-visibility, failure-repair, cleanup and withdrawal checks. Keep `ec.identity_handoff.enabled = false` until those gates pass. @@ -112,7 +112,7 @@ cargo test --package trusted-server-cli --target "$HOST_TARGET" - The foundation requests revision-conditional invalidation tests but defers `invalidated_at`, invalidation storage and active resolver to later work. Does the first capture delivery need an invalidation marker at all? This plan avoids dead state and blocks source-specific revoke until scope is approved. Native LiveRamp and its intake/refresh questions remain deferred in [#1245](https://github.com/IABTechLab/trusted-server/issues/1245). Cross-browser linkage needs a separate approved design; #1245 is not authorization to build it. - Implemented ownership policy: reject owned sources with legacy pull enabled. Keep `api_token` read-only for `identify`; reject owned-source batch writes with 403 before limiter/storage. Ownership is validated through deploy/config-push/runtime/startup paths, even with acquisition disabled. Operator ownership migration and invocation drain still need confirmation. -- Implemented `/_ts/admin/eids` retirement as authenticated local 410/no-store/nosniff on all four adapters; unknown admin subpaths remain locally denied. Cookie readers/writer and diagnostic emission are retired; inbound strips remain. External diagnostic consumers must be checked before deployment. +- Restored `/_ts/admin/eids` as authenticated local cookie diagnostics on all four adapters; no-store/nosniff and unknown-admin fallback denial remain. Browser cookie readers/writer and diagnostic emission are retained alongside body persistence and the authenticated capture handoff. Transport retirement and external consumer migration are deferred to a separate PR. - Confirm deployment drain/rollback for mixed binary versions, external uses of `x-ts-eids` headers, Lockr fixtures and each enabled provider mapping, Lockr consent requirements and revoke semantics/scope. Clarify overlapping SDK response ordering. None is settled by local tests. ## Local implementation verification (2026-10-09) @@ -146,3 +146,29 @@ Not verified: real provider acceptance, sanitized provider-approved mappings/fix - Executed consistency limit: a simulated stale live snapshot can deliver its pending handoff despite an authoritative withdrawal elsewhere. The subsequent CAS rejects replay and preserves the tombstone. This is the existing snapshot-read consistency limit, not a strong-read or instant cross-POP withdrawal guarantee; deployed propagation/frequency remains unverified. Not verified: remote CI, live provider acceptance, actual cross-POP visibility/withdrawal timing, operational key provisioning/rotation, provider-approved fixture sizes, consent/revoke scope, overlapping SDK response ordering or metadata-compatible deployment drain. Keep handoff default-off and Lockr capture explicitly disabled until the applicable rollout gates pass. + +## Expired-source auction fallback verification + +Tested state: `16b6d09a` plus the expired-source fallback worktree changes in the two handoff Rust files, both specs, this plan, configuration/Lockr guides and the example configuration. Native checks ran on `x86_64-unknown-linux-gnu`; Fastly checks ran on `wasm32-wasip1` through Viceroy. SHA-256 fingerprints of the Rust inputs are `61ba327154ad9c0669f6b17c3c199250d3343c6c4eeca0a4c9eff17a8ef2f16f` for `identity_handoff.rs` and `df8a85b886f347b2307be2cc6a0a5a330fd60251739c01d43376b740fec501db` for `identity_handoff_tests.rs`. + +- Three new regressions first failed against the old behavior: expired-source delivery, issuance over an expired stored record, and the exact expiry boundary. A fourth failing regression identified that a matching UID without refreshed provider expiry must not bypass its known stored expiry. All four now pass. +- `cargo test -p trusted-server-core --lib ec::identity`: 44 passed. `cargo test -p trusted-server-core --lib`: 2,995 passed. +- `cargo test-fastly`: 219 adapter and 2,989 core tests passed, with six native-only core tests ignored; shared-crate and doc tests also passed. `cargo test-fastly-reuse`: 219 passed. +- `cargo test-axum`, `cargo test-cloudflare`, `cargo test-spin`, `cargo test --manifest-path crates/trusted-server-integration-tests/Cargo.toml --test parity`, `./scripts/test-cli.sh` including all required ignored browser-fixture filters, and `cargo test -p trusted-server-core --test template_build_digest` passed. +- All target-matched Clippy aliases in `AGENTS.md`, native build-digest Clippy, Rust formatting, full JS tests/build/format, docs formatting and non-docs Markdown formatting passed. JS results remain reusable after the final Rust-only expiry guard: 50 files and 1,209 tests passed. Gate commands, outcomes, elapsed times and logs are recorded in `/tmp/pr1260-expired-gates-summary.txt` and `/tmp/pr1260-expired-other-gates-summary.txt`. +- Executed delivery proof: auction resolution and browser merging use authenticated capture for expired matching or different UIDs, preserve Lockr provenance, and reject expired handoffs, empty records, unknown clocks, request denial and disabled policy. A matching known-expired UID needs a known refreshed expiry. Usable and unknown-expiry stored records retain precedence. +- Executed persistence proof: reconciliation keeps the original signed deadline over expired stored state without staging a replay write. Read-back after send preserves stored UID, expiry, writer, revision and generation. Fresh direct capture still writes after send, then a later request clears its acknowledged cookie. Existing absent-source replay and concurrent withdrawal/CAS tests remain passing. + +No cookie codec, revision-ordering mechanism, persistence replacement policy or rollout default changed. First-auction timing, live provider acceptance, cross-POP consistency, key operations and deployment drain remain unverified. Keep rollout gates unchanged. + +## Browser-cookie and diagnostic restoration verification + +Checks ran against `16b6d09a` plus the expiry fallback and browser-cookie restoration worktree changes before commit and push. This checkpoint supersedes the earlier transport-retirement decision, not the capture or lifecycle contracts. + +- Restored the bounded one-day browser `ts-eids` writer, body-or-cookie auction fallback, publisher/SPA readers, response-time cookie ingestion, authenticated local admin diagnostics and bounded EID response headers. Retirement is deferred to a separate PR. Adapter cookie threading was not reintroduced; request-local cookie state is private/redacted in `EcContext`. +- Executed safety proof: body beats cookie, direct `sharedId` wins last, registered cookie-only sources persist, matching expired cookie UIDs do not reach bidders, and browser input cannot replace managed UID/expiry/writer/revision metadata. A cookie matching authenticated pending capture cannot persist an unknown-expiry browser record before send; trusted replay retains provider expiry and the Lockr writer after send. Existing consent, root, ownership, CAS and withdrawal gates remain intact. +- Native core: 3,008 tests passed. Fastly: 219 adapter and 3,002 core tests passed, with six native-only core tests ignored; shared-crate and doc tests passed. Reusable Fastly, Axum, Cloudflare, Spin, integration parity, native build-digest and CLI tests passed. The CLI helper executed all four required ignored browser-fixture filters. +- JS: 50 files and 1,210 tests passed, with no type errors. Build and formatting passed. Restoring the cookie writer raised the measured shim to 41,007 bytes; its documented size guard is now 42,000 bytes rather than 41,000. Separate Prebid-free artifact assertions remain intact. +- All target-matched Clippy aliases, native build-digest Clippy, Rust formatting, docs formatting and non-docs Markdown formatting passed. Initial Axum checks found a test-only `serde_json` dependency assumption, removed without adding a dependency. Clippy found one missing documentation backtick, corrected before successful rechecks. Logs and final outcomes are in `/tmp/pr1260-restore-rust-gates-summary.txt` and `/tmp/pr1260-restore-*.log`. + +Not verified: remote CI, live/provider acceptance, cross-POP visibility/withdrawal timing, key operations and metadata-compatible deployment drain. Restoring browser transport does not guarantee first-auction availability or change rollout defaults. diff --git a/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md b/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md index 2aaeb2edb..6d5d2b3d7 100644 --- a/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md +++ b/docs/superpowers/specs/2026-10-08-lockr-identity-capture-design.md @@ -12,8 +12,8 @@ Issue: [#1246](https://github.com/IABTechLab/trusted-server/issues/1246). The foundation owns capability registration, common identity outcomes, source ownership, lifecycle records, consent/mutation policy and adapter execution. -It also owns auction-body ingestion and retirement of `ts-eids` and the EID -diagnostic headers. This spec defines Lockr's capture behavior using those +It also owns auction-body ingestion while preserving `ts-eids` and the EID +diagnostic headers; retirement is deferred to a separate PR. This spec defines Lockr's capture behavior using those contracts; it does not introduce a separate identity service or KV writer. Related: [KV EID request snapshot and EC recovery](2026-07-10-kv-eid-request-snapshot-ec-recovery-design.md) @@ -42,11 +42,11 @@ identity pointer. Reintroduce `ts-eids-s` as an optional, short-lived authentica capture handoff through the shared identity service, not a long-lived identity store or restoration of the browser-written `ts-eids` cookie. -For a source absent from a live, consenting EC row, a request carrying the handoff -can use the captured ID before its post-send write becomes visible and can stage -a bounded persistence retry. An auction that starts before the capture response -installs the cookie can still miss it. Existing-source refresh still waits for -the direct capture update; stale cookie replay must not replace stored state. +For a source absent or expired in a live, consenting EC row, a request carrying +a valid handoff can use the captured ID before its post-send write becomes visible. +Only absent sources can stage a bounded persistence retry. An auction that starts +before the capture response installs the cookie can still miss it. A usable stored +ID wins; persistent existing-source refresh still requires direct capture. Without handoff opt-in, the implemented KV-only behavior remains unchanged. There is no durable capture retry queue. @@ -111,8 +111,8 @@ and attaches `EcFinalizeState`; Fastly already has post-send execution. Today the upstream response body is not inspected. `/auction` body EIDs are used for that auction, while cookie EIDs are persisted by finalization. The -foundation changes the latter contract to direct auction-body ingestion before -retiring the cookie path. +foundation adds direct auction-body ingestion while retaining the browser +cookie path under the same conservative lifecycle policy. ## 3. Requirements and non-goals @@ -133,9 +133,9 @@ retiring the cookie path. foundation. A failed capture must not break the SDK call. 8. Reconstruct the fixed ID5 OpenRTB provenance on later auctions only for usable records whose persisted acquisition writer is `lockr`. -9. Offer immediate absent-source delivery and bounded retry through the - foundation's authenticated capture handoff, without weakening live-root, - consent, ownership, expiry or stale-replay rules. +9. Offer absent-source and expired-source auction fallback through the + foundation's authenticated capture handoff, with bounded persistence retry + only for absent sources. Preserve live-root, consent, ownership and expiry gates. Out of scope: server-to-server calls to lockr, identifier intake, cross-browser linking, changing Prebid provider discovery, direct ID5 or LiveRamp proxies, @@ -311,17 +311,26 @@ if cookie creation fails. A `NotRead` snapshot at capture does not force an adde pre-send lookup, but it also does not prove root eligibility for later delivery. On later requests, validate the handoff against current policy and the existing -full-ID-bound snapshot. An absent source on a live, consenting root may use the -handoff UID for a bidstream-enabled auction and stage fill-absent-only persistence -after send. Recheck absence, consent, provider expiry and the original handoff -deadline at mutation time. A matching persisted UID, writer and effective expiry -acknowledges the entry. A conflicting existing record wins and -retires the handoff; expired or invalidated records also block replay. No handoff -replaces a stored UID, extends its expiry, changes its writer or creates a root. -This first version does not accelerate refresh of an existing source. +full-ID-bound snapshot. An absent or known-expired source on a live, consenting +root may use the valid handoff UID for a bidstream-enabled auction. This includes +a same-UID capture with a known refreshed expiry and a different valid captured +UID. Missing expiry cannot revive a matching known-expired UID. Different UIDs +retain the capture policy's optional provider expiry and bounded handoff deadline. +Usable stored IDs retain precedence; empty or invalidated records block fallback. +A fresh capture can issue a handoff over an expired record, and reconciliation +retains it until its original deadline while the stored source remains expired. +This is auction delivery only, not a stored-record replacement or expiry change. + +Only absent sources may stage fill-absent-only persistence after send. Recheck +absence, consent, provider expiry and the original handoff deadline at mutation +time. A matching usable persisted UID, writer and effective expiry acknowledges +the entry; another usable stored record retires the conflicting handoff. Every +existing record, including an expired record, still blocks persistence replay. +Existing-source persistence requires direct capture; no replay changes its UID, +expiry or writer. No handoff creates a root or needs a provider-ordering mechanism. Snapshot eligibility retains existing KV read consistency. A stale live snapshot -that lacks the source can still deliver the handoff before it observes a newer +with an absent or expired source can still deliver the handoff before it observes a newer record or withdrawal elsewhere. Replay uses generation-checked writes and cannot replace that newer record or revive its tombstone. Current request consent always applies; the handoff does not promise instant cross-POP withdrawal propagation. @@ -344,12 +353,21 @@ to 30 days, with cookie-first precedence and generic cookie ingestion. Do not restore it as-is. It omitted expiry needed by its own merge rules and offered no authenticated full-EC binding or stale-replay contract. The revised handoff restores useful -absent-source availability and retry while keeping lifecycle policy in EC. - -The existing `ts-eids` fallback was largely redundant with normal `/auction` -body submission and did not merge missing sources into a valid body. Restoring -its late-browser-ID ingestion benefit is a separate decision, not part of this -capture handoff. +absent-source and expired-source auction availability, with absent-source-only +persistence retry, while keeping lifecycle policy in EC. + +The cookie was initially removed because it could not guarantee first-auction +availability: an auction can leave before the Lockr response installs the cookie. +That timing limitation remains. The restored authenticated handoff improves +later auction coverage and can retry an absent-source write; it does not claim +an unconditional next-auction guarantee. + +The existing `ts-eids` fallback is largely redundant with normal `/auction` +body submission and does not merge missing sources into a valid body. It remains +in this PR, including late-browser-ID ingestion on later navigation, publisher +and SPA cookie readers, authenticated admin diagnostics and auction EID headers. +Retirement is deferred to a separate PR. These unsigned browser inputs are not +capture authority and cannot erase stored or pending capture lifecycle metadata. ## 5. Data flow and freshness @@ -362,18 +380,19 @@ flowchart TD E --> F[Shared service applies authoritative updates after send] F --> G[EC identity graph stores IDs and lifecycle metadata] E --> H[Later request validates handoff against live EC root] - H --> I[Absent-source auction delivery and bounded after-send retry] - I --> G + H --> I[Absent or expired source gets valid capture for auction] + H --> K[Only absent sources stage bounded after-send retry] + K --> G G --> J[Later requests use KV state and clear acknowledged handoff] ``` -| Event | Persistence | Auction availability | -| ---------------------------------- | ------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | -| First issue, source absent | Authoritative update after send; eligible handoff can stage a later fill-absent retry | After cookie installation on a qualifying live-root request, or once KV exposes the record | -| SDK refresh, source already stored | Authoritative replacement or expiry update after send; replay cannot replace existing state | Existing usable KV record until the new state is visible | -| Provider or handoff expiry | No read-triggered cleanup write | Expired handoff is rejected; known-expired stored IDs remain omitted | -| Explicit global revoke | EC and handoff cookies expired; existing-key-only tombstone | Later requests stop using the identity; dispatched auctions cannot be recalled | -| Returning visitor, no new token | No new capture effect; valid pending handoff can independently stage replay | Existing usable KV state, or an eligible absent-source handoff before its original deadline | +| Event | Persistence | Auction availability | +| ---------------------------------- | ------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | +| First issue, source absent | Authoritative update after send; eligible handoff can stage a later fill-absent retry | After cookie installation on a qualifying live-root request, or once KV exposes the record | +| SDK refresh, source already stored | Authoritative replacement or expiry update after send; replay cannot replace existing state | Usable KV record wins; an expired record allows valid handoff auction fallback until visibility | +| Provider or handoff expiry | No read-triggered cleanup write | Expired handoff is rejected; known-expired stored IDs remain omitted | +| Explicit global revoke | EC and handoff cookies expired; existing-key-only tombstone | Later requests stop using the identity; dispatched auctions cannot be recalled | +| Returning visitor, no new token | No new capture effect; only absent sources may stage handoff replay | Existing usable KV state, or valid handoff fallback for an absent or expired source before its original deadline | Inspection can retain 64 KiB of original bytes and 64 KiB of decoded JSON, plus bounded stream/decoder overhead. While pending, the handoff adds up to 2,048 @@ -468,11 +487,14 @@ prove that this consumer reaches those paths correctly: - Capture effects execute after Fastly send through the foundation, not a separate Lockr adapter implementation; persisted snapshot handoff is retained. - Capture handoff defaults off; enabling it emits only the bounded authenticated - `ts-eids-s`, never the retired `ts-eids` or diagnostic EID headers. Provider - browser stores and existing publisher credential stripping remain unchanged. + `ts-eids-s` on proxy responses, not browser `ts-eids` or auction diagnostic + headers. The separate legacy browser writer/readers and auction diagnostics + remain enabled. Provider stores and publisher credential stripping are unchanged. - With capture KV visibility held back, an installed valid handoff reaches a - later live-root auction for an absent registered source. An earlier auction, - root-read failure or existing stored source receives no handoff guarantee. + later live-root auction for an absent or expired registered source. Same-UID + expiry refresh and different captured UID both work without changing KV. + Usable stored records win. An earlier auction or root-read failure still has + no handoff guarantee; expired-source persistence still requires direct capture. - No-token requests can repair pending absent-source capture after send; changed UID/expiry/writer, expired stored records, CAS conflicts and withdrawal cannot be overwritten by replay. Acknowledgement cleanup waits for a later response. @@ -484,7 +506,8 @@ Live acceptance on an approved publisher test route: 1. Fresh profile, grant consent and wait for `page-view`. 2. Confirm unchanged upstream body/encoding and, with handoff explicitly enabled, authenticated `ts-eids-s` plus private/no-store response handling. Confirm - `ts-eids` and diagnostic EID headers remain absent. + the proxy emits neither browser `ts-eids` nor auction diagnostic headers. + Their independent browser/auction paths remain available. 3. Allow for post-send work and KV visibility. Inspect the session's admin EC record for the registered source and lifecycle metadata, without exporting IDs. 4. Trigger a later auction and inspect bidder-facing payload evidence, including @@ -497,7 +520,9 @@ Live acceptance on an approved publisher test route: deadline or overwriting an existing record. 6. Trigger SDK refresh: confirm changed UID or expiry persists after visibility. 7. Exercise known expiry in a controlled fixture and confirm omission, including - a stale matching UID resubmitted in a browser auction body. + a stale matching UID resubmitted in a browser auction body. With a valid capture + handoff, verify expired-source auction fallback for matching and different UIDs, + unchanged stored metadata and deadline, and no expired-source replay write. 8. Trigger confirmed global revoke: EC and handoff cookies expire, tombstone exists and later enrichment or handoff replay cannot revive it. 9. On an approved controlled fixture, delay first-issue KV visibility and prove @@ -511,7 +536,8 @@ committed validation notes. Keep sanitized status/source/count evidence only. ## 9. Rollout 1. Deliver the foundation, including metadata-preserving writers/readers, - source ownership, browser-body migration and `ts-eids` cookie/header retirement. + source ownership and browser-body ingestion alongside retained `ts-eids` + cookie readers/writer and diagnostic headers. 2. Confirm Lockr path, provider token shapes, consent and revoke contracts using sanitized fixtures; leave unconfirmed source mappings disabled. 3. Register the capture consumer and its `capture_identity` kill switch. diff --git a/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md b/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md index 17f79f413..d53817a0a 100644 --- a/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md +++ b/docs/superpowers/specs/2026-10-09-server-side-identity-foundation-design.md @@ -319,9 +319,12 @@ existing caps and consent gates across all auction paths. When the optional handoff is enabled, resolve eligible captured IDs under section 8.2 before the existing browser merge. Handoff entries require a live, -consenting root and an absent stored source. Stored records retain precedence, -including expired or invalidated records that block replay. Handoff delivery -does not relax current-request browser consent gates. +consenting root and an absent or known-expired stored source. A usable stored +record retains precedence. A valid handoff can supply the same refreshed UID or +a different captured UID when the stored record has expired, without changing +that record. Empty or invalidated records still block delivery; every existing +record blocks persistence replay. Handoff delivery does not relax current-request +browser consent gates. ## 7. Consent and execution timing @@ -368,44 +371,45 @@ Other adapters use the shared service only where EC KV support exists. Document before-send behavior where no post-send execution hook exists; do not claim persistence on unsupported adapters. -## 8. Browser-body migration and capture handoff +## 8. Browser EID transport and capture handoff -### 8.1 Retire the redundant browser EID cookie +### 8.1 Preserve browser cookie and diagnostics alongside body ingestion Use validated `/auction` body EIDs both for current-request forwarding and for registered-source browser enrichment. Expose the existing EID-to-update helper without round-tripping through a fake cookie. Reuse the auction snapshot in finalization instead of performing a second unconditional lookup. -Remove the `ts-eids` writer and server readers only after body ingestion is -verified. Final behavior: - -- `/auction` gets current-request client EIDs from the body, with no `ts-eids` - fallback. Unknown browser sources remain usable only in that current auction. -- Publisher and SPA page-bid auctions read registered IDs from KV. All auction - paths may also use eligible `ts-eids-s` observations under section 8.2. -- The separate `sharedId` cookie retains its existing ingestion behavior; - combine browser inputs into one finalization mutation attempt. -- Orphan recovery no longer seeds IDs from a `ts-eids` cookie. -- Late browser IDs need another auction submission for persistence. No new - page-exit beacon or browser EID cookie fallback is introduced. `ts-eids-s` - contains validated server capture only; it does not collect late browser IDs. - -Normal `/auction` callers already submit available browser EIDs in the body. -The old `ts-eids` reader used body-or-cookie fallback, not missing-source merge. -Its current-auction role was therefore largely redundant. It could also ingest -IDs discovered after request construction on a later navigation. Restoring that -behavior remains a separate benefit decision; this handoff does not restore it. - -Retire `x-ts-eids` and `x-ts-eids-truncated` emission, checking external diagnostic -tooling first. Keep both names on the inbound internal-header strip list. -Do not remove EC or consent headers as part of this change. - -A transition JavaScript release expires the old host-only `Path=/` cookie; -its existing max-age is one day. New servers ignore it even if cached older -bundles still write it. Remove the cleanup shim once the transition artifacts -have aged out. Do not add unconditional server cookie cleanup that changes -cache privacy on every response. +Retain the existing `ts-eids` writer, readers and diagnostic paths. Their +retirement belongs in a separate PR, not this foundation delivery: + +- `/auction` prefers usable body EIDs as a complete set, falling back to `ts-eids` + only when no usable body set exists. Do not merge missing cookie sources into + a valid body. Unknown browser sources remain current-request-only. +- Publisher and SPA page-bid auctions merge permitted cookie EIDs with usable + registered KV IDs. All auction paths may also use eligible `ts-eids-s` + observations under section 8.2, without reviving matching expired browser UIDs. +- Capture the raw browser cookie as redacted request-local context, not new + adapter argument plumbing. Finalization combines decoded cookie updates, + staged body updates and direct `sharedId` in one existing snapshot/CAS mutation. + Body beats cookie; direct `sharedId` wins last. +- All browser inputs remain UID-only, registered-source enrichment. They cannot + confer acquisition authority, erase lifecycle metadata, overwrite expired + records, revive tombstones or strip authenticated pending capture metadata. + Normal EC creation and publisher-only orphan-recovery policy remain unchanged. +- The Prebid bid callback writes a one-day host-only `Path=/; Secure; SameSite=Lax` + cookie, capped at 3,072 base64 bytes by deterministic tail UID/source trimming. + The auction body remains untrimmed. Clear stale empty/unavailable IDs, not the + cookie unconditionally at installation. Late IDs may reach a later navigation; + no page-exit beacon or first-auction guarantee is added. + +Keep `x-ts-eids` diagnostic emission bounded at 4,096 base64 bytes and emit +`x-ts-eids-truncated: true` only when truncated. Retain authenticated local +`/_ts/admin/eids` cookie inspection, no-store/nosniff responses and admin fallback +denial on all adapters. Diagnostics confer no inbound identity authority: both +header names stay on the internal-header strip list. EC/consent headers and +expiry-aware `/_ts/admin/ec` remain independent. Identify external diagnostic +consumers and agree transition behavior before any later retirement. ### 8.2 Short-lived authenticated capture handoff @@ -465,8 +469,8 @@ auction delivery nor handoff persistence. No cookie creates or recovers a root. An unusable clock cannot authorize a time-bounded handoff. Eligibility reflects the row observed in this request, not a strongly consistent -or globally current read. A stale live snapshot that still lacks the source can -deliver a handoff while another POP has newer stored state or a withdrawal. +or globally current read. A stale live snapshot with an absent or expired source +can deliver a handoff while another POP has newer stored state or a withdrawal. This retains the foundation's existing KV read-consistency limit. Current request consent still gates delivery; post-send replay checks the authoritative generation and cannot overwrite the newer record or revive its tombstone. No cross-POP @@ -474,12 +478,13 @@ instant-withdrawal guarantee is made. For each registered source: -| Stored source state | Handoff action | -| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------- | -| Absent on a live, consenting root | Allow the unexpired captured UID for a bidstream-enabled source and stage a fill-absent-only retry. | -| Expired or invalidated record | Retire the handoff without delivery or retry for that source; the retained record is not an empty slot. | -| Usable record with the same UID, writer and effective expiry | Acknowledge the handoff and remove it; use the stored record. | -| Any other existing record | Use the stored record if usable and retire the conflicting handoff; do not replace the UID, extend expiry or change its writer. | +| Stored source state | Handoff action | +| ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Absent on a live, consenting root | Allow the unexpired captured UID for a bidstream-enabled source and stage a fill-absent-only retry. | +| Known-expired record with a non-empty UID | Allow the unexpired captured UID for this auction and retain its original handoff deadline. Do not replay persistence or change the stored record. | +| Empty or invalidated record | Retire the handoff without delivery or retry. Expiration is not permission to bypass explicit invalidation. | +| Usable record with the same UID, writer and effective expiry | Acknowledge the handoff and remove it; use the stored record. | +| Any other usable existing record | Use the stored record and retire the conflicting handoff, even if its UID or expiry differs. Do not replace the UID, extend expiry or change its writer. | Source/UID equality alone cannot acknowledge a handoff with different expiry or writer metadata. When capture omitted expiry, a stored same-UID known expiry is @@ -488,11 +493,21 @@ metadata into browser authority. Provenance reconstruction uses only the validated acquisition module and its approved source mapping; mixed-origin browser merges follow section 6.3. -This conservative first version accelerates absent-source capture only. A refresh -of an existing source still depends on the direct authoritative capture write. -It does not make an older cookie beat a newer record observed in the snapshot. Supporting immediate -replacement or expiry extension from replay requires a separate revision-bound -contract; edge timestamps and signatures do not prove provider ordering. +Auction fallback depends on usability, not on establishing which overlapping +provider response was newest. An absent or known-expired stored ID may use a +valid authenticated capture, including a same-UID expiry refresh. A usable stored +ID always wins. Do not add cookie revisions or response-time ordering for this +fallback. A matching known-expired UID requires a known, unexpired provider +expiry in the handoff; omitted expiry cannot prove a refresh or erase the stored +expiry. A different captured UID retains the existing optional-expiry policy. +Unknown stored expiry retains the existing legacy usability rule. + +Cookie issuance and reconciliation follow the same source policy. A fresh capture +can issue a handoff over an observed expired record. Retain an eligible incoming +handoff until its original deadline while that record remains expired. Only an +absent source can stage persistence replay; all existing-source refreshes still +persist through direct authoritative capture. Replay never replaces a stored UID +or extends its expiry. The shared service owns retry. A new direct capture observation supersedes any incoming handoff for the same source; do not replay the older entry alongside it. @@ -517,8 +532,8 @@ best-effort and requires another qualifying request before expiry. The capture response must install the cookie before a later request can carry it. An auction starting earlier can still miss the ID. The handoff does not guarantee -the first auction, immediate refresh of an existing source, delivery during a -root-read failure, or repair after its deadline. +the first auction, replacement of a usable stored ID, delivery during a root-read +failure, persistent repair of an expired source, or repair after its deadline. Shared handoff issuance, delivery and retries default off behind explicit publisher opt-in. The integration capture kill switch also stops handoff delivery @@ -600,7 +615,8 @@ Delivery stages: mutation plumbing with managed acquisition inactive. 2. Add common outcomes, source ownership and capture registration. Keep authoritative and browser mutation classes distinct. -3. Add browser-body ingestion, verify it, then retire `ts-eids` cookie/header paths. +3. Add browser-body ingestion while retaining `ts-eids` cookie/header paths; + defer their retirement to a separate PR. 4. Add bounded capture handling and shared Fastly effect execution, using synthetic integration fixtures before enabling Lockr. 5. Implement the separate Lockr consumer, then the optional authenticated @@ -620,7 +636,7 @@ Foundation tests must cover: - Authoritative replacement and same-UID expiry changes; unchanged metadata skips writes; unrelated sources survive CAS conflicts. - Known expiry boundaries across publisher, `/auction` and SPA auction paths, - including an expired UID resubmitted in the auction body. + including an expired UID resubmitted in the auction body or browser cookie. - Writer-aware Lockr ID5 provenance through KV reads and bidder-facing auctions, including browser/legacy/unknown writers, configured-owner-only records, unrelated sources, matching versus different body UIDs and explicit body @@ -632,7 +648,9 @@ Foundation tests must cover: - Independent original/decoded size limits, compressed expansion, unsupported encodings, decode failure and byte-identical capture pass-through. - Browser updates surviving handled auction-provider failures; one combined - browser finalization mutation with `sharedId`; body-only migration behavior. + browser finalization mutation with cookie/body/`sharedId`; body-only persistence, + cookie fallback, late-ID ingestion, authenticated admin diagnostics and bounded + EID response headers. - Fastly send-before-capture-write ordering and resulting snapshot handoff to pull sync; no-token and failed-update paths; supported adapter behavior. - Redacted errors and no private data in logs, including provider query URLs. @@ -642,8 +660,11 @@ Foundation tests must cover: - Handoff auction eligibility across publisher, `/auction` and SPA paths before capture KV visibility; no delivery for missing/failed/withdrawn roots, denied consent, expired entries, disabled capture or changed ownership. -- Absent-source-only replay, exact acknowledgement, different-UID and same-UID - changed-expiry conflicts, retained expired records, duplicate requests and CAS +- Expired-source auction fallback for matching and different captured UIDs, + including the exact expiry boundary, unchanged stored metadata, retained + original deadlines, and rejection of expired handoffs or empty records. +- Absent-source-only replay, exact acknowledgement, usable different-UID and + same-UID changed-expiry precedence, retained expired records, duplicate requests and CAS races with acquisition or withdrawal. No stale replacement or metadata loss. - Send-before-replay-write ordering, combined replay bounds, acknowledgement cleanup on a later response, no-token request repair and failure retention only diff --git a/trusted-server.example.toml b/trusted-server.example.toml index 85a57a272..7639123b2 100644 --- a/trusted-server.example.toml +++ b/trusted-server.example.toml @@ -602,9 +602,9 @@ geo_query_parameters = false # capture_global_withdrawal = false # require confirmed global revoke semantics # The separate [ec.identity_handoff] opt-in stays off until its synthetic and -# live acceptance gates pass. It is absent-source-only, expires within five -# minutes, and provides only best-effort after-send retry. It does not restore -# the retired browser-written ts-eids cookie. +# live acceptance gates pass. It supplies auction IDs for absent or expired +# sources, expires within five minutes, and retries persistence only for absent +# sources. The browser-written ts-eids cookie remains a separate legacy path. # DataDome bot protection. Proxies tags.js + signal-collection API first-party. # Kept active but disabled so `ts audit` can flip `enabled` when detected.