fix: cap the total size of activity attributes

[themarolt]

Summary

Caps the serialized size of an activity's attributes at 16 KiB for every platform. Since #4934 each string is trimmed to 500 characters, but an activity with many attributes or long arrays could still be arbitrarily large in total. Activities under the cap are serialized exactly as before; larger ones lose their largest values first instead of everything.

Changes

• serializeAttributes replaces objectToBytes: the existing 500-character string trim, then a UTF-8 byte measurement against ATTRIBUTES_MAX_BYTES (16 KiB, which keeps a 10-message Kafka batch far below the 1 MB broker default). Under the cap the output is unchanged.
• Over the cap, pruneToFit removes the largest entries first: objects and arrays shrink from the inside, other values are dropped. Ties go alphabetically for keys and from the end for array items, so the output is deterministic. Small scalars that pipes read, like isIndirectFork, are the last to go.
• A warning is logged for every pruned activity with its id, platform, type, sourceId and the size before pruning, never the attribute contents.
• Tests: exact output for under-cap attributes, pruning order, nested shrinking, top-level arrays, UTF-8 measurement, determinism, and a seeded random check that every output is valid JSON under the cap.
• Protected path: the data-access layer.

Type of change

• Bug fix
• New feature
• Refactor / cleanup
• Performance improvement
• Chore / dependency update
• Documentation

Related issue

Refs linuxfoundation/lfx-cdp-insights-ops#748

[cursor]

PR Summary

Medium Risk
Changes what gets stored and sent to Kafka for oversized activities—some attribute data may be dropped or truncated, though behavior under 16 KiB is unchanged.

Overview
Activity attributes on insert are now bounded to 16 KiB (UTF-8) after the existing per-string 500-character trim, replacing the old objectToBytes path with serializeAttributes.

Payloads under the cap serialize the same as before; larger ones go through pruneToFit, which shrinks nested objects/arrays from the inside or drops the largest top-level entries first (deterministic tie-breaking), so Kafka batches stay small and small scalars like isIndirectFork are preserved longer than huge arrays or strings.

insertActivities stores the serialized JSON string and emits a warning (activity id, platform, type, sourceId, original size—no attribute bodies) when pruning runs. Tests cover under-cap parity, pruning order, nested/top-level arrays, UTF-8 sizing, key-order stability, and random oversized shapes.

^{Reviewed by Cursor Bugbot for commit 527268a. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

🔵 Needs a closer look

Lossy pruning changes shared ingestion behavior across all platforms and warrants code-owner validation.

0 open findings

What changed in this PR

Adds a 16 KiB total limit to serialized activity attributes across all platforms, extending the existing per-string trimming.

Changes:

• Prunes oversized attributes deterministically, shrinking nested containers while preserving smaller values where possible.
• Logs pruning events with activity metadata, without attribute contents.
• Adds tests for size limits, pruning order, nested structures, UTF-8 sizing, and randomized inputs.

File	Description
services/​libs/​data-access-layer/​src/​activities/​insert.ts	Implements capped serialization, recursive pruning, and warnings.
services/​libs/​data-access-layer/​src/​activities/​insert.test.ts	Tests serialization compatibility and pruning behavior.

🧠 Review effort: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.