Skip to content

[common] Add shared-shredding write helpers for MAP columns#8332

Merged
JingsongLi merged 2 commits into
apache:masterfrom
lxy-9602:add-shredding-write-helpers
Jun 23, 2026
Merged

[common] Add shared-shredding write helpers for MAP columns#8332
JingsongLi merged 2 commits into
apache:masterfrom
lxy-9602:add-shredding-write-helpers

Conversation

@lxy-9602

Copy link
Copy Markdown
Contributor

Purpose

This is a sub-PR for MAP shared-shredding write support. The full shared-shredding write path will be wired in later PRs.

  • Add MapSharedShreddingContext to maintain cross-file shared-shredding state and compute the next physical column width K.
  • Add MapSharedShreddingFieldDict to assign stable field ids for MAP keys within a shared-shredding field.
  • Add MapSharedShreddingColumnAllocator to allocate MAP entries into physical shared columns for each row.
    • This is currently a simple temporary implementation based on row order.
    • A later version will support a more sophisticated LRU-style allocation strategy.
  • Add MapSharedShreddingRowConverter to lazily wrap logical InternalRow records and convert shared-shredding MAP fields into their physical representation only when those fields are accessed.

Related design:
https://cwiki.apache.org/confluence/display/PAIMON/PIP-43%3A+Columnar+Storage+Optimization+for+MAP+Type+in+Paimon

Tests

mvn -pl paimon-common -am -Pfast-build -DfailIfNoTests=false \
  -Dtest=MapSharedShreddingColumnAllocatorTest,MapSharedShreddingContextTest,MapSharedShreddingFieldDictTest,MapSharedShreddingRowConverterTest test

@JingsongLi

Copy link
Copy Markdown
Contributor

Looks no problem, wait CI.

@JingsongLi

Copy link
Copy Markdown
Contributor

+1

@JingsongLi JingsongLi merged commit 0644a72 into apache:master Jun 23, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants