Skip to content

[spark] Support timestamp_ntz columns in zorder and hilbert clustering - #10011

Merged
JingsongLi merged 1 commit into
apache:masterfrom
cxzl25:spark-cluster-timestamp-ntz
Sep 21, 2026
Merged

JingsongLi merged 1 commit into
apache:masterfrom
cxzl25:spark-cluster-timestamp-ntz

Conversation

@cxzl25

@cxzl25 cxzl25 commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Purpose

Clustering a paimon table on a TIMESTAMP column fails on Spark 3.4 and above:

Cannot use column ts of type TimestampNTZType in ZOrdering, the type is unsupported
java.lang.IllegalArgumentException: Cannot use column ts of type TimestampNTZType in ZOrdering, the type is unsupported
	at org.apache.paimon.spark.sort.SparkZOrderUDF.sortedLexicographically(SparkZOrderUDF.java:354)
	at org.apache.paimon.spark.sort.ZorderSorter.lambda$zValue$0(ZorderSorter.java:71)

Tests

Add UT

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requirement fit: SUPPORTED (triage: GO)\nImplementation: CLEAN\n\nThis fixes an end-to-end Spark clustering failure for the TimestampNTZ type produced by current Paimon timestamp mappings. I reviewed null handling, monotonic conversion, Z-order and Hilbert dispatch, and Spark-version compatibility. The seconds-level representation matches the existing TimestampType cast path; no actionable P0/P1/P2 issue found.

@JingsongLi
JingsongLi merged commit bb5498f into apache:master Sep 21, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants