[parquet] Add regression test for TIMESTAMP(n<=3) reading MICROS-annotated INT64 files#8238
Conversation
…v2 compatibility) Paimon emits TIMESTAMP(MILLIS) for precision <= 3 columns. The Iceberg v2 spec requires INT64 MICROS for timestamp/timestamptz; MILLIS is only valid under Iceberg v3. This causes Iceberg-aware engines (Athena, Trino, Spark) to reject Parquet files with a schema compatibility error. - ParquetSchemaConverter.createTimestampWithLogicalType: emit MICROS for precision <= 3 instead of MILLIS. - ParquetRowDataWriter.TimestampMillsWriter.writeTimestamp: call value.toMicros() so the stored INT64 matches the MICROS annotation unit. The reader path (MILLIS -> precision=3, MICROS -> precision=6) is left unchanged so files written by older versions remain readable. Existing tables with precision<=3 columns should be rebuilt after upgrading. Tests: testLowPrecisionTimestampUseMicrosAnnotation verifies MICROS annotation for precision 0-3; testPaimonParquetSchemaConvert updated for the widened round-trip precision.
…of micros ParquetSimpleStatsExtractor.toTimestampStats called fromEpochMillis for precision <= 3, but footer statistics for those columns now contain INT64 microseconds (matching the MICROS annotation). Switch to fromMicros so that Parquet column bounds are decoded correctly.
VectorizedColumnReader has a lazy dictionary fast path for INT64/ LongColumnVector: the raw Parquet dictionary is stored on the vector directly, bypassing LongTimestampUpdater.longTimestamp() which normalises on-disk microseconds to the milliseconds that ParquetTimestampVector. getTimestamp expects. The result is timestamps ~1000x too far in the future for any dictionary-encoded page (triggered when rowGroupSize is large enough to activate dictionary encoding). Exclude precision <= 3 timestamp types from lazy decoding via a new isLowPrecisionTimestamp helper so the eager path (decodeDictionaryIds) is always taken, applying the correct /1000 normalisation.
…vs epoch_µs After the MICROS annotation change, ParquetRowDataWriter stores TIMESTAMP(n<=3) values as epoch microseconds. ParquetFilters.convertLiteral was still using getMillisecond() (epoch_ms) for those columns, so the Parquet row-group statistics comparison always failed against the new epoch_µs statistics — causing WHERE predicates on low-precision timestamp columns to filter out all row groups and return empty results. Fix: use toMicros() for all INT64 timestamp precisions (0-6) in ParquetFilters.convertLiteral, matching the storage unit written by the writer. Update ParquetFiltersTest assertions accordingly.
…poch-milliseconds
…tated Parquet files
q8webmaster
changed the title
| return new SimpleColStats( | ||
| Timestamp.fromEpochMillis(longStats.getMin()), | ||
| Timestamp.fromEpochMillis(longStats.getMax()), | ||
| Timestamp.fromMicros(longStats.getMin()), |
There was a problem hiding this comment.
After this change the extractor decodes TIMESTAMP(0..3) footer stats as micros solely from the Paimon field precision. Existing Paimon Parquet files written before this PR use TIMESTAMP_MILLIS and store min/max in epoch milliseconds, so extracting stats for those files (for example during migrate/clone or any metadata regeneration) would turn 2024-01-01T00:00:00.123 into a 1970 timestamp and write incorrect file stats. Can we derive the unit from stats.type().getLogicalTypeAnnotation() / the column metadata, like the reader does, and keep MILLIS for legacy files?
| } else if (precision <= 6) { | ||
| // microseconds | ||
| if (precision <= 6) { | ||
| return timestamp.toMicros(); |
There was a problem hiding this comment.
This predicate literal is now always epoch micros for TIMESTAMP(0..6), but precision 0..3 files written by existing Paimon versions store epoch milliseconds. The filter is created in ParquetFileFormat before ParquetReaderFactory opens each file, so it cannot see whether a particular file schema is TIMESTAMP_MILLIS or TIMESTAMP_MICROS. For old files, ts = 2024-01-01 becomes 1704067200000000 while the row-group stats/data are around 1704067200000, and Parquet can incorrectly drop matching row groups. We should either build the timestamp filter after reading the file schema or disable/avoid this pushdown for legacy low-precision timestamp files.
2 participants
Problem
After PR #8230 lands,
TIMESTAMP(n<=3)columns will be written asINT64with aMICROSParquet annotation and epoch-microsecond values. If the vectorized reader fails to respect the annotation's time unit when decoding those columns, it returns timestamps ~1000× too large (year ~58xxx) or throwsArithmeticException: Millis overflow.This scenario had no test coverage.
Root cause
The fix is already present on master:
LongTimestampUpdater.timestampUnit()(introduced in #7845 for NANOS support) reads the actual Parquet annotation and normalises the stored value to epoch-milliseconds before it reachesParquetTimestampVector.getTimestamp(). Without that normalisation (e.g. paimon 1.4.1, which lackstimestampUnit()), the raw epoch-µs value is passed toTimestamp.fromEpochMillis()— 1000× wrong.Fix
No production code change — the fix is already in
LongTimestampUpdater. This PR adds a regression test that would catch any future regression in this code path.The test mirrors
testReadTimestampNanosWrittenByParquet: it writes a Parquet file externally withINT64 MICROSannotation for aTIMESTAMP(3)column, reads it back viaParquetReaderFactorywith aTimestampType(3)row type, and asserts the decoded values matchTimestamp.fromMicros().Prior art
PR #8230 introduced the MICROS writer path this test covers. PR #7845 introduced
timestampUnit(), which is the reader fix this test guards.Changes
ParquetReadWriteTest.java:testReadTimestampMicrosWrittenByParquetForLowPrecision— reads externally-writtenINT64 MICROSParquet with aTIMESTAMP(3)schema and verifies correct decoding