fix(hudi): Fall back to parquet-footer stats when the metadata table returns degenerate value counts - #951
Open
vamsikarnika wants to merge 2 commits into
Open
fix(hudi): Fall back to parquet-footer stats when the metadata table returns degenerate value counts#951vamsikarnika wants to merge 2 commits into
vamsikarnika wants to merge 2 commits into
Conversation
…returns degenerate value counts tryEnrichWithMetadataStats() accepted the Hudi metadata table's column-stats entry for a file as valid whenever the entry existed at all, even when every column's value count came back non-positive (or absent). In that case getMaxFromColumnStats() falls through to .orElse(0L), silently emitting recordCount=0 for a file that has real data -- the existing fallback to parquet-footer stats only triggers when the metadata table has no entry for the file at all, not when it has an entry with unusable values. Observed in practice on files written before column-stats indexing was enabled for their table: real Parquet files with correct row counts in their own footers, but zero-valued entries in the metadata table's column-stats index, ended up with recordCount=0 (and empty column stats) in the synced Iceberg table's manifest -- real data, wrong per-file statistics. Fix: treat an absent/non-positive max the same as "no stats" and fall back to reading the file's actual row count from its Parquet footer, reusing the existing (already correct) fallback path. Adds a regression test reproducing the exact scenario: metadata table returns a non-empty entry with valueCount=0 for every column; asserts the result falls back to the parquet footer (real row count) instead of emitting recordCount=0.
vamsikarnika
force-pushed
the
fix/hudi-iceberg-stats-fallback
branch
from
September 24, 2026 15:45
79bec97 to
69025f1
Compare
vinishjail97
left a comment
vinishjail97
left a comment
Contributor
There was a problem hiding this comment.
Thanks for picking this up @vamsikarnika, added few comments.
| long recordCount = getMaxFromColumnStats(columnStats).orElse(0L); | ||
| return Optional.of(file.toBuilder().columnStats(columnStats).recordCount(recordCount).build()); | ||
| Optional<Long> recordCount = getMaxFromColumnStats(columnStats); | ||
| if (!recordCount.isPresent()) { |
Contributor
There was a problem hiding this comment.
These files now also reach the log.warn in addStatsToFiles that says they "had no column stats". Would it be reasonable to reword it to "had missing or unusable column stats", so the log matches both cases?
Contributor
Author
There was a problem hiding this comment.
Thanks @vinishjail97 for pointing this out. I've fixed the log now.
| // Must have fallen back to the parquet footer (real row count), not the degenerate | ||
| // metadata-table stats (which would previously have produced recordCount=0). | ||
| assertEquals(2, result.getRecordCount()); | ||
| assertFalse(result.getColumnStats().isEmpty()); |
Contributor
There was a problem hiding this comment.
Would it work to assert the column stats match the footer-derived ones (for example, a non-zero numValues)? isEmpty() would still pass if the zero-count metadata stats leaked through.
Contributor
Author
There was a problem hiding this comment.
Makes sense. Updated the test to add more checks.
Comment on lines
+204
to
+213
| Optional<Long> recordCount = getMaxFromColumnStats(columnStats); | ||
| if (!recordCount.isPresent()) { | ||
| // The metadata table has an entry for this file, but every column's value count is | ||
| // absent or non-positive (observed for files written before column-stats indexing was | ||
| // enabled). Treat this the same as "no stats" instead of emitting recordCount=0, so the | ||
| // caller falls back to reading the authoritative row count from the Parquet footer. | ||
| return Optional.empty(); | ||
| } | ||
| return Optional.of( | ||
| file.toBuilder().columnStats(columnStats).recordCount(recordCount.get()).build()); |
Contributor
There was a problem hiding this comment.
Nitpick: you could simplify this to:
getMaxFromColumnStats(columnStats).map(recordCount -> file.toBuilder().columnStats(columnStats).recordCount(recordCount).build());
Contributor
Author
There was a problem hiding this comment.
Thanks @the-other-tim-brown for pointing this out. I have addressed it. Change looks cleaner now.
- Reword the log.warn in addStatsToFiles() to say files had 'missing or unusable' column stats, since it now covers both the absent-entry case and the degenerate-zero-value-count case, not just the former. - Simplify tryEnrichWithMetadataStats's if/return into a single getMaxFromColumnStats(...).map(...) expression. - Strengthen the new regression test's assertion: instead of only checking columnStats isn't empty (which degenerate zero-count stats would also satisfy if they leaked through), assert the actual footer-derived values for a specific column (numValues, numNulls, min/max range).
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Found while syncing a real 24-table, 1TB, thousands-of-partitions-per-table Hudi dataset to Iceberg via
RunSync.tryEnrichWithMetadataStats()accepted the Hudi metadata table's column-stats entry for a file as valid whenever the entry existed at all — even when every column's value count came back non-positive. In that case it silently emitsrecordCount=0for a file that has real data, instead of falling back to the (already-existing) Parquet-footer-based stats path, which is only triggered when the metadata table has no entry at all, not when it has an entry with unusable values.Observed on real files written before column-stats indexing was enabled for their table: correct data, correct row counts in the actual Parquet footers, but zero-valued entries in the metadata table's index — resulting in
recordCount=0(and empty column stats) written into the synced Iceberg table's manifest.Fix
Treat an absent/non-positive max the same as "no stats" and fall back to the footer-based path. Added a regression test reproducing the exact scenario (metadata table returns a non-empty entry with
valueCount=0for every column; asserts the result falls back to the real row count from the Parquet footer).Testing
columnStatsWithMetadataTableZeroValueCountFallsBackToParquetFooterspasses.xtable-coremodule compiles andTestHudiFileStatsExtractorsuite passes against the real, publicly-publishedorg.apache.hudi:hudi-common:1.2.0(verified with a from-scratch~/.m2resolution — one pre-existing, unrelated failure incolumnStatsWithMetadataTable, aNoSuchFieldErrordeep in Hudi's own bulk-insert internals on a code path this PR doesn't touch).COUNT(*) = 0via both Trino's and Presto's Iceberg connectors despite having real data; after the fix, all 24 tables report correct row counts matching the source.🤖 Generated with Claude Code
https://claude.ai/code/session_01DFemyCQKjw9RTX3P3vixY1