docs: add guide for ingesting PostgreSQL partitioned tables - #38199
Draft
tylerhartwig wants to merge 3 commits into
Draft
docs: add guide for ingesting PostgreSQL partitioned tables#38199tylerhartwig wants to merge 3 commits into
tylerhartwig wants to merge 3 commits into
Conversation
Partitioned tables are a recurring source of confusion: PostgreSQL expands them to their leaf partitions in a publication, and `publish_via_partition_root` is not supported by the PostgreSQL source. Adds a guide covering the two supported approaches: - Ingest the leaf partitions and union them into a materialized view, using replacement views to add and remove partitions without recreating dependent objects. - Flatten the table onto a dedicated replica, as an extension of the existing dedicated-replica guide. Also adds a "Partitioned tables" entry to the PostgreSQL source considerations, which warns against `publish_via_partition_root` and links to the guide. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dedicated-replica approach deferred the CREATE SUBSCRIPTION step to the linked guide and never explained why changes to a leaf partition land in a flat table on the replica, which is the key mechanism. Adds a 'How the flattening works' section covering name-based relation matching, shows the subscription step inline, and adds a verification step. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The subscription names no tables, which reads as though something is missing. Explain that the table set comes from the publication and that slot creation, initial copy, and streaming are defaults. Also make the ordering dependency explicit: the table must exist on the replica first, since DDL is not replicated. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
Partitioned PostgreSQL tables come up repeatedly with customers and we have no
docs for them. PostgreSQL expands a partitioned table to its leaf partitions in
a publication, and the PostgreSQL source does not support
publish_via_partition_root, so users need a documented strategy.Adds a guide under Ingest data → PostgreSQL, plus a "Partitioned tables"
entry in the PostgreSQL source considerations that warns against
publish_via_partition_rootand links to it.What's covered
Approach 1 — ingest the leaf partitions. One table per partition, unioned
into a materialized view so that partitions can be added and removed with
CREATE REPLACEMENT MATERIALIZED VIEW/ALTER MATERIALIZED VIEW ... APPLY REPLACEMENT, without recreating dependent objects. (Per @sjwiesman's andNate Stewart's suggestion — using a plain view means every rollover breaks
downstream objects.)
Approach 2 — flatten onto a dedicated replica. Written as an extension of
the existing dedicated replica
guide
rather than a duplicate: only the partition-specific deltas
(
publish_via_partition_root = trueon the primary→replica hop, flat table withthe parent's primary key on the replica, mirroring retention with
DELETE).It also explains how the flattening actually happens, since that was the least
obvious part in review: logical replication matches publisher to subscriber by
schema-qualified name, and
publish_via_partition_rootchanges which name goeson the wire, so partitions are never named and the subscriber's table can be an
ordinary one.
Please verify before merge
Three claims come from local testing (Postgres 17 → Materialize v26.37.0), not
from existing docs. Flagging them so review effort lands in the right place:
"Materialize does not support
publish_via_partition_root... can produceincorrect results." Reproduced locally: the source is created and appears
to work, then diverges once partitions are attached/detached, and an
ATTACHof a populated partition followed by anUPDATEof one of its rowsproduced a negative accumulation (
Non-positive multiplicity in DistinctBy).Matches SS-384 and SS-394. Verify: is "can produce incorrect results" the
wording we want, and should this instead be a hard error at
CREATE SOURCE?The add/remove-partition runbooks. Verified end to end on v26.37.0,
including that dependent materialized views survive both operations and that
retiring a partition in Materialize before dropping it upstream retracts the
rows rather than stranding them. Verify: the ordering in "Remove a
partition" is the sequence we want to recommend.
REPLICA IDENTITY FULLdoes not cascade to partitions, and a partitionwith neither a replica identity nor a primary key causes PostgreSQL to reject
UPDATE/DELETEagainst it on the primary. Verified on PostgreSQL 17.Open question
Approach 1 leans on replacement views, which are Public Preview and
currently annotated "only recommended for use in development environments." That
sits awkwardly with recommending them as the primary path in an ingest guide.
Options: ship as-is with the preview annotation (current state), soften Approach
1 to plain views and note replacement views as the better option once GA, or
lead with Approach 2. Happy to take direction here.
Testing
hugobuilds clean; all shortcodes expand and internal links resolve.doc/user/.prompts/docs_pre_pr_review.md.