Skip to content

docs: add guide for ingesting PostgreSQL partitioned tables - #38199

Draft
tylerhartwig wants to merge 3 commits into
mainfrom
docs/postgres-partitioned-tables
Draft

docs: add guide for ingesting PostgreSQL partitioned tables#38199
tylerhartwig wants to merge 3 commits into
mainfrom
docs/postgres-partitioned-tables

Conversation

@tylerhartwig

@tylerhartwig tylerhartwig commented Aug 13, 2026

Copy link
Copy Markdown

Motivation

Partitioned PostgreSQL tables come up repeatedly with customers and we have no
docs for them. PostgreSQL expands a partitioned table to its leaf partitions in
a publication, and the PostgreSQL source does not support
publish_via_partition_root, so users need a documented strategy.

Adds a guide under Ingest data → PostgreSQL, plus a "Partitioned tables"
entry in the PostgreSQL source considerations that warns against
publish_via_partition_root and links to it.

What's covered

Approach 1 — ingest the leaf partitions. One table per partition, unioned
into a materialized view so that partitions can be added and removed with
CREATE REPLACEMENT MATERIALIZED VIEW / ALTER MATERIALIZED VIEW ... APPLY REPLACEMENT, without recreating dependent objects. (Per @sjwiesman's and
Nate Stewart's suggestion — using a plain view means every rollover breaks
downstream objects.)

Approach 2 — flatten onto a dedicated replica. Written as an extension of
the existing dedicated replica
guide

rather than a duplicate: only the partition-specific deltas
(publish_via_partition_root = true on the primary→replica hop, flat table with
the parent's primary key on the replica, mirroring retention with DELETE).

It also explains how the flattening actually happens, since that was the least
obvious part in review: logical replication matches publisher to subscriber by
schema-qualified name, and publish_via_partition_root changes which name goes
on the wire, so partitions are never named and the subscriber's table can be an
ordinary one.

Please verify before merge

Three claims come from local testing (Postgres 17 → Materialize v26.37.0), not
from existing docs. Flagging them so review effort lands in the right place:

  1. "Materialize does not support publish_via_partition_root ... can produce
    incorrect results."
    Reproduced locally: the source is created and appears
    to work, then diverges once partitions are attached/detached, and an
    ATTACH of a populated partition followed by an UPDATE of one of its rows
    produced a negative accumulation (Non-positive multiplicity in DistinctBy).
    Matches SS-384 and SS-394. Verify: is "can produce incorrect results" the
    wording we want, and should this instead be a hard error at CREATE SOURCE?

  2. The add/remove-partition runbooks. Verified end to end on v26.37.0,
    including that dependent materialized views survive both operations and that
    retiring a partition in Materialize before dropping it upstream retracts the
    rows rather than stranding them. Verify: the ordering in "Remove a
    partition" is the sequence we want to recommend.

  3. REPLICA IDENTITY FULL does not cascade to partitions, and a partition
    with neither a replica identity nor a primary key causes PostgreSQL to reject
    UPDATE/DELETE against it on the primary. Verified on PostgreSQL 17.

Open question

Approach 1 leans on replacement views, which are Public Preview and
currently annotated "only recommended for use in development environments." That
sits awkwardly with recommending them as the primary path in an ingest guide.
Options: ship as-is with the preview annotation (current state), soften Approach
1 to plain views and note replacement views as the better option once GA, or
lead with Approach 2. Happy to take direction here.

Testing

  • hugo builds clean; all shortcodes expand and internal links resolve.
  • Reviewed against doc/user/.prompts/docs_pre_pr_review.md.

tylerhartwig and others added 3 commits August 13, 2026 11:14
Partitioned tables are a recurring source of confusion: PostgreSQL expands
them to their leaf partitions in a publication, and `publish_via_partition_root`
is not supported by the PostgreSQL source.

Adds a guide covering the two supported approaches:

- Ingest the leaf partitions and union them into a materialized view, using
  replacement views to add and remove partitions without recreating dependent
  objects.
- Flatten the table onto a dedicated replica, as an extension of the existing
  dedicated-replica guide.

Also adds a "Partitioned tables" entry to the PostgreSQL source considerations,
which warns against `publish_via_partition_root` and links to the guide.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dedicated-replica approach deferred the CREATE SUBSCRIPTION step to the
linked guide and never explained why changes to a leaf partition land in a
flat table on the replica, which is the key mechanism.

Adds a 'How the flattening works' section covering name-based relation
matching, shows the subscription step inline, and adds a verification step.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The subscription names no tables, which reads as though something is missing.
Explain that the table set comes from the publication and that slot creation,
initial copy, and streaming are defaults.

Also make the ordering dependency explicit: the table must exist on the replica
first, since DDL is not replicated.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant