Sql-150: Temporary objects to the catalog by SangJunBak · Pull Request #38151 · MaterializeInc/materialize

SangJunBak · 2026-08-10T22:07:02Z

This PR is meant to be merged with everything else in the stack and is split for review purposes

Motivation

sql-150

Description

1. design: update durable temporary objects design doc (`de137ed708c8`)

Initially the design doc stated that we wanted to persist sessions in the Catalog. Through benchmarking, it was found to create majoir regressions on CPS, even with optimizations. Thus we keep mz_sessions as a builtin table but will continue to persist durable objects.

We also change the requirement of temporary schemas being durable to being a followup given we don't need them to move builtin tables.

2. catalog: bump catalog version to 91 (`88dd791836fd`)

Generic catalog migration version bump. Copies everything and is intended to make the review easier.

3. catalog: add ephemeral_owner_session to durable items (`fa94bb99cb93`)

We add the ephemeral_owner_session property in durable Items to indicate if an object is temporary
We remove all ephemeral items in savepoint/write open of the catalog. We can't do this for readonly mode since readonly followers would remove it in memory but then eventually panic when it sees a retraction for it

4. catalog: make temporary items durable in the catalog shard (`fd2f72fddd15`)

Temporary items now write real durable Item rows, marked with ephemeral_owner_session = the creating session's UUID
The duplicated TemporaryItem is consolidated
- Replaces catalog entries' From implementation with durable_item since we now need access to the connection <-> session mapping.
We introduce two map state variables: ephemeral_owner_conns_by_uuid and ephemeral_owner_uuids_by_conn. Both serve to create a session UUID <-> Conneciton ID mapping. These are used for two purposes:
- When resolving a temporary object, routing it to its temporary schema via the connection ID <-> Temporary schema mapping usingtemporary_schemas
- When editing an object, as opposed to creating an object, to store the new object durably, we don't have access to the session UUID but have access to the connection ID. Thus we use these maps to grab it.
- In a future commit, we'll be merging temporary_schemas with these map variables since all three have the same lifecycle and are all used together

5. adapter: unify temporary schemas with ephemeral owner registration (`7b83fefba86e`)

The lifecycle and purpose of our ephemeral_* map state variables are very similar to temporary_schemas. That is, all lazily initialize in-memory temporary item metadata to resolve temporary items. Thus we unify all three in a struct TemporaryNamespaces
Gets rid of extraneous lazy initialization of the mz_temp schema inside apply_item_update from before given the lazy initialization in the inner insert_entry is all we need
Gets rid of eager mz_system temporary schema initialization.

Verification

Future PR in the stack has a more in-depth test suite for testing cleanup, multiple generations, as well as our current tests for temporary objects
I tested nightlies on a branch with all commits combined https://buildkite.com/materialize/nightly/builds/17923. I've since added the ci-nightly label to the latest PR in this stack

Initially the design doc stated that we wanted to persist sessions in the Catalog. Through benchmarking, it was found to create majoir regressions on CPS, even with optimizations. Thus we keep mz_sessions as a builtin table but will continue to persist durable objects. We also change the requirement of temporary schemas being durable to being a followup given we don't need them to move builtin tables.

Generic catalog migration version bump. Copies everything and is intended to make the review easier.

- We add the ephemeral_owner_session property in durable Items to indicate if an object is temporary - We remove all ephemeral items in savepoint/write open of the catalog. We can't do this for readonly mode since readonly followers would remove it in memory but then eventually panic when it sees a retraction for it

- Temporary items now write real durable Item rows, marked with ephemeral_owner_session = the creating session's UUID - The duplicated TemporaryItem is consolidated - Replaces catalog entries' `From` implementation with durable_item since we now need access to the connection <-> session mapping. - We introduce two map state variables: ephemeral_owner_conns_by_uuid and ephemeral_owner_uuids_by_conn. Both serve to create a session UUID <-> Conneciton ID mapping. These are used for two purposes: - When resolving a temporary object, routing it to its temporary schema via the `connection ID <-> Temporary schema mapping` using`temporary_schemas` - When editing an object, as opposed to creating an object, to store the new object durably, we don't have access to the session UUID but have access to the connection ID. Thus we use these maps to grab it. - In a future commit, we'll be merging temporary_schemas with these map variables since all three have the same lifecycle and are all used together

- The lifecycle and purpose of our `ephemeral_*` map state variables are very similar to `temporary_schemas`. That is, all lazily initialize in-memory temporary item metadata to resolve temporary items. Thus we unify all three in a struct `TemporaryNamespaces` - Gets rid of extraneous lazy initialization of the mz_temp schema inside `apply_item_update` from before given the lazy initialization in the inner `insert_entry` is all we need - Gets rid of eager mz_system temporary schema initialization.

linear-code · 2026-08-11T12:31:42Z

SQL-150

SangJunBak · 2026-08-11T12:34:18Z

+            // durable catalog to allocate a new OID for every temporary schema. Instead, we give
+            // them all the same invalid OID. This matches the semantics of temporary schema
+            // `GlobalId`s which are all -1.
+            let oid = INVALID_OID;


The body here is copied from the deleted create_temporary_schema below

ggevay · 2026-08-11T16:50:22Z

+            // dead. Reclaim the temporary items they owned here, before
+            // anything else reads the catalog.
+            if mode != Mode::Readonly {
+                txn.remove_ephemeral_items();


The fencing rationale is sound, and #38159/#38160 now cover the single-writer crash paths well (kill -9 item reclamation, read-only non-reclamation, and the pre-existing shard leak). In today's architecture there is no leak path at all: every crash is followed by some process's writable open, which purges.

The open question is the multi-process endgame this series builds toward: with several serving envds, one of them crashing leaks its sessions' temporary items until the next deploy, since a peer crash triggers no fence and no writable open. The design doc's mz_sessions follow-up sketches a durable envd-heartbeat to identify rows left by dead processes. Is ephemeral-item reclamation intended to ride on that same mechanism? A sentence in the design doc's follow-up section would settle it.

I think in addition to a comment in the design doc a comment here about a necessary change for a multi-envd world would be helpful

mtabebe

LGTM, a few questions and a few comments.

I am happy with how much code this removed.

mtabebe · 2026-08-12T19:54:29Z


 Multi-envd cleanup is follow-up work. Once several envds can run concurrently, we will need a durable envd-heartbeat table so any envd can identify dead peers and drop temp items owned by their sessions.

+### Follow-up: Multi-writer `mz_sessions`


Thanks for writing this

mtabebe · 2026-08-12T19:54:39Z

-Two additions to the durable catalog.

-(1) Durable session records: a new `StateUpdateKind::Session { uuid, deploy_generation, connection_id, role_id, client_ip, connected_at }`. The envd owning a session writes this on connect and deletes it on graceful close. This lets `mz_sessions` become a catalog-derived MV, and gives GC the durable session inventory it needs.
+An ephemeral-owner field on items: `ephemeral_owner_session: Option<Uuid>` on `ItemValue` in `src/catalog-protos`. `None` means a normal durable item. `Some(uuid)` means a temp item, visible only to the session with that UUID.


Appreciate the update based on the code

mtabebe · 2026-08-12T19:59:18Z

                schema_unique_fn,
                schema_unique_fn,
            )?,
+            // Temporary items from different sessions may share a name in the


mtabebe · 2026-08-12T20:02:26Z

+            // dead. Reclaim the temporary items they owned here, before
+            // anything else reads the catalog.
+            if mode != Mode::Readonly {
+                txn.remove_ephemeral_items();


I think in addition to a comment in the design doc a comment here about a necessary change for a multi-envd world would be helpful

mtabebe · 2026-08-12T20:03:40Z

Wahooo 🥳 I love deleting special cases!

mtabebe · 2026-08-12T20:21:08Z

 using the batching oracle's existing mechanism to serve more callers per batch,
 or relaxing the isolation level for queries that opt in.
+
+### Session records in the durable catalog


Nice, thanks for updating

mtabebe · 2026-08-12T20:24:09Z

        let mut catalog_updates = Vec::new();

        for state_update in updates {
+            // Applying a temporary item whose owning session is not connected


I'm not sure I understand this... can you explain? (Not blocking, just trying to understand)

I agree it's a bit confusing without context 😅 The important part is "Applying a temporary item whose owning session is not connected must be a complete no-op", where we never want read-only environments to listen to temporary item updates from the durable catalog and apply it in memory. However in practice, I believe read-only environments never actually hit this filter. The reason being read only envs open the catalog in savepoint mode and in persist.rs, we make sure we do tx.remove_ephemeral_items that effectively filters all temp items before it gets to this code path. But for readonly mode of the catalog, we don't call tx.remove_ephemeral_items and thus need the filter here.

Maybe it's cleaner to simplify and just change this to:

// Do not apply temporary item updates from other processes since temporary items are scoped per session which must be in the same process

I think that makes sense

mtabebe · 2026-08-12T20:24:46Z

-    fn merge_item_updates(
+    // Temporary and non-temporary items sort together: the type groups order
+    // cross-type dependencies and the topological sorts order dependencies
+    // within a group, regardless of which items are temporary.


Cool, glad we can remove more code

mtabebe · 2026-08-12T20:25:53Z

                        )));
                    }
-                    let oid = tx.allocate_oid(&temporary_oids)?;
+                    // The durable owner of a temporary item is the uuid of


Anyway to extract this to a helper, I don't love having all the error handling in here, maybe it is just a taste thing though

mtabebe · 2026-08-12T20:30:43Z

+                .iter()
+                .map(JsonCompatible::convert)
+                .collect(),
+            ephemeral_owner_session: None,


Makes sense to backfill as none, but ... why do we need to do this since they weren't in the catalog in the first place?

I guess we get their state in memory and have to write it in?

I guess we get their state in memory and have to write it in?

Exactly! I think serde actually backfills the property automatically to None when we read it back into memory without this explicit backfill. However, the problem is when we write with value ephemeral_owner_session:None … Because the durable catalog doesn't actually have ephemeral_owner_session on the entry, an edit on the entry causes a negative multiplicity. A pretty dangerous source of bugs imo for catalog migrations.

A non-empty temporary schema at unregistration means drop_temp_items didn't clean up the session's items, which can leak shards. Surface the invariant violation via soft_panic_or_log instead of returning an error the caller can only log.

Reclaiming on writable open only covers the single-writer world, where every crash is followed by some process's writable open. With several serving envds, a peer crash triggers no fence, so its sessions' items would leak until the next deploy. Dead-peer reclamation rides the durable envd-heartbeat mechanism sketched for mz_sessions.

ggevay

LGTM from my side, but I suggest waiting for also @mtabebe's final review.

mtabebe

LGTM thanks Jun!

SangJunBak added 5 commits August 10, 2026 10:44

catalog: bump catalog version to 91

88dd791

Generic catalog migration version bump. Copies everything and is intended to make the review easier.

SangJunBak changed the title ~~design: update durable temporary objects design doc~~ Sql-150: Temporary objects to the catalog Aug 11, 2026

SangJunBak requested a review from mtabebe August 11, 2026 12:31

SangJunBak commented Aug 11, 2026

View reviewed changes

SangJunBak marked this pull request as ready for review August 11, 2026 12:35

SangJunBak requested review from a team as code owners August 11, 2026 12:35

ggevay reviewed Aug 11, 2026

View reviewed changes

mtabebe reviewed Aug 12, 2026

View reviewed changes

SangJunBak added 5 commits August 12, 2026 23:05

adapter: simplify the nonlocal ephemeral item filter comment

4c024af

adapter: extract temporary item owner session resolution to a helper

5ec5c57

catalog: document why the v91 backfill rewrites item records

437a326

ggevay approved these changes Aug 13, 2026

View reviewed changes

SangJunBak requested a review from mtabebe August 13, 2026 15:22

mtabebe approved these changes Aug 13, 2026

View reviewed changes

SangJunBak merged commit f3b4f3f into main Aug 13, 2026
85 checks passed

SangJunBak deleted the jun/move-temp-to-catalog-split branch August 13, 2026 21:17


		Multi-envd cleanup is follow-up work. Once several envds can run concurrently, we will need a durable envd-heartbeat table so any envd can identify dead peers and drop temp items owned by their sessions.

		### Follow-up: Multi-writer `mz_sessions`

Uh oh!

Conversation

SangJunBak commented Aug 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Description

1. design: update durable temporary objects design doc (de137ed708c8)

2. catalog: bump catalog version to 91 (88dd791836fd)

3. catalog: add ephemeral_owner_session to durable items (fa94bb99cb93)

4. catalog: make temporary items durable in the catalog shard (fd2f72fddd15)

5. adapter: unify temporary schemas with ephemeral owner registration (7b83fefba86e)

Verification

Uh oh!

linear-code Bot commented Aug 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtabebe left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SangJunBak Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ggevay left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtabebe left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

SangJunBak commented Aug 10, 2026 •

edited

Loading

1. design: update durable temporary objects design doc (`de137ed708c8`)

2. catalog: bump catalog version to 91 (`88dd791836fd`)

3. catalog: add ephemeral_owner_session to durable items (`fa94bb99cb93`)

4. catalog: make temporary items durable in the catalog shard (`fd2f72fddd15`)

5. adapter: unify temporary schemas with ephemeral owner registration (`7b83fefba86e`)

SangJunBak Aug 12, 2026 •

edited

Loading

ggevay left a comment •

edited

Loading