Skip to content

Commit c84b03d

Browse files
d-csd-cs
committed
fix(webapp): hold the active mint-shard list in the database, not the environment
A rolling deploy takes hours, so two pods run different values of RUN_OPS_MINT_SHARDS at the same time. The grace window is sized in seconds, so new pods left it long before old pods were gone: for the rest of the rollout the two placed the same environment on different shards. That is the divergence the grace exists to close. The environment variable is now a ceiling that changes only by deploy. It says which shard keys this deployment can mint into. The live list moves to the control-plane database as runOpsMintShardSet, so every pod reads one shared value whatever config generation it is running. Resolution intersects the two, so a stored key this deployment cannot route is never minted into. RUN_OPS_MINT_SHARDS_PREV and RUN_OPS_MINT_SHARDS_FLIPPED_AT are gone. An environment variable cannot record its own flip time, and an operator cannot know a rollout's end in advance. The stamp is now written server-side against the control-plane clock, under an advisory lock, on a genuine change. Stamping generalizes to N graced flag groups in one transaction under one lock, covering the existing mint-kind trio and the new list. That closes a hole on the global admin flags page, which wrote any catalog key with a bare upsert: a graced key could be set with no stamp, or swept away by a save that omitted it. applyGlobalMintKindFlip stays as a thin wrapper so its route and its test keep working unchanged. Operational rule this creates: every change to RUN_OPS_MINT_SHARDS must land across the whole fleet before the flag selects a key it adds. Routing before minting, which is how the shard topology is already gated.
1 parent f2d9670 commit c84b03d

10 files changed

Lines changed: 623 additions & 138 deletions

‎apps/webapp/app/env.server.ts‎

Lines changed: 5 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2016,14 +2016,12 @@ const EnvironmentSchema = z
20162016
// (stale or fresh) resolves to the same kind for the whole window. See mintFlipGrace.ts.
20172017
RUN_OPS_MINT_FLIP_GRACE_MS: z.coerce.number().int().default(90_000),
20182018

2019-
// Gen-2 mint shards — CSV of single-char [a-z0-9] keys eligible for ROOT minting. Unset or
2020-
// empty means no gen-2 minting, which is today's behaviour. Validated at boot: an invalid
2021-
// key would mint an id that cannot be routed. _PREV + _FLIPPED_AT stamp a set change so
2022-
// every process crosses the cutover together; set both, or the grace never applies.
2023-
// Removing a key stops new roots on it and never stops routing it. See mintShardGrace.ts.
2019+
// Gen-2 mint shards — CSV of single-char [a-z0-9] keys this deployment can mint roots into.
2020+
// Unset or empty means no gen-2 minting, which is today's behaviour. Validated at boot: an
2021+
// invalid key would mint an id that cannot be routed. This is a CEILING, not the live list:
2022+
// it changes only by deploy, and the runOpsMintShardSet flag selects from it at runtime.
2023+
// A rolling deploy runs two values of this var at once, so it must never be the ramp lever.
20242024
RUN_OPS_MINT_SHARDS: shardCsvString(),
2025-
RUN_OPS_MINT_SHARDS_PREV: shardCsvString(),
2026-
RUN_OPS_MINT_SHARDS_FLIPPED_AT: z.string().datetime().optional(),
20272025

20282026
// Session replication (Postgres → ClickHouse sessions_v1). Shares Redis
20292027
// with the runs replicator for leader locking but has its own slot and

‎apps/webapp/app/routes/admin.api.v1.feature-flags.ts‎

Lines changed: 6 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ import { json } from "@remix-run/server-runtime";
33
import { prisma } from "~/db.server";
44
import { env } from "~/env.server";
55
import { requireAdminApiRequest } from "~/services/personalAccessToken.server";
6-
import { applyGlobalMintKindFlip, makeSetMultipleFlags } from "~/v3/featureFlags.server";
6+
import { applyGlobalGracedFlips, makeSetMultipleFlags } from "~/v3/featureFlags.server";
77
import { validatePartialFeatureFlags } from "~/v3/featureFlags";
88

99
export async function action({ request }: ActionFunctionArgs) {
@@ -29,14 +29,15 @@ export async function action({ request }: ActionFunctionArgs) {
2929
const {
3030
runOpsMintKindPrev: _ignoredPrev,
3131
runOpsMintKindFlippedAt: _ignoredFlippedAt,
32+
runOpsMintShardSetPrev: _ignoredSetPrev,
33+
runOpsMintShardSetFlippedAt: _ignoredSetFlippedAt,
3234
...requestedFlags
3335
} = validationResult.data;
3436

35-
// A global mint-kind flip stamps its grace window under a lock (applyGlobalMintKindFlip);
36-
// any other flag save writes directly.
37+
// A change to a graced group stamps its window under a lock; any other save writes directly.
3738
const updatedFlags =
38-
requestedFlags.runOpsMintKind !== undefined
39-
? await applyGlobalMintKindFlip(prisma, requestedFlags, env.RUN_OPS_MINT_FLIP_GRACE_MS)
39+
requestedFlags.runOpsMintKind !== undefined || requestedFlags.runOpsMintShardSet !== undefined
40+
? await applyGlobalGracedFlips(prisma, requestedFlags, env.RUN_OPS_MINT_FLIP_GRACE_MS)
4041
: await makeSetMultipleFlags(prisma)(requestedFlags);
4142

4243
return json({

‎apps/webapp/app/routes/admin.feature-flags.tsx‎

Lines changed: 11 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,18 @@ import { json } from "@remix-run/server-runtime";
55
import { typedjson, useTypedLoaderData } from "remix-typedjson";
66
import { z } from "zod";
77
import { LockClosedIcon } from "@heroicons/react/20/solid";
8-
import { boundedIn, prisma } from "~/db.server";
8+
import { prisma } from "~/db.server";
99
import { env } from "~/env.server";
1010
import { dashboardAction, dashboardLoader } from "~/services/routeBuilders/dashboardBuilder";
1111
import {
1212
FEATURE_FLAG,
1313
GLOBAL_LOCKED_FLAGS,
14+
type FeatureFlagKey,
1415
type FlagControlType,
1516
getAllFlagControlTypes,
1617
validatePartialFeatureFlags,
1718
} from "~/v3/featureFlags";
18-
import { flags as getGlobalFlags } from "~/v3/featureFlags.server";
19+
import { flags as getGlobalFlags, replaceGlobalFeatureFlags } from "~/v3/featureFlags.server";
1920
import { featuresForRequest } from "~/features.server";
2021
import { Button } from "~/components/primitives/Buttons";
2122
import { Callout } from "~/components/primitives/Callout";
@@ -116,39 +117,15 @@ export const action = dashboardAction(
116117
);
117118
}
118119

119-
const validatedFlags = validationResult.data as Record<string, unknown>;
120-
const controlTypes = getAllFlagControlTypes();
121-
const catalogKeys = Object.keys(controlTypes);
122-
123-
const keysToDelete: string[] = [];
124-
const upsertOps: ReturnType<typeof prisma.featureFlag.upsert>[] = [];
125-
126-
for (const key of catalogKeys) {
127-
if (key in validatedFlags) {
128-
upsertOps.push(
129-
prisma.featureFlag.upsert({
130-
where: { key },
131-
create: { key, value: validatedFlags[key] as any },
132-
update: { value: validatedFlags[key] as any },
133-
})
134-
);
135-
} else {
136-
// On cloud, never delete locked flags (they're not in the payload
137-
// because the UI doesn't include them). Locally, delete everything
138-
// the user didn't include - full control.
139-
const isProtected = isManagedCloud && GLOBAL_LOCKED_FLAGS.includes(key);
140-
if (!isProtected) {
141-
keysToDelete.push(key);
142-
}
143-
}
144-
}
120+
const catalogKeys = Object.keys(getAllFlagControlTypes()) as FeatureFlagKey[];
145121

146-
await prisma.$transaction([
147-
...upsertOps,
148-
...(keysToDelete.length > 0
149-
? [prisma.featureFlag.deleteMany({ where: { key: { in: boundedIn(keysToDelete) } } })]
150-
: []),
151-
]);
122+
await replaceGlobalFeatureFlags(prisma, {
123+
requestedFlags: validationResult.data,
124+
catalogKeys,
125+
// On cloud, never delete locked flags (the UI omits them). Locally, full control.
126+
isProtected: (key) => isManagedCloud && GLOBAL_LOCKED_FLAGS.includes(key),
127+
graceMs: env.RUN_OPS_MINT_FLIP_GRACE_MS,
128+
});
152129

153130
return json({ success: true });
154131
}

‎apps/webapp/app/v3/featureFlags.server.ts‎

Lines changed: 105 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,8 @@ import {
88
FeatureFlagCatalog,
99
} from "~/v3/featureFlags";
1010
import { stampMintKindFlip } from "~/v3/runOpsMigration/mintFlipGrace";
11+
import { stampMintShardSetFlip } from "~/v3/runOpsMigration/mintShardGrace";
12+
import { boundedIn } from "~/db.server";
1113

1214
export type FlagsOptions<T extends FeatureFlagKey> = {
1315
key: T;
@@ -182,41 +184,125 @@ export function makeSetMultipleFlags(_prisma: PrismaClientOrTransaction = prisma
182184
// Read -> stamp -> write the global mint-kind grace metadata in one transaction. The three
183185
// FeatureFlag rows may not exist yet, so a row FOR UPDATE can't lock them; an advisory xact lock
184186
// serializes concurrent global flips so one can't clobber another's grace stamp (mirrors per-org).
185-
export async function applyGlobalMintKindFlip(
187+
// Every group of global flags whose value carries its own grace stamp. One transaction and one
188+
// lock cover all of them, so a save that flips two groups can never stamp one and lose the other.
189+
const GRACED_GLOBAL_GROUPS = [
190+
{
191+
keys: [
192+
FEATURE_FLAG.runOpsMintKind,
193+
FEATURE_FLAG.runOpsMintKindPrev,
194+
FEATURE_FLAG.runOpsMintKindFlippedAt,
195+
] as FeatureFlagKey[],
196+
stamp: stampMintKindFlip,
197+
},
198+
{
199+
keys: [
200+
FEATURE_FLAG.runOpsMintShardSet,
201+
FEATURE_FLAG.runOpsMintShardSetPrev,
202+
FEATURE_FLAG.runOpsMintShardSetFlippedAt,
203+
] as FeatureFlagKey[],
204+
stamp: stampMintShardSetFlip,
205+
},
206+
] as const;
207+
208+
// Keys the graced path owns. They never take a bare upsert and never enter the replace sweep,
209+
// because a server-computed stamp must not be written from a request body nor swept away.
210+
const GRACED_GLOBAL_KEYS: FeatureFlagKey[] = GRACED_GLOBAL_GROUPS.flatMap((g) => g.keys);
211+
212+
export async function applyGlobalGracedFlips(
186213
client: PrismaClient,
187214
requestedFlags: Partial<z.infer<typeof FeatureFlagCatalogSchema>>,
188215
graceMs: number
189216
): Promise<{ key: string; value: any }[]> {
190217
return client.$transaction(async (tx) => {
191-
await tx.$executeRaw`SELECT pg_advisory_xact_lock(hashtext('runops-global-mint-kind-flip'))`;
218+
await tx.$executeRaw`SELECT pg_advisory_xact_lock(hashtext('runops-global-graced-flag-flip'))`;
192219

193220
const existingRows = await tx.featureFlag.findMany({
194-
where: {
195-
key: {
196-
in: [
197-
FEATURE_FLAG.runOpsMintKind,
198-
FEATURE_FLAG.runOpsMintKindPrev,
199-
FEATURE_FLAG.runOpsMintKindFlippedAt,
200-
],
201-
},
202-
},
221+
where: { key: { in: GRACED_GLOBAL_KEYS } },
203222
select: { key: true, value: true },
204223
});
205224
const existingGlobal: Record<string, unknown> = {};
206225
for (const row of existingRows) {
207226
existingGlobal[row.key] = row.value;
208227
}
209228

210-
// Anchor the cutover to the control-plane DB clock, not this process's wall clock.
229+
// Anchor the cutover to the control-plane DB clock, not this process's wall clock. A rolling
230+
// deploy spans hours, so every pod must date the window against one shared clock.
211231
const [{ now }] = await tx.$queryRaw<{ now: Date }[]>`SELECT now() AS now`;
212232

213-
const stamped = stampMintKindFlip(
214-
existingGlobal,
215-
{ ...requestedFlags },
216-
now.getTime(),
217-
graceMs
218-
) as Partial<z.infer<typeof FeatureFlagCatalogSchema>>;
233+
let stamped: Record<string, unknown> = { ...requestedFlags };
234+
for (const group of GRACED_GLOBAL_GROUPS) {
235+
stamped = group.stamp(existingGlobal, stamped, now.getTime(), graceMs);
236+
}
219237

220-
return makeSetMultipleFlags(tx)(stamped);
238+
return makeSetMultipleFlags(tx)(stamped as Partial<z.infer<typeof FeatureFlagCatalogSchema>>);
221239
});
222240
}
241+
242+
/** @deprecated Prefer applyGlobalGracedFlips, which stamps every graced group in one lock. */
243+
export async function applyGlobalMintKindFlip(
244+
client: PrismaClient,
245+
requestedFlags: Partial<z.infer<typeof FeatureFlagCatalogSchema>>,
246+
graceMs: number
247+
): Promise<{ key: string; value: any }[]> {
248+
return applyGlobalGracedFlips(client, requestedFlags, graceMs);
249+
}
250+
251+
// Replace-semantics write for the global admin flags page: upsert submitted catalog flags, delete
252+
// omitted ones unless protected, and route any graced group through the stamped path.
253+
export async function replaceGlobalFeatureFlags(
254+
client: PrismaClient,
255+
params: {
256+
requestedFlags: Partial<z.infer<typeof FeatureFlagCatalogSchema>>;
257+
catalogKeys: FeatureFlagKey[];
258+
isProtected: (key: FeatureFlagKey) => boolean;
259+
graceMs: number;
260+
}
261+
): Promise<void> {
262+
// Derived stamp fields are computed server-side; never trust them from the body.
263+
const requestedFlags: Record<string, unknown> = { ...params.requestedFlags };
264+
for (const group of GRACED_GLOBAL_GROUPS) {
265+
for (const derived of group.keys.slice(1)) {
266+
delete requestedFlags[derived];
267+
}
268+
}
269+
270+
const touchesGracedGroup = GRACED_GLOBAL_GROUPS.some(
271+
(group) => requestedFlags[group.keys[0]] !== undefined
272+
);
273+
if (touchesGracedGroup) {
274+
await applyGlobalGracedFlips(
275+
client,
276+
requestedFlags as Partial<z.infer<typeof FeatureFlagCatalogSchema>>,
277+
params.graceMs
278+
);
279+
}
280+
281+
const upsertOps: ReturnType<typeof client.featureFlag.upsert>[] = [];
282+
const keysToDelete: string[] = [];
283+
284+
for (const key of params.catalogKeys) {
285+
if (GRACED_GLOBAL_KEYS.includes(key)) {
286+
continue;
287+
}
288+
if (key in requestedFlags) {
289+
const value = requestedFlags[key];
290+
upsertOps.push(
291+
client.featureFlag.upsert({
292+
where: { key },
293+
create: { key, value: value as any },
294+
update: { value: value as any },
295+
})
296+
);
297+
} else if (!params.isProtected(key)) {
298+
keysToDelete.push(key);
299+
}
300+
}
301+
302+
await client.$transaction([
303+
...upsertOps,
304+
...(keysToDelete.length > 0
305+
? [client.featureFlag.deleteMany({ where: { key: { in: boundedIn(keysToDelete) } } })]
306+
: []),
307+
]);
308+
}

‎apps/webapp/app/v3/featureFlags.ts‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,11 @@ export const FEATURE_FLAG = {
2929
// Gen-2 mint shard pins, read from the org override blob only. See runOpsMintShard.server.ts.
3030
runOpsMintShard: "runOpsMintShard",
3131
runOpsMintShardEnvPins: "runOpsMintShardEnvPins",
32+
// The active mint-shard list, global only. Lives here rather than in the environment because a
33+
// rolling deploy runs two environment values at once for hours. See mintShardGrace.ts.
34+
runOpsMintShardSet: "runOpsMintShardSet",
35+
runOpsMintShardSetPrev: "runOpsMintShardSetPrev",
36+
runOpsMintShardSetFlippedAt: "runOpsMintShardSetFlippedAt",
3237
queueMetricsUiEnabled: "queueMetricsUiEnabled",
3338
// Per-organization rollout for creating additional environment API keys.
3439
additionalApiKeysEnabled: "additionalApiKeysEnabled",
@@ -118,6 +123,21 @@ export const FeatureFlagCatalog = {
118123
}
119124
}
120125
}),
126+
// CSV of the shard keys eligible for root minting right now, bounded by RUN_OPS_MINT_SHARDS.
127+
// Empty means no gen-2 minting. Reserved keys are rejected: "new" already means gen-1.
128+
[FEATURE_FLAG.runOpsMintShardSet]: z.string().refine(
129+
(v) =>
130+
v
131+
.split(",")
132+
.map((s) => s.trim())
133+
.filter(Boolean)
134+
.every((k) => /^[a-z0-9]$/.test(k)),
135+
"must be a CSV of single [a-z0-9] chars"
136+
),
137+
// Grace stamp: the previously-effective list and the flip time, written by
138+
// stampMintShardSetFlip on a genuine change. Display-only (see ORG_LOCKED_FLAGS).
139+
[FEATURE_FLAG.runOpsMintShardSetPrev]: z.string(),
140+
[FEATURE_FLAG.runOpsMintShardSetFlippedAt]: z.string().datetime(),
121141
// Per-org access to the Queue Metrics dashboard UI (view only; emission is global and
122142
// separate). Off unless enabled for the org.
123143
[FEATURE_FLAG.queueMetricsUiEnabled]: z.coerce.boolean(),
@@ -151,6 +171,10 @@ export const ORG_LOCKED_FLAGS: FeatureFlagKey[] = [
151171
// System-wide only — orgs must not be able to override these kill switches.
152172
FEATURE_FLAG.additionalApiKeyIssuanceEnabled,
153173
FEATURE_FLAG.additionalApiKeyLookupEnabled,
174+
// The active mint-shard list is deployment-wide; only the pins are per-org.
175+
FEATURE_FLAG.runOpsMintShardSet,
176+
FEATURE_FLAG.runOpsMintShardSetPrev,
177+
FEATURE_FLAG.runOpsMintShardSetFlippedAt,
154178
];
155179

156180
// Create a Zod schema from the existing catalog

0 commit comments

Comments
 (0)