diff --git a/SUMMARY.md b/SUMMARY.md index 7ac4f08..041b7b3 100644 --- a/SUMMARY.md +++ b/SUMMARY.md @@ -49,6 +49,7 @@ * [Cloud Events Support](/contents/CloudEventsSupport.md) * [CloudEvents Reference](/contents/CloudEventsReference.md) * [Claim Check](/contents/ClaimCheck.md) + * [Handle a Large Message](/contents/HandlingLargeMessages.md) * [S3 Luggage Store](/contents/S3LuggageStore.md) * [Compression](/contents/Compression.md) * [Dynamic Message Deserialization](/contents/DynamicMessageDeserialization.md) @@ -72,6 +73,7 @@ * [PostgreSQL Broker Trade-Offs](/contents/PostgreSQLBrokerTradeOffs.md) * [PostgreSQL for Transport and Outbox](/contents/PostgreSQLTransportAndOutbox.md) * [MSSQL Message Broker](/contents/MSSQLMessageBroker.md) + * [MSSQL for Transport, Outbox and Inbox](/contents/MSSQLTransportInboxAndOutbox.md) * [GCP Pub/Sub Configuration](/contents/GcpPubSubConfiguration.md) * [RocketMQ Configuration](/contents/RocketMQConfiguration.md) * [MQTT Configuration](/contents/MQTTConfiguration.md) diff --git a/contents/ClaimCheck.md b/contents/ClaimCheck.md index 49ff8e1..26b1727 100644 --- a/contents/ClaimCheck.md +++ b/contents/ClaimCheck.md @@ -21,11 +21,16 @@ We provide a **WrapWithAttribute** of **ClaimCheck** that will use the **ClaimCh In the following example we add the **ClaimCheck** attribute to the *Message Mapper* with a trigger at 256Kb -``` csharp -[ClaimCheck(step:0, thresholdInKb: 256)] -public Message MapToMessage(GreetingEvent request) +```csharp +using System.Text.Json; +using Paramore.Brighter; +using Paramore.Brighter.JsonConverters; +using Paramore.Brighter.Transforms.Attributes; + +[ClaimCheck(step: 0, thresholdInKb: 256)] +public Message MapToMessage(GreetingEvent request, Publication publication) { - var header = new MessageHeader(messageId: request.Id, topic: typeof(GreetingEvent).FullName.ToValidSNSTopicName(), messageType: MessageType.MT_EVENT); + var header = new MessageHeader(messageId: request.Id, topic: publication.Topic!, messageType: MessageType.MT_EVENT); var body = new MessageBody(JsonSerializer.Serialize(request, JsonSerialisationOptions.Options)); var message = new Message(header, body); return message; @@ -34,15 +39,19 @@ public Message MapToMessage(GreetingEvent request) We provide a matching **UnwrapWithAttribute** of **RetrieveClaim** that will use the **ClaimCheckTransformer** to download the body of your **Message** from a luggage store and replace the existing body (likely a claim check reference) with the downloaded content. -``` csharp -[RetrieveClaim(0, retain:false)] +```csharp +using System.Text.Json; +using Paramore.Brighter; +using Paramore.Brighter.JsonConverters; +using Paramore.Brighter.Transforms.Attributes; + +[RetrieveClaim(step: 0, retain: false)] public GreetingEvent MapToRequest(Message message) { var greetingCommand = JsonSerializer.Deserialize(message.Body.Value, JsonSerialisationOptions.Options); - return greetingCommand; + return greetingCommand!; } - ``` An optional parameter 'retain' determines if we keep the body in storage after it is retrieved or delete it. The default is to delete it. @@ -54,22 +63,49 @@ The outcome of these attributes is that the uploading of the body to the *luggag The *luggage store* is where we store the body of the message for later retrieval. We provide implementations of the Luggage Store interface for popular distributed stores, but you can implement the interface for any that we do not provide. ```csharp +using System.IO; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter.Observability; - public interface IAmAStorageProviderAsync - { - Task DeleteAsync(string claimCheck, CancellationToken cancellationToken); - Task DownloadAsync(string claimCheck, CancellationToken cancellationToken); - Task HasClaimAsync(string claimCheck, CancellationToken cancellationToken); - Task UploadAsync(Stream stream, CancellationToken cancellationToken); - } - +public interface IAmAStorageProviderAsync +{ + IAmABrighterTracer? Tracer { get; set; } + Task EnsureStoreExistsAsync(CancellationToken cancellationToken = default); + Task DeleteAsync(string claimCheck, CancellationToken cancellationToken = default); + Task RetrieveAsync(string claimCheck, CancellationToken cancellationToken = default); + Task HasClaimAsync(string claimCheck, CancellationToken cancellationToken = default); + Task StoreAsync(Stream stream, CancellationToken cancellationToken = default); +} ``` -* DeleteAsync: Deletes a item from the store -* DownloadAsync: Creates a stream for a download from the store -* HasClaimAsync: Does the claim check exist in the store -* UploadAsync: Uploads a stream to the store and returns a claim, an identifier that can later be used to delete, download or check for the existence of the file uploaded to the store. +* `Tracer`: the tracer used to capture telemetry. You do not set this — the registration does +* `EnsureStoreExistsAsync`: creates the store, or checks that it is there, according to `StorageOptions.Strategy` +* `DeleteAsync`: deletes an item from the store +* `RetrieveAsync`: creates a stream for a download from the store +* `HasClaimAsync`: does the claim check exist in the store +* `StoreAsync`: puts a stream into the store and returns a claim, an identifier that can later be used to delete, retrieve or check for the existence of what was stored + +There is a synchronous `IAmAStorageProvider` alongside it carrying the same operations, and +**every store implements both** — which is what registration requires. + +## Luggage Store Implementations + +Seven implementations ship with V10: + +| Store | Package | +|---|---| +| `S3LuggageStore` | `Paramore.Brighter.Transformers.AWS`, and `Paramore.Brighter.Transformers.AWS.V4` for AWS SDK v4 | +| `AzureBlobLuggageStore` | `Paramore.Brighter.Transformers.Azure` | +| `GcsLuggageStore` | `Paramore.Brighter.Transformers.Gcp` | +| `MongoDbLuggageStore` | `Paramore.Brighter.Transformers.MongoGridFS` | +| `FileSystemStorageProvider` | `Paramore.Brighter` (core) | +| `InMemoryStorageProvider` | `Paramore.Brighter` (core) | +| `NullLuggageStore` | `Paramore.Brighter` (core) — the default, and every method throws | -We provide the following implementations of **IAmAStorageProviderAsync: +**Registering one of them is a step of its own**, and `AddBrighter` leaves you with the null +store until you do. See +[Put a Large Payload Behind a Claim Check](/contents/HandlingLargeMessages.md), which covers the +registration, the threshold and how to tell whether the payload really left. -* [S3LuggageStore](/contents/S3LuggageStore.md) +* [S3 Luggage Store](/contents/S3LuggageStore.md) diff --git a/contents/HandlingLargeMessages.md b/contents/HandlingLargeMessages.md new file mode 100644 index 0000000..425df25 --- /dev/null +++ b/contents/HandlingLargeMessages.md @@ -0,0 +1,295 @@ +--- +description: "When a message body outgrows what your transport will carry, a claim check stores the payload elsewhere and sends a token in its place." +layout: + description: + visible: false +--- + +# Put a Large Payload Behind a Claim Check + +> **How-to** · Applies to **Brighter V10** · Prerequisites: [Claim Check](/contents/ClaimCheck.md), [Message Mappers](/contents/MessageMappers.md) + +When a message body outgrows what your transport will carry, a claim check stores the payload +elsewhere and sends a token in its place. + +[Claim Check](/contents/ClaimCheck.md) explains the pattern and the two attributes that +implement it. This guide is the wiring: which store to pick, how to register it, where the +threshold is measured, and how to tell whether the payload really left. + +**Registering the store is the step people miss.** `AddBrighter` finishes by registering a +`NullLuggageStore`, so a mapper carrying `[ClaimCheck]` with no store configured compiles, +starts, and throws `NotImplementedException` the first time it maps a message. Step 3 is that +registration. + +## Step 1: Find Your Transport's Message Size Limit + +The limit is your broker's, not Brighter's. **Brighter neither enforces a maximum nor reports +one** — an oversized message is rejected by the transport, at publish time, with whatever that +broker's client throws. + +These are the published limits for the transports Brighter ships against. Check them against +your broker's own documentation and your own plan or tier, because several are configurable and +one of them changes with what you pay: + +| Transport | Limit | Configurable? | +|---|---|---| +| AWS SQS and SNS | 256 KiB | No, for standard delivery | +| Azure Service Bus | 256 KB Standard, 100 MB Premium | By tier | +| Kafka | 1 MiB by default (`message.max.bytes`) | Yes, broker and topic | +| RabbitMQ | 128 MiB default cap since 3.8 (`max-message-size`) | Yes | +| Redis | 512 MB per value | Effectively no | +| PostgreSQL, MSSQL, MySQL | governed by the column type | Yes, by schema | + +**One thing you cannot set through Brighter, despite appearances.** Azure Service Bus's +administration wrapper takes a `maxMessageSizeInKilobytes` argument on `CreateQueueAsync` and +`CreateTopicAsync` (`AzureServiceBusWrappers/AdministrationClientWrapper.cs:69`, `:140`), but +**nothing in the product ever supplies it** — there is no call site outside the wrapper and its +interface. Set the entity's maximum in Azure, not in your Brighter configuration. + +**The size that counts is the serialized body.** The claim check compares +`message.Body.Memory.Length` — the bytes your mapper produced — against the threshold. Headers +are not included in that comparison but *are* carried by the broker, so a body sitting just +under a hard transport limit can still be rejected once its headers are added. Leave room. + +## Step 2: Choose a Luggage Store + +Seven implementations of `IAmAStorageProvider` and `IAmAStorageProviderAsync` ship with V10. +Every store implements both interfaces, which is what the registration in step 3 requires: + +| Store | Package | Options type | +|---|---|---| +| `S3LuggageStore` | `Paramore.Brighter.Transformers.AWS` | `S3LuggageOptions(AWSS3Connection connection, string bucketName)` | +| `S3LuggageStore` | `Paramore.Brighter.Transformers.AWS.V4` | the same, built against AWS SDK v4 | +| `AzureBlobLuggageStore` | `Paramore.Brighter.Transformers.Azure` | `AzureBlobLuggageOptions` — `ContainerUri` and `Credential`, or `ConnectionString` and `ContainerName` | +| `GcsLuggageStore` | `Paramore.Brighter.Transformers.Gcp` | `GcsLuggageOptions` — `ProjectId`, `Bucket`, optional `Credential` | +| `MongoDbLuggageStore` | `Paramore.Brighter.Transformers.MongoGridFS` | `MongoDbLuggageStoreOptions(string connectionString, string database, string bucketName)` | +| `FileSystemStorageProvider` | `Paramore.Brighter` (core) | `FileSystemOptions(string path)` | +| `InMemoryStorageProvider` | `Paramore.Brighter` (core) | none — a default constructor | + +**The two AWS packages are one store with two SDK generations**, not two features. Take +`Paramore.Brighter.Transformers.AWS.V4` if the rest of your application is on AWS SDK v4, and +the unsuffixed package otherwise. Referencing both puts two types called `S3LuggageStore` in +scope and you will be qualifying namespaces for the rest of the file. + +**`InMemoryStorageProvider` is for tests**, and it is genuinely useful there — step 6 uses it. +It holds the luggage in the process that stored it, so a separate consumer process finds +nothing. + +**There is also a `NullLuggageStore`, and you never register it deliberately.** It is what +`AddBrighter` leaves you with, and every one of its methods throws. See step 3. + +**Every store shares one option**, from the `StorageOptions` base: `Strategy`, which is +`StorageStrategy.CreateIfMissing` by default and `StorageStrategy.Validate` if you would rather +the store be provisioned by your infrastructure and Brighter merely check that it is there. + +## Step 3: Register the Luggage Store + +`UseExternalLuggageStore` extends `IBrighterBuilder`, so it chains off +`AddBrighter`. Three overloads, differing only in who constructs the store: + +```csharp +using Amazon; +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter.Extensions.DependencyInjection; +using Paramore.Brighter.Transformers.AWS; +using Paramore.Brighter.Transforms.Storage; + +// 1. Brighter constructs it — the store needs a public constructor the container can call +services.AddBrighter() + .UseExternalLuggageStore(); + +// 2. You construct it, and hand over the instance +services.AddBrighter() + .UseExternalLuggageStore(new FileSystemStorageProvider( + new FileSystemOptions("/var/brighter/luggage"))); + +// 3. A factory, when the store needs something from the container +services.AddBrighter() + .UseExternalLuggageStore(provider => new S3LuggageStore( + new S3LuggageOptions( + new AWSS3Connection(awsCredentials, RegionEndpoint.EUWest1), + bucketName: "my-brighter-luggage") + { + HttpClientFactory = provider.GetRequiredService() + })); +``` + +**Register after `AddBrighter`, and yours wins.** `AddBrighter` ends with +`UseExternalLuggageStore()` (`ServiceCollectionExtensions.cs:222-223`), so a +store is always registered. The overloads use `AddSingleton` rather than `TryAdd`, and +`GetRequiredService` resolves the **last** registration, so calling +`UseExternalLuggageStore` after `AddBrighter` displaces the null store. This is the opposite of +`AddBrighterDefault`'s "register yours first" rule — the two use different registration methods +and the order that works for one is the order that fails for the other. + +**Skip this step and every method of the null store throws.** Resolving +`IAmAStorageProvider` from a container configured with `AddBrighter()` and nothing else gives +you: + +```text +System.NotImplementedException: This is a null store, you must register a real store after Brighter + at Paramore.Brighter.Transforms.Storage.NullLuggageStore.EnsureStoreExists() +``` + +**Note where that throw comes from — it is on resolution, not on threshold.** The registration +wraps your store in a factory that sets its `Tracer` and calls `EnsureStoreExists()` *before +handing it out*, so the failure lands the first time anything resolves the store, which is when +the transform pipeline for a `[ClaimCheck]` mapper is first built. Whatever size that first +message happens to be, a small one buys you no reprieve. + +The same eager `EnsureStoreExists()` is what provisions a *real* store, so a missing bucket or +container surfaces at the same moment, under whatever `StorageOptions.Strategy` you chose. + +## Step 4: Attach the Claim Check to Your Mapper + +The claim check is transform middleware, so it attaches to a **message mapper** — one attribute +on the way out, one on the way back: + +```csharp +using System.Text.Json; +using Paramore.Brighter; +using Paramore.Brighter.JsonConverters; +using Paramore.Brighter.Transforms.Attributes; + +public class LargeOrderMessageMapper : IAmAMessageMapper +{ + public IRequestContext? Context { get; set; } + + [ClaimCheck(step: 0, thresholdInKb: 200)] + public Message MapToMessage(LargeOrderPlaced request, Publication publication) + { + var header = new MessageHeader( + messageId: request.Id, + topic: publication.Topic!, + messageType: MessageType.MT_EVENT); + + var body = new MessageBody( + JsonSerializer.Serialize(request, JsonSerialisationOptions.Options)); + + return new Message(header, body); + } + + [RetrieveClaim(step: 0, retain: false)] + public LargeOrderPlaced MapToRequest(Message message) + { + return JsonSerializer.Deserialize( + message.Body.Value, JsonSerialisationOptions.Options)!; + } +} +``` + +**You need a mapper of your own to attach these to.** An attribute goes on a method of a type +you own, and you do not own `JsonMessageMapper` — so a claim check means writing the +mapper out, as above, rather than leaning on +[default message mappers](/contents/DefaultMessageMappers.md). That is the same constraint +[Message Transforms](/contents/MessageTransforms.md) states for any transform of your own. + +**`retain: false` is the default and it deletes the luggage** once the receiver has read it. +Set `retain: true` when more than one consumer reads the same message, or the second reader +will find the claim check pointing at nothing. + +**Register the mapper**, or none of this runs: + +```csharp +using Paramore.Brighter.Extensions.DependencyInjection; +using Paramore.Brighter.Transforms.Storage; + +services.AddBrighter() + .UseExternalLuggageStore() + .AutoFromAssemblies([typeof(LargeOrderMessageMapper).Assembly]); +``` + +## Step 5: Choose a Threshold + +`thresholdInKb` is compared as `thresholdInKb * 1024` bytes against the serialized body, and the +comparison is `body.Length < threshold` — so a body **exactly** on the threshold is checked into +the store, not sent inline. + +**`thresholdInKb: 0` checks every message**, because no body is shorter than zero bytes. That is +occasionally what you want — uniform behaviour is easier to reason about than a size-dependent +branch — but it means a round trip to your store for a 200-byte event. + +Pick a threshold **below your transport's limit with room for headers**, not at it. On a +256 KiB transport, something like 200 KiB leaves the headers, the CloudEvents attributes and the +claim check itself somewhere to live. + +**What the message looks like once it is checked**: the body is replaced with the literal text +`Claim Check {id}`, the id goes into the header bag under `claim_check_header`, and +`MessageHeader.DataRef` is set to the same id. `DataRef` is the CloudEvents +[`dataref`](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/extensions/dataref.md) +extension, which is what makes the claim check readable by consumers that are not Brighter — +and it is a deliberate break from V9, which used the header bag alone. On the way back the +transformer reads the bag first and falls back to `DataRef`, so it understands both. + +## Step 6: Verify the Payload Went to the Store + +The visible symptom of a claim check that is not working is *nothing* — the message arrives, the +handler runs, and you never learn the body travelled inline. So assert on the store. + +`HasClaimAsync` answers the question directly, and `InMemoryStorageProvider` makes it a test +rather than an integration: + +```csharp +using System.Text.Json; +using System.Threading.Tasks; +using Paramore.Brighter; +using Paramore.Brighter.Transforms.Storage; +using Paramore.Brighter.Transforms.Transformers; + +var store = new InMemoryStorageProvider(); +var transformer = new ClaimCheckTransformer(store, store); +transformer.InitializeWrapFromAttributeParams(5); // 5Kb threshold + +var big = new string('x', 10 * 1024); +var message = new Message( + new MessageHeader(Id.Random(), new RoutingKey("large.order"), MessageType.MT_EVENT), + new MessageBody(JsonSerializer.Serialize(big))); + +var wrapped = await transformer.WrapAsync(message, new Publication()); + +// The body no longer carries the payload... +Assert.StartsWith("Claim Check", wrapped.Body.Value); + +// ...and the store does. +var claim = (string)wrapped.Header.Bag[ClaimCheckTransformer.CLAIM_CHECK]; +Assert.True(await store.HasClaimAsync(claim)); +Assert.Equal(claim, wrapped.Header.DataRef); +``` + +Against a real store, the same three assertions work with the store swapped: the body starts +with `Claim Check`, `HasClaimAsync` returns `true`, and your bucket or container has an object +whose name is the claim. For S3 that object sits under the `LuggagePrefix`, which defaults to +`BRIGHTER_CHECKED_LUGGAGE`. + +## Claim Check Failures + +**`NotImplementedException: This is a null store…`** — no store registered. Step 3. + +**`NotImplementedException` on the consumer only** — the store was registered in the producer's +service collection and not the consumer's. Both ends need it: one to check the luggage in, the +other to claim it. + +**The consumer gets a body of `Claim Check `** — the mapper's `MapToRequest` has no +`[RetrieveClaim]`, or its step ordering puts it after something that already tried to +deserialize. The claim check unwraps at `step: 0` by convention because it has to run before +anything that reads the body. + +**The second consumer of the same message finds nothing** — `retain` defaulted to `false` and +the first consumer deleted the luggage. Set `retain: true`, and take on the deletion yourself. + +**`InMemoryStorageProvider` works in tests and not between processes** — it is a dictionary in +the process that wrote it. Use `FileSystemStorageProvider` for a local multi-process run, and a +real store beyond that. + +**The payload never leaves, silently** — the body is under the threshold. Print +`message.Body.Memory.Length` and compare it with `thresholdInKb * 1024`, remembering the +comparison is on the serialized bytes rather than the size of your object. + +## Further Reading + +- [Claim Check](/contents/ClaimCheck.md) — the pattern, and the two attributes +- [S3 Luggage Store](/contents/S3LuggageStore.md) — the AWS store in detail +- [Message Transforms](/contents/MessageTransforms.md) — how transform middleware is composed +- [Message Mappers](/contents/MessageMappers.md) — writing the mapper the attributes attach to +- [Compression](/contents/Compression.md) — the other answer to a body that is too big +- [Cloud Events Support](/contents/CloudEventsSupport.md) — the `dataref` attribute the claim check sets diff --git a/contents/MSSQLInbox.md b/contents/MSSQLInbox.md index a3d6bf8..66f3669 100644 --- a/contents/MSSQLInbox.md +++ b/contents/MSSQLInbox.md @@ -49,7 +49,7 @@ Brighter ships a library that creates the Inbox table on first start and evolves **Option B — Manage the DDL yourself.** -Use `MsSqlInboxBuilder.GetDDL()` to obtain the DDL Brighter ships and apply it via your own tooling (FluentMigrator, Flyway, Liquibase, or hand-rolled scripts). +Use `SqlInboxBuilder.GetDDL()` to obtain the DDL Brighter ships and apply it via your own tooling (FluentMigrator, Flyway, Liquibase, or hand-rolled scripts). **There is no `MsSqlInboxBuilder`** — MSSQL is the one provider whose builders carry no prefix, in `Paramore.Brighter.Inbox.MsSql`. Choose based on fit; neither option is deprecated. diff --git a/contents/MSSQLOutbox.md b/contents/MSSQLOutbox.md index 4769697..626b800 100644 --- a/contents/MSSQLOutbox.md +++ b/contents/MSSQLOutbox.md @@ -21,7 +21,7 @@ Brighter ships a library that creates the table on first start and evolves its s **Option B — Manage the DDL yourself (recommended where you have schema-change governance).** -Use `MsSqlOutboxBuilder.GetDDL()` to obtain the same DDL Brighter ships, then drive it through your own change-management tooling — FluentMigrator, Flyway, Liquibase, an enterprise change-window pipeline, or hand-rolled scripts. The rest of this page describes this option. +Use `SqlOutboxBuilder.GetDDL()` to obtain the same DDL Brighter ships, then drive it through your own change-management tooling — FluentMigrator, Flyway, Liquibase, an enterprise change-window pipeline, or hand-rolled scripts. The rest of this page describes this option. Neither option is deprecated. Choose based on fit: small teams and greenfield apps benefit from startup-time provisioning; teams with DBA approval workflows or change windows often prefer to drive the same DDL through their own tooling. @@ -41,29 +41,33 @@ Install-Package Paramore.Brighter.MsSql.EntityFrameworkCore ## MSSQL Outbox Database Table Schema -The MSSQL Outbox requires a specific table in your database to store messages before they are dispatched. You can generate the necessary SQL Data Definition Language (DDL) script to create this table using the `MsSqlOutboxBuilder` helper class. +The MSSQL Outbox requires a specific table in your database to store messages before they are dispatched. You can generate the necessary SQL Data Definition Language (DDL) script to create this table using the `SqlOutboxBuilder` helper class. **Note:** When you choose Option B, you are responsible for creating the table and applying schema changes when upgrading to new versions of Brighter. Option A handles both for you — see [Database Provisioning](/contents/BoxProvisioning.md). Either way, application-level concerns like additional indexes for query performance remain your responsibility. ### **Generating the DDL** -The `MsSqlOutboxBuilder.GetDDL()` method creates the SQL script for you. You can execute this script against your database to create the outbox table. +The `SqlOutboxBuilder.GetDDL()` method creates the SQL script for you. You can execute this script against your database to create the outbox table. + +**Mind the type name.** Every other provider prefixes its builder — `PostgreSqlOutboxBuilder`, `MySqlOutboxBuilder`, `SqliteOutboxBuilder`, `SpannerOutboxBuilder` — and MSSQL alone does not. There is no `MsSqlOutboxBuilder`; the type is `SqlOutboxBuilder`, in `Paramore.Brighter.Outbox.MsSql`. The Inbox builder is `SqlInboxBuilder` for the same reason. ```csharp +using Paramore.Brighter.Outbox.MsSql; + // The table name can be whatever you choose. string tableName = "Outbox"; // The DDL for a table that stores the message body as NVARCHAR(MAX) -string ddl = MsSqlOutboxBuilder.GetDDL(tableName); +string ddl = SqlOutboxBuilder.GetDDL(tableName); // The DDL for a table that stores the message body as VARBINARY(MAX) // Useful if your message body is binary -string binaryDdl = MsSqlOutboxBuilder.GetDDL(tableName, hasBinaryMessagePayload: true); +string binaryDdl = SqlOutboxBuilder.GetDDL(tableName, hasBinaryMessagePayload: true); ``` ### **Example SQL Script** -Running `MsSqlOutboxBuilder.GetDDL("Outbox")` will generate the following SQL script: +Running `SqlOutboxBuilder.GetDDL("Outbox")` will generate the following SQL script: ```sql CREATE TABLE Outbox ( diff --git a/contents/MSSQLTransportInboxAndOutbox.md b/contents/MSSQLTransportInboxAndOutbox.md new file mode 100644 index 0000000..1d60460 --- /dev/null +++ b/contents/MSSQLTransportInboxAndOutbox.md @@ -0,0 +1,422 @@ +--- +description: "One SQL Server database can carry your message queue, your Outbox and your Inbox together, sharing a single connection string and one configuration object." +layout: + description: + visible: false +--- + +# Use MSSQL for Transport, Outbox and Inbox + +> **How-to** · Applies to **Brighter V10** · Prerequisites: [MSSQL Message Broker](/contents/MSSQLMessageBroker.md), [MSSQL Outbox](/contents/MSSQLOutbox.md) + +One SQL Server database can carry your message queue, your Outbox and your Inbox together, sharing a single connection string and one configuration object. + +[Use PostgreSQL for Both Transport and Outbox](/contents/PostgreSQLTransportAndOutbox.md) makes the case for composing a broker and an Outbox in one database: the business write and the message announcing it commit together, so there is no window in which the row exists and the message does not. That argument is not about PostgreSQL, and this guide is the demonstration — the same composition on SQL Server, with an Inbox added so the receiving end is idempotent too. + +The shape is deliberately the same as the PostgreSQL guide's. Where the two differ, the difference is called out, and **step 2 is the one that matters** — SQL Server will not create your queue table for you. + +A working version is in the Brighter repository at `Brighter/samples/TaskQueue/MsSqlMessagingGateway/GreetingsSender/`, with the consumer beside it in `GreetingsReceiverConsole/`. + +## Step 1: Install the MSSQL Packages + +The transport, the Outbox, the Inbox, the transaction provider, the Sweeper and the provisioner are separate packages: + +```bash +dotnet add package Paramore.Brighter.MessagingGateway.MsSql +dotnet add package Paramore.Brighter.Outbox.MsSql +dotnet add package Paramore.Brighter.Inbox.MsSql +dotnet add package Paramore.Brighter.MsSql +dotnet add package Paramore.Brighter.Outbox.Hosting +dotnet add package Paramore.Brighter.BoxProvisioning.MsSql +``` + +`Paramore.Brighter.MsSql` is the one people miss. It holds `MsSqlConnectionProvider` and `MsSqlTransactionProvider`, which are what let your handler and the Outbox share a transaction — without it there is no composition, only three subsystems pointed at the same database. + +## Step 2: Create the Queue, Outbox and Inbox Tables + +**The MSSQL transport does not create its queue table, and this is the one place the PostgreSQL pattern does not carry over.** `OnMissingChannel.Create` is accepted on an MSSQL publication and subscription and then never acted on — the gateway has no provisioning path at all, where the PostgreSQL gateway has one. Set the table up yourself before anything runs, or the first send fails against a table that is not there. + +Brighter ships the DDL for you to run, in `MsSqlQueueBuilder`. Nothing in the product calls it, which is exactly why it is public: + +```csharp +using Paramore.Brighter.MessagingGateway.MsSql; + +// The queue table, and the index the consumer's topic lookup wants +string queueDdl = MsSqlQueueBuilder.GetDDL("QueueData"); +string queueIndexDdl = MsSqlQueueBuilder.GetIndexDDL("QueueData"); + +// And the test for whether it is already there +string exists = MsSqlQueueBuilder.GetExistsQuery("QueueData", schemaName: "dbo"); +``` + +That produces: + +```sql +CREATE TABLE [QueueData] +( + [Id] [BIGINT] IDENTITY(1,1) NOT NULL PRIMARY KEY, + [Topic] [NVARCHAR](255) NOT NULL, + [MessageType] [NVARCHAR](1024) NOT NULL, + [Payload] [NVARCHAR](MAX) NOT NULL +); + +CREATE NONCLUSTERED INDEX [IX_QueueData_Topic] ON [QueueData] ([Topic] ASC); +``` + +**The Outbox and Inbox tables** are created and migrated by [Box Provisioning](/contents/BoxProvisioning.md) at startup, which is the call in step 9. To manage them yourself instead, ask Brighter for the same DDL: + +```csharp +using Paramore.Brighter.Inbox.MsSql; +using Paramore.Brighter.Outbox.MsSql; + +string outboxDdl = SqlOutboxBuilder.GetDDL("Outbox"); +string inboxDdl = SqlInboxBuilder.GetDDL("InboxMessages"); +``` + +**Mind those two type names.** Every other provider prefixes its builders — `PostgreSqlOutboxBuilder`, `MySqlOutboxBuilder`, `SqliteOutboxBuilder`, `SpannerOutboxBuilder` — and MSSQL alone does not. `MsSqlOutboxBuilder` and `MsSqlInboxBuilder` do not exist. + +Your own tables are yours. Brighter does not create, migrate or know about them. + +## Step 3: Describe All Three Tables in One Configuration + +This is the pivot the whole guide turns on. The queue store, the Outbox and the Inbox are three parameters on **one** object, so there is no second configuration to keep in step: + +```csharp +using Paramore.Brighter; + +const string connectionString = + "Server=localhost,14330;Database=BrighterTests;User Id=sa;Password=Password1!;Encrypt=false"; + +// One object, three tables. +var configuration = new RelationalDatabaseConfiguration( + connectionString, + databaseName: "BrighterTests", + outBoxTableName: "Outbox", + inboxTableName: "InboxMessages", + queueStoreTable: "QueueData"); +``` + +**There is no connection wrapper here, and that is a difference from PostgreSQL.** PostgreSQL's transport takes a `PostgresMessagingGatewayConnection` holding the configuration; MSSQL's takes the `RelationalDatabaseConfiguration` directly. One less type, and one less thing to construct. + +For every option this type carries — including `schemaName` and the payload flags — see [Relational Database Configuration Reference](/contents/RelationalDatabaseConfigurationReference.md#relational-database-configuration-options). + +## Step 4: Register the MSSQL Configuration + +Put the same object in the container: + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter; + +builder.Services.AddSingleton(configuration); +``` + +**This line is easy to leave out and the failure lands nowhere near it.** `TransactionProvider` in the next step is given as a *type*, so the container activates `MsSqlTransactionProvider`, and its constructor asks for exactly this interface. Omit the registration and your application starts, provisions the boxes, and only then throws — see [Failures](#mssql-transport-outbox-and-inbox-failures). + +## Step 5: Wire the MSSQL Producer and the Outbox + +The publication says where the event goes; the three lines after `ProducerRegistry` are what make the Outbox durable and transactional: + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter; +using Paramore.Brighter.Extensions.DependencyInjection; +using Paramore.Brighter.MessagingGateway.MsSql; +using Paramore.Brighter.MsSql; +using Paramore.Brighter.Outbox.MsSql; + +var producerRegistry = new MsSqlProducerRegistryFactory( + configuration, + [ + new Publication + { + Topic = new RoutingKey("greeting.event") + } + ]).Create(); + +builder.Services + .AddBrighter() + .AddProducers(configure => + { + configure.ProducerRegistry = producerRegistry; + + configure.Outbox = new MsSqlOutbox(configuration); + configure.ConnectionProvider = typeof(MsSqlConnectionProvider); + configure.TransactionProvider = typeof(MsSqlTransactionProvider); + }) + .AutoFromAssemblies(); +``` + +`MsSqlProducerRegistryFactory` takes the `RelationalDatabaseConfiguration` from step 3 directly. **There is no `MsSqlPublication`** — the publications are plain `Publication`, because the MSSQL transport has no per-publication settings of its own. + +**The transaction provider is not optional here.** It is what fixes the transaction type Brighter matches the Outbox against, and `MsSqlOutbox` only satisfies that match when the provider is SQL Server's. Leave it out and registration itself fails, again in [Failures](#mssql-transport-outbox-and-inbox-failures). + +## Step 6: Wire the MSSQL Consumer + +The consumer reads from the same queue store table. Note the order: `AddConsumers` extends `IServiceCollection`, while `AddProducers` extends the builder it returns, so a consumer registration comes first and everything else chains off it. + +```csharp +using System; +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter; +using Paramore.Brighter.MessagingGateway.MsSql; +using Paramore.Brighter.ServiceActivator.Extensions.DependencyInjection; +using Paramore.Brighter.ServiceActivator.Extensions.Hosting; + +var subscriptions = new Subscription[] +{ + new MsSqlSubscription( + new SubscriptionName("paramore.example.greeting"), + new ChannelName("greeting.event"), + new RoutingKey("greeting.event"), + timeOut: TimeSpan.FromMilliseconds(200), + messagePumpType: MessagePumpType.Reactor) +}; + +builder.Services.AddConsumers(options => + { + options.Subscriptions = subscriptions; + options.DefaultChannelFactory = new ChannelFactory( + new MsSqlMessageConsumerFactory(configuration)); + }) + .AutoFromAssemblies(); + +builder.Services.AddHostedService(); +``` + +**Use `MsSqlSubscription`, not `Subscription`.** `ChannelFactory` casts what it is given down to `MsSqlSubscription` and throws on failure: + +```text +Paramore.Brighter.ConfigurationException: MS SQL ChannelFactory We expect an MsSqlSubscription or MsSqlSubscription as a parameter +``` + +It does this in all three of its channel-creation methods. A plain `Subscription` compiles perfectly and then dies when the Dispatcher starts reading — which is worse than a compile error, because everything looks right until the moment it runs. + +**`ChannelFactory` is a name ten transports share.** This one is `Paramore.Brighter.MessagingGateway.MsSql.ChannelFactory`, so mind the `using` directive if your solution talks to more than one broker. + +## Step 7: Add the Inbox + +The Inbox is what makes the *consumer* idempotent: it records the messages a handler has already seen, so a redelivery is recognised rather than reprocessed. Configure it on the same `AddConsumers` call, against the same database: + +```csharp +using Microsoft.Extensions.DependencyInjection; +using Paramore.Brighter; +using Paramore.Brighter.Inbox; +using Paramore.Brighter.Inbox.MsSql; +using Paramore.Brighter.MessagingGateway.MsSql; +using Paramore.Brighter.ServiceActivator.Extensions.DependencyInjection; + +builder.Services.AddConsumers(options => + { + options.Subscriptions = subscriptions; + options.DefaultChannelFactory = new ChannelFactory( + new MsSqlMessageConsumerFactory(configuration)); + + options.InboxConfiguration = new InboxConfiguration( + new MsSqlInbox(configuration), + scope: InboxScope.Commands, + onceOnly: true, + actionOnExists: OnceOnlyAction.Warn); + }) + .AutoFromAssemblies(); +``` + +**`MsSqlInbox` takes the same configuration object**, reading `inboxTableName` from it — the third of the three names step 3 set. + +The three arguments after the Inbox are the ones worth deciding rather than defaulting: + +| Option | Default | What it does | +|---|---|---| +| `scope` | `InboxScope.All` | `Commands`, `Events`, or `All`. Commands are the usual choice — an event delivered twice is often harmless, a command rarely is | +| `onceOnly` | `true` | Whether to de-duplicate at all. `false` records without suppressing, which is useful as an audit log | +| `actionOnExists` | `OnceOnlyAction.Throw` | What a duplicate does. `Throw` surfaces it, `Warn` logs and drops it | + +`OnceOnlyAction.Throw` is the default and it is the safe one, but on a transport that redelivers it will fill your logs with exceptions for messages that are being handled correctly. `Warn` is usually what you want once you trust the Inbox. + +See [Brighter Inbox Support](/contents/BrighterInboxSupport.md) for the `[UseInbox]` attribute, which scopes an Inbox to a single handler instead of globally. + +## Step 8: Deposit and Clear Inside Your Transaction + +Ask the transaction provider for the connection and the transaction rather than opening your own. That shared pair is what makes two writes one atomic act: + +```csharp +using System; +using System.Data.Common; +using System.Threading; +using System.Threading.Tasks; +using Paramore.Brighter; + +public class AddGreetingHandlerAsync : RequestHandlerAsync +{ + private readonly IAmATransactionConnectionProvider _transactionProvider; + private readonly IAmACommandProcessor _postBox; + + public AddGreetingHandlerAsync( + IAmATransactionConnectionProvider transactionProvider, + IAmACommandProcessor postBox) + { + _transactionProvider = transactionProvider; + _postBox = postBox; + } + + public override async Task HandleAsync( + AddGreeting addGreeting, + CancellationToken cancellationToken = default) + { + DbConnection connection = await _transactionProvider.GetConnectionAsync(cancellationToken); + DbTransaction transaction = await _transactionProvider.GetTransactionAsync(cancellationToken); + + try + { + // 1. Your write, to your table. + await using (DbCommand command = connection.CreateCommand()) + { + command.Transaction = transaction; + command.CommandText = "insert into Greeting (Message) values (@message)"; + + DbParameter message = command.CreateParameter(); + message.ParameterName = "message"; + message.Value = addGreeting.Greeting; + command.Parameters.Add(message); + + await command.ExecuteNonQueryAsync(cancellationToken); + } + + // 2. Brighter's write, to the Outbox, on that same transaction. Nothing has + // reached the queue store table yet. + await _postBox.DepositPostAsync( + new GreetingEvent(addGreeting.Greeting), + _transactionProvider, + cancellationToken: cancellationToken); + + // 3. Both, or neither. + await _transactionProvider.CommitAsync(cancellationToken); + } + catch (Exception) + { + await _transactionProvider.RollbackAsync(cancellationToken); + throw; + } + finally + { + _transactionProvider.Close(); + } + + return await base.HandleAsync(addGreeting, cancellationToken); + } +} +``` + +There is no `ClearOutboxAsync` call here on purpose: the message waits in the Outbox for the Sweeper. Call `ClearOutboxAsync` after the commit instead if you would rather dispatch immediately, at the cost of doing the send on the request thread. + +## Step 9: Provision the Boxes and Run the Sweeper + +The Sweeper is a hosted service that wakes on a timer, finds undispatched messages and sends them — here, into the queue store table in the same database. Chain both calls off the registration in step 5, and provision the Inbox alongside the Outbox: + +```csharp +using System; +using Paramore.Brighter.BoxProvisioning; +using Paramore.Brighter.BoxProvisioning.MsSql; +using Paramore.Brighter.Extensions.DependencyInjection; +using Paramore.Brighter.Outbox.Hosting; + +builder.Services + .AddBrighter() + .AddProducers(configure => + { + // ... as in step 5 + }) + .AutoFromAssemblies() + + // Creates and migrates both tables at startup. Needs rights to CREATE TABLE. + .UseBoxProvisioning(options => + { + options.AddMsSqlOutbox(configuration); + options.AddMsSqlInbox(configuration); + }) + + .UseOutboxSweeper(options => + { + options.TimerInterval = 5; + options.MinimumMessageAge = TimeSpan.FromSeconds(5); + }); +``` + +Both Sweeper values above are the defaults. Together they mean a message is picked up on the first tick after it is five seconds old, so expect a five to ten second delay before it appears on the queue. + +**Provisioning covers the Outbox and the Inbox and not the queue.** There is no `AddMsSqlQueue`, for the reason in step 2 — that table is yours to create. + +Running more than one instance? Configure a [distributed lock](/contents/MsSqlDistributedLock.md) so only one Sweeper runs at a time. + +## Step 10: Verify It Worked + +Send one message, then look at the three tables. + +Immediately after the send, the Outbox holds the message and the queue does not: + +```sql +select MessageId, Topic, case when Dispatched is null then 0 else 1 end as Dispatched from Outbox; +select Id, Topic from QueueData; +``` + +Five to ten seconds later the Sweeper has dispatched it, and the same two queries show the message dispatched *and* sitting on the queue: + +```text +MessageId Topic Dispatched +------------------------------------ --------------- ---------- +01a07aff-20b1-722b-a12a-c5ecb3c466f9 greeting.event 1 + +Id Topic +--- --------------- +1 greeting.event +``` + +Start the consumer and the row leaves the queue table. Send the same message again and the Inbox is what you watch instead: + +```sql +select CommandId, CommandType, ContextKey, Timestamp from InboxMessages; +``` + +`ContextKey` is what scopes a message to a handler. It is generated from the handler's class name unless you pass a `context` function to `InboxConfiguration`, which is why two different handlers can each record the same `CommandId` without either seeing the other's row. + +A second delivery adds no row and, with `actionOnExists: OnceOnlyAction.Warn`, logs rather than throws — the handler does not run twice. + +**The check that matters is the failure case**, because it is the reason for all of this. Throw inside the handler between the two writes and the commit, and both disappear together — the row counts in your table and in `Outbox` are unchanged, and nothing reaches the queue. + +## MSSQL Transport, Outbox and Inbox Failures + +**`Invalid object name 'QueueData'`** + +The queue table does not exist, and nothing was ever going to create it. `OnMissingChannel.Create` is inert on this transport. Run the DDL from step 2. + +**`Paramore.Brighter.ConfigurationException: MS SQL ChannelFactory We expect an MsSqlSubscription or MsSqlSubscription as a parameter`** + +A `Subscription` where an `MsSqlSubscription` was needed. It compiles; it fails when the Dispatcher builds its channels. Step 6. + +**`Unable to register outbox of type MsSqlOutbox - no transaction provider has been registered that matches the outbox's transaction type`** + +A `ConfigurationException`, thrown by `AddProducers` while your application is still starting. Brighter takes the transaction type from `TransactionProvider`, falling back to `InMemoryTransactionProvider` when you do not set one, and then checks that the Outbox you supplied implements the Outbox interfaces for *that* transaction type. Set both `ConnectionProvider` and `TransactionProvider` as step 5 shows. + +**`Unable to resolve service for type 'Paramore.Brighter.IAmARelationalDatabaseConfiguration' while attempting to activate 'Paramore.Brighter.MsSql.MsSqlTransactionProvider'`** + +You skipped step 4. What makes this one expensive is how healthy everything looks first: the host starts, both boxes are provisioned, and the exception arrives only on the first attempt to resolve a command processor, naming a type your code never mentions. + +**Every message is handled twice** + +The Inbox is configured with `scope: InboxScope.Events` while the redelivered request is a command, or the other way round. Check the scope against what you are actually sending; `InboxScope.All` covers both while you work out which. + +**`The type or namespace name 'MsSqlOutboxBuilder' could not be found`** + +There is no such type. It is `SqlOutboxBuilder`, and the Inbox one is `SqlInboxBuilder` — MSSQL is the only provider whose DDL builders carry no prefix. Step 2. + +## Further Reading + +- [Use PostgreSQL for Both Transport and Outbox](/contents/PostgreSQLTransportAndOutbox.md) — the same composition on PostgreSQL, and the argument for doing it at all +- [MSSQL Message Broker](/contents/MSSQLMessageBroker.md) — the transport on its own, with every subscription option +- [MSSQL Outbox](/contents/MSSQLOutbox.md) — the Outbox on its own, and its DDL +- [MSSQL Inbox](/contents/MSSQLInbox.md) — the Inbox on its own +- [Brighter Inbox Support](/contents/BrighterInboxSupport.md) — the `[UseInbox]` attribute, and Inbox behaviour in general +- [Relational Database Configuration Reference](/contents/RelationalDatabaseConfigurationReference.md) — every option on the configuration object step 3 builds +- [Outbox Pattern](/contents/OutboxPattern.md) — why an Outbox, and what it does and does not guarantee +- [Box Provisioning](/contents/BoxProvisioning.md) — startup provisioning and migration for the Outbox and Inbox tables +- [MSSQL Distributed Lock](/contents/MsSqlDistributedLock.md) — required once more than one instance runs a Sweeper diff --git a/contents/PostgreSQLTransportAndOutbox.md b/contents/PostgreSQLTransportAndOutbox.md index 896486e..1eac607 100644 --- a/contents/PostgreSQLTransportAndOutbox.md +++ b/contents/PostgreSQLTransportAndOutbox.md @@ -262,6 +262,7 @@ The Sweeper is a hosted service that wakes on a timer, finds undispatched messag using System; using Paramore.Brighter.BoxProvisioning; using Paramore.Brighter.BoxProvisioning.PostgreSql; +using Paramore.Brighter.Extensions.DependencyInjection; using Paramore.Brighter.Outbox.Hosting; builder.Services diff --git a/spec/013-howto_guides/tasks.md b/spec/013-howto_guides/tasks.md index a3cc81a..6b0ea93 100644 --- a/spec/013-howto_guides/tasks.md +++ b/spec/013-howto_guides/tasks.md @@ -801,12 +801,12 @@ arriving inside the phase whose findings are about exactly that. **Goal:** the claim-check recipe, and the proof that design §2.5's composition generalises. **Eight tasks. One PR.** -- [ ] **Task 4.1:** `HandlingLargeMessages.md` — front matter, H1, banner, opening sentence +- [x] **Task 4.1:** `HandlingLargeMessages.md` — front matter, H1, banner, opening sentence - Input: design §4.3 - Output: H1 *Put a Large Payload Behind a Claim Check*; prerequisites `ClaimCheck.md` and `MessageMappers.md` -- [ ] **Task 4.2:** Steps 1–2 — the size limit, and the six luggage stores +- [x] **Task 4.2:** Steps 1–2 — the size limit, and the six luggage stores - Input: `git grep -l 'IAmAStorageProviderAsync' 10.7.0 -- src/` - Output: `## Step 1: Find Your Transport's Message Size Limit`, `## Step 2: Choose a Luggage Store` @@ -817,7 +817,7 @@ tasks. One PR.** an **unclosed bold** — `**IAmAStorageProviderAsync:` — which is how long it has been since anyone read the bottom of that page. -- [ ] **Task 4.3:** Step 3 — register the luggage store. **This is why the page exists** +- [x] **Task 4.3:** Step 3 — register the luggage store. **This is why the page exists** - Input: `ServiceCollectionExtensions.cs:951`, `:971`, `:992` - Output: `## Step 3: Register the Luggage Store` - Notes: **`UseExternalLuggageStore` is on 0 of 157 pages**, against a control @@ -825,7 +825,7 @@ tasks. One PR.** following `ClaimCheck.md` attaches the attribute and gets **no store**. Written from the type, three overloads. -- [ ] **Task 4.4:** Steps 4–6, the failures section, and `ClaimCheck.md`'s pointer +- [x] **Task 4.4:** Steps 4–6, the failures section, and `ClaimCheck.md`'s pointer - Input: `ClaimCheck.md:25`, `:38`; `MessageTransforms.md`'s ruling - Output: `## Step 4: Attach the Claim Check to Your Mapper`, `## Step 5: Choose a Threshold`, `## Step 6: Verify the Payload Went to the Store`, `## Claim Check Failures`; @@ -835,21 +835,21 @@ tasks. One PR.** not own. The default `JsonMessageMapper` already carries `[CloudEvents(0)]`, so "default mappers do not run transforms" is **false** and must not be written. -- [ ] **Task 4.5:** `MSSQLTransportInboxAndOutbox.md` — front matter, H1, banner, opening sentence +- [x] **Task 4.5:** `MSSQLTransportInboxAndOutbox.md` — front matter, H1, banner, opening sentence - Input: design §4.4 - Output: H1 naming transport, Inbox and Outbox together; prerequisites `MSSQLMessageBroker.md` and `MSSQLOutbox.md` - Notes: **`MSSQLOutbox.md` must already be repaired by task 1.10** — it is a prerequisite this guide links, and it named a type that has never existed. -- [ ] **Task 4.6:** The MSSQL steps, mirroring P0-1 with an Inbox step inserted after step 5 +- [x] **Task 4.6:** The MSSQL steps, mirroring P0-1 with an Inbox step inserted after step 5 - Input: design §4.4; `MSSQLMessageBroker.md:107` - Output: the step sequence, and *Further Reading* pointing back at P0-1 so the two read as one pattern - Notes: **divergence from P0-1's shape is a defect here, not variety** — the value of this page is that the pattern generalises. -- [ ] **Task 4.7:** The `MsSqlSubscription` caveat — load-bearing, from Brighter#4302 +- [x] **Task 4.7:** The `MsSqlSubscription` caveat — load-bearing, from Brighter#4302 - Input: `MessagingGateway.MsSql/ChannelFactory.cs:46`, `:65`, `:88`; `MSSQLMessageBroker.md:142` - Output: every subscription typed `MsSqlSubscription`, **with the reason stated** @@ -858,7 +858,7 @@ tasks. One PR.** dies at `dispatcher.Receive()`** — strictly worse than a compile error, because the page looks authoritative right up to the throw. Mirror the MsSql gateway tests (obligation 3). -- [ ] **Task 4.8:** Both `SUMMARY.md` entries, both `pagetypes.tsv` rows, compile, gates +- [x] **Task 4.8:** Both `SUMMARY.md` entries, both `pagetypes.tsv` rows, compile, gates - Output: link 162 → **164**, pagelint 160 → **162**, shape 159 → **161** with widest still **12 of 20**, redirects and optioncheck **unmoved**; `--verify` **161/161** after publication - Notes: two nested pages in one PR — **assert the widest and the redirect count individually @@ -866,6 +866,94 @@ tasks. One PR.** --- +## Phase 4 as executed — 2026-09-10, `f8e769b` on `docs/013-phase4-large-messages-mssql`, PR #155 + +**Two new nested pages, four repaired, one upstream sample PR.** All eight tasks done, **39 / 43**. + +### The gates landed on task 4.8's prediction + +link **162 → 164**, pagelint **160 → 162 pages**, shape **159 → 161** with the widest section +**unmoved at 12 of 20**; redirects (**77 / 7858**), versioncheck (**0 stale of 18**) and +optioncheck (**0 across 59 tables, 519 rows**) all unmoved. `--changed origin/master` read +**`6 documentation page(s), 25 code block(s) strict`**, 0 errors — the scope line was read, not +just the verdict. + +**The warning count fell 772 → 768**, which task 4.8 did *not* predict. It was bought: four +previously-bare blocks on `ClaimCheck.md` and `MSSQLOutbox.md` earned real `using` directives +while their defects were being repaired. + +### Findings + +**A. THE NULL LUGGAGE STORE THROWS ON RESOLUTION, NOT ON THRESHOLD — AND THE FIRST DRAFT OF THE +PAGE SAID OTHERWISE.** `AddBrighter` ends with `UseExternalLuggageStore()` +(`ServiceCollectionExtensions.cs:222-223`), and `RegisterLuggageStore`'s factory calls +`EnsureStoreExists()` **before handing the store out**. So the failure lands when anything +resolves `IAmAStorageProvider` — when a `[ClaimCheck]` mapper's transform pipeline is first +built — whatever size that message is. I wrote *"at the first message over the threshold"*, +compiled it happily, and only running it found the error. **Lesson 13 again, in a phase whose +whole subject is registration.** + +**B. REGISTRATION ORDER FOR THE LUGGAGE STORE IS THE OPPOSITE OF `AddBrighterDefault`'s.** +`UseExternalLuggageStore` uses `AddSingleton`, so the **last** registration wins and yours must +come **after** `AddBrighter`. `AddBrighterDefault` uses `TryAddBuilder`, so yours must come +**first**. Two rules, opposite directions, and neither is discoverable from the call site. +Measured with a control: `AddBrighter()` alone resolves to a throwing `NullLuggageStore`; +the same container with `.UseExternalLuggageStore()` resolves to +`InMemoryStorageProvider`. + +**C. NINE OF TEN PROVIDERS PREFIX THE DDL BUILDER AND MSSQL DOES NOT — THE DOCS WROTE THE +PATTERN.** `MsSqlOutboxBuilder` (6 sites, `MSSQLOutbox.md`) and `MsSqlInboxBuilder` (1 site, +`MSSQLInbox.md`) have **never existed**; the types are `SqlOutboxBuilder` and `SqlInboxBuilder`. +The census that settles it is one command over the ten Outbox/Inbox packages — `PostgreSql…`, +`MySql…`, `Sqlite…`, `Spanner…` all prefixed, MsSql alone bare. **This is lesson 12 inverted: +there, one file was checked and generalised; here, the family was assumed and never checked at +all.** Design §4.4 asserted task 1.10 had already repaired `MSSQLOutbox.md`; it had not, because +phase 1 swept *call sites*, not *DDL builder names*. + +**D. `ClaimCheck.md` PRINTED AN INTERFACE WITH THREE MEMBERS THAT DO NOT EXIST.** +`UploadAsync`/`DownloadAsync` where the product has `StoreAsync`/`RetrieveAsync`, and no +`EnsureStoreExistsAsync` or `Tracer` at all. The page also carried a V9 one-argument +`MapToMessage` — the defect review 6 found on Brighter#4302 — and a store list that stopped +after **one of seven**, mid-`**`. It had been green under every gate for as long as the gates +have existed, because nothing about it was *malformed*. + +**E. THE MSSQL GATEWAY HAS NO PROVISIONING PATH, SO `OnMissingChannel.Create` IS INERT.** +Control: the Postgres gateway has **12** `OnMissingChannel` references and a `CREATE TABLE`; +the MSSQL gateway has **0** beyond storing the value on the publication. `MsSqlQueueBuilder` +— the type that would give you the DDL — has **no production call site**, which is exactly why +it is public. **This is the one place design §2.5's composition does not generalise**, and the +page says so rather than smoothing it over. + +**F. THE MSSQL SAMPLE'S TWO RECEIVERS COULD NEVER HAVE STARTED.** Both built +`Subscription` where `ChannelFactory` downcasts to `MsSqlSubscription` and throws. Measured +with a control — `Subscription` → `ConfigurationException`, `MsSqlSubscription` → +accepted. Fixed in **Brighter#4331**, along with the Outbox and Inbox the sample now +demonstrates. **A sample that compiles is not a sample that runs**, and nothing in either +repository was checking. + +**G. `## Step N:` HEADINGS ARE NOT UNIQUE ACROSS PAGES ONCE TWO HOW-TOS SHARE A SEQUENCE.** +`CLAUDE.md` justifies the step-heading convention partly on the claim that a step heading *"is +unique across pages"*. That held only while no two how-tos shared a shape — and design §4.4 +**required** P1-2 to mirror P0-1's. Four headings collided and rule 3a failed the build. +Resolved by qualifying the four **on the new page only**, so no published PostgreSQL anchor +moves. **Phase 5 and spec 014 both want this**: the convention's stated rationale has an +unstated precondition. + +### What the harnesses were + +Two, both `PackageReference` to **10.7.0** (phase 2's finding F), both `net9.0`, +`disable`, **no ``** (lesson 12). `largecheck` +extracted **4** fences and `mssqlcheck` **9**; both reached **0 errors**, and the only warning +classes were `CS0105` and `CS0414`, both harness artefacts — **no `CS0618`**, so nothing on +either page uses an obsolete API. + +**The extractor learned to hoist `using` directives.** A page's `using` lines must be *in* the +block — that is rule 6 — but are only legal at file scope, so the harness moves them rather +than supplying them. A block missing one still fails to resolve its types, which is how the two +missing directives on the MSSQL page were caught. + +--- + ## Phase 5 — Acceptance **Goal:** walk AC1–AC10 with evidence, and find what the phases did not. **Four tasks. One PR.**