feat(mailbox): file inbound mail, and block the senders you mark as spam - #27
Merged
Merged
Conversation
The inbound path parsed each message, published an email.received event and kept nothing. That was right for a Datamailer-era consumer holding its own system of record and wrong for a mailbox, where the reader is a person in this console. aishippinglabs.com is still forwarded by a Lambda in main/aisl, so every cold pitch for the domain lands in a personal inbox, and adding an address means editing Terraform and applying it. This makes Relay the place where both of those happen. A message becomes an inbound_messages row: headers, a snippet and the SES verdicts in Postgres, bodies and attachments in object storage. Bodies stay in S3 because inbound mail is unbounded in size and retention, and the disk fill that disqualified inbound in production was a host-disk problem. Headers are what the console lists and searches on, so they are cheap to index; bodies are read one at a time, by an operator who has decided to open one. A receiving address is a row in inbound_addresses, created in the console. That is the whole change: no Terraform edit, no apply, no redeploy. Addresses fall back to INBOUND_EMAIL_ROUTES when nothing matches, so the sandbox keeps working and an estate that has created no addresses is not silently dark, and provision_inbound_addresses copies the routes across. Mark as spam blocks the sender, and the block is checked before storage rather than after. Blocking should stop future mail, not filter a mailbox an operator has to read. A discarded message is recorded as blocked with no body, so the block is auditable and unblocking does not resurrect something nobody read. Address and domain are separate buttons, so one correspondent at a shared sending domain can be blocked alone. Two things got smaller rather than larger. Idempotency moved from a DynamoDB conditional write to a unique constraint on message_id: the same guarantee, inside the transaction that was already happening, with one fewer service to fail. And the inbound-email event no longer carries the body and every attachment, only the id of the stored message -- a second full copy over SNS was the thing this path was built to stop doing. docs/worker-contracts.md says so, and a consumer reading body or attachments has to change. The inbound drain now runs outside the sandbox too, with an explicit memory limit. It needed one: the limit table had no entry for it, so enabling the container without adding it would have started an unbounded process on the 2 GiB host that inbound was last blamed for taking down.
Rewrite app.css on the dataops foundation: white canvas, muted 268px sidebar, CMP blue accent, Inter + IBM Plex Mono (self-hosted, SIL OFL), bordered tables, 34px controls, light/dark themes. Class names are unchanged, so templates need no edits. Design tokens documented in docs/design-system.md.
Add a DevAutoLoginMiddleware that signs requests in as the seeded superuser when DEBUG is on, so local development skips the Django admin login. Deployed environments run with DEBUG off and keep the OIDC flow in relay.oidc; the test suite runs with DEBUG off too, so the middleware is dormant there.
Resolve the deploy-script overlap in favor of main's newer inbound topology: relay-inbound-ingress is sandbox-only and production conditionally drains the sandbox queues instead. This supersedes this branch's both-environments inbound drain, its production required-container entry, and the memory limit for a container production never starts. All other overlaps (urls, views, settings) kept both sides.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second half of moving
aishippinglabs.commail into Relay. PR #60 built the transport; this makes the application able to receive, file and filter.The feature
Receiving addresses are rows. Create one in the console under Configure → Receiving addresses and it starts receiving. No Terraform edit, no apply, no redeploy — which is the thing that made adding an address painful before.
"Mark as spam" blocks the sender. Two buttons on the message page: block that address, or block the whole sending domain, so one correspondent at a shared domain can be stopped without silencing everyone else on it.
Two design decisions worth arguing with
The blocklist is checked before storage, not after. Blocking should stop future mail. Filtering a mailbox that already has to be read is the same problem as a filter you have to keep applying by hand, which is what it replaces. A discarded message is still recorded, as
blockedwith no body — so the block is auditable, and unblocking doesn't resurrect something nobody read.Headers in Postgres, bodies in S3. Inbound mail is unbounded in size and retention. The disk fill that disqualified inbound in production was a host-disk problem, and keeping bodies out of the database is what stops that recurring. Headers are what the console lists, searches and sorts on, so they're cheap to index; a body is read one at a time, by someone who has already decided to open that message. HTML bodies are never rendered — inbound HTML is untrusted third-party markup and an authenticated page is a poor place to execute it.
Things that got smaller
message_id. Same guarantee, inside the transaction that was already happening, with one fewer service to fail and nothing to half-succeed.INBOUND_EMAIL_IDEMPOTENCY_TABLEis no longer read.inbound-emailSNS event lostbodyandattachments. It now carriesinbound_message_idand the raw MIME pointer. A second full copy over SNS was the thing this path was built to stop doing. This is a contract change — any consumer reading those keys has to read them from Relay instead.docs/worker-contracts.mdsays so explicitly.INBOUND_EMAIL_EVENTS_TOPIC_ARNis now optional. A Relay that owns a mailbox doesn't need to announce anything, and requiring it would make the mailbox depend on a consumer that may not exist.The drain now runs outside the sandbox
It needed an explicit memory limit to do that. The
set_memory_argstable had no entry forrelay-inbound-ingress, so removing theif [[ "$environment" == sandbox ]]guard alone would have started an unbounded container on the 2 GiB production host — the exact failure inbound was last blamed for. Added at 128m, matching the SES ingress drain, and added torequired_containersso a deploy fails if it isn't up.Not in this PR
aishippinglabs.comreceipt rule still points at the forwarding Lambda, so no mail arrives here yet. That's the fourth step: repoint the rule inmain/aisland drop the Lambda, which is the change that actually stops the spam. It's one line once this lands, because PR #60 already created the bucket it writes to.INBOUND_EMAIL_ROUTESstill works as a fallback, andprovision_inbound_addressescopies those routes into the table so an existing domain doesn't go dark on cutover.Verification
uv run pytest— 701 passed (48 of them new: 22 storage/blocking, 26 console/permissions)uv run ruff check .— cleanuv run python manage.py check— no issuesuv run python manage.py makemigrations --check --dry-run— no changesbash -n scripts/deploy_relay_sandbox.sh— cleanThree bugs the new tests caught during development, all now fixed: the mailbox list ignored its default filter (an absent
stateparam meant show everything, not show unread); blocking a domain silently blocked nothing, because a bare domain has no@to split on; and body artifacts were keyed without the message id, so the second message overwrote the first's.One deliberate change to the previous test file worth flagging in review: it asserted SNS publishing and the DynamoDB claim, both of which are gone by design, so it was rewritten rather than extended.