Architecture blueprint · living document
A full service-by-service map of mail-engine-serverless's AWS SAM stack onto Oracle
Cloud Infrastructure's native services — every Lambda-to-Function mapping, the DynamoDB-to-
Autonomous-Database data layer, illustrative monthly costs on both clouds at 100,000 emails/day,
and what's still an open gap versus what's since been closed by real, live-verified work.
Looking for a plain-language walkthrough of how the whole system works instead? See the App Guide.
Every managed AWS service this platform uses, and its OCI counterpart. "Direct" means the OCI service covers the same job with a comparable operational model; "Partial" means the concept exists but something material changes — expanded in the deep-dive below. Two rows (Connector Hub, Resource Scheduler) have no AWS-side equivalent at all — they only exist because OCI has no native "queue triggers a function" or "cron triggers a function" primitive the way SQS and EventBridge do. (Resource Scheduler has since been removed from the platform entirely — its hourly floor was the reason; see Section 05's resolved entry.)
| AWS service | OCI equivalent | Fit | What it does here |
|---|---|---|---|
| Lambda (9 functions) | OCI Functions (11 functions) | Direct | All compute — ingestion, sending, API handlers, authorizer, bounce polling, plus campaign_sweeper with no AWS-side equivalent |
| API Gateway (HTTP API) | OCI API Gateway | Direct | Public/protected routes, CORS, custom auth — needs 2 separate gateways (one deployment per path-prefix limit), not 1 |
| Lambda Authorizer | Custom Authorizer Function | Direct | Same SHA-256 X-API-Key hash lookup, but invoked in a distinct named-parameter format with no method/path visible to it |
| S3 | Object Storage | Direct | content.json/recipients.csv staging, console static assets |
| S3 Event Notifications | Object Storage Events + Events Service | Direct | New recipient list triggers ingestion |
| DynamoDB (8 tables + 3 GSIs) | Autonomous Database, ATP Always Free tier (10 tables) | Direct | A real relational engine with native transactions — MERGE replaces DynamoDB's ConditionExpression for every atomic pattern this platform needs, real ORDER BY/batch IN (...) queries work, one-way TLS with no wallet file baked into Function images |
| SQS (EmailSendQueue + DLQ) | OCI Queue | Direct | Per-recipient send jobs; dead-letter behavior is built into the same queue resource, not a second queue |
| SQS-to-Lambda event source | Connector Hub (Service Connector Hub) | Partial | No native queue→function trigger exists; a separate connector resource is required, and each connector invokes serially — concurrency means running N connectors, within a real tenancy limit (2 on Always Free; 20 since the Pay-As-You-Go upgrade) |
| SNS (SES events, alarms) | Notifications (ONS) | Direct | Alarm → email notification works the same way |
| SES (SendRawEmail, identities, DKIM) | Email Delivery | Direct | REST send call (submit_raw_email, not SMTP), domain approval, DKIM via CNAME |
| SES Configuration Set → SNS events | — (no push equivalent) | Partial | No real-time bounce/complaint stream; replaced with a scheduled poller — confirmed ~1 min real latency, not push-instant |
| SES native Tenants (reputation isolation) | — (no equivalent) | Partial | Approximated via one Approved Sender per tenant domain; whether throttling itself is domain-scoped is unconfirmed |
| Lambda Function URLs | API Gateway route | Partial | No bare function URL — every public entry (including Unsubscribe, tracking pixels) needs a Gateway route |
| EventBridge cron → Lambda | Resource Scheduler | Partial | No native cron-to-function trigger; a separate schedule resource with its own IAM condition originally drove bounce_poller and campaign_sweeper — real confirmed minimum granularity is hourly, not finer, which is why it was later replaced by the compute-worker instance's 5-minute loop and removed outright (Section 05, fixed 2026-07-15) |
| CloudWatch Logs | Logging | Direct | Retention only in 30-day increments, not AWS's arbitrary day counts |
| CloudWatch Alarms/Metrics | Monitoring | Direct | Bounce/complaint-rate alarms → Notifications; thresholds are real percentages (HardBounceRate/ComplaintRate, computed in code from a new EmailsSent metric, matching SES's own 5%/0.1% levels), not raw counts — see Section 05's own resolved gap |
| IAM Roles/Policies | IAM Dynamic Groups + Policies | Direct | Per-function least-privilege access; 3 distinct grant mechanisms needed (dynamic group, service X, request.principal.type=serviceconnector) |
| SAM / CloudFormation | Resource Manager (Terraform) | Direct | Declarative infra-as-code, OCI's Terraform provider |
| Deployment package (zip) | Container image (OCIR) | Partial | OCI Functions ship as linux/amd64 Docker images; a shared base image bakes in common/, rebuilt before every dependent function image |
Same shape as the AWS original's own diagram, redrawn with the OCI services actually deployed
today — including the two additions (campaign_sweeper, 2 parallel Connector Hub
connectors) that came out of the scaling pass, not the original port plan.
EmailsSent metricHow each functional area of the AWS codebase maps over, component by component.
9 Python 3.12 functions, zip deployment via SAM, shared src/common/ via a wide CodeUri
Same handlers, packaged as container images — a shared base image bakes in common/, per-function Dockerfiles just FROM it
Rich pre-parsed event — routeKey, pathParameters, requestContext.authorizer, all automatic
No pre-parsed event at all — a compatibility shim (common/http_event.py) reconstructs the same AWS-shaped dict from raw ctx.Method()/ctx.RequestURL()/ctx.Headers(), letting every handler's business logic stay unchanged
Every handler's actual Python logic — validation regexes, MIME building, chunked ingestion, quota reservation, idempotency — is cloud-agnostic stdlib/plain-Python code. The AWS repo's zero-pip-dependency discipline turned out to be a real portability asset: nothing depends on a boto3-specific behavior without an OCI SDK equivalent. The real migration cost sat almost entirely in the data-access layer and each handler's cloud-SDK calls, not the business logic.
Unbounded OCIR image growthFixed 2026-07-12
A container-image deploy has no AWS zip-deployment equivalent for this specific failure mode:
every deploy pushes only to the :latest tag, but the digest that tag previously
pointed at isn't deleted when the tag moves — it just becomes untagged and stays in OCIR
forever. Confirmed live: ~255-306 images across all 12 repos (base + 11 functions), ~72GB
total, after roughly 90 real deploys — genuinely unbounded growth, not a one-time cost.
No native retention-policy resource exists — confirmed by a full
scan of the OCI Terraform provider's own schema (neither
oci_artifacts_container_repository nor
oci_artifacts_container_configuration has a retention attribute) and the CLI's
own command tree (create/get/list/delete/update only, no lifecycle-policy subcommand) — this
had to be a script step in deploy.yml, not a declarative resource.
deploy.yml now prunes every repo down to its newest 10 images after each deploy.
Safety verified live before merging: each docker buildx push actually
creates 3 image records per repo, not 1 (buildx's default build-provenance/SBOM attestations
wrap even a single-platform image in a manifest-list index plus 2 untagged children) — the
tagged :latest digest is always the single newest record in the repo, confirmed
by cross-checking it directly against a real deployed Function's own image URI, so keeping the
newest 10 by creation time always protects the live tag's full manifest cluster plus ~2 more
clusters of rollback buffer. Confirmed live (2026-07-12): the first real run processed
the entire historical backlog in one pass, successfully — every one of the 12 repos landed on
exactly 10 images, down from 255-312 each (~3,400 total images deleted, down to 120), with the
deploy job itself completing all 17 steps green, 0 partial failures. Deletions are best-effort
so a single failed delete couldn't have failed an otherwise-successful deploy either way.
ObjectCreated on recipients.csv invokes Ingest directly; chunked self-invoke (CHUNK_SIZE=2000) via lambda:InvokeFunction
Same event-on-create trigger; self-invoke becomes a detached Functions invocation — but a raw timeout (not a clean return) never fires it, which is what campaign_sweeper exists to fix
PAY_PER_REQUEST, batch_get_item/batch_write_item, conditional update_item with ConditionExpression for atomic quota/webhook claims, TTL on 2 tables
Declared columns + a flexible data JSON overflow column per table, real MERGE-based atomic writes, real ORDER BY/FETCH FIRST pagination, real batch WHERE ... IN (...) queries
Atomic conditional writes — a single MERGE statement, not a retry loop
DynamoDB's ConditionExpression + ReturnValues=ALL_NEW backed 3
correctness-critical patterns (reserve_tenant_quota, update_job,
the webhook-completion claim). Oracle's MERGE statement supports a
WHERE clause on both its WHEN MATCHED and
WHEN NOT MATCHED branches — every one of these patterns is a single atomic
statement in common/atomic.py, with row-level locking handling concurrency
natively. No retry loop at all — a real improvement over DynamoDB's own
conditional-update-plus-retry shape. Confirmed correct under real concurrent load — a real
over-limit quota reservation left the stored value provably unchanged.
Batch suppression checks — one real IN (...) query
batch_is_suppressed is a single WHERE "tenantEmail" IN (...) query
per chunk (Oracle's real IN-list limit is 1,000 expressions, confirmed via
ORA-01795), not N individual round trips.
message_jobs_table row expiry
Oracle has no native row-TTL feature — expiresAt is the real source of truth for
expiry, and a real sweep (common/db.py's reap_expired_rows(), wired
into campaign_sweeper's existing hourly schedule) enforces it across every
TTL-bearing table.
send_message_batch (10/call), per-entry partial-failure reporting, native Lambda event-source mapping, ReservedConcurrentExecutions=5
Batch enqueue has no per-entry failure reporting (fixed via bisection-retry); no native trigger (Connector Hub required); concurrency means N separate connectors, capped at a real tenancy limit — 2 on Always Free (not the 5 originally planned), 20 since the Pay-As-You-Go upgrade
Custom MIME via stdlib email, Configuration Set → SNS push events, native per-tenant "Tenant" resource for reputation isolation
Same stdlib MIME, sent via EmailDPClient.submit_raw_email (REST, not SMTP); DKIM-via-CNAME, but DKIM verification is hard-required to send (SPF is not) and took ~50-100 min live; no native reputation-isolation concept
Still open — per-tenant reputation isolationUnconfirmed, real mitigation shipped instead
The mitigation (one Approved Sender per tenant domain) is implemented, and Monitoring confirms bounce metrics are tracked per sending domain. Whether OCI actually throttles per domain rather than per account was never conclusively proven — a real 10-bounce test was too small a volume for any anti-abuse system to react to, and a larger test was judged too risky to this tenancy's live sending access. This is the exact capability gap the original OCI-based mail-engine was retired over in favor of AWS — carried forward deliberately, not resolved.
Practical mitigation shipped instead (2026-07-13), not an answer to
the question above: rather than keep trying to prove OCI's own isolation behavior, this
makes it stop mattering. bounce_poller's new evaluate_tenant_suspensions
computes each tenant's own rolling 24h hard-bounce/complaint rate — the real denominator is a
new tenantId dimension added to send_email_worker's existing
EmailsSent metric, the numerator a direct query against
suppression_table's existing tenant-scoped index — and sets a real
sendingSuspended flag (in tenants_table's flexible data
column, no schema change needed) the moment that one tenant's own rate crosses the same
5%/0.1% thresholds the account-wide alarms already use. api_campaigns' new
check_tenant_not_suspended enforces it with a 403 on every campaign-creating
route, checked before any other validation work runs. A tiny sample floor (20 real sends
minimum before a rate is even evaluated) keeps one unlucky address from tripping this on a
legitimate small tenant. Deliberately not auto-lifted once the rate recovers —
lifting a real suspension is left as a manual admin action, not automatic, to avoid a
suspend/re-offend/auto-lift loop for a genuinely abusive tenant. This still doesn't prove or
disprove OCI's own per-domain throttling — it just guarantees no single tenant can ever
accumulate enough bad sends to be the one that finds out.
Verified live end-to-end (2026-07-13), not just deployed: exercised
the real deployed code against the real production database and real Monitoring, scoped to one
disposable, clearly-marked fake tenant — a real EmailsSent publish, real
suppression_table rows, then the real evaluate_tenant_suspensions and
check_tenant_not_suspended functions, never Email Delivery's actual send/
suppression APIs, so this carried zero real sending-reputation risk. Confirmed correct on every
count: not suspended before any bounces, correctly suspended once its own rate hit 6.7% (4
bounces / 60 sends, over the 5% threshold) with the right reason text stored, then correctly
blocked by check_tenant_not_suspended afterward. Every trace of the test data
(tenant rows, suppression rows) was deleted afterward — confirmed via a direct re-query showing
nothing left behind.
One real, useful operational finding along the way, unrelated to any code defect:
Monitoring's query-side propagation for a brand-new dimension combination (a tenantId
value never queried before) can take meaningfully longer than for an already-warm series — over
10 minutes for one freshly-generated test tenant ID in this session, while a previously-queried
one stayed reliably queryable within moments. Independently confirmed via direct SDK calls that
this is a first-time-indexing latency quirk, not a bug in the interval/dimension-filter syntax
itself ([1h]/[24h] both work correctly once a series has been queried
at least once). Worth remembering if a tenant's very first suspension evaluation ever needs
debugging — its very first real send's denominator may take longer to become queryable than
steady-state behavior would suggest.
Resolved — no real-time bounce/complaint events
SES's Configuration Set → SNS pipeline made bounce handling near-instant; OCI Email Delivery
has no push equivalent. A scheduled poller against ListSuppressions, diffed against
a stored high-water mark, was built and confirmed live: a real hard bounce reflected into
SuppressionTable in about 1 minute — slower than push, but not the "minutes to
hours" originally feared. Bounce-reason granularity turned out better than expected too — OCI's
6-value reason enum (HARDBOUNCE/SOFTBOUNCE/COMPLAINT/UNSUBSCRIBE/MANUAL/UNKNOWN) is
close to SES's own taxonomy, though the 3-strike soft-bounce counter was simplified away since
OCI appears to track repeat occurrences internally.
Resolved — per-recipient delivery outcomes ("which email soft-bounced?")Fixed 2026-07-16/17
SES's Configuration Sets emit per-message delivery/bounce/complaint events; OCI's
deliverability dashboard is aggregate counts only — a real soft bounce showed up as an
anonymous "1" with no way to tell which recipient deferred (found live 2026-07-16: 1 soft
bounce in 32 sends, unattributable). Fixed with Email Delivery's service logs
(outboundaccepted/outboundrelayed, log group
mail-engine-oci-email-logs): every send now records an accepted and a relayed (or
bounced) entry carrying the recipient address, the receiving mail server, and its literal SMTP
response — verified live the same day (4 recipients, 8 events, each individually attributable).
The catch: these logs attach to individual Email Domain OCIDs, which this platform
creates at runtime, not in Terraform — so Terraform only bootstraps the domains that predated
logging, and api_domains creates the log pair itself for every domain registered
after (best-effort by design: a logging failure warns loudly but never fails a registration).
Retention is 30 days (the OCI minimum — there is no shorter option), and billing is on
ingested volume: ~2 KB per send is negligible even at the full 50k/day cap.
3-day log retention per function, bounce/complaint-rate alarms → SNS → email
Retention floors at 30 days (no 3-day option); alarms publish a custom metric explicitly (no free Configuration-Set-style pipeline); thresholds are real percentages (fixed 2026-07-12 — see Section 05); per-domain Email Delivery service logs added 2026-07-16 record every send's accepted/relayed outcome per recipient, incl. the receiving server's SMTP response
Resolved — concurrency/throttling model
ReservedConcurrentExecutions=5 has no direct OCI Functions equivalent — OCI's
Connector Hub invokes a target function serially, one connector at a time. Fixed by running N
independent connectors against the same queue.
Real limit found live: the plan called for 5 connectors to match AWS's setting; a real
tenancy cap of 2 Service Connector Hub connectors per region surfaced as a
400 LimitExceeded on the 3rd create. Shipped at 2 — the achievable ceiling on the
Always Free tier. The Pay-As-You-Go upgrade (2026-07-15) later raised the limit to 20
(Section 05); the platform still runs 2 deliberately, since that already outpaces the daily
sending cap.
Per-function execution roles, deployed via sam build && sam deploy, OIDC-federated GitHub Actions
3 distinct grant mechanisms for 3 different invoking services (dynamic group for Functions, service cloudEvents for Events, request.principal.type='serviceconnector' for Connector Hub/Scheduler); Terraform via Resource Manager replaces CloudFormation
A Lambda reads a private S3 bucket via IAM, served through API Gateway since S3 website hosting is HTTP-only
Identical shape — a Function reads a private Object Storage bucket, served through OCI API Gateway's default HTTPS endpoint. This piece of the AWS architecture already was the OCI-shaped solution; nothing to redesign
Both tables are order-of-magnitude estimates from each provider's public pricing, not a bill from either account — this project's own real spend is far lower (its live tenancy is capped at 50,000 emails/day, half this volume). Figures marked "not priced" are ones where the public pricing page didn't publish an exact per-unit rate at the time of writing; rather than invent a number, they're called out explicitly.
| Item | Why | Monthly cost |
|---|---|---|
| SES sending | $0.10 per 1,000 emails — the one cost that scales directly with volume | ~$300 |
| Lambda (9 functions) | Billed per invocation + per millisecond used | ~$5–8 |
| DynamoDB (8 tables) | On-demand mode — no idle capacity to pay for | ~$5–8 |
| SQS + SNS + S3 | Comfortably inside the AWS free tier at this volume | ~$1–2 |
| Total | ~$311–318/mo |
| Item | Why | Monthly cost |
|---|---|---|
| Email Delivery sending | A paid service — $0.085 per 1,000 emails, with a $0 monthly billing allowance for the first 3,100. Not an Always Free service (it isn't on Oracle's Always Free services list at all); this is a free usage tier within a metered service, the same shape as the Functions/Queue/Notifications rows below | ~$255 |
| Functions (12 functions) | A paid service with a free monthly allowance: $0.0000002/invocation + GB-seconds, first 2M invocations + 400,000 GB-seconds/mo free — invocation cost alone is negligible at this volume | ~$5–8 |
| Compute worker (A1.Flex 1 OCPU/6GB, 24/7) | Runs the 5-minute bounce-poller/sweeper loop. Metered on provisioned shape-hours (~744 OCPU-hrs/mo), all inside the tenancy-wide A1 Always Free allotment — and unlike everything else in this table, this cost doesn't change with send volume at all | $0 (Always Free allotment) |
| API Gateway (both gateways, incl. open/click tracking) | First 1M calls/mo free, then ~$3.00 per million. At 100k emails/day the dominant traffic is tracking pixels/clicks, not API calls: a typical 20–40% open rate ≈ 0.6–1.2M hits/mo, hovering right at the free line | ~$0–3 |
| Logging (per-recipient delivery logs + function invoke logs) | First 10GB/mo free, then $0.05/GB. The Email Delivery service logs added 2026-07-16 contribute ~2KB per send (accepted + relayed entries) ≈ ~6GB/mo at 3M sends; invoke logs add a few GB more — straddling the free allowance, and pennies per GB past it | ~$0–1 |
| Autonomous Database (ATP, Always Free tier, 10 tables) | This one genuinely is on Oracle's Always Free services list — a real, distinct instance you can run indefinitely at $0, not a usage allowance inside a paid service. $0/mo at this tenancy's real current data volume (~6GB of a hard, non-negotiable 20GB ceiling). A genuine sustained 100,000/day workload would likely need to size up past this single free instance to a real paid ADB shape; Oracle's per-OCPU/per-GB rate beyond Always Free wasn't estimated here since this tenancy has never needed to cross that line | $0 (Always Free) |
| OCI Queue (send + engagement queues) | A paid service with a free monthly allowance: first 1M requests/mo free (64KB/request); the per-million rate beyond that still wasn't published on Oracle's pricing page as of 2026-07-17 (re-checked). At 100k emails/day expect roughly 6–10M requests/mo (enqueue + delivery + connector polling + engagement events) — likely single-digit dollars at any plausible per-million rate, but deliberately not invented here | not priced (likely ~$1–10) |
| Connector Hub | The connector itself is free in every OCI region — pay only for the Functions invocations it triggers, already counted above | $0 |
| Notifications | An Always Free service — within its free-service ceiling (1M notifications/mo) at this alarm volume regardless | ~$0–1 |
| Total | ~3.04M emails/mo at 100k/day; Email Delivery is ~95% of the bill | ~$260–272/mo (+ unpriced Queue, est. ~$1–10) |
Two different "free" concepts are mixed together in this table — worth being precise about. Oracle's Always Free services (a fixed, named list — Autonomous Database's ATP shape and Notifications both qualify; Email Delivery, Functions, and Queue do not) are entire services usable indefinitely at $0 within real limits. Separately, several otherwise-paid services (Email Delivery, Functions, Queue) each carry their own small free monthly allowance before metered billing starts — a $0 amount within a paid service, not the same thing as being on the Always Free list. Only the Autonomous Database row above is the first kind; everything else marked "free" in this table is the second kind.
Bottom line (updated 2026-07-17, with everything now live measured or corroborated): a genuine 100,000-emails/day month costs roughly $265–280 all-in on this platform — about $0.087–0.092 per 1,000 emails, versus ~$311–318 for the AWS original. Sending itself is ~95% of that bill; every standing piece of infrastructure (compute worker, database, NAT, gateways, connectors, the per-recipient delivery logging added 2026-07-16) either rides an Always Free allotment or a free monthly allowance, which the measured-at-idle table below confirms with real Usage API data rather than estimates. Sending is the dominant line item on both clouds and OCI's confirmed per-email rate is roughly 15% cheaper than SES's. The real constraint neither table can price around: this tenancy's actual Email Delivery cap is 50,000 emails/day (the Pay-As-You-Go tier limit, confirmed live 2026-07-15 after the account upgrade — previously 200/day on the sandbox tier) — reaching a literal 100,000/day needs a real Oracle Support quota-increase request, independent of any Terraform or application change; at the current 50,000/day cap the same math halves to roughly $130–140/mo.
Sources: OCI Email Delivery FAQ, OCI Free Tier (the canonical Always Free services list), OCI Autonomous Database pricing, OCI Queue pricing, OCI Connector Hub pricing. The $0.085/1,000-emails and 3,100/mo-free Email Delivery figures could not be independently confirmed against Oracle's own pricing page directly (automated fetches of oracle.com return a 403) — corroborated instead via 2 independent secondary-source searches agreeing on the same numbers, not a primary-source citation. The API Gateway (first 1M calls/mo free, ~$3.00/M after) and Logging (first 10GB/mo free, $0.05/GB after) figures added 2026-07-17 were corroborated the same secondary-source way, against API Gateway pricing and Logging pricing.
Unlike the illustrative table above, this one is measured, not estimated — pulled from the tenancy's own Usage API on 2026-07-15, the day the account upgraded to Pay-As-You-Go, with every current resource live: 11 Functions, both API Gateways, the Autonomous Database, OCI Queue + 2 connectors, the A1.Flex compute worker, Object Storage (including ~24GB of OCIR images), NAT Gateway, and the full Monitoring/Logging/Notifications stack. With no campaigns being sent, the standing platform's run rate is $0/month.
| Item | Why it's $0 at idle | Measured |
|---|---|---|
| Compute worker (A1.Flex 1 OCPU/6GB, 24/7) | ~744 OCPU-hours + ~4,464 GB-hours/month — ~50% of the tenancy-wide A1 Always Free allotment since Oracle quietly halved it to 2 OCPU/12GB (1,500 OCPU-hrs + 9,000 GB-hrs/mo) effective 2026-06-15; still $0, re-verified against real Usage API data 2026-07-17. Metered on provisioned shape-hours, so the mostly-sleeping loop doesn't change this either way | $0.00 |
| Autonomous Database | The genuinely-Always-Free ATP instance, unchanged by the account upgrade | $0.00 |
| Functions, Queue, Object Storage (incl. OCIR), Block Storage, Logging, Monitoring, Notifications | Each inside its own free monthly allowance — including the connectors' constant queue polling and the worker's every-5-minute heartbeat metric, both confirmed metering at $0 in the real data | $0.00 |
| NAT Gateway, Connector Hub, VCN | No standing charge on OCI at all (OCI's NAT Gateway is free, unlike AWS's) | $0.00 |
| Email Delivery | Purely per-email; zero sends means zero cost | $0.00 |
| API Gateway | The one metered item that ever registers: ~$0.002–0.013/day when the console or API is actually used, $0 at true zero traffic | ~$0.00 |
The same Usage API pull also showed two historical month-to-date amounts worth understanding rather than worrying about: ~25 SGD of NoSQL Database (Jul 6–11 only — the old NoSQL tables during the ADB migration, deleted Jul 11, zero every day since and confirmed absent from every compartment) and ~2.85 SGD of Compute (Jul 1–6 instance experiments, zero since). Both predate the Pay-As-You-Go upgrade, when the tenancy was still Always Free and couldn't actually be charged — the Usage API reports rack-rate computed value regardless of billing status. Real billing starts 2026-07-15, from which date everything above measures $0. Housekeeping from the same review: the one genuinely orphaned resource found (a detached 47GB boot volume left behind by a terminated test instance) was deleted the same day.
A full audit (2026-07-10) of every concrete thing standing between this system and a genuine, sustained 100,000 requests/day: hard account-level quotas that need an Oracle Support ticket (not Terraform), config values still tuned for today's near-zero test volume, and what's simply never been tested at anything close to this scale. Updated 2026-07-12 following a real staged DRY_RUN load test (5k/10k/25k/100k recipients, plus a follow-up unthrottled 25k measurement stage) — 165,000 total messages processed end-to-end through the real pipeline, 0 failures, 0 timeouts. Two items below are now resolved as a direct result and have been removed; see the updated note on the Connector Hub quota for what that test found there.
A 2026-07-12 load test now confirms the send pipeline itself (ingestion chunking, the rate limiter, Connector Hub, the ADB connection pool) holds up cleanly at 100,000 recipients in a single campaign — 0 failures across 165,000 total DRY_RUN messages. What's not yet confirmed is real (non-DRY_RUN) sending at that volume, since Email Delivery's own account cap made that impossible to test directly — the items below are what's left once the pipeline itself is no longer the open question.
Hard account-level quotas — need a real Oracle Support ticket, not Terraform
Email Delivery: 50,000 emails/dayAccount quota, raised 2026-07-15
This tenancy's real, confirmed sending limit — raised from 200/day to 50,000/day on
2026-07-15 when the account upgraded to Pay-As-You-Go (verified live via
oci limits value list; the send-rate limit rose to 18,000/minute at the same
time). Every pipeline fix in this document raises how much volume the system can
move — none of it changes what Email Delivery is allowed to hand off. 100,000/day
is now only a 2× increase over the confirmed cap, but that last doubling still needs
a real Oracle Support limit-increase request.
App-level guard shipped ahead of the upgrade (2026-07-12), now
matching the real live cap: ACCOUNT_DAILY_EMAIL_QUOTA (50,000) is reserved
before the existing per-tenant TENANT_DAILY_RECIPIENT_QUOTA on every
campaign submission, via the same atomic MERGE-based reservation primitive against a
sentinel row in tenant_quota_table — no new table needed. The per-tenant quota
alone (also 50,000) only ever bounded one tenant at a time; with more than one active
tenant, their individual quotas could combine to exceed the real account-wide ceiling even
though each stayed under their own limit. With the Pay-As-You-Go cap now live, this guard
guarantees the platform can never collectively schedule more than the account is actually
allowed to send. Verified live: the env var is confirmed present in the Functions
Application's shared config post-deploy.
Service Connector Hub: limit raised to 20 connectors/regionAccount quota, raised 2026-07-15
A single connector invokes its target Function serially (confirmed via OCI's own docs), so
send-side parallelism means running N connectors. The tenancy limit was 2 on Always Free —
the Pay-As-You-Go upgrade (2026-07-15) raised it to 20 (confirmed live via
oci limits value list), with no support ticket needed. The platform deliberately
still runs 2: measured live (2026-07-12), unthrottled, 2 connectors sustain ~50
sends/sec combined (peak steady-state; ~42/sec averaged across a full run), which drains the
full 50,000/day cap in ~17 minutes — more connectors buy nothing until the daily cap itself
moves toward 100,000/day or a single very large campaign needs near-instant draining, neither
of which is a real requirement today. When that changes, raising
send_email_worker_connector_count is now just a Terraform variable change, up to
the new limit of 20.
Autonomous Database: 20GB storage, 1 OCPU/ECPU — hard, non-negotiableAccount quota, confirmed non-blocking
The Always Free tier's real ceiling, confirmed via this tenancy's own
adb-free-count limit. Unlike a capacity-based service where you can raise a
declared limit, this is a single fixed-size instance — there's no Terraform-only way past
20GB once that's used; the only path is sizing up to a paid ADB shape, a real cost this
document's cost table doesn't estimate (see Section 04).
Confirmed live (2026-07-12): real usage is 6.09GB of 20GB (30%), measured
after the 165,000-message load test (Section 05's own load-test note above) --
those rows barely moved the needle, since DRY_RUN rows carry almost no payload. Row-level TTL
is also already fully implemented and hourly-swept (campaign_sweeper →
reap_expired_rows()): campaigns_table/message_jobs_table
at 10 days, tenant_quota_table at 2 days, send_rate_limit_table at 1
day. The remaining tables (tenants_table, api_keys_table,
domains_table, suppression_table, consent_audit_table,
poller_state_table) are deliberately permanent, not an oversight -- suppression
and consent records specifically must never silently expire. With real headroom and no
unbounded growth path, this is no longer a practical blocker.
Configuration still tuned for today's near-zero test volume
Resource Scheduler's hourly floor — replaced by a compute-worker instance, then removed entirelyFixed 2026-07-15
The original gap, confirmed live: OCI Resource Scheduler rejects any CRON frequency finer
than hourly ("Frequency cannot be higher than HOURLY") — not a config choice, a hard service
ceiling — so bounce_poller and campaign_sweeper could only run once
per hour, bounding both bounce-suppression lag and stalled-campaign detection lag at up to an
hour regardless of throughput or storage headroom.
Fixed exactly the way the recommendation recorded here said to: one Always
Free-eligible VM.Standard.A1.Flex (1 OCPU/6GB) compute instance
(terraform/modules/compute-worker) runs both jobs' existing, unchanged
Python app.py logic in a plain 5-minute systemd loop
(compute-worker/worker.py) — a 12× latency improvement, no code rewrite,
no per-invocation billing (it rides the A1 free allotment at $0, see Section 04's real-cost
table), and the API/send/ingest path stayed on Functions untouched. Verified live 2026-07-15:
first cycles clean, Instance Principal + the NAT-routed database path confirmed working, and
a per-cycle WorkerHeartbeat metric with a 15-minute absence alarm pages if the
loop or instance dies for any reason. Getting the shape right took two attempts — the first
try (E2.1.Micro, PR #95/#96) OOM-died during first-boot provisioning on that shape's ~500MB
usable RAM; the Pay-As-You-Go upgrade made A1 capacity obtainable and the retry shipped with
swap-before-dnf and no-weak-deps defenses baked into cloud-init anyway.
Resource Scheduler is now fully removed from the platform (schedules + their IAM policies, same day, once the worker was verified) — the worker loop is both jobs' only trigger, an explicit trade: an instance outage now stops the jobs until fixed (the heartbeat alarm pages within ~15 minutes; both Functions remain deployed for ad hoc manual invocation as a bridge) instead of degrading to an hourly backstop cadence. The alternate Monitoring-alarm + ONS → Function trigger chain investigated on 2026-07-12 remains documented in git history as a compute-free fallback design, unbuilt — its one unverified prerequisite (a reliable every-minute heartbeat metric to alarm on) is moot now that the worker itself publishes exactly such a metric.
API Gateway rate limiting: 5 req/sec per client IPAccepted 2026-07-15 — workaround investigated, deliberately not applied
Deployment-wide (OCI has no route-level throttling), and applies to every route on the
public gateway — POST /tenants, GET /unsubscribe, both tracking-pixel
routes, and docs_server's catch-all — not just the one bootstrap route it was sized
for. 5/sec is plenty tight to stop someone scripting a flood of fake tenant signups (the actual
threat POST /tenants needs protecting from), but the same number gets forced onto
totally different, much higher-volume, completely legitimate traffic it was never sized for.
Concrete example of why that breaks things: picture 50 employees at one company all
opening the same campaign email around 9am. Behind a typical corporate NAT, their traffic often
looks like one shared IP to the outside world. Only the first 5 "I opened this" pings in that
second get through — the other 45 silently vanish, no error, and openedCount
just under-reports real engagement. Worse on /unsubscribe: someone who
explicitly asked to stop receiving mail could have their request silently dropped and still be
on the list, a real compliance risk in a lot of places, not just an analytics gap. Unlike the
Connector Hub/ADB gaps above, this triggers on a burst from one place at one moment,
not total daily volume — it could happen even at fairly low overall volume.
Workaround investigated (2026-07-12): split the public gateway's one deployment into
two — /tenants keeping the tight 5/sec, everything else on a much higher limit.
Confirmed via OCI's own docs this is structurally possible in general ("one deployment per
path prefix per gateway," not "one deployment per gateway"), but blocked here specifically:
docs_server genuinely needs the bare / prefix (serves
index.html at root plus arbitrary asset paths), and Oracle's own docs confirm a
/-prefixed deployment must be the only deployment on its gateway — it
can't coexist with a second /tenants deployment alongside it. Independently
re-verified route-level rate limiting doesn't exist either (dumped the real, current
oci_apigateway_deployment provider schema directly rather than trusting an old
code comment — confirmed no rate_limiting block under routes, only
at the deployment level).
The real fix, identified but not applied: give POST
/tenants its own third gateway (mirroring the exact pattern this project already used
once to solve an almost identical "one deployment per prefix" conflict — that's why public and
protected are 2 separate gateways today). No branding needed for a signup API call, so it could
reuse OCI's own generated hostname like the protected gateway does. Decision: not worth the
added complexity for today's real, still-low traffic — revisit if/when real recipient
volume grows enough that the corporate-NAT scenario above becomes a live concern, not a
theoretical one. The protected gateway (every authenticated tenant/domain/campaign/recipient
route) currently has no rate limiting configured at all either — fine today, same
"revisit at real scale" reasoning applies there too.
No lifecycle policy on the campaigns bucketFixed 2026-07-12
Every campaign's content.json/recipients.csv used to stay in Object
Storage indefinitely — no expiration or archival rule existed. Fixed with a real 10-day DELETE
lifecycle rule (terraform/modules/object-storage), matching
campaigns_table/message_jobs_table's own existing 10-day TTL exactly
— once those DB rows are reaped, the underlying files serve no purpose either.
Live finding along the way: the first apply hit a real
400 InsufficientServicePermissions — Object Storage's own lifecycle-management
engine needs its own explicit IAM grant (Allow service objectstorage-<region> to
manage object-family...), separate from whatever already lets Functions read/write the
bucket. Fixed and confirmed live (oci os object-lifecycle-policy get shows the
rule active).
Never tested or confirmed at anything close to this scale
Custom-authorizer decisions appear cached for ~12-30 secondsSecurity window — bounded but not trivial
Confirmed live: a just-revoked API key kept authenticating successfully for a bounded window
after revocation, then reliably stopped working by the 30s mark — even with API Gateway's
cache_key left unset in Terraform, which should mean no caching at all per OCI's
own schema. A real, minor, bounded security window, not yet mitigated (no confirmed way to
disable it found).
Why this isn't fully trivial (2026-07-12 review): checked create_api_key
(src/api_tenants/app.py) directly — it trusts the authorizer's cached decision
completely, with no additional re-check of the specific key against the database. That means a
just-revoked key used within this window to call POST /tenants/{tenantId}/
api-keys can mint a brand-new, genuinely valid key that was never revoked — not subject
to this caching window at all, since it's fresh. A bounded 30-second problem can become
silent, effectively permanent persistence: whoever revoked the original key has no
reason to know a second one got minted in that window, so wouldn't think to revoke it too.
Real-world severity, honestly assessed: this requires an attacker
who already holds a valid-but-about-to-be-revoked key and is actively watching for
revocation closely enough to slip a request in during that exact window — not a way in from
nothing, just a way to potentially outlive getting caught. Given this platform's current real
scale (low volume, mostly test tenants, no high-value target profile yet), practical risk is
low today. Not urgent, but not something to consider fully closed either — worth a real fix
(e.g. re-validating the specific key against the database inside create_api_key
specifically, not just trusting the authorizer's cache) once real customer data is at stake.
Tracking-burst database exhaustion (Functions Application concurrency)Fixed 2026-07-15
The original gap: Terraform doesn't set an explicit concurrency limit on the shared
Functions Application. The
2026-07-12 load test did run ingest_campaign's 50 self-invoked chunks and 2
send_email_worker connectors concurrently for the full ~83-minute 100k run with 0
issues — real evidence the Application handles this platform's own real concurrent invocation
pattern fine. What that never exercised: a real burst of inbound traffic (
tracking-pixel opens/clicks arriving all at once), a different concurrency shape than anything
this load test exercised.
The calculated risk that motivated the fix (2026-07-12): Oracle's own docs confirm this tenancy's
Always Free ADB has a hard ceiling of 20 total simultaneous database sessions — separate
from, and far smaller than, anything Connector Hub or ADB storage were bound by. Each Function
container holds its own connection pool (min=1, max=4,
common/db.py), and OCI Functions containers each handle one invocation at a
time — so a real burst of simultaneous opens/clicks/unsubscribes forces that many separate
containers to cold-start in parallel, each grabbing at least one DB connection, against a
ceiling that's already partly consumed by baseline activity (the 2
send_email_worker connectors, any in-flight ingest_campaign chunk).
The math suggests something as small as ~15-20 simultaneous tracking/unsubscribe
requests — not hundreds — could plausibly exceed real capacity and throw genuine
ORA-00018: maximum number of sessions exceeded errors for the overflow. That's an
easy threshold to hit in practice (20 people opening the same email within the same second is
completely ordinary for even a modest company), unlike the Connector Hub/storage gaps where the
math found huge headroom instead.
Fixed architecturally (2026-07-15), not by paying for more sessions:
the public tracking/unsubscribe Function no longer touches the database from the request path
at all. Every route (/track/open, /track/click,
/unsubscribe, /tracking-optout) now enqueues its event to a second
OCI Queue (the engagement queue) — an HTTP put, no database session — and a new
engagement_worker Function applies the actual writes behind 2 Connector Hub
connectors, the identical serial-per-connector pattern the send path already uses. Database
concurrency for engagement writes is therefore bounded at the connector count (2) no matter
how many pixels load simultaneously — the burst scenario above physically cannot reach the
20-session ceiling anymore. Failure semantics are deliberate: a failed enqueue on
open/click is swallowed (the pixel/redirect must always render; a lost open record is
harmless), while unsubscribe/tracking-optout enqueue failures return a 5xx rather than
falsely confirming a compliance action that was never recorded. The engagement queue's
dead-letter count rides the existing DeadLetteredMessages metric/alarm under its
own queue dimension. The cost is seconds of write latency on data nobody reads
synchronously — the same poll-interval-bound tolerance suppression itself already has. A paid
ADB tier (300 sessions at 1 OCPU) remains available as pure headroom if some other
workload ever crowds the ceiling, but is no longer needed for this scenario.
Verified live the same day (2026-07-15), keeping this document's
confirm-before-relying discipline: a real burst of 80 simultaneous requests against the
live /track/open//track/click endpoints (garbage
campaign/recipient pairs — try_claim_once no-ops on rows that don't exist, so
nothing was written and no email sent) produced 13 instant successes and 67 clean
429s from the public gateway's own 5 req/sec per-client-IP rate limit — itself
a second, outer layer of burst protection this scenario's original math didn't credit — with
zero 5xx and zero database errors. A follow-up 2-minute sustained run: 82/82 succeeded.
Every one of the 99 events reconciled 1:1 against engagement-worker's
FunctionInvocationCount, its FunctionErrorCount stream is empty,
and the engagement queue's DLQ held 0 throughout — since Connector Hub only deletes a
message on a successful invocation, a drained queue plus an empty DLQ is positive proof
every event applied to the database. The compliance routes
(/unsubscribe, /tracking-optout) were exercised end-to-end too,
landing real rows via the worker. One caveat honestly noted: a single test machine cannot
simulate a genuinely distributed burst (many client IPs at once) — the per-IP rate
limit blocks that by design — but the protection doesn't depend on the gateway: whatever
reaches the platform from any number of IPs lands on the queue, and the database only ever
sees the 2 connectors' sessions.
No DLQ-specific alarmFixed 2026-07-12
The 2 Monitoring alarms used to watch bounce/complaint rate only — nothing watched OCI Queue's dead-letter portion, so a systematic send failure (a bad Email Delivery response, a misconfigured sender) could silently pile messages there with no alert firing at all.
Fixed with a real custom metric, not a native one — confirmed live there's no built-in
Monitoring metric for this at all (DroppedMessagesCount in the
oci_queue namespace is specifically about filter-mismatch drops, unrelated to
dead-lettering). common/queue.py's new get_dlq_message_count() calls
QueueClient.get_stats directly (confirmed real response shape against the
installed SDK's own model source: separate .queue/.dlq stats
objects). campaign_sweeper now publishes
mail_engine_oci/DeadLetteredMessages every hourly run — unconditionally, including
zero, since an alarm needs a continuous data point to evaluate reliably, unlike
SuppressionsReflected's own only-when-nonzero choice. A new alarm
(DeadLetteredMessages[2h].max() > 0) watches it, same topic/severity as the
existing bounce/complaint alarms. A new read queues IAM grant was needed for
GetStats specifically — confirmed via OCI's own Queue policy reference as a
distinct resource-type/verb from the existing queue-push/queue-pull grants.
Verified live, not just deployed: manually invoked
campaign_sweeper in production and confirmed it actually published
DeadLetteredMessages = 0.0 to Monitoring — the real value right now, consistent
with this session's DRY_RUN testing never having produced a genuine failure. One honest caveat
still open: this closes the alerting gap, but the underlying failure/retry/dead-letter
path itself still hasn't been exercised by a real send failure — the alarm now exists
to catch it whenever that first happens, but it hasn't been tested against a real trigger yet.
Alarm thresholds are absolute counts, not volume-normalizedFixed 2026-07-12
bounce_poller had no visibility into total send volume to compute a real rate
against (unlike SES's own CloudWatch percentage-based thresholds). Thresholds tuned as
reasonable at near-zero current volume would either fire constantly or never mean anything at
real 100k/day volume.
Fixed by computing the rate in code, not in the alarm query — confirmed against OCI's
own MQL reference docs that Monitoring's query language can't divide two different metrics
against each other, so a real percentage has to be computed before it's published, not at
alarm-evaluation time. send_email_worker now publishes a new
mail_engine_oci/EmailsSent metric on every real successful send (never during
DRY_RUN). bounce_poller reads that back via a second Monitoring
client (the read-back needs the regular telemetry endpoint, not the
ingestion-only one already in use for writes) and publishes real
HardBounceRate/ComplaintRate percentages every poll cycle,
unconditionally — including zero, since an alarm needs a continuous data point. The 2 alarms
now query HardBounceRate[1h].mean() > 0.05 and
ComplaintRate[1h].mean() > 0.001, matching AWS SES's own published
thresholds (5%/0.1%) instead of raw counts. A new read metrics IAM grant was
needed for the read-back specifically, distinct from the existing use metrics
write grant.
Verified live, not just deployed: the first real invocation after
deploy surfaced a genuine production bug — OCI's PostMetricData rejects any
datapoint with empty dimensions (a requirement undocumented anywhere obvious),
which crashed bounce_poller outright and would have caused
send_email_worker to mark real successful sends as failed, since the metric
publish sat before the final status write with no isolation. Fixed same-session: added a
real source dimension to all 3 new metrics, and wrapped both publish call sites
in their own try/except so a metrics failure can never affect core send/suppression
correctness. Re-verified after that fix: manually invoked bounce_poller in
production, confirmed HardBounceRate/ComplaintRate now exist as
real registered metrics in the namespace, and pulled back actual datapoints
(HardBounceRate = 0.0, correctly zero against zero recent sends) with the
correct dimension attached.
DKIM verification: ~50-100 min per new tenant domainOnboarding gap
Not a live-sending bottleneck, but a real onboarding-velocity one: confirmed live across multiple domains, with meaningfully variable timing (the widest observed retry budget needed was 100s just for the DKIM-creation call itself to succeed, separate from the ~50-100 minute background verification). Onboarding many tenants at once to ramp toward real volume means each one individually waits on this.
Investigated, not fixed — the wait itself is a real OCI-side
constraint, same category as every ESP's own DNS-based domain verification (AWS SES
included). No Terraform config or app code changes that window. What's already in place,
confirmed by re-reading the actual code: a real POST /domains/{id}/verify
status-check endpoint (returns verified/dkimStatus, documented in
openapi.yaml with explicit "normal to call this more than once" guidance), a
console Domains panel with a live verified/not-verified pill, a Verify button, an FAQ
explaining the real ~50-55 min wait, and a campaign-form domain dropdown that flags
unverified domains so a tenant can't accidentally pick one that'll fail to send. One small
honest gap found, left as-is: the post-registration success banner says generic "DNS
propagation can take minutes to hours" rather than the more specific ~50-55 min figure (that
number currently lives only in the FAQ and OpenAPI docs) — a minor copy polish, not a
missing capability, not worth a deploy on its own.
Single region, single Functions ApplicationAvailability gap
No multi-AD or multi-region failover story exists anywhere in this stack today. Not evaluated as part of any scaling pass so far — a real gap if 100,000 requests/day comes with any uptime expectation, separate from raw throughput.
Deliberately deferred, not overlooked — and no longer blocked:
this tenancy was on OCI's Always Free tier, which restricted every resource to its one home
region (us-ashburn-1) and made a second region impossible to stand up at all.
That blocker was removed on 2026-07-15, when the account upgraded to Pay-As-You-Go
(the same upgrade that raised the Email Delivery cap to 50,000/day). Multi-region/multi-AD
failover is now buildable in principle — it remains an unbuilt, planned item, but the reason
is now prioritization, not an account-tier restriction.
Sizing past the Always Free database tier is unpricedCost unknown
Section 04's own cost table already flags this — the $0/mo Always Free ADB shape is a single fixed-size instance (1 OCPU/ECPU, 20GB), and closing the 20GB gap above means moving to a real paid ADB shape. No real cost model exists yet for what that would cost monthly, on either database compute/storage or Logging volume at 100k/day worth of function invocations.
Deliberately left unpriced, not overlooked: the plan has always been to price this from a real bill, not OCI's list pricing — matching the "confirm live, don't guess" discipline used for every other limit in this document (Section 04's own cost table, Known Gap #4's load-test numbers, etc.). The account-level Pay-As-You-Go upgrade this was waiting on happened on 2026-07-15, so real measurement is now possible — the ADB itself still runs on its Always Free shape (which persists unchanged under Pay-As-You-Go) and hasn't needed to size up yet; the real cost number gets recorded here when it actually does, from this platform's own measured usage against a real paid shape.
Found via a self-audit of this session's own new features (2026-07-13)
After the per-tenant reputation-isolation mitigation shipped, the platform was re-audited for concrete gaps its own new code introduced or left unaddressed, rather than only re-checking the same list already known. 6 found, all real and verifiable against the actual code — 2 fixed same-session, 4 documented and deliberately left open.
Tenant auto-suspension was completely silentFixed 2026-07-13
evaluate_tenant_suspensions only did print(f"SUSPENDED tenant...")
on a real suspension — that reaches Function logs and nowhere else. Unlike the bounce/
complaint-rate and DLQ alarms (each a real Monitoring alarm + ONS topic), nobody was actually
notified when a tenant got suspended — a human had to go looking, or wait for the tenant to
complain. Undermined the whole point of the feature: a safety mechanism nobody finds out
about isn't really a safety mechanism.
Fixed by publishing directly to the same ONS topic every other alarm
already uses — not a 4th Monitoring alarm watching a new metric, deliberately: alarm
evaluation has its own real lag (the account-wide alarms evaluate on roughly a 1-minute cycle
at best), and a real suspension deserves an immediate notification, not one waiting for the
next evaluation cycle. notify_tenant_suspended calls
NotificationDataPlaneClient.publish_message directly, right after the suspension
write succeeds, isolated in its own try/except (the suspension itself, already fully applied
by that point, must never be undone or masked by a notification failure). A new
use ons-topics IAM grant was needed — confirmed against OCI's own IAM policy
reference that use (not manage) is the documented verb covering
PublishMessage specifically.
No way to lift a suspension except raw SQLDeliberate, but a real operational gap
Confirmed live: no admin API route, no console UI, nothing — grepped both
api_tenants/app.py/api_campaigns/app.py and docs-site/
index.html directly. This was a deliberate design choice when the suspension feature
was built (avoiding a suspend/re-offend/auto-lift loop for a genuinely abusive tenant needs a
human in the loop somewhere), but it's still a real gap: whoever needs to lift one has to know
the exact database update by hand, via the same ad hoc local-ADB-access procedure documented
in this project's own Operations section — no self-service, no admin authentication surface
at all.
DLQ has alerting but no reprocessing pathFixed 2026-07-13
The DLQ-specific alarm (an earlier fix this session) correctly detects and pages on dead- lettered messages — but nothing in the codebase could actually recover one. A message that landed there just sat, permanently manual-only recovery.
Fixed with a real proxy, not direct DLQ access — because direct access
genuinely doesn't exist. Confirmed directly against the installed OCI SDK:
QueueClient exposes only delete_message(s)/get_messages/
get_stats/list_channels/put_messages/
update_message(s) — get_messages has no dead-letter-specific mode
(only channel_filter/consumer_group_id), and get_stats
returns just a count. There is genuinely no API path to inspect or requeue one specific
dead-lettered message by identity — a real OCI service limitation, not a missed SDK call.
campaign_sweeper's new reprocess_dead_lettered_messages uses the
real achievable proxy instead: message_jobs_table already has full per-recipient
visibility into every failure, and a message only ever reaches the actual dead-letter portion
after 3 straight transient failures (a genuinely permanent one never gets redelivered at all,
confirmed via send_email_worker's own is_permanent_send_failure
logic) — so re-queuing a campaign's own FAILED rows is a safe, real proxy for
retrying exactly what could plausibly be dead-lettered, without ever needing to read the DLQ's
own content. Bounded to retry each row at most once (a new sweepRequeuedAt
marker, in message_jobs_table's existing flexible data column) —
otherwise a genuinely permanent failure, which looks identical to a transient one in this
table, would get re-queued every single hourly sweep forever.
Suppression is permanently one-wayNo reversal path, self-service or admin
Confirmed live: no route anywhere removes an address from suppression_table.
Once suppressed — a hard bounce, a mistaken unsubscribe click, a mailbox that was temporarily
full and is fine now — that address is blocked from every future campaign, from every tenant,
forever. No admin override, no self-service "I'd like to receive mail again" path. Deliberately
conservative by default (matching how aggressively this platform treats suppression
elsewhere — see Section 09's "no 3-strike counter" reasoning), but a real, undocumented
one-way door nonetheless.
No real-infrastructure testing in CIStructural — every fix this session needed manual live verification
Confirmed by reading .github/workflows/ci.yml directly: it runs the mocked-SDK
pytest suite plus terraform validate/fmt, nothing more. No stage
anywhere exercises real OCI services automatically. This isn't a hidden gap so much as the
explanation for a pattern visible throughout this whole document — nearly every fix in this
field guide required a manual "verified live" step after deploying, precisely because the
pipeline itself has no automated way to catch a live-infrastructure bug before it ships.
No deploy rollback or canary mechanismEvery deploy replaces 100% of production immediately
Confirmed by reading deploy.yml directly: terraform apply then a
force-redeploy loop, straight to every function, no staged/canary rollout, no automated
rollback on error-rate spike. If a bad deploy ships, the only safety net is whatever the
mocked test suite happened to catch beforehand — nothing in the pipeline itself would notice
or revert a live regression after the fact.
No separate PROD environment — "dev" is the only environment, and it's livePlanned: stand up a real PROD, keep this one as DEV
Confirmed by the repo's own layout: terraform/envs/dev/ is, in this project's
own words, "the one real environment" — but it's also the only one, and it's the one
actually serving real traffic (mail.rissolv.com, real tenants, real Email
Delivery sends). Every merge to main runs terraform apply and a
force-redeploy directly against this same environment, with nothing standing between a
Terraform change and real production infrastructure. This compounds the deploy-rollback gap
immediately above — not only does a bad deploy go straight to 100% of production with no
rollback, there's also no separate environment where that risk could be caught first.
Planned, not yet built: a genuinely separate
terraform/envs/prod/ environment, with the current live one continuing on as
DEV going forward rather than being renamed or replaced in place. Real open questions this
still needs, not yet decided: whether PROD starts fresh (empty tenants/data, DEV keeps all of
today's real tenants and history) or the current live tenants/data get migrated into the new
PROD and DEV is reset to a clean slate — a real data-migration decision, not just a
copy-pasted Terraform directory. Also needs its own remote state (a second S3-backend prefix
or bucket, distinct from mail-engine-oci-tfstate's current single-environment
state), its own set of GitHub secrets/OCI resources (a second ADB, second set of Email
Delivery domains, etc. — none of it can safely share the live DEV/current environment's real
infrastructure), and a real decision on how deploy.yml should target one
environment or the other going forward.
Compliance & regulatory — reviewed 2026-07-13, starting from "erasure already exists, what else?"
A follow-up review specifically asking what's still missing beyond the existing GDPR/CCPA
erasure route (api_recipients) and the CAN-SPAM footer/one-click-unsubscribe
already in common/mail_sender.py. 3 found, all real — all three closed on
2026-07-15.
No "right of access" (GDPR Article 15)Fixed 2026-07-15
The original gap: api_recipients had exactly two routes —
erase_recipient (delete) and list_suppressions — so a data
subject could have their records erased (Article 17) but never request a copy of what's
held about them (Article 15), two separate, both-required GDPR rights.
Fixed exactly the way this entry predicted: GET
/tenants/{tenantId}/recipients/{email} mirrors erase_recipient's own
design (same tenant-scoped message_jobs_table lookup, same case-insensitive
fallback so a casing mismatch can't read as "nothing is held") and returns everything:
per-campaign send history including open/click tracking timestamps — exactly the data a
recipient most plausibly doesn't know exists — suppression status and reason,
tracking-opt-out status, and the consent basis archived for each campaign that included
them.
consentBasis was unvalidated free textFixed 2026-07-15
The original gap: every campaign required a consentBasis string,
permanently archived in consent_audit_table — but anything typed was accepted
and archived as if it were a real legal basis. Mattered most for CASL (Canada),
whose genuine-opt-in standard is materially stricter than CAN-SPAM's opt-out model.
Fixed: consentBasis is now a closed, canonical vocabulary
(express_consent, implied_consent_existing_relationship,
implied_consent_inquiry, transactional_relationship,
legitimate_interest) — chosen specifically to distinguish the standards that
differ by jurisdiction (CASL's express-vs-implied split; GDPR's enumerated bases). Free
text is rejected with a 400 naming the allowed values; specifics move to the optional
consentBasisDetail field, archived alongside — and required for
legitimate_interest, since naming that basis without stating the interest is
exactly the empty claim this fix exists to reject. The console's consent dropdown now
submits the canonical values (its free-text "Other" escape hatch is gone).
No disclosure or opt-out for open/click trackingFixed 2026-07-15
The original gap: every sent email got an invisible open-tracking pixel and every link rewritten for click-tracking, with no visible disclosure anywhere a recipient would see it, and no way to keep receiving mail while declining tracking specifically — only the all-or-nothing unsubscribe.
Fixed, both halves: every tracked email now carries a visible disclosure footer
stating plainly that it uses an open-tracking image and measured links, with a "keep
receiving these emails without tracking" link to the new public
GET /tracking-optout route (same unauthenticated recipient-facing contract as
unsubscribe). An opt-out lands in the new tracking_optout_table (deliberately
NOT extra rows in suppression_table, whose rows mean "never send at all") and
is honored per recipient at send time: their emails go out with no pixel, no rewritten
links, and no disclosure footer making claims that no longer apply. The opt-out link itself
is deliberately never click-tracked — a tracked "stop tracking me" link would contradict
itself.
Written after the first real 16-recipient campaign (2026-07-16, campaign
058bbbed…): every email delivered successfully, zero bounces — and every one
landed in the spam folder. This is not a platform bug, and it's worth understanding why
before assuming something needs fixing.
Authentication was verified correct at the time, live against public DNS: SPF
(v=spf1 include:rp.oracleemaildelivery.com ~all on rissolv.com),
DMARC published (p=none with rua reporting), and per-domain DKIM
signing (the thing submit_raw_email hard-requires anyway). Passing
SPF/DKIM/DMARC is the prerequisite for inbox placement, not a guarantee of it —
what's left is reputation, and a new sender has none.
Why a correctly-authenticated email still goes to spam, in this platform's exact situation:
The warm-up playbook, in order of impact:
p=none toward
quarantine/reject (a trust signal in itself), and revisit content
shape — more plain personal text, fewer links — for anything that still folders.The concrete week-by-week plan for rissolv.com (sized to the actual
starting point: a cold domain, Oracle's shared IP pool, and a seed list of ~16 friendly
mailboxes). The core rule throughout: volume grows only as fast as positive engagement
does, and Postmaster Tools is the scoreboard.
p=none →
p=quarantine.None of this is a platform change — the pipeline's job (authenticated, compliant, suppression-checked delivery with disclosed tracking) is done and verified. Inbox placement from here is a sender-reputation game played over weeks of real, wanted sending.