fix: retry Google's transient domain-wide-delegation propagation-delay error

Live Google Workspace delivery validation surfaced a real, self-resolving
Google-side condition: freshly-authorized domain-wide delegation can take
up to about a minute to propagate across Google's auth infrastructure,
producing intermittent unauthorized_client on the OAuth token exchange
even with correct configuration. gws-provider.js now classifies
unauthorized_client as a temporary (retryable) error instead of permanent,
so the existing bounded-backoff worker absorbs it automatically.

Also documents the alias-vs-delegated-user env passthrough in
docker-compose.dev.yml, and records both real findings (this one and the
GOOGLE_DELEGATED_USER alias distinction) plus the full live-delivery
validation outcome in the runbook and Phase 5C doc.

Validated live: isolated gws-email-test.js send (DKIM/SPF passing,
delivered) and a full real lifecycle test (create student, enrollment,
real activation email, activate, login, see program, real password-reset
email, reset, login with new password) via [email protected].

docs/PHASE5C_ENROLLMENT_COMMUNICATIONS.md

@@ -1,7 +1,9 @@
 # Phase 5C — Enrollment Operations & Communications
 
 **Status: PHASE 5C COMPLETE. ENROLLMENT COMMUNICATIONS COMPLETE. STUDENT ACTIVATION PIPELINE READY.**
-**LIVE GOOGLE WORKSPACE DELIVERY VALIDATION PENDING** — see §14.
+**LIVE GOOGLE WORKSPACE DELIVERY VALIDATED** — real activation and password-reset emails
+delivered end to end via `[email protected]` (a Workspace alias), DKIM/SPF passing,
+inbox delivery confirmed. See §14.
 
 Phase 5B built the enrollment/entitlement model and the website integration boundary. What was
 still missing for a real student: nobody ever told them. This phase closes that gap — real
@@ -221,8 +223,26 @@ is exhausted, then `failed` too. Never an infinite loop; every attempt is persis
   then this phase's migration applied — every prior record intact; a new notification correctly
   links to the pre-existing student/enrollment; re-running the migration set is a no-op.
 
-**Live Google Workspace delivery has not been validated** — no service account/domain-wide
-delegation exists yet for this RootNode Workspace domain. The adapter (`gws-provider.js`) is
-fully implemented and unit-testable at the code level; `infra/scripts/gws-email-test.js` is
-ready to run the moment credentials exist (opt-in, `RUN_GWS_EMAIL_TEST=true`, sends exactly one
-message to one operator-configured test address — see the runbook).
+**Live Google Workspace delivery has been validated for real**, both in isolation
+(`infra/scripts/gws-email-test.js`, one message, DKIM/SPF passing, delivered) and through the
+full genuine flow: a real student created, enrolled, receiving a real activation email, setting
+their password from the real link, logging in, seeing their enrolled program, requesting and
+receiving a real password-reset email, and completing the reset — every step against the live
+Platform API, `EMAIL_PROVIDER=gws`, nothing simulated.
+
+**One real Google Workspace limitation found and handled:** newly-authorized domain-wide
+delegation does not take effect instantly across all of Google's auth infrastructure — the
+identical token request succeeded, then failed with `unauthorized_client`, then succeeded again,
+over the span of about a minute, before settling. This is Google-side propagation delay, not a
+configuration error, and it self-resolved. `gws-provider.js` now classifies `unauthorized_client`
+as a *temporary* failure (retried with the existing bounded backoff) rather than permanent —
+this is exactly the failure mode that backoff exists for, and it means a real deployment doesn't
+need a human to notice and manually retry during that propagation window. See
+`docs/RUNBOOK_GOOGLE_WORKSPACE_EMAIL.md`'s troubleshooting section.
+
+**A second real-world detail found during setup, not a bug:** the sending address
+(`[email protected]`) is a Workspace *alias* of an existing account rather than its
+own paid seat — a deliberate cost-saving choice. Domain-wide delegation authorizes a real
+account, not an arbitrary alias string, so `GOOGLE_DELEGATED_USER` (the alias's real owner) was
+added as a setting distinct from `EMAIL_FROM_ADDRESS` (the visible sending address) — see §2 and
+the runbook §5.

docs/RUNBOOK_GOOGLE_WORKSPACE_EMAIL.md

@@ -29,6 +29,16 @@ you set `EMAIL_FROM_ADDRESS` to (§5).
    on it anyway, even though RootNode's application will never use that password — this account
    still exists as a real Workspace identity and should be secured like any other.
 
+**If the sending address is an ALIAS of an existing account (a real cost-saving option — you
+don't need a whole extra paid seat just for this):** that's fine and supported, but note it for
+step 4 below. Domain-wide delegation authorizes the service account to act as a real Workspace
+**user account**, not an arbitrary alias string — if `[email protected]` is an alias of,
+say, `[email protected]`, the delegation in step 4 is granted for `chase@`'s account
+(the alias's owner), and RootNode's config keeps the alias as the visible sending address
+separately (see §5's `GOOGLE_DELEGATED_USER` vs. `EMAIL_FROM_ADDRESS`). Gmail already lets any
+account send as its own verified aliases, so this works cleanly — you just need both pieces of
+information (the alias, and which account owns it) when configuring RootNode.
+
 ## 3. REQUIRES GOOGLE WORKSPACE ADMIN ACTION: create the service account (Google Cloud Console)
 
 1. In Google Cloud Console (console.cloud.google.com), create or choose a GCP project associated
@@ -54,8 +64,10 @@ you set `EMAIL_FROM_ADDRESS` to (§5).
    granting them would violate least-privilege for no benefit.
 4. Authorize.
 
-This is the step that lets the service account act as `EMAIL_FROM_ADDRESS` specifically — it
-does not grant access to any other mailbox in the domain.
+This grants the service account permission to act as **one specific Workspace user account**.
+If the sending address is that account's own primary login, that's the account. If the sending
+address is an ALIAS (§2), this authorization is inherently for the alias's owner account — there
+is no separate "authorize an alias" step, because delegation only ever targets real accounts.
 
 ## 5. RootNode-side environment variables
 
@@ -66,11 +78,16 @@ dev: `platform/.env`, already gitignored):
 |---|---|---|
 | `EMAIL_PROVIDER` | `gws` | Unset/anything else defaults to the dev provider — never real mail. |
 | `GOOGLE_SERVICE_ACCOUNT_JSON` | the full JSON key file's contents, as one string | The sensitive credential — see §8. |
-| `EMAIL_FROM_ADDRESS` | the mailbox from §2, e.g. `norepl[email protected]` | Also the delegated-user (`sub`) claim. |
+| `EMAIL_FROM_ADDRESS` | the address students see mail arrive from, e.g. `academ[email protected]` | Always the visible `From:` header. |
+| `GOOGLE_DELEGATED_USER` | **only needed if `EMAIL_FROM_ADDRESS` is an alias** — the real account that alias belongs to, e.g. `[email protected]` | This is what domain-wide delegation actually authorized in step 4. If `EMAIL_FROM_ADDRESS` is itself a primary account, leave this unset — the code falls back to `EMAIL_FROM_ADDRESS` automatically. |
 | `EMAIL_FROM_NAME` | e.g. `RootNode Academy` | Display name only, not sensitive. |
 | `EMAIL_REPLY_TO` | optional, e.g. `[email protected]` | Where a student's reply actually goes, if different from the sending address. |
 | `PUBLIC_APP_ORIGIN` | e.g. `https://learn.rootnodeacademy.com` | The origin activation/reset links are built from (§7 of the Phase 5C authorization) — required for any token-bearing email to render a working link. |
 
+**Concrete example matching the alias setup described above:** `[email protected]`
+(what students see), `[email protected]` (whose account actually
+owns that alias and was granted delegation in step 4).
+
 ## 6. DNS: SPF, DKIM, DMARC (REQUIRES DNS ADMIN ACTION — not performed from this repository)
 
 Google Workspace's own documentation is authoritative here; this is a summary of what must
@@ -104,13 +121,16 @@ network. `platform/test/e2e-phase5c.js` and the Phase 5C drills run entirely thi
 RUN_GWS_EMAIL_TEST=true \
 TEST_EMAIL_RECIPIENT=[email protected] \
 GOOGLE_SERVICE_ACCOUNT_JSON="$(cat /path/to/service-account-key.json)" \
-EMAIL_FROM_ADDRESS=norepl[email protected] \
+EMAIL_FROM_ADDRESS=academ[email protected] \
+GOOGLE_DELEGATED_USER=[email protected] \
 EMAIL_FROM_NAME="RootNode Academy" \
 node infra/scripts/gws-email-test.js
 ```
-Sends exactly one clearly-marked test message to `TEST_EMAIL_RECIPIENT` — never inferred from
-production/student data, never run automatically as part of any regression suite. Without
-`RUN_GWS_EMAIL_TEST=true` it does nothing and exits cleanly.
+(Omit `GOOGLE_DELEGATED_USER` entirely if `EMAIL_FROM_ADDRESS` is itself a primary account
+rather than an alias — see §5.) Sends exactly one clearly-marked test message to
+`TEST_EMAIL_RECIPIENT` — never inferred from production/student data, never run automatically as
+part of any regression suite. Without `RUN_GWS_EMAIL_TEST=true` it does nothing and exits
+cleanly.
 
 ## 8. Production secret handling
 
@@ -151,7 +171,7 @@ individual key — rotating keys does not require re-authorizing delegation.
 | Symptom | Likely cause |
 |---|---|
 | Every send fails with `gws_not_configured` | `GOOGLE_SERVICE_ACCOUNT_JSON` or `EMAIL_FROM_ADDRESS` unset/malformed in the Platform API's environment. |
-| `gws_auth_failed` | Domain-wide delegation not authorized for this Client ID/scope, or the service account key was deleted/revoked. Re-check §4. |
+| `gws_auth_failed` | Domain-wide delegation not authorized for this Client ID/scope, or the service account key was deleted/revoked. Re-check §4. **If this happens intermittently right after newly authorizing delegation** (the exact same request succeeds, then fails with `unauthorized_client`, then succeeds again, over roughly a minute) — that's Google's own auth infrastructure propagation delay, not a config error. It self-resolves; the notification worker's existing retry/backoff already absorbs it (`gws-provider.js` classifies `unauthorized_client` as retryable), so no action is needed beyond waiting. |
 | `gws_http_403` | The delegated `sub` (EMAIL_FROM_ADDRESS) isn't a real mailbox in this Workspace domain, or the Gmail API isn't enabled on the GCP project (§3.2). |
 | `gws_rate_limited` (HTTP 429) | Gmail API per-user sending limits hit — see §12. The notification worker retries this automatically with backoff. |
 | Mail sends successfully but lands in spam | SPF/DKIM/DMARC (§6) not fully configured/passing yet — this is a deliverability issue, not an application error. |

platform/api/src/email/gws-provider.js

@@ -1,6 +1,16 @@
 // Phase 5C §2 — the Google Workspace adapter. Sends via the Gmail API
 // (users.messages.send), authenticated as a Google Cloud service account using Workspace
-// domain-wide delegation to impersonate a dedicated sending mailbox (EMAIL_FROM_ADDRESS).
+// domain-wide delegation to impersonate a Workspace user.
+//
+// GOOGLE_DELEGATED_USER vs. EMAIL_FROM_ADDRESS — two different things, deliberately kept
+// separate: domain-wide delegation authorizes the service account to act as one real Workspace
+// USER ACCOUNT (the "sub" claim below) — if the sending address is an ALIAS of a real account
+// rather than its own paid seat (a deliberate cost-saving choice, not a mistake), the delegated
+// user must be the ALIAS'S OWNER (the real account it belongs to), not the alias string itself.
+// EMAIL_FROM_ADDRESS is only ever the visible From: header — Gmail lets any account send as its
+// own verified aliases, so this can safely be the alias while GOOGLE_DELEGATED_USER is the real
+// account underneath it. If the sending address IS a primary account already,
+// GOOGLE_DELEGATED_USER can be left unset and this falls back to EMAIL_FROM_ADDRESS.
 //
 // WHY this mechanism, not SMTP or a human login: Google's own guidance for a backend
 // application sending mail as a specific Workspace address is a service account + domain-wide
@@ -42,8 +52,8 @@ function loadServiceAccount() {
 // implements.
 async function getAccessToken() {
   const sa = loadServiceAccount();
-  const impersonate = process.env.EMAIL_FROM_ADDRESS;
-  if (!impersonate) throw configError('EMAIL_FROM_ADDRESS is not set (required as the delegated-user "sub" claim)');
+  const impersonate = process.env.GOOGLE_DELEGATED_USER || process.env.EMAIL_FROM_ADDRESS;
+  if (!impersonate) throw configError('Set GOOGLE_DELEGATED_USER (or EMAIL_FROM_ADDRESS if the sending address is itself a primary account) — required as the delegated-user "sub" claim');
 
   const now = Math.floor(Date.now() / 1000);
   const header = Buffer.from(JSON.stringify({ alg: 'RS256', typ: 'JWT' })).toString('base64url');
@@ -61,7 +71,15 @@ async function getAccessToken() {
   const body = await res.json().catch(() => ({}));
   if (!res.ok) {
     const e = new Error(`Google OAuth token exchange failed: ${body.error_description || body.error || res.status}`);
-    e.errorClass = res.status >= 500 ? 'temporary' : 'permanent';
+    // 'unauthorized_client' is retried, not treated as permanent — real-world observation
+    // during Phase 5C setup: freshly-authorized domain-wide delegation can take a few minutes
+    // to propagate across Google's auth infrastructure, producing this EXACT error
+    // intermittently (the identical request succeeded, then failed, then failed again, over a
+    // span of about a minute) even though the delegation is correctly configured. A config
+    // that's genuinely wrong still ends up 'failed' once MAX_ATTEMPTS is exhausted — this only
+    // buys the propagation-delay case a real chance to self-resolve instead of giving up on
+    // the very first attempt.
+    e.errorClass = (res.status >= 500 || body.error === 'unauthorized_client') ? 'temporary' : 'permanent';
     e.errorCode = 'gws_auth_failed';
     throw e;
   }

platform/docker-compose.dev.yml

@@ -80,6 +80,13 @@ services:
       # network call no matter what recipient it's given.
       PUBLIC_APP_ORIGIN: ${PUBLIC_APP_ORIGIN:-http://localhost:4200}
       EMAIL_FROM_NAME: ${EMAIL_FROM_NAME:-RootNode Academy}
+      # Empty/unset by default (fail-safe: dev provider, no real email — see email/provider.js).
+      # Set all four in platform/.env only for a deliberate one-off live-delivery test; unset
+      # again afterward so routine dev/test runs never risk a real send.
+      EMAIL_PROVIDER: ${EMAIL_PROVIDER:-}
+      GOOGLE_SERVICE_ACCOUNT_JSON: ${GOOGLE_SERVICE_ACCOUNT_JSON:-}
+      EMAIL_FROM_ADDRESS: ${EMAIL_FROM_ADDRESS:-}
+      GOOGLE_DELEGATED_USER: ${GOOGLE_DELEGATED_USER:-}
       PLATFORM_API_PORT: "4300"
       # No ADMIN_PASSWORD anymore (Phase 3A) — instructor accounts live in Postgres, created via
       # `npm run bootstrap-owner` for the first owner and the admin portal after that.