Files
rootnode-academy/docs/RUNBOOK_GOOGLE_WORKSPACE_EMAIL.md
ced4568 4c0a950afb
Release to Docker Hub / Build & Push Docker Image (push) Canceled after 0s
Release to Docker Hub / Create GitHub Release (push) Canceled after 0s
fix: retry Google's transient domain-wide-delegation propagation-delay error
Live Google Workspace delivery validation surfaced a real, self-resolving
Google-side condition: freshly-authorized domain-wide delegation can take
up to about a minute to propagate across Google's auth infrastructure,
producing intermittent unauthorized_client on the OAuth token exchange
even with correct configuration. gws-provider.js now classifies
unauthorized_client as a temporary (retryable) error instead of permanent,
so the existing bounded-backoff worker absorbs it automatically.

Also documents the alias-vs-delegated-user env passthrough in
docker-compose.dev.yml, and records both real findings (this one and the
GOOGLE_DELEGATED_USER alias distinction) plus the full live-delivery
validation outcome in the runbook and Phase 5C doc.

Validated live: isolated gws-email-test.js send (DKIM/SPF passing,
delivered) and a full real lifecycle test (create student, enrollment,
real activation email, activate, login, see program, real password-reset
email, reset, login with new password) via [email protected].
2026-08-25 21:36:12 -05:00

12 KiB

Runbook: Google Workspace Email Configuration

Everything in this document happens OUTSIDE this repository, in the Google Admin Console and Google Cloud Console, and is marked REQUIRES GOOGLE WORKSPACE ADMIN ACTION. Nothing here can be scripted or automated from the codebase — Chase (or whoever holds Workspace super-admin) has to do these steps by hand, once, before RootNode Academy can send real email.

See docs/PHASE5C_ENROLLMENT_COMMUNICATIONS.md §2 for why this mechanism was chosen over SMTP or a human login.

1. Selected mechanism

Gmail API (users.messages.send) via a Google Cloud service account using Workspace domain-wide delegation, impersonating one dedicated sending mailbox. No human password is ever stored by RootNode. The delegated scope is exactly https://www.googleapis.com/auth/ gmail.send — nothing broader.

2. REQUIRES GOOGLE WORKSPACE ADMIN ACTION: create the sending mailbox

Decide the address RootNode sends as — e.g. [email protected] or [email protected]. This repository does not assume or hard-code one; it's whatever you set EMAIL_FROM_ADDRESS to (§5).

  1. In Google Admin Console (admin.google.com) → Directory → Users, create this mailbox as a real Workspace user (or a Google Group with "Collaborative Inbox" if you'd rather route replies to a shared team inbox — either works with this mechanism, but a Group needs its own delegation setup; a plain user mailbox is simpler and is what this runbook assumes).
  2. Consider giving it a strong, unique, never-reused password and enabling 2-Step Verification on it anyway, even though RootNode's application will never use that password — this account still exists as a real Workspace identity and should be secured like any other.

If the sending address is an ALIAS of an existing account (a real cost-saving option — you don't need a whole extra paid seat just for this): that's fine and supported, but note it for step 4 below. Domain-wide delegation authorizes the service account to act as a real Workspace user account, not an arbitrary alias string — if [email protected] is an alias of, say, [email protected], the delegation in step 4 is granted for chase@'s account (the alias's owner), and RootNode's config keeps the alias as the visible sending address separately (see §5's GOOGLE_DELEGATED_USER vs. EMAIL_FROM_ADDRESS). Gmail already lets any account send as its own verified aliases, so this works cleanly — you just need both pieces of information (the alias, and which account owns it) when configuring RootNode.

3. REQUIRES GOOGLE WORKSPACE ADMIN ACTION: create the service account (Google Cloud Console)

  1. In Google Cloud Console (console.cloud.google.com), create or choose a GCP project associated with your Workspace organization.
  2. Enable the Gmail API for that project (APIs & Services → Library → search "Gmail API" → Enable).
  3. IAM & Admin → Service Accounts → Create Service Account. Name it something identifiable (e.g. rootnode-academy-mailer). No project-level IAM roles are needed for this — it only ever calls the Gmail API as the impersonated user, not as itself.
  4. On that service account, create a JSON key (Keys → Add Key → Create new key → JSON). This downloads a file containing client_email and private_key — this is the sensitive credential, treat it exactly like a password (see §8).
  5. Copy the service account's Client ID (a long numeric string, on the service account's detail page) — needed for the next step.

4. REQUIRES GOOGLE WORKSPACE ADMIN ACTION: authorize domain-wide delegation

  1. Google Admin Console → Security → Access and data control → API controls → Domain-wide Delegation → Add new.
  2. Client ID: the numeric ID from step 3.5 above.
  3. OAuth Scopes: https://www.googleapis.com/auth/gmail.send — only this scope. Do not add broader Gmail scopes (readonly, modify, etc.) — RootNode's adapter never needs them and granting them would violate least-privilege for no benefit.
  4. Authorize.

This grants the service account permission to act as one specific Workspace user account. If the sending address is that account's own primary login, that's the account. If the sending address is an ALIAS (§2), this authorization is inherently for the alias's owner account — there is no separate "authorize an alias" step, because delegation only ever targets real accounts.

5. RootNode-side environment variables

Set these on the Platform API's environment (production: real secret storage — see §8; local dev: platform/.env, already gitignored):

Variable Value Notes
EMAIL_PROVIDER gws Unset/anything else defaults to the dev provider — never real mail.
GOOGLE_SERVICE_ACCOUNT_JSON the full JSON key file's contents, as one string The sensitive credential — see §8.
EMAIL_FROM_ADDRESS the address students see mail arrive from, e.g. [email protected] Always the visible From: header.
GOOGLE_DELEGATED_USER only needed if EMAIL_FROM_ADDRESS is an alias — the real account that alias belongs to, e.g. [email protected] This is what domain-wide delegation actually authorized in step 4. If EMAIL_FROM_ADDRESS is itself a primary account, leave this unset — the code falls back to EMAIL_FROM_ADDRESS automatically.
EMAIL_FROM_NAME e.g. RootNode Academy Display name only, not sensitive.
EMAIL_REPLY_TO optional, e.g. [email protected] Where a student's reply actually goes, if different from the sending address.
PUBLIC_APP_ORIGIN e.g. https://learn.rootnodeacademy.com The origin activation/reset links are built from (§7 of the Phase 5C authorization) — required for any token-bearing email to render a working link.

Concrete example matching the alias setup described above: [email protected] (what students see), [email protected] (whose account actually owns that alias and was granted delegation in step 4).

6. DNS: SPF, DKIM, DMARC (REQUIRES DNS ADMIN ACTION — not performed from this repository)

Google Workspace's own documentation is authoritative here; this is a summary of what must exist before mail sent via the Gmail API from a Workspace mailbox reliably lands in inboxes instead of spam:

  • SPF: your domain's TXT record must include Google's SPF include (include:_spf.google.com) if Workspace is your MX provider — check Admin Console → Apps → Google Workspace → Gmail → Authenticate email for the exact current record Google recommends for your domain.
  • DKIM: Admin Console → Apps → Google Workspace → Gmail → Authenticate email → generate a DKIM key for the domain, then publish the TXT record it gives you, then return to that page and turn DKIM signing ON. Gmail API sends from a Workspace mailbox are signed the same way regular Gmail sends are once this is on.
  • DMARC: publish a _dmarc TXT record (start with p=none to monitor before enforcing) once SPF and DKIM both pass consistently.

Do not change DNS from this repository or these instructions — this is exactly the kind of external, admin-console/DNS-provider action that stays outside the codebase. Verify current exact record values in Google's own Workspace admin help before publishing anything.

7. Testing procedure

Local/dev — no live credentials needed at all. EMAIL_PROVIDER unset (or anything other than gws) uses the deterministic dev provider (email/dev-provider.js) — every notification still goes through the full queue/worker/retry pipeline, but nothing ever reaches a real network. platform/test/e2e-phase5c.js and the Phase 5C drills run entirely this way.

Live delivery — opt-in, one message, once credentials exist:

RUN_GWS_EMAIL_TEST=true \
TEST_EMAIL_RECIPIENT=[email protected] \
GOOGLE_SERVICE_ACCOUNT_JSON="$(cat /path/to/service-account-key.json)" \
EMAIL_FROM_ADDRESS=[email protected] \
GOOGLE_DELEGATED_USER=[email protected] \
EMAIL_FROM_NAME="RootNode Academy" \
node infra/scripts/gws-email-test.js

(Omit GOOGLE_DELEGATED_USER entirely if EMAIL_FROM_ADDRESS is itself a primary account rather than an alias — see §5.) Sends exactly one clearly-marked test message to TEST_EMAIL_RECIPIENT — never inferred from production/student data, never run automatically as part of any regression suite. Without RUN_GWS_EMAIL_TEST=true it does nothing and exits cleanly.

8. Production secret handling

  • GOOGLE_SERVICE_ACCOUNT_JSON is exactly as sensitive as a password — store it in whatever secret manager/environment-injection mechanism the production deployment uses (see docs/PRODUCTION_DEPLOYMENT.md for the general pattern this repo follows for other secrets like PROVISIONER_SHARED_SECRET). Never commit it, never put it in a Docker image layer, never log it. gws-provider.js reads it fresh from the environment on every send and never writes it to disk.
  • Nothing in this codebase ever returns GOOGLE_SERVICE_ACCOUNT_JSON's contents, an access token, or any part of it through an API response, log line, or audit event.

9. Credential rotation

  1. Google Cloud Console → the service account → Keys → Add Key → Create new key (JSON) — this creates an ADDITIONAL valid key without invalidating the old one yet.
  2. Update GOOGLE_SERVICE_ACCOUNT_JSON in production's secret store to the new key, redeploy/ restart the Platform API.
  3. Confirm delivery still works (§7's live test, or just watch /api/admin/notifications for a real send after the restart).
  4. Google Cloud Console → the service account → Keys → delete the OLD key.

Domain-wide delegation authorization (§4) is tied to the service account's Client ID, not to any individual key — rotating keys does not require re-authorizing delegation.

10. Revocation (compromised credential / offboarding)

  1. Google Cloud Console → the service account → Keys → delete the affected key immediately — this invalidates it instantly, before anything else.
  2. If the whole service account should stop being able to send at all: Google Admin Console → Security → API controls → Domain-wide Delegation → remove that Client ID's authorization.
  3. Rotate EMAIL_FROM_ADDRESS's own Workspace login password as a precaution even though the application never used it — a compromised service-account key doesn't imply the mailbox's own credentials were exposed, but treat it as a "review this account" trigger.

11. Troubleshooting

Symptom Likely cause
Every send fails with gws_not_configured GOOGLE_SERVICE_ACCOUNT_JSON or EMAIL_FROM_ADDRESS unset/malformed in the Platform API's environment.
gws_auth_failed Domain-wide delegation not authorized for this Client ID/scope, or the service account key was deleted/revoked. Re-check §4. If this happens intermittently right after newly authorizing delegation (the exact same request succeeds, then fails with unauthorized_client, then succeeds again, over roughly a minute) — that's Google's own auth infrastructure propagation delay, not a config error. It self-resolves; the notification worker's existing retry/backoff already absorbs it (gws-provider.js classifies unauthorized_client as retryable), so no action is needed beyond waiting.
gws_http_403 The delegated sub (EMAIL_FROM_ADDRESS) isn't a real mailbox in this Workspace domain, or the Gmail API isn't enabled on the GCP project (§3.2).
gws_rate_limited (HTTP 429) Gmail API per-user sending limits hit — see §12. The notification worker retries this automatically with backoff.
Mail sends successfully but lands in spam SPF/DKIM/DMARC (§6) not fully configured/passing yet — this is a deliverability issue, not an application error.

12. Google Workspace quotas/limits relevant to this mechanism

Gmail API sending inherits the same per-user daily sending limits as regular Gmail/Workspace sending (limits vary by Workspace edition — check current figures in Google's own Workspace admin documentation, as these change over time and are not something to hard-code here). For RootNode Academy's transactional volume (activation/enrollment/reset emails, not bulk/marketing — explicitly out of scope, see the Phase 5C authorization §5/§28), this should sit comfortably under any Workspace edition's per-user limit; revisit if enrollment volume grows substantially.