Skip to content

CLOUDFLARE-EDGE.md

Latest commit

 

History

History
158 lines (136 loc) · 10.5 KB

CLOUDFLARE-EDGE.md

File metadata and controls

158 lines (136 loc) · 10.5 KB
Raw

Cloudflare edge — WAF, zero-trust login, origin certs (⛔ ROLLED BACK 2026-07-19)

STATUS: DISABLED — rolled back the same day it went live (owner call: the Access OTP gate added friction and the login callback intermittently failed in-browser with ERR_QUIC_PROTOCOL_ERROR; will revisit later, likely as SSO). The app serves DIRECT to origin again, exactly as before. Everything below documents the built integration so it can be re-enabled quickly.

What was reverted (2026-07-19):

  • acme + apex A records set back to DNS-only (grey) — no proxy, no WAF, no Access in path.
  • The Access application was DELETED (opengp platform, facf955a-…) along with its policy.
  • Caddy restored to Let's Encrypt on all hosts (backups /opt/opengp/{Caddyfile,docker-compose.yml}.bak-origincert were the pre-edge configs; the live files are now those). Verified serving LE, app 307 → /login.
  • Product-info pages (Architecture/Security) reverted to not claim an edge gate.

What REMAINS in place (harmless, ready for re-enable):

  • Nameservers were reverted at Namecheap to Route53 (the 4 awsdns servers) on 2026-07-19 — Route53 is authoritative again and the Terraform dns_records.tf records are live. The Cloudflare zone still exists as an inactive mirror (it will show "moved" — that's fine; it makes a future re-enable easier: switch NS back to heather/yahir and wait for zone re-activation).
  • Zero Trust account (team opengp) + the One-time-PIN login method.
  • The 15-yr origin cert + key still on the box at /opt/opengp/certs/origin.{pem,key} (unused).
  • The scoped API token opengp-edge-automation (rotate it anyway).

To RE-ENABLE the edge later (~10 min):

  1. Cloudflare → Zero Trust → Access → Applications → re-create the self-hosted app on app.yourdomain.example + yourdomain.example, One-time PIN login, allow-policy = customer email domains (or wire an SSO/OIDC IdP instead of OTP — the original owner intent).
  2. SSL/TLS mode should still be Full (Strict); re-check it.
  3. On the box: re-apply the origin-cert Caddy config (add the tls /etc/caddy/certs/origin.pem /etc/caddy/certs/origin.key line to the acme/apex block + the /opt/opengp/certs volume mount in docker-compose — see §Origin certificate below) → docker compose up -d caddy.
  4. Flip acme + apex A records to Proxied (orange).
  5. Verify: 302 → Access login → OTP → app; api stays grey and direct.
  6. If NS were reverted to Route53 in the meantime: first switch NS back to Cloudflare at Namecheap (heather/yahir) and wait for the zone to go Active.

Original build documentation (as-shipped 2026-07-19, pre-rollback)

yourdomain.example sits behind Cloudflare for app.yourdomain.example + the apex yourdomain.example: a WAF/DDoS layer and a Cloudflare Access email-OTP login gate in front of the app, plus a 15-year Cloudflare Origin Certificate on Caddy so origin TLS never has to renew under the proxy. api.yourdomain.example is deliberately left OFF Cloudflare (grey / DNS-only) — it's machine-to-machine (webhooks + the frontend BFF) and keeps its publicly-trusted Let's Encrypt cert.

Two independent gates now protect the app: (1) Cloudflare Access (email OTP — "are you even allowed to load this site?") then (2) the app's own Cognito login. They don't cross-check — passing OTP with an allowed-domain email lets you reach the Cognito screen, where any valid app credentials (incl. the demo account) work.

Account / zone / team

  • Cloudflare account: 9b0f167c3f01a60ae1c4843847df5ff7 (maintainers@opengp.example)
  • Zone yourdomain.example: 8cbc2daa88c1366dd42c4183954c39f3 — plan Free, status active (activated 2026-07-19T11:33Z)
  • Zero Trust team: opengp → OTP login domain opengp.cloudflareaccess.com (renamed from the auto-assigned fragrant-cake-6236; rename lives in Zero Trust → Settings → Team name)
  • Zero Trust plan: Free (≤50 users, $0 — but a card is required on file to activate)

DNS — authoritative on Cloudflare (was Route53)

Registrar is Namecheap; it delegates the domain to Cloudflare's nameservers.

  • Current NS (live): heather.ns.cloudflare.com, yahir.ns.cloudflare.com
  • OLD NS (Route53 — restore these to roll back): ns-1494.awsdns-58.org, ns-1715.awsdns-22.co.uk, ns-409.awsdns-51.com, ns-875.awsdns-45.net

⚠️ The Route53 hosted zone <ROUTE53_ZONE_ID> still exists (its records mirror what Cloudflare serves), it's just no longer authoritative. So the Terraform Route53 records in infra/terraform/dns_records.tf are now dormant — Cloudflare is the source of truth for DNS. Changing a record in Route53 does nothing live until/unless the nameservers are pointed back.

Records in the Cloudflare zone (10 — a mirror of the old Route53 set):

Name Type Value Proxy
yourdomain.example A <APP_SERVER_EIP> Proxied (orange)
app.yourdomain.example A <APP_SERVER_EIP> Proxied (orange)
api.yourdomain.example A <APP_SERVER_EIP> DNS-only (grey)
3× SES DKIM CNAME (SES DKIM targets) DNS-only
inbound-smtp / feedback-smtp MX (SES/Resend) DNS-only
resend DKIM, SPF TXT (email auth) DNS-only

The apex A once pointed at the wrong IP (69.46.46.78) during setup — corrected to the app-server EIP <APP_SERVER_EIP>. Email (MX/TXT/DKIM) is unaffected by proxying the A records.

SSL/TLS

  • Zone SSL mode = Full (Strict) — Cloudflare validates the origin certificate.
  • Proxied hosts (acme, apex) present a Cloudflare Origin CA cert at the origin (§ Origin cert).
  • api (grey) presents Let's Encrypt (publicly trusted) — required because external webhook senders connect to it directly and validate the chain.

Access — the login gate

  • Self-hosted app opengp platform — id facf955a-4c5b-4e80-9fc8-9bd3667366f4, aud f61b8ba…
  • Covers app.yourdomain.example + yourdomain.example; session duration 24h.
  • Login method = One-time PIN (email OTP) ONLY. The auto-added "Cloudflare" IdP was deleted so users get the email→code flow, not a Cloudflare-account login. (Integrations → Identity providers.)
  • Allow-policy 40afc060-… (decision = allow): email domains acme.com, bridgewood.com, gsequity.com + the specific email maintainers@opengp.example. Add a domain here when a new customer joins (Access → Applications → opengp platform → Policies).
  • WAF on the Free plan = DDoS + Cloudflare Free Managed Ruleset only; the full customizable OWASP WAF is a Pro/Business upgrade. The Access login gate is fully featured on Free.

Origin certificate on Caddy

Because the proxied hosts route through Cloudflare, Caddy can no longer complete Let's Encrypt ACME challenges for them (Cloudflare intercepts the challenge) → they would 526 at cert expiry. Fixed with a 15-year Cloudflare Origin cert, installed the secure way (private key never left the box):

  • Key generated on the box: /opt/opengp/certs/origin.key (600, root). A CSR was signed by Cloudflare Origin CA → cert /opt/opengp/certs/origin.pem (issuer "CloudFlare Origin SSL Certificate Authority", valid to 2041-07-15).
  • SAN = yourdomain.example, app.yourdomain.example — deliberately NOT a wildcard. A *.yourdomain.example cert would also match api.yourdomain.example and make Caddy serve the (publicly-untrusted) origin cert there, breaking webhooks. Non-wildcard keeps api on Let's Encrypt.
  • docker-compose.yml caddy service mounts /opt/opengp/certs:/etc/caddy/certs:ro.
  • Caddyfile: the app.yourdomain.example, yourdomain.example { … } block has tls /etc/caddy/certs/origin.pem /etc/caddy/certs/origin.key; the api.yourdomain.example { … } block is unchanged (auto Let's Encrypt).
  • Backups on the box: /opt/opengp/Caddyfile.bak-origincert, /opt/opengp/docker-compose.yml.bak-origincert.
  • When adding a new proxied workspace subdomain (e.g. bridgewood.yourdomain.example): reissue the origin cert (SSL/TLS → Origin Server → Create Certificate → use the existing CSR) with the new hostname added, replace origin.pem, docker compose up -d caddy. Same for adding it to the Access app.

API token used

  • Scoped user token opengp-edge-automation (cfut_…): Zone→DNS:Edit + Account→Access:Apps & Policies:Edit only. It CANNOT set SSL mode, run activation checks, manage identity providers, or sign origin certs — those steps were done in the dashboard. Gotcha: user/tokens/verify reports the token "Invalid" even though it works — test scope with GET /zones.
  • ⚠️ Rotate this token and the Cloudflare/R2 tokens pasted in chat during setup.

Rollback — revert to direct-to-origin (and optionally Route53)

Order matters: the origin cert is only trusted by Cloudflare, so you must stop proxying and restore Let's Encrypt on Caddy before browsers ever hit the origin directly — otherwise acme/apex would serve the untrusted origin cert and throw TLS errors.

Partial rollback (fastest — keeps the Cloudflare zone, just turns the edge off):

  1. Cloudflare → DNS → set acme + yourdomain.example A records back to DNS-only (grey). (This alone removes the WAF + Access gate. But those hosts now serve the origin cert directly → do step 2 immediately.)
  2. On the box, restore the pre-origin-cert Caddy config so it goes back to Let's Encrypt: 
    ssh -i ~/.ssh/opengp-boxes.pem ec2-user@<APP_SERVER_EIP>
sudo cp /opt/opengp/Caddyfile.bak-origincert /opt/opengp/Caddyfile
sudo cp /opt/opengp/docker-compose.yml.bak-origincert /opt/opengp/docker-compose.yml
cd /opt/opengp && sudo docker compose up -d caddy
    Caddy re-obtains Let's Encrypt for acme/apex now that they're reachable directly. Verify: curl -I https://app.yourdomain.example returns 200/307 with a Let's Encrypt cert.

Full rollback (also hand DNS authority back to Route53): 3. Namecheap → Domain List → yourdomain.example → Nameservers (Custom DNS) → replace heather.ns.cloudflare.com + yahir.ns.cloudflare.com with the 4 awsdns servers listed above → Save. Route53 becomes authoritative again (its records already point at the origin).

To temporarily disable just the login gate but keep the WAF: Cloudflare → Access → Applications → opengp platform → delete/disable the app (proxy stays on, cert unaffected).