docs(deployment): document Cloudflare proxy, SendGrid SMTP, and Caddyfile updates

Captures all server-side work completed on 2026-02-16: - Cloudflare Full setup for wizard.lu, omsflow.lu, rewardflow.lu (NS, SSL, origin certs) - SendGrid SMTP configured for Alertmanager and app transactional emails - Caddyfile updated with origin certs and tls issuer acme for git.wizard.lu - Alertmanager v2 API for test alerts, multi-domain email strategy documented - Cloudflare security: bot protection, DDoS, rate limiting on /api/ paths Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-16 20:20:17 +01:00
parent eaab47f2f8
commit 3a7cf29386
1 changed files with 134 additions and 26 deletions
--- a/docs/deployment/hetzner-server-setup.md
+++ b/docs/deployment/hetzner-server-setup.md
@@ -102,6 +102,36 @@ Complete step-by-step guide for deploying Orion on a Hetzner Cloud VPS.

    **Steps 1–24 fully complete.** Enterprise infrastructure hardening done.

+!!! success "Progress — 2026-02-16"
+    **Completed:**
+
+    - **Step 21: Cloudflare Domain Proxy** — all three domains active on Cloudflare (Full setup):
+        - `wizard.lu` — DNS records configured (6 A + 6 AAAA), old CNAME records removed, NS switched at Netim, SSL/TLS set to Full (Strict), Always Use HTTPS enabled, AI crawlers blocked
+        - `omsflow.lu` — DNS records configured (2 A + 2 AAAA), NS switched at Netim, SSL/TLS Full (Strict) + Always Use HTTPS
+        - `rewardflow.lu` — DNS records configured (2 A + 2 AAAA), NS switched at Netim, SSL/TLS Full (Strict) + Always Use HTTPS
+        - `git.wizard.lu` stays DNS-only (grey cloud) for SSH access on port 2222
+        - DNSSEC disabled at registrar (will re-enable via Cloudflare later)
+        - Registrar: Netim (`netim.com`)
+        - Origin certificates generated (non-wildcard, specific subdomains) and installed on server
+        - Caddyfile updated: origin certs for proxied domains, `tls { issuer acme }` for `git.wizard.lu`
+        - Access logging enabled for fail2ban (`/var/log/caddy/access.log`)
+        - All domains verified working: `wizard.lu`, `omsflow.lu`, `rewardflow.lu`, `api.wizard.lu`, `git.wizard.lu`
+    - **Step 19: SendGrid SMTP** — fully configured and tested:
+        - SendGrid account created (free trial, 60-day limit)
+        - `wizard.lu` domain authenticated (5 CNAME + 1 TXT in Cloudflare DNS)
+        - Link branding enabled
+        - API key `orion-production` created
+        - Alertmanager SMTP configured (`alerts@wizard.lu` → SendGrid)
+        - App email configured (`EMAIL_PROVIDER=sendgrid`, `noreply@wizard.lu`)
+        - Test alert sent and received successfully
+
+    - **Cloudflare security** — configured on all three domains:
+        - Bot Fight Mode enabled
+        - DDoS protection active (default)
+        - Rate limiting: 100 req/10s on `/api/` paths, block for 10s
+
+    **Steps 1–24 fully deployed and operational.**
+

 ## Installed Software Versions

@@ -1180,7 +1210,7 @@ The `docker-compose.yml` includes:

 Alertmanager needs SMTP to send email notifications. SendGrid handles both transactional emails and marketing campaigns under one account — set it up once and use it for everything.

-**Free tier**: 100 emails/day (~3,000/month). Covers alerting + transactional emails through launch.
+**Free trial**: 100 emails/day for 60 days. Covers alerting + transactional emails through launch. After 60 days, upgrade to a paid plan (Essentials starts at ~$20/mo for 50K emails/mo).

 **1. Create SendGrid account:**

@@ -1238,21 +1268,14 @@ curl -s http://localhost:9093/-/healthy  # Should return OK
 **5. Test by triggering a test alert (optional):**

 ```bash
-# Send a test alert to alertmanager
-curl -X POST http://localhost:9093/api/v1/alerts \
-  -H "Content-Type: application/json" \
-  -d '[{
-    "labels": {"alertname": "TestAlert", "severity": "warning"},
-    "annotations": {"summary": "Test alert — please ignore"},
-    "startsAt": "'$(date -u +%Y-%m-%dT%H:%M:%SZ)'",
-    "endsAt": "'$(date -u -d '+5 minutes' +%Y-%m-%dT%H:%M:%SZ)'"
-  }]'
+# Send a test alert to alertmanager (v2 API)
+curl -X POST http://localhost:9093/api/v2/alerts -H "Content-Type: application/json" -d '[{"labels":{"alertname":"TestAlert","severity":"warning"},"annotations":{"summary":"Test alert - please ignore"},"startsAt":"'$(date -u +%Y-%m-%dT%H:%M:%SZ)'","endsAt":"'$(date -u -d '+5 minutes' +%Y-%m-%dT%H:%M:%SZ)'"}]'
 ```

 Check your inbox within 30 seconds. Then verify the alert resolved:

 ```bash
-curl -s http://localhost:9093/api/v1/alerts | python3 -m json.tool
+curl -s http://localhost:9093/api/v2/alerts | python3 -m json.tool
 ```

 !!! tip "Alternative SMTP providers"
@@ -1285,6 +1308,28 @@ curl -s http://localhost:9090/api/v1/alerts | python3 -m json.tool
 curl -s http://localhost:9090/api/v1/targets | python3 -m json.tool | grep alertmanager
 ```

+### 19.8 Multi-Domain Email Strategy
+
+SendGrid supports multiple authenticated domains on a single account. This enables sending emails from client domains (e.g., `orders@acme.lu`) without clients needing their own SendGrid plan.
+
+**Current setup:**
+
+- `wizard.lu` authenticated — used for platform emails (`alerts@`, `noreply@`)
+
+**Future: client domain onboarding**
+
+When a client wants emails sent from their domain (e.g., `acme.lu`):
+
+1. In SendGrid: **Settings** > **Sender Authentication** > **Authenticate a Domain** → add `acme.lu`
+2. SendGrid provides CNAME + TXT records
+3. Client adds the DNS records to their domain
+4. Verify in SendGrid
+
+This is the professional approach — emails come from the client's domain with proper SPF/DKIM, not from `wizard.lu`. Build an admin flow to automate this as part of store onboarding.
+
+!!! note "Volume planning"
+    The free trial allows 100 emails/day. Once clients start sending marketing campaigns, upgrade to a paid SendGrid plan based on total volume across all client domains.
+
 ---

 ## Step 20: Security Hardening
@@ -1487,27 +1532,47 @@ Save the output — you'll need to verify these exist after Cloudflare import.

 1. Log in to [Cloudflare Dashboard](https://dash.cloudflare.com)
 2. **Add a site** for each domain: `wizard.lu`, `omsflow.lu`, `rewardflow.lu`
-3. Cloudflare auto-scans and imports existing DNS records
-4. **Verify MX/SPF/DKIM/DMARC records are present** before changing NS
-5. Email records must stay as **DNS-only (grey cloud)** — never proxy MX records
+3. Select **Free** plan → choose **Full setup** (nameserver-based, not CNAME/partial)
+4. Block AI crawlers on all pages
+5. Cloudflare auto-scans and imports existing DNS records — **review carefully**:
+    - Delete any stale CNAME records (leftover from partial setup)
+    - Add missing A/AAAA records manually (Cloudflare scan may miss some)
+    - Verify MX/SPF/DKIM/DMARC records are present before changing NS
+    - Email records (MX, TXT) must stay as **DNS-only (grey cloud)** — never proxy MX records
+6. Set proxy status:
+    - **Orange cloud (proxied)**: `@`, `www`, `api`, `flower`, `grafana` — gets WAF + CDN
+    - **Grey cloud (DNS only)**: `git` — needs direct access for SSH on port 2222

 ### 21.3 Change Nameservers

-At your domain registrar, update NS records to Cloudflare's assigned nameservers. Cloudflare will show which NS to use (e.g., `ns1.cloudflare.com`, `ns2.cloudflare.com`).
+At your domain registrar (Netim), update NS records to Cloudflare's assigned nameservers. Cloudflare shows the exact pair during activation (e.g., `name1.ns.cloudflare.com`, `name2.ns.cloudflare.com`).
+
+Disable DNSSEC at the registrar before switching NS — re-enable later via Cloudflare.

 ### 21.4 Generate Origin Certificates

 Cloudflare Origin Certificates (free, 15-year validity) avoid ACME challenge issues when traffic is proxied:

 1. In Cloudflare: **SSL/TLS** > **Origin Server** > **Create Certificate**
-2. Generate for `*.wizard.lu, wizard.lu` (repeat for each domain)
-3. Download the certificate and private key
+2. Generate for each domain with **specific subdomains** (not wildcards):
+    - `wizard.lu`: `wizard.lu, api.wizard.lu, www.wizard.lu, flower.wizard.lu, grafana.wizard.lu`
+    - `omsflow.lu`: `omsflow.lu, www.omsflow.lu`
+    - `rewardflow.lu`: `rewardflow.lu, www.rewardflow.lu`
+3. Download the certificate and private key (private key is shown only once)
+
+!!! warning "Do NOT use wildcard origin certs for wizard.lu"
+    A `*.wizard.lu` wildcard cert will match `git.wizard.lu`, which needs a Let's Encrypt cert (DNS-only, not proxied through Cloudflare). Use specific subdomains instead.

 Install on the server:

 ```bash
 sudo mkdir -p /etc/caddy/certs/{wizard.lu,omsflow.lu,rewardflow.lu}
-# Copy cert.pem and key.pem to each directory
+
+# For each domain, create cert.pem and key.pem:
+sudo nano /etc/caddy/certs/wizard.lu/cert.pem    # paste certificate
+sudo nano /etc/caddy/certs/wizard.lu/key.pem      # paste private key
+# Repeat for omsflow.lu and rewardflow.lu
+
 sudo chown -R caddy:caddy /etc/caddy/certs/
 sudo chmod 600 /etc/caddy/certs/*/key.pem
 ```
@@ -1517,22 +1582,50 @@ sudo chmod 600 /etc/caddy/certs/*/key.pem
 For Cloudflare-proxied domains, use explicit TLS with origin certs. Keep auto-HTTPS for `git.wizard.lu` (DNS-only, grey cloud):

 ```caddy
-# ─── Cloudflare-proxied domains (origin certs) ──────────
+{
+    log {
+        output file /var/log/caddy/access.log {
+            roll_size 100MiB
+            roll_keep 5
+        }
+        format json
+    }
+}
+
+# ─── Platform 1: Main (wizard.lu) ───────────────────────────
 wizard.lu {
    tls /etc/caddy/certs/wizard.lu/cert.pem /etc/caddy/certs/wizard.lu/key.pem
    reverse_proxy localhost:8001
 }

+www.wizard.lu {
+    tls /etc/caddy/certs/wizard.lu/cert.pem /etc/caddy/certs/wizard.lu/key.pem
+    redir https://wizard.lu{uri} permanent
+}
+
+# ─── Platform 2: OMS (omsflow.lu) ───────────────────────────
 omsflow.lu {
    tls /etc/caddy/certs/omsflow.lu/cert.pem /etc/caddy/certs/omsflow.lu/key.pem
    reverse_proxy localhost:8001
 }

+www.omsflow.lu {
+    tls /etc/caddy/certs/omsflow.lu/cert.pem /etc/caddy/certs/omsflow.lu/key.pem
+    redir https://omsflow.lu{uri} permanent
+}
+
+# ─── Platform 3: Loyalty+ (rewardflow.lu) ──────────────────
 rewardflow.lu {
    tls /etc/caddy/certs/rewardflow.lu/cert.pem /etc/caddy/certs/rewardflow.lu/key.pem
    reverse_proxy localhost:8001
 }

+www.rewardflow.lu {
+    tls /etc/caddy/certs/rewardflow.lu/cert.pem /etc/caddy/certs/rewardflow.lu/key.pem
+    redir https://rewardflow.lu{uri} permanent
+}
+
+# ─── Services (wizard.lu origin cert) ───────────────────────
 api.wizard.lu {
    tls /etc/caddy/certs/wizard.lu/cert.pem /etc/caddy/certs/wizard.lu/key.pem
    reverse_proxy localhost:8001
@@ -1548,8 +1641,11 @@ grafana.wizard.lu {
    reverse_proxy localhost:3001
 }

-# ─── DNS-only domain (auto-HTTPS via Let's Encrypt) ─────
+# ─── DNS-only domain (Let's Encrypt, not proxied by Cloudflare) ─
 git.wizard.lu {
+    tls {
+        issuer acme
+    }
    reverse_proxy localhost:3000
 }
 ```
@@ -1563,13 +1659,25 @@ sudo systemctl status caddy

 ### 21.6 Cloudflare Settings (per domain)

-| Setting | Value |
+Configure these in the Cloudflare dashboard for each domain (`wizard.lu`, `omsflow.lu`, `rewardflow.lu`):
+
+| Setting | Location | Value |
+|---|---|---|
+| SSL mode | SSL/TLS > Overview | Full (Strict) |
+| Always Use HTTPS | SSL/TLS > Edge Certificates | On |
+| Bot Fight Mode | Security > Settings | On |
+| DDoS protection | Security > Security rules > DDoS | Active (enabled by default) |
+| AI crawlers | Security (during setup) | Blocked on all pages |
+
+**Rate limiting rule** (Security > Security rules > Create rule):
+
+| Field | Value |
 |---|---|
-| SSL mode | Full (Strict) |
-| Always Use HTTPS | On |
-| WAF Managed Rules | On |
-| Bot Fight Mode | On |
-| Rate Limiting | 100 req/min on `/api/*` |
+| Match | URI Path contains `/api/` |
+| Characteristics | IP |
+| Rate | 100 requests per 10 seconds |
+| Action | Block |
+| Duration | 10 seconds |

 ### 21.7 Production Environment