feat(loyalty): Phase 1 production launch hardening

Phase 1 of the loyalty production launch plan: config & security
hardening, dropped-data fix, DB integrity guards, rate limiting, and
constant-time auth compare. 362 tests pass.

- 1.4 Persist customer birth_date (new column + migration). Enrollment
  form was collecting it but the value was silently dropped because
  create_customer_for_enrollment never received it. Backfills existing
  customers without overwriting.
- 1.1 LOYALTY_GOOGLE_SERVICE_ACCOUNT_JSON validated at startup (file
  must exist and be readable; ~ expanded). Adds is_google_wallet_enabled
  and is_apple_wallet_enabled derived flags. Prod path documented as
  ~/apps/orion/google-wallet-sa.json.
- 1.5 CHECK constraints on loyalty_cards (points_balance, stamp_count
  non-negative) and loyalty_programs (min_purchase, points_per_euro,
  welcome_bonus non-negative; stamps_target >= 1). Mirrored as
  CheckConstraint in models. Pre-flight scan showed zero violations.
- 1.3 @rate_limit on store mutating endpoints: stamp 60/min,
  redeem/points-earn 30-60/min, void/adjust 20/min, pin unlock 10/min.
- 1.2 Constant-time hmac.compare_digest for Apple Wallet auth token
  (pulled forward from Phase 9 — code is safe whenever Apple ships).

See app/modules/loyalty/docs/production-launch-plan.md for the full
launch plan and remaining phases.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

app/modules/loyalty/docs/production-launch-plan.md

@@ -0,0 +1,330 @@
+# Loyalty Module — Production Launch Plan
+
+**Created:** 2026-04-09
+**Owner:** Samir
+**Status:** Planning locked, awaiting Phase 1 kickoff
+
+This is the active execution plan for taking the Loyalty module to production. It complements the earlier [Production Readiness review](production-readiness.md) by sequencing the work into phases and recording the decisions made.
+
+---
+
+## Locked Decisions
+
+| # | Topic | Decision |
+|---|---|---|
+| 1 | Apple Wallet | **Deferred to Phase 9 (post-launch)**. Google Wallet only at launch. |
+| 2 | Secrets path | Google service account JSON lives at `~/apps/orion/google-wallet-sa.json` in production (app user, mode 600). Different from Caddy TLS certs at `/etc/caddy/certs/`. |
+| 3 | Notification channels | **Email only.** No SMS at launch. |
+| 4 | Notification scope | Enrollment, welcome bonus, points-expiring (14d warning), points-expired, reward-ready, **birthday**, **re-engagement** (inactive). 7 templates × 4 locales = 28 seed rows. |
+| 5 | T&C strategy | **CMS integration scoped to the program's owning store.** New `terms_cms_page_slug` column on `loyalty_programs`. Legacy `terms_text` kept one release as fallback. |
+| 6 | Coverage target | **80%** on `app/modules/loyalty/*` (enforced in CI for loyalty job only). |
+| 7 | GDPR deletion | **Anonymize** (null `customer_id`, scrub PII fields, keep aggregate transactions). |
+| 8 | Bulk actions | Primary endpoints on **merchant router**; admin "act on behalf of" endpoints under admin router accept `merchant_id` and stamp `acting_admin_id` in audit log. |
+
+---
+
+## Verified Reuse (no new infra)
+
+| Need | Existing component |
+|---|---|
+| Email | `app/modules/messaging/services/email_service.py` — multi-provider, DB templates, 4 locales, EmailLog. SMTP enabled. |
+| Background tasks | **Celery** (`@shared_task`, beat schedule in `app/modules/loyalty/definition.py`). |
+| Rate limiting | `middleware/decorators.py @rate_limit` decorator (already used in `routes/api/storefront.py`). |
+| Migrations | Alembic loyalty branch (`loyalty_001`, `loyalty_002`). |
+| pytest-cov | Wired in `pyproject.toml`; currently `--cov-fail-under=0` (disabled). |
+
+---
+
+## Migration Chain
+
+```
+loyalty_002 (existing)
+loyalty_003 — CHECK constraints (points_balance >= 0, stamp_count >= 0, etc.)
+loyalty_004 — seed 28 notification email templates
+loyalty_005 — add columns: last_expiration_warning_at, last_reengagement_at on cards;
+              acting_admin_id on transactions
+loyalty_006 — terms_cms_page_slug on programs
+loyalty_007 — birth_date on customers (P0 — Phase 1.4 fix, dropped data bug)
+```
+
+---
+
+## Phases
+
+### Phase 0 — Decisions & prerequisites *(done)*
+All 8 decisions locked. No external blockers.
+
+### Phase 1 — Config, Secrets & Security Hardening *(✅ DONE 2026-04-09)*
+
+#### 1.1 Google Wallet config cleanup
+- Move `LOYALTY_GOOGLE_SERVICE_ACCOUNT_JSON` from dev path to `~/apps/orion/google-wallet-sa.json` in prod `.env`.
+- Add Pydantic `@field_validator` in `app/modules/loyalty/config.py:39-43` that checks file exists and is readable when `google_issuer_id` is set. Fail fast at import time if not.
+- **Validation:** unit test using `tmp_path`; manual: missing file → clear startup error.
+
+#### 1.2 Apple Wallet timing-safe token compare *(pull-forward from Phase 9)*
+- Replace `!=` with `hmac.compare_digest()` at `app/modules/loyalty/services/apple_wallet_service.py:98`.
+- 15 min, code is safe whenever Apple ships.
+- **Validation:** unit test with wrong token.
+
+#### 1.3 Rate-limit store mutating endpoints
+- Apply `@rate_limit` to `routes/api/store.py:600-795`:
+  - `POST /stamp` 60/min
+  - `POST /stamp/redeem` 30/min
+  - `POST /stamp/void` 20/min
+  - `POST /points/earn` 60/min
+  - `POST /points/redeem` 30/min
+  - `POST /points/void` 20/min
+  - `POST /cards/{card_id}/points/adjust` 20/min
+  - `POST /pins/{pin_id}/unlock` 10/min
+- Caps confirmed.
+- **Validation:** httpx loop exceeds limit → 429.
+
+#### 1.4 Fix dropped birthday data (P0 bug)
+**Background:** The enrollment form (`enroll.html:87`) collects a birthday, the schema accepts `customer_birthday` (`schemas/card.py:34`), and `card_service.resolve_customer_id` has the parameter (`card_service.py:172`) — but the call to `customer_service.create_customer_for_enrollment` at `card_service.py:215-222` does not pass it, and the `Customer` model has no `birth_date` column at all. Every enrollment silently loses the birthday. Not live yet, so no backfill needed.
+
+- New migration `loyalty_007_add_customer_birth_date.py` (or place under customers module if that's the convention) adds `birth_date: Date | None` to `customers`.
+- Update `customer_service.create_customer_for_enrollment` to accept and persist `birth_date`.
+- Update `card_service.py:215-222` to pass `customer_birthday` through, with `date.fromisoformat()` parsing and validation (must be a real past date, sensible age range).
+- Update `customer_service.update_customer` to allow backfill.
+- Add unit + integration tests for enrollment-with-birthday and enrollment-without-birthday paths.
+- **Validation:** enroll with birthday → row in `customers` has `birth_date` set; enrollment without birthday still works.
+
+#### 1.5 DB CHECK constraints migration `loyalty_003`
+- Add: `loyalty_cards.points_balance >= 0`, `loyalty_cards.stamp_count >= 0`, `loyalty_programs.minimum_purchase_cents >= 0`, `loyalty_programs.points_per_euro >= 0`, `loyalty_programs.stamps_target >= 1`, `loyalty_programs.welcome_bonus_points >= 0`.
+- Mirror as `CheckConstraint` in models.
+- Pre-flight: scan dev DB for existing violations; include data fix in `upgrade()` if found.
+- **Validation:** migration runs clean; direct UPDATE to negative value fails.
+
+---
+
+### Phase 2 — Notifications Infrastructure *(4d)*
+
+#### 2.1 `LoyaltyNotificationService`
+- New `app/modules/loyalty/services/notification_service.py` with methods:
+  - `send_enrollment_confirmation(db, card)`
+  - `send_welcome_bonus(db, card, points)` *(may merge into enrollment)*
+  - `send_points_expiration_warning(db, card, days_before, expiring_points, expiration_date)`
+  - `send_points_expired(db, card, expired_points)`
+  - `send_reward_available(db, card, reward_description)`
+  - `send_birthday_message(db, card)`
+  - `send_reengagement(db, card, days_inactive)`
+- All methods enqueue via Celery (2.2), never call EmailService inline.
+
+#### 2.2 Celery dispatch task
+- New `app/modules/loyalty/tasks/notifications.py` with `@shared_task(name="loyalty.send_notification_email", bind=True, max_retries=3, default_retry_delay=60)`.
+- Opens fresh `SessionLocal`, calls `EmailService(db).send_template(...)`, retries on SMTP errors.
+
+#### 2.3 Seed templates `loyalty_004`
+- 7 templates × 4 locales (en, fr, de, lb) = **28 rows** in `email_templates`.
+- Template codes: `loyalty_enrollment`, `loyalty_welcome_bonus`, `loyalty_points_expiring`, `loyalty_points_expired`, `loyalty_reward_ready`, `loyalty_birthday`, `loyalty_reengagement`.
+- **Copywriting needs sign-off** before applying to prod.
+
+#### 2.4 Wire enrollment confirmation
+- In `card_service.enroll_customer_for_store` (~lines 480-540), call notification service **after** `db.commit()`.
+
+#### 2.5 Wire expiration warning into expiration task
+- Migration `loyalty_005` adds `last_expiration_warning_at` to prevent duplicates.
+- In rewritten `tasks/point_expiration.py` (see 3.1), find cards 14 days from expiry, fire warning, stamp timestamp.
+- **Validation:** time-mocked test — fires once at 14-day mark.
+
+#### 2.6 Wire reward-ready notification
+- In `services/stamp_service.add_stamp`, after commit, if `card.stamp_count == program.stamps_target`, dispatch.
+
+#### 2.7 Birthday Celery beat task
+- New `@shared_task loyalty.send_birthday_emails`, daily `0 8 * * *` in `definition.py`.
+- Queries cards where `customer.birth_date` matches today (month/day), respects timezone.
+- Depends on Phase 1.4 (column + persistence already in place).
+
+#### 2.8 Re-engagement Celery beat task
+- Weekly schedule. Finds cards inactive > N days (default 60, configurable).
+- Throttled via `last_reengagement_at` (added in `loyalty_005`) — once per quarter per card.
+
+---
+
+### Phase 3 — Task Reliability *(1.5d)*
+
+#### 3.1 Batched point expiration
+- Rewrite `tasks/point_expiration.py:154-185` from per-card loop to set-based SQL:
+  - Per program, chunk card IDs with `LIMIT 500 FOR UPDATE SKIP LOCKED`
+  - Build expiration `loyalty_transactions` rows in bulk per chunk
+  - `UPDATE loyalty_cards SET points_balance=0, total_points_voided=total_points_voided+points_balance WHERE id = ANY(...)`
+  - Commit per chunk inside savepoint
+- Done same pass as 2.5 (warning logic).
+- **Validation:** seed 10k cards, time the run, verify flat memory and chunked commits.
+
+#### 3.2 Wallet sync exponential backoff
+- Replace `time.sleep(2)` retry at `tasks/wallet_sync.py:71-95` with exponential backoff `[1s, 4s, 16s]`.
+- Per-card try/except (one bad card doesn't kill the batch).
+- Log `failed_card_ids` to observability.
+- Use `tenacity` if available, otherwise small helper.
+
+---
+
+### Phase 4 — Accessibility & T&C *(2d)*
+
+#### 4.1 T&C via store CMS integration
+- Migration `loyalty_006`: add `terms_cms_page_slug: str | None` to `loyalty_programs`.
+- Update `schemas/program.py` (`ProgramCreate`, `ProgramUpdate`).
+- `program-form.html:251` — CMS page picker scoped to the program's owning store.
+- `enroll.html:99-160` — resolve slug to CMS page URL/content; legacy `terms_text` fallback.
+- Service layer: `program_service.py:491,948`.
+- JS: `loyalty-program-form.js:34,75,106`.
+- Pre-flight: confirm CMS module exposes "get pages by store" API.
+
+#### 4.2 Accessibility audit
+- All loyalty templates: icon-only buttons → `aria-label`; modals → `role="dialog"` + `aria-modal="true"` + `aria-labelledby` + focus trap + Escape key + return-focus-on-close.
+- PIN entry modal: `inputmode="numeric"`, `autocomplete="one-time-code"`, `aria-live="polite"` for errors.
+- Pattern reference: existing Escape handler in `enroll.html:147`.
+- **Validation:** axe-core on the 5 main loyalty pages, zero critical violations.
+
+---
+
+### Phase 5 — Google Wallet Production Hardening *(1d)*
+
+#### 5.1 Cert deployment
+- Place service account JSON at `~/apps/orion/google-wallet-sa.json`, app user, mode 600.
+- `.env` line 196 updated.
+- Startup validators from 1.1 confirm load.
+
+#### 5.2 Conditional UI flag
+- Storefront API returns `google_wallet_enabled: bool` (derived from `config.is_google_wallet_enabled`).
+- `enroll.html:99-125`: hide Google button when disabled.
+- Apple Wallet button removed entirely (or hidden behind feature flag for Phase 9 reintroduction).
+
+#### 5.3 End-to-end real-device test
+- Android device → enroll → "Add to Google Wallet" → verify pass renders → trigger stamp → verify pass auto-updates within ~60s.
+
+---
+
+### Phase 6 — Admin UX, GDPR, Bulk *(3d)*
+
+#### 6.1 Admin trash UI
+- Trash tab on programs list and cards list, calling existing `?only_deleted=true` API.
+- Restore button per row + hard-delete destructive action.
+- Templates: `templates/loyalty/admin/programs.html`, `cards.html` + Alpine JS.
+
+#### 6.2 Cascade restore
+- `program_service.restore_for_merchant(db, merchant_id)` — restores soft-deleted programs and cards under a merchant.
+- Hooked into tenancy module's restore flow via event subscriber or direct call.
+- **Validation:** soft-delete merchant + 2 programs + 50 cards; restore merchant; all 52 rows un-deleted.
+
+#### 6.3 Customer GDPR delete (anonymize)
+- `DELETE /api/v1/loyalty/cards/customer/{customer_id}` on admin router.
+- New service method `card_service.anonymize_cards_for_customer`:
+  - `card.customer_id = NULL`
+  - Scrub PII fields, scrub `notes` if PII suspected
+  - Keep transaction rows (aggregate data)
+  - Audit log entry with admin ID
+
+#### 6.4 Bulk endpoints
+- **Merchant router** (`routes/api/merchant.py`):
+  - `POST /merchants/loyalty/cards/bulk/deactivate` — `{card_ids, reason}`
+  - `POST /merchants/loyalty/pins/bulk/assign` — `{store_id, pins}`
+  - `POST /merchants/loyalty/cards/bulk/bonus` — `{card_ids | filter, points, reason}`, fires notifications
+- **Admin "act on behalf"** (`routes/api/admin.py`):
+  - `POST /admin/loyalty/merchants/{merchant_id}/cards/bulk/deactivate` (etc.)
+  - Shared service layer; route stamps `acting_admin_id` in audit log
+- New `loyalty_transactions.acting_admin_id` column in `loyalty_005`.
+
+#### 6.5 Manual override: restore expired points
+- `POST /admin/loyalty/cards/{card_id}/restore_points` with `{points, reason}`.
+- Creates `ADMIN_ADJUSTMENT` transaction via existing `points_service.adjust_points`.
+- UI button on admin card detail page.
+
+---
+
+### Phase 7 — Advanced Analytics *(2.5d)*
+
+#### 7.1 Cohort retention
+- New `services/analytics_service.py` (or extend `program_service`).
+- Group cards by enrollment month, track % active (transaction in last 30d) per subsequent month.
+- Endpoint `/merchants/loyalty/analytics/cohorts` returns matrix.
+- Chart.js heatmap on `analytics.html`.
+
+#### 7.2 Simple churn rule
+- "At risk" = inactive > 2× normal inter-transaction interval.
+- Expose count + list.
+
+#### 7.3 Revenue attribution
+- Join `loyalty_transactions` (where `points_earned.order_reference` set) with orders module.
+- Loyalty-customer revenue vs non-loyalty per store per month.
+- **Cross-module rule:** use orders module's query helper, do not reach into its tables directly.
+
+---
+
+### Phase 8 — Tests, Docs, Observability *(2d)*
+
+#### 8.1 Coverage enforcement
+- Loyalty CI job: `pytest app/modules/loyalty/tests --cov=app/modules/loyalty --cov-fail-under=80`.
+- Audit current with `--cov-report=term-missing`, fill biggest gaps (notifications, batched expiration, rate-limited endpoints).
+
+#### 8.2 OpenAPI polish
+- Audit all loyalty routes for `summary`, `description`, `response_model`, example payloads, `tags=["Loyalty - Store"]` etc.
+- Export static `loyalty-openapi.json` under `app/modules/loyalty/docs/`.
+
+#### 8.3 Runbooks
+Three new docs under `docs/modules/loyalty/`:
+- `runbook-wallet-certs.md` — obtain, rotate, install, expiry monitoring, rollback.
+- `runbook-expiration-task.md` — re-run for specific merchant, partial failure handling, link to 6.5 manual restore.
+- `runbook-wallet-sync.md` — interpreting `failed_card_ids`, manual re-sync, common failures.
+
+#### 8.4 Monitoring & alerting
+- Celery `loyalty.expire_points` not succeeded in 26h → page
+- `loyalty.sync_wallet_passes` failure rate > 5% → warn
+- Loyalty email failure rate > 1% → warn
+- Google Wallet service account JSON cert expiry < 30d → warn
+- Rate-limit 429s per store > 100/min → investigate
+- Wire into `app/core/observability.py` (or whatever stack is present).
+
+---
+
+### Phase 9 — Apple Wallet *(post-launch, 3d)*
+
+Tracked separately, not blocking launch.
+
+- 9.1 Apple Developer account + Pass Type ID + WWDR cert
+- 9.2 Apple config wiring + all-or-nothing validators in `config.py:33-37`
+- 9.3 Cert deployment under `~/apps/orion/apple-wallet/` (matching the Google secrets convention)
+- 9.4 *(already done in 1.2 — timing-safe compare)*
+- 9.5 UI feature flag flip + Apple button re-enabled
+- 9.6 iOS Simulator + real device E2E test
+
+---
+
+## Critical Path
+
+```
+Phase 0 (done) ──┬─► Phase 1 ──┬─► Phase 3 ──┐
+                 ├─► Phase 2 ──┤             ├─► Phase 8 ──► LAUNCH
+                 └─► Phase 5 ──┘             │
+                                             │
+   Phase 4, 6, 7 (parallelizable) ───────────┘
+
+   Phase 9 — post-launch
+```
+
+Phases 4, 6, 7 can run in parallel with 2/3/5 if multiple developers are available.
+
+## Effort Summary
+
+| Phase | Days |
+|---|---|
+| 0 — Decisions | done |
+| 1 — Config & security | 2 |
+| 2 — Notifications | 4 |
+| 3 — Task reliability | 1.5 |
+| 4 — A11y + CMS T&C | 2 |
+| 5 — Google Wallet hardening | 1 |
+| 6 — Admin / GDPR / bulk | 3 |
+| 7 — Analytics | 2.5 |
+| 8 — Tests / docs / observability | 2 |
+| **Launch total** | **~18 days sequential, ~10 with 2 parallel tracks** |
+| 9 — Apple Wallet (post-launch) | 3 |
+
+---
+
+## Open Items Needing Sign-off
+
+1. ~~**Rate limit caps**~~ — confirmed.
+2. **Email copywriting** for the 7 templates × 4 locales (Phase 2.3) — flow: I draft EN, Samir reviews, then translate.
+3. ~~**`birth_date` column**~~ — confirmed missing; addressed in Phase 1.4. No backfill needed (not yet live).