docs: update infrastructure docs to reflect Celery/Redis implementation

- Update "Current State" table: Celery, Beat, Flower, Redis now production-ready - Update "What We Need to Add": removed Celery/Redis, added S3 and Sentry as priorities - Add Celery commands section to development environment - Update services table with Redis, Celery, Flower ports - Update Docker Compose services section with all Celery services - Update Migration Path: Phase 1 & 2 marked as COMPLETE - Update Next Steps: focus on S3, Sentry, CloudFlare - Fix Celery app path in systemd services (app.core.celery_config) - Add Flower systemd service configuration - Add queue flags to Celery worker command Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-11 19:13:40 +01:00
parent 0d204f39a8
commit 12b79c1ff0
2 changed files with 93 additions and 29 deletions
--- a/docs/deployment/infrastructure.md
+++ b/docs/deployment/infrastructure.md
@@ -83,18 +83,20 @@ This guide documents the complete infrastructure for the Wizamart platform, from
 | Authentication | JWT (PyJWT) | ✅ Production Ready |
 | Email | SMTP/SendGrid/Mailgun/SES | ✅ Production Ready |
 | Payments | Stripe | ✅ Production Ready |
-| Background Jobs | - | ⏳ Planned (Celery) |
-| Caching | - | ⏳ Planned (Redis) |
+| Task Queue | Celery 5.3 + Redis | ✅ Production Ready |
+| Task Scheduler | Celery Beat | ✅ Production Ready |
+| Task Monitoring | Flower | ✅ Production Ready |
+| Caching | Redis 7 | ✅ Production Ready |
 | File Storage | Local filesystem | ⏳ Needs S3 for prod |

 ### What We Need to Add

 | Component | Priority | Reason |
 |-----------|----------|--------|
-| Redis | High | Session cache, Celery broker |
-| Celery | High | Background jobs (imports, emails, reports) |
-| S3/MinIO | Medium | Scalable file storage |
-| Sentry | Medium | Error tracking |
+| S3/MinIO | High | Scalable file storage |
+| Sentry | High | Error tracking |
+| CloudFlare | Medium | CDN + DDoS protection |
+| Load Balancer | Medium | Horizontal scaling |
 | Prometheus/Grafana | Low | Metrics and dashboards |

 ---
@@ -104,7 +106,7 @@ This guide documents the complete infrastructure for the Wizamart platform, from
 ### Local Setup (Recommended)

 ```bash
-# 1. Start PostgreSQL
+# 1. Start PostgreSQL and Redis
 make docker-up

 # 2. Run migrations
@@ -116,7 +118,10 @@ make init-prod
 # 4. Start development server
 make dev

-# 5. (Optional) Run tests
+# 5. (Optional) Start Celery worker for background tasks
+make celery-dev  # Worker + Beat together
+
+# 6. (Optional) Run tests
 make test
 ```

@@ -127,16 +132,45 @@ make test
 | FastAPI | localhost | 8000 | Main application |
 | PostgreSQL | localhost | 5432 | Development database |
 | PostgreSQL (test) | localhost | 5433 | Test database |
-| MkDocs | localhost | 8001 | Documentation |
+| Redis | localhost | 6380 | Cache and task broker |
+| Celery Worker | - | - | Background task processing |
+| Celery Beat | - | - | Scheduled task scheduler |
+| Flower | localhost | 5555 | Task monitoring dashboard |
+| MkDocs | localhost | 9991 | Documentation |

 ### Docker Compose Services

 ```yaml
 # docker-compose.yml
 services:
-  db:          # PostgreSQL for development
-  redis:       # Redis for cache/queue (coming soon)
-  api:         # FastAPI application (optional)
+  db:            # PostgreSQL 15 for development
+  redis:         # Redis 7 for cache/queue
+  api:           # FastAPI application (profile: full)
+  celery-worker: # Background task processor (profile: full)
+  celery-beat:   # Scheduled task scheduler (profile: full)
+  flower:        # Task monitoring UI (profile: full)
+```
+
+### Celery Commands
+
+```bash
+# Start worker only
+make celery-worker
+
+# Start scheduler only
+make celery-beat
+
+# Start worker + scheduler together (development)
+make celery-dev
+
+# Start Flower monitoring
+make flower
+
+# Check worker status
+make celery-status
+
+# Purge pending tasks
+make celery-purge
 ```

 ---
@@ -804,42 +838,47 @@ python -c "from app.core.database import engine; print(engine.connect())"

 ## Migration Path

-### Phase 1: Current (Development)
- ✅ PostgreSQL (Docker)
+### Phase 1: Current (Development) ✅ COMPLETE
+- ✅ PostgreSQL 15 (Docker)
 - ✅ FastAPI + Uvicorn
 - ✅ Local file storage

-### Phase 2: Production MVP
+### Phase 2: Production MVP ✅ COMPLETE
 - ✅ PostgreSQL (managed or VPS)
 - ✅ FastAPI + Uvicorn (systemd or Docker)
- ⏳ Redis (session cache)
- ⏳ Celery (background jobs)
+- ✅ Redis 7 (cache + task broker)
+- ✅ Celery 5.3 (background jobs)
+- ✅ Celery Beat (scheduled tasks)
+- ✅ Flower (task monitoring)
 - ⏳ S3/MinIO (file storage)
+- ⏳ Sentry (error tracking)

 ### Phase 3: Scale
 - Horizontal app scaling (multiple Uvicorn instances)
+- Load balancer (Nginx/HAProxy)
 - PostgreSQL read replicas
- Redis cluster
- CDN for static assets
+- Redis Sentinel/cluster
+- CDN for static assets (CloudFlare)
 - Dedicated Celery workers per queue

 ### Phase 4: High Availability
 - Multi-region deployment
 - Database failover
 - Container orchestration (Kubernetes)
- Full monitoring stack
+- Full monitoring stack (Prometheus/Grafana/Loki)

 ---

 ## Next Steps

-1. **Add Redis to docker-compose.yml** - For session cache
-2. **Implement Celery** - Start with email and import tasks
-3. **Configure S3/MinIO** - For production file storage
-4. **Set up Sentry** - Error tracking
+1. **Configure S3/MinIO** - For production file storage (high priority)
+2. **Set up Sentry** - Error tracking (high priority)
+3. **Add CloudFlare** - CDN + DDoS protection (medium priority)
+4. **Configure load balancer** - When scaling horizontally
 5. **Choose production deployment** - VPS or Docker based on team preference

 See also:
 - [Production Deployment Guide](production.md)
 - [Docker Deployment](docker.md)
 - [Environment Configuration](environment.md)
+- [Background Tasks Architecture](../architecture/background-tasks.md)
--- a/docs/deployment/production.md
+++ b/docs/deployment/production.md
@@ -81,7 +81,7 @@ StandardError=journal
 WantedBy=multi-user.target
 ```

-### Celery Worker (when implemented)
+### Celery Worker

 ```bash
 sudo nano /etc/systemd/system/wizamart-celery.service
@@ -98,7 +98,7 @@ Group=wizamart
 WorkingDirectory=/home/wizamart/app
 Environment="PATH=/home/wizamart/app/.venv/bin"
 EnvironmentFile=/home/wizamart/app/.env
-ExecStart=/home/wizamart/app/.venv/bin/celery -A app.celery worker --loglevel=info --concurrency=4
+ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config worker --loglevel=info -Q default,long_running,scheduled --concurrency=4
 Restart=always
 RestartSec=3
 StandardOutput=journal
@@ -125,7 +125,32 @@ Group=wizamart
 WorkingDirectory=/home/wizamart/app
 Environment="PATH=/home/wizamart/app/.venv/bin"
 EnvironmentFile=/home/wizamart/app/.env
-ExecStart=/home/wizamart/app/.venv/bin/celery -A app.celery beat --loglevel=info
+ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config beat --loglevel=info
+Restart=always
+RestartSec=3
+
+[Install]
+WantedBy=multi-user.target
+```
+
+### Flower (Task Monitoring)
+
+```bash
+sudo nano /etc/systemd/system/wizamart-flower.service
+```
+
+```ini
+[Unit]
+Description=Wizamart Flower Task Monitor
+After=network.target redis.service
+
+[Service]
+User=wizamart
+Group=wizamart
+WorkingDirectory=/home/wizamart/app
+Environment="PATH=/home/wizamart/app/.venv/bin"
+EnvironmentFile=/home/wizamart/app/.env
+ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config flower --port=5555
 Restart=always
 RestartSec=3

@@ -137,8 +162,8 @@ WantedBy=multi-user.target

 ```bash
 sudo systemctl daemon-reload
-sudo systemctl enable wizamart wizamart-celery wizamart-celery-beat
-sudo systemctl start wizamart wizamart-celery wizamart-celery-beat
+sudo systemctl enable wizamart wizamart-celery wizamart-celery-beat wizamart-flower
+sudo systemctl start wizamart wizamart-celery wizamart-celery-beat wizamart-flower
 ```

 ---