From 12b79c1ff02e04db2d2790ac96f0167fa3f51065 Mon Sep 17 00:00:00 2001 From: Samir Boulahtit Date: Sun, 11 Jan 2026 19:13:40 +0100 Subject: [PATCH] docs: update infrastructure docs to reflect Celery/Redis implementation - Update "Current State" table: Celery, Beat, Flower, Redis now production-ready - Update "What We Need to Add": removed Celery/Redis, added S3 and Sentry as priorities - Add Celery commands section to development environment - Update services table with Redis, Celery, Flower ports - Update Docker Compose services section with all Celery services - Update Migration Path: Phase 1 & 2 marked as COMPLETE - Update Next Steps: focus on S3, Sentry, CloudFlare - Fix Celery app path in systemd services (app.core.celery_config) - Add Flower systemd service configuration - Add queue flags to Celery worker command Co-Authored-By: Claude Opus 4.5 --- docs/deployment/infrastructure.md | 87 ++++++++++++++++++++++--------- docs/deployment/production.md | 35 +++++++++++-- 2 files changed, 93 insertions(+), 29 deletions(-) diff --git a/docs/deployment/infrastructure.md b/docs/deployment/infrastructure.md index c251c383..394c6233 100644 --- a/docs/deployment/infrastructure.md +++ b/docs/deployment/infrastructure.md @@ -83,18 +83,20 @@ This guide documents the complete infrastructure for the Wizamart platform, from | Authentication | JWT (PyJWT) | ✅ Production Ready | | Email | SMTP/SendGrid/Mailgun/SES | ✅ Production Ready | | Payments | Stripe | ✅ Production Ready | -| Background Jobs | - | ⏳ Planned (Celery) | -| Caching | - | ⏳ Planned (Redis) | +| Task Queue | Celery 5.3 + Redis | ✅ Production Ready | +| Task Scheduler | Celery Beat | ✅ Production Ready | +| Task Monitoring | Flower | ✅ Production Ready | +| Caching | Redis 7 | ✅ Production Ready | | File Storage | Local filesystem | ⏳ Needs S3 for prod | ### What We Need to Add | Component | Priority | Reason | |-----------|----------|--------| -| Redis | High | Session cache, Celery broker | -| Celery | High | Background jobs (imports, emails, reports) | -| S3/MinIO | Medium | Scalable file storage | -| Sentry | Medium | Error tracking | +| S3/MinIO | High | Scalable file storage | +| Sentry | High | Error tracking | +| CloudFlare | Medium | CDN + DDoS protection | +| Load Balancer | Medium | Horizontal scaling | | Prometheus/Grafana | Low | Metrics and dashboards | --- @@ -104,7 +106,7 @@ This guide documents the complete infrastructure for the Wizamart platform, from ### Local Setup (Recommended) ```bash -# 1. Start PostgreSQL +# 1. Start PostgreSQL and Redis make docker-up # 2. Run migrations @@ -116,7 +118,10 @@ make init-prod # 4. Start development server make dev -# 5. (Optional) Run tests +# 5. (Optional) Start Celery worker for background tasks +make celery-dev # Worker + Beat together + +# 6. (Optional) Run tests make test ``` @@ -127,16 +132,45 @@ make test | FastAPI | localhost | 8000 | Main application | | PostgreSQL | localhost | 5432 | Development database | | PostgreSQL (test) | localhost | 5433 | Test database | -| MkDocs | localhost | 8001 | Documentation | +| Redis | localhost | 6380 | Cache and task broker | +| Celery Worker | - | - | Background task processing | +| Celery Beat | - | - | Scheduled task scheduler | +| Flower | localhost | 5555 | Task monitoring dashboard | +| MkDocs | localhost | 9991 | Documentation | ### Docker Compose Services ```yaml # docker-compose.yml services: - db: # PostgreSQL for development - redis: # Redis for cache/queue (coming soon) - api: # FastAPI application (optional) + db: # PostgreSQL 15 for development + redis: # Redis 7 for cache/queue + api: # FastAPI application (profile: full) + celery-worker: # Background task processor (profile: full) + celery-beat: # Scheduled task scheduler (profile: full) + flower: # Task monitoring UI (profile: full) +``` + +### Celery Commands + +```bash +# Start worker only +make celery-worker + +# Start scheduler only +make celery-beat + +# Start worker + scheduler together (development) +make celery-dev + +# Start Flower monitoring +make flower + +# Check worker status +make celery-status + +# Purge pending tasks +make celery-purge ``` --- @@ -804,42 +838,47 @@ python -c "from app.core.database import engine; print(engine.connect())" ## Migration Path -### Phase 1: Current (Development) -- ✅ PostgreSQL (Docker) +### Phase 1: Current (Development) ✅ COMPLETE +- ✅ PostgreSQL 15 (Docker) - ✅ FastAPI + Uvicorn - ✅ Local file storage -### Phase 2: Production MVP +### Phase 2: Production MVP ✅ COMPLETE - ✅ PostgreSQL (managed or VPS) - ✅ FastAPI + Uvicorn (systemd or Docker) -- ⏳ Redis (session cache) -- ⏳ Celery (background jobs) +- ✅ Redis 7 (cache + task broker) +- ✅ Celery 5.3 (background jobs) +- ✅ Celery Beat (scheduled tasks) +- ✅ Flower (task monitoring) - ⏳ S3/MinIO (file storage) +- ⏳ Sentry (error tracking) ### Phase 3: Scale - Horizontal app scaling (multiple Uvicorn instances) +- Load balancer (Nginx/HAProxy) - PostgreSQL read replicas -- Redis cluster -- CDN for static assets +- Redis Sentinel/cluster +- CDN for static assets (CloudFlare) - Dedicated Celery workers per queue ### Phase 4: High Availability - Multi-region deployment - Database failover - Container orchestration (Kubernetes) -- Full monitoring stack +- Full monitoring stack (Prometheus/Grafana/Loki) --- ## Next Steps -1. **Add Redis to docker-compose.yml** - For session cache -2. **Implement Celery** - Start with email and import tasks -3. **Configure S3/MinIO** - For production file storage -4. **Set up Sentry** - Error tracking +1. **Configure S3/MinIO** - For production file storage (high priority) +2. **Set up Sentry** - Error tracking (high priority) +3. **Add CloudFlare** - CDN + DDoS protection (medium priority) +4. **Configure load balancer** - When scaling horizontally 5. **Choose production deployment** - VPS or Docker based on team preference See also: - [Production Deployment Guide](production.md) - [Docker Deployment](docker.md) - [Environment Configuration](environment.md) +- [Background Tasks Architecture](../architecture/background-tasks.md) diff --git a/docs/deployment/production.md b/docs/deployment/production.md index 473537cf..c5e56b18 100644 --- a/docs/deployment/production.md +++ b/docs/deployment/production.md @@ -81,7 +81,7 @@ StandardError=journal WantedBy=multi-user.target ``` -### Celery Worker (when implemented) +### Celery Worker ```bash sudo nano /etc/systemd/system/wizamart-celery.service @@ -98,7 +98,7 @@ Group=wizamart WorkingDirectory=/home/wizamart/app Environment="PATH=/home/wizamart/app/.venv/bin" EnvironmentFile=/home/wizamart/app/.env -ExecStart=/home/wizamart/app/.venv/bin/celery -A app.celery worker --loglevel=info --concurrency=4 +ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config worker --loglevel=info -Q default,long_running,scheduled --concurrency=4 Restart=always RestartSec=3 StandardOutput=journal @@ -125,7 +125,32 @@ Group=wizamart WorkingDirectory=/home/wizamart/app Environment="PATH=/home/wizamart/app/.venv/bin" EnvironmentFile=/home/wizamart/app/.env -ExecStart=/home/wizamart/app/.venv/bin/celery -A app.celery beat --loglevel=info +ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config beat --loglevel=info +Restart=always +RestartSec=3 + +[Install] +WantedBy=multi-user.target +``` + +### Flower (Task Monitoring) + +```bash +sudo nano /etc/systemd/system/wizamart-flower.service +``` + +```ini +[Unit] +Description=Wizamart Flower Task Monitor +After=network.target redis.service + +[Service] +User=wizamart +Group=wizamart +WorkingDirectory=/home/wizamart/app +Environment="PATH=/home/wizamart/app/.venv/bin" +EnvironmentFile=/home/wizamart/app/.env +ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config flower --port=5555 Restart=always RestartSec=3 @@ -137,8 +162,8 @@ WantedBy=multi-user.target ```bash sudo systemctl daemon-reload -sudo systemctl enable wizamart wizamart-celery wizamart-celery-beat -sudo systemctl start wizamart wizamart-celery wizamart-celery-beat +sudo systemctl enable wizamart wizamart-celery wizamart-celery-beat wizamart-flower +sudo systemctl start wizamart wizamart-celery wizamart-celery-beat wizamart-flower ``` ---