docs: update infrastructure docs to reflect Celery/Redis implementation

- Update "Current State" table: Celery, Beat, Flower, Redis now production-ready
- Update "What We Need to Add": removed Celery/Redis, added S3 and Sentry as priorities
- Add Celery commands section to development environment
- Update services table with Redis, Celery, Flower ports
- Update Docker Compose services section with all Celery services
- Update Migration Path: Phase 1 & 2 marked as COMPLETE
- Update Next Steps: focus on S3, Sentry, CloudFlare
- Fix Celery app path in systemd services (app.core.celery_config)
- Add Flower systemd service configuration
- Add queue flags to Celery worker command

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
2026-01-11 19:13:40 +01:00
parent 0d204f39a8
commit 12b79c1ff0
2 changed files with 93 additions and 29 deletions

View File

@@ -83,18 +83,20 @@ This guide documents the complete infrastructure for the Wizamart platform, from
| Authentication | JWT (PyJWT) | ✅ Production Ready | | Authentication | JWT (PyJWT) | ✅ Production Ready |
| Email | SMTP/SendGrid/Mailgun/SES | ✅ Production Ready | | Email | SMTP/SendGrid/Mailgun/SES | ✅ Production Ready |
| Payments | Stripe | ✅ Production Ready | | Payments | Stripe | ✅ Production Ready |
| Background Jobs | - | Planned (Celery) | | Task Queue | Celery 5.3 + Redis | Production Ready |
| Caching | - | Planned (Redis) | | Task Scheduler | Celery Beat | Production Ready |
| Task Monitoring | Flower | ✅ Production Ready |
| Caching | Redis 7 | ✅ Production Ready |
| File Storage | Local filesystem | ⏳ Needs S3 for prod | | File Storage | Local filesystem | ⏳ Needs S3 for prod |
### What We Need to Add ### What We Need to Add
| Component | Priority | Reason | | Component | Priority | Reason |
|-----------|----------|--------| |-----------|----------|--------|
| Redis | High | Session cache, Celery broker | | S3/MinIO | High | Scalable file storage |
| Celery | High | Background jobs (imports, emails, reports) | | Sentry | High | Error tracking |
| S3/MinIO | Medium | Scalable file storage | | CloudFlare | Medium | CDN + DDoS protection |
| Sentry | Medium | Error tracking | | Load Balancer | Medium | Horizontal scaling |
| Prometheus/Grafana | Low | Metrics and dashboards | | Prometheus/Grafana | Low | Metrics and dashboards |
--- ---
@@ -104,7 +106,7 @@ This guide documents the complete infrastructure for the Wizamart platform, from
### Local Setup (Recommended) ### Local Setup (Recommended)
```bash ```bash
# 1. Start PostgreSQL # 1. Start PostgreSQL and Redis
make docker-up make docker-up
# 2. Run migrations # 2. Run migrations
@@ -116,7 +118,10 @@ make init-prod
# 4. Start development server # 4. Start development server
make dev make dev
# 5. (Optional) Run tests # 5. (Optional) Start Celery worker for background tasks
make celery-dev # Worker + Beat together
# 6. (Optional) Run tests
make test make test
``` ```
@@ -127,16 +132,45 @@ make test
| FastAPI | localhost | 8000 | Main application | | FastAPI | localhost | 8000 | Main application |
| PostgreSQL | localhost | 5432 | Development database | | PostgreSQL | localhost | 5432 | Development database |
| PostgreSQL (test) | localhost | 5433 | Test database | | PostgreSQL (test) | localhost | 5433 | Test database |
| MkDocs | localhost | 8001 | Documentation | | Redis | localhost | 6380 | Cache and task broker |
| Celery Worker | - | - | Background task processing |
| Celery Beat | - | - | Scheduled task scheduler |
| Flower | localhost | 5555 | Task monitoring dashboard |
| MkDocs | localhost | 9991 | Documentation |
### Docker Compose Services ### Docker Compose Services
```yaml ```yaml
# docker-compose.yml # docker-compose.yml
services: services:
db: # PostgreSQL for development db: # PostgreSQL 15 for development
redis: # Redis for cache/queue (coming soon) redis: # Redis 7 for cache/queue
api: # FastAPI application (optional) api: # FastAPI application (profile: full)
celery-worker: # Background task processor (profile: full)
celery-beat: # Scheduled task scheduler (profile: full)
flower: # Task monitoring UI (profile: full)
```
### Celery Commands
```bash
# Start worker only
make celery-worker
# Start scheduler only
make celery-beat
# Start worker + scheduler together (development)
make celery-dev
# Start Flower monitoring
make flower
# Check worker status
make celery-status
# Purge pending tasks
make celery-purge
``` ```
--- ---
@@ -804,42 +838,47 @@ python -c "from app.core.database import engine; print(engine.connect())"
## Migration Path ## Migration Path
### Phase 1: Current (Development) ### Phase 1: Current (Development) ✅ COMPLETE
- ✅ PostgreSQL (Docker) - ✅ PostgreSQL 15 (Docker)
- ✅ FastAPI + Uvicorn - ✅ FastAPI + Uvicorn
- ✅ Local file storage - ✅ Local file storage
### Phase 2: Production MVP ### Phase 2: Production MVP ✅ COMPLETE
- ✅ PostgreSQL (managed or VPS) - ✅ PostgreSQL (managed or VPS)
- ✅ FastAPI + Uvicorn (systemd or Docker) - ✅ FastAPI + Uvicorn (systemd or Docker)
- Redis (session cache) - Redis 7 (cache + task broker)
- Celery (background jobs) - Celery 5.3 (background jobs)
- ✅ Celery Beat (scheduled tasks)
- ✅ Flower (task monitoring)
- ⏳ S3/MinIO (file storage) - ⏳ S3/MinIO (file storage)
- ⏳ Sentry (error tracking)
### Phase 3: Scale ### Phase 3: Scale
- Horizontal app scaling (multiple Uvicorn instances) - Horizontal app scaling (multiple Uvicorn instances)
- Load balancer (Nginx/HAProxy)
- PostgreSQL read replicas - PostgreSQL read replicas
- Redis cluster - Redis Sentinel/cluster
- CDN for static assets - CDN for static assets (CloudFlare)
- Dedicated Celery workers per queue - Dedicated Celery workers per queue
### Phase 4: High Availability ### Phase 4: High Availability
- Multi-region deployment - Multi-region deployment
- Database failover - Database failover
- Container orchestration (Kubernetes) - Container orchestration (Kubernetes)
- Full monitoring stack - Full monitoring stack (Prometheus/Grafana/Loki)
--- ---
## Next Steps ## Next Steps
1. **Add Redis to docker-compose.yml** - For session cache 1. **Configure S3/MinIO** - For production file storage (high priority)
2. **Implement Celery** - Start with email and import tasks 2. **Set up Sentry** - Error tracking (high priority)
3. **Configure S3/MinIO** - For production file storage 3. **Add CloudFlare** - CDN + DDoS protection (medium priority)
4. **Set up Sentry** - Error tracking 4. **Configure load balancer** - When scaling horizontally
5. **Choose production deployment** - VPS or Docker based on team preference 5. **Choose production deployment** - VPS or Docker based on team preference
See also: See also:
- [Production Deployment Guide](production.md) - [Production Deployment Guide](production.md)
- [Docker Deployment](docker.md) - [Docker Deployment](docker.md)
- [Environment Configuration](environment.md) - [Environment Configuration](environment.md)
- [Background Tasks Architecture](../architecture/background-tasks.md)

View File

@@ -81,7 +81,7 @@ StandardError=journal
WantedBy=multi-user.target WantedBy=multi-user.target
``` ```
### Celery Worker (when implemented) ### Celery Worker
```bash ```bash
sudo nano /etc/systemd/system/wizamart-celery.service sudo nano /etc/systemd/system/wizamart-celery.service
@@ -98,7 +98,7 @@ Group=wizamart
WorkingDirectory=/home/wizamart/app WorkingDirectory=/home/wizamart/app
Environment="PATH=/home/wizamart/app/.venv/bin" Environment="PATH=/home/wizamart/app/.venv/bin"
EnvironmentFile=/home/wizamart/app/.env EnvironmentFile=/home/wizamart/app/.env
ExecStart=/home/wizamart/app/.venv/bin/celery -A app.celery worker --loglevel=info --concurrency=4 ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config worker --loglevel=info -Q default,long_running,scheduled --concurrency=4
Restart=always Restart=always
RestartSec=3 RestartSec=3
StandardOutput=journal StandardOutput=journal
@@ -125,7 +125,32 @@ Group=wizamart
WorkingDirectory=/home/wizamart/app WorkingDirectory=/home/wizamart/app
Environment="PATH=/home/wizamart/app/.venv/bin" Environment="PATH=/home/wizamart/app/.venv/bin"
EnvironmentFile=/home/wizamart/app/.env EnvironmentFile=/home/wizamart/app/.env
ExecStart=/home/wizamart/app/.venv/bin/celery -A app.celery beat --loglevel=info ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config beat --loglevel=info
Restart=always
RestartSec=3
[Install]
WantedBy=multi-user.target
```
### Flower (Task Monitoring)
```bash
sudo nano /etc/systemd/system/wizamart-flower.service
```
```ini
[Unit]
Description=Wizamart Flower Task Monitor
After=network.target redis.service
[Service]
User=wizamart
Group=wizamart
WorkingDirectory=/home/wizamart/app
Environment="PATH=/home/wizamart/app/.venv/bin"
EnvironmentFile=/home/wizamart/app/.env
ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config flower --port=5555
Restart=always Restart=always
RestartSec=3 RestartSec=3
@@ -137,8 +162,8 @@ WantedBy=multi-user.target
```bash ```bash
sudo systemctl daemon-reload sudo systemctl daemon-reload
sudo systemctl enable wizamart wizamart-celery wizamart-celery-beat sudo systemctl enable wizamart wizamart-celery wizamart-celery-beat wizamart-flower
sudo systemctl start wizamart wizamart-celery wizamart-celery-beat sudo systemctl start wizamart wizamart-celery wizamart-celery-beat wizamart-flower
``` ```
--- ---