docs: update infrastructure docs to reflect Celery/Redis implementation
- Update "Current State" table: Celery, Beat, Flower, Redis now production-ready - Update "What We Need to Add": removed Celery/Redis, added S3 and Sentry as priorities - Add Celery commands section to development environment - Update services table with Redis, Celery, Flower ports - Update Docker Compose services section with all Celery services - Update Migration Path: Phase 1 & 2 marked as COMPLETE - Update Next Steps: focus on S3, Sentry, CloudFlare - Fix Celery app path in systemd services (app.core.celery_config) - Add Flower systemd service configuration - Add queue flags to Celery worker command Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -83,18 +83,20 @@ This guide documents the complete infrastructure for the Wizamart platform, from
|
|||||||
| Authentication | JWT (PyJWT) | ✅ Production Ready |
|
| Authentication | JWT (PyJWT) | ✅ Production Ready |
|
||||||
| Email | SMTP/SendGrid/Mailgun/SES | ✅ Production Ready |
|
| Email | SMTP/SendGrid/Mailgun/SES | ✅ Production Ready |
|
||||||
| Payments | Stripe | ✅ Production Ready |
|
| Payments | Stripe | ✅ Production Ready |
|
||||||
| Background Jobs | - | ⏳ Planned (Celery) |
|
| Task Queue | Celery 5.3 + Redis | ✅ Production Ready |
|
||||||
| Caching | - | ⏳ Planned (Redis) |
|
| Task Scheduler | Celery Beat | ✅ Production Ready |
|
||||||
|
| Task Monitoring | Flower | ✅ Production Ready |
|
||||||
|
| Caching | Redis 7 | ✅ Production Ready |
|
||||||
| File Storage | Local filesystem | ⏳ Needs S3 for prod |
|
| File Storage | Local filesystem | ⏳ Needs S3 for prod |
|
||||||
|
|
||||||
### What We Need to Add
|
### What We Need to Add
|
||||||
|
|
||||||
| Component | Priority | Reason |
|
| Component | Priority | Reason |
|
||||||
|-----------|----------|--------|
|
|-----------|----------|--------|
|
||||||
| Redis | High | Session cache, Celery broker |
|
| S3/MinIO | High | Scalable file storage |
|
||||||
| Celery | High | Background jobs (imports, emails, reports) |
|
| Sentry | High | Error tracking |
|
||||||
| S3/MinIO | Medium | Scalable file storage |
|
| CloudFlare | Medium | CDN + DDoS protection |
|
||||||
| Sentry | Medium | Error tracking |
|
| Load Balancer | Medium | Horizontal scaling |
|
||||||
| Prometheus/Grafana | Low | Metrics and dashboards |
|
| Prometheus/Grafana | Low | Metrics and dashboards |
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -104,7 +106,7 @@ This guide documents the complete infrastructure for the Wizamart platform, from
|
|||||||
### Local Setup (Recommended)
|
### Local Setup (Recommended)
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 1. Start PostgreSQL
|
# 1. Start PostgreSQL and Redis
|
||||||
make docker-up
|
make docker-up
|
||||||
|
|
||||||
# 2. Run migrations
|
# 2. Run migrations
|
||||||
@@ -116,7 +118,10 @@ make init-prod
|
|||||||
# 4. Start development server
|
# 4. Start development server
|
||||||
make dev
|
make dev
|
||||||
|
|
||||||
# 5. (Optional) Run tests
|
# 5. (Optional) Start Celery worker for background tasks
|
||||||
|
make celery-dev # Worker + Beat together
|
||||||
|
|
||||||
|
# 6. (Optional) Run tests
|
||||||
make test
|
make test
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -127,16 +132,45 @@ make test
|
|||||||
| FastAPI | localhost | 8000 | Main application |
|
| FastAPI | localhost | 8000 | Main application |
|
||||||
| PostgreSQL | localhost | 5432 | Development database |
|
| PostgreSQL | localhost | 5432 | Development database |
|
||||||
| PostgreSQL (test) | localhost | 5433 | Test database |
|
| PostgreSQL (test) | localhost | 5433 | Test database |
|
||||||
| MkDocs | localhost | 8001 | Documentation |
|
| Redis | localhost | 6380 | Cache and task broker |
|
||||||
|
| Celery Worker | - | - | Background task processing |
|
||||||
|
| Celery Beat | - | - | Scheduled task scheduler |
|
||||||
|
| Flower | localhost | 5555 | Task monitoring dashboard |
|
||||||
|
| MkDocs | localhost | 9991 | Documentation |
|
||||||
|
|
||||||
### Docker Compose Services
|
### Docker Compose Services
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
# docker-compose.yml
|
# docker-compose.yml
|
||||||
services:
|
services:
|
||||||
db: # PostgreSQL for development
|
db: # PostgreSQL 15 for development
|
||||||
redis: # Redis for cache/queue (coming soon)
|
redis: # Redis 7 for cache/queue
|
||||||
api: # FastAPI application (optional)
|
api: # FastAPI application (profile: full)
|
||||||
|
celery-worker: # Background task processor (profile: full)
|
||||||
|
celery-beat: # Scheduled task scheduler (profile: full)
|
||||||
|
flower: # Task monitoring UI (profile: full)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Celery Commands
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start worker only
|
||||||
|
make celery-worker
|
||||||
|
|
||||||
|
# Start scheduler only
|
||||||
|
make celery-beat
|
||||||
|
|
||||||
|
# Start worker + scheduler together (development)
|
||||||
|
make celery-dev
|
||||||
|
|
||||||
|
# Start Flower monitoring
|
||||||
|
make flower
|
||||||
|
|
||||||
|
# Check worker status
|
||||||
|
make celery-status
|
||||||
|
|
||||||
|
# Purge pending tasks
|
||||||
|
make celery-purge
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
@@ -804,42 +838,47 @@ python -c "from app.core.database import engine; print(engine.connect())"
|
|||||||
|
|
||||||
## Migration Path
|
## Migration Path
|
||||||
|
|
||||||
### Phase 1: Current (Development)
|
### Phase 1: Current (Development) ✅ COMPLETE
|
||||||
- ✅ PostgreSQL (Docker)
|
- ✅ PostgreSQL 15 (Docker)
|
||||||
- ✅ FastAPI + Uvicorn
|
- ✅ FastAPI + Uvicorn
|
||||||
- ✅ Local file storage
|
- ✅ Local file storage
|
||||||
|
|
||||||
### Phase 2: Production MVP
|
### Phase 2: Production MVP ✅ COMPLETE
|
||||||
- ✅ PostgreSQL (managed or VPS)
|
- ✅ PostgreSQL (managed or VPS)
|
||||||
- ✅ FastAPI + Uvicorn (systemd or Docker)
|
- ✅ FastAPI + Uvicorn (systemd or Docker)
|
||||||
- ⏳ Redis (session cache)
|
- ✅ Redis 7 (cache + task broker)
|
||||||
- ⏳ Celery (background jobs)
|
- ✅ Celery 5.3 (background jobs)
|
||||||
|
- ✅ Celery Beat (scheduled tasks)
|
||||||
|
- ✅ Flower (task monitoring)
|
||||||
- ⏳ S3/MinIO (file storage)
|
- ⏳ S3/MinIO (file storage)
|
||||||
|
- ⏳ Sentry (error tracking)
|
||||||
|
|
||||||
### Phase 3: Scale
|
### Phase 3: Scale
|
||||||
- Horizontal app scaling (multiple Uvicorn instances)
|
- Horizontal app scaling (multiple Uvicorn instances)
|
||||||
|
- Load balancer (Nginx/HAProxy)
|
||||||
- PostgreSQL read replicas
|
- PostgreSQL read replicas
|
||||||
- Redis cluster
|
- Redis Sentinel/cluster
|
||||||
- CDN for static assets
|
- CDN for static assets (CloudFlare)
|
||||||
- Dedicated Celery workers per queue
|
- Dedicated Celery workers per queue
|
||||||
|
|
||||||
### Phase 4: High Availability
|
### Phase 4: High Availability
|
||||||
- Multi-region deployment
|
- Multi-region deployment
|
||||||
- Database failover
|
- Database failover
|
||||||
- Container orchestration (Kubernetes)
|
- Container orchestration (Kubernetes)
|
||||||
- Full monitoring stack
|
- Full monitoring stack (Prometheus/Grafana/Loki)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Next Steps
|
## Next Steps
|
||||||
|
|
||||||
1. **Add Redis to docker-compose.yml** - For session cache
|
1. **Configure S3/MinIO** - For production file storage (high priority)
|
||||||
2. **Implement Celery** - Start with email and import tasks
|
2. **Set up Sentry** - Error tracking (high priority)
|
||||||
3. **Configure S3/MinIO** - For production file storage
|
3. **Add CloudFlare** - CDN + DDoS protection (medium priority)
|
||||||
4. **Set up Sentry** - Error tracking
|
4. **Configure load balancer** - When scaling horizontally
|
||||||
5. **Choose production deployment** - VPS or Docker based on team preference
|
5. **Choose production deployment** - VPS or Docker based on team preference
|
||||||
|
|
||||||
See also:
|
See also:
|
||||||
- [Production Deployment Guide](production.md)
|
- [Production Deployment Guide](production.md)
|
||||||
- [Docker Deployment](docker.md)
|
- [Docker Deployment](docker.md)
|
||||||
- [Environment Configuration](environment.md)
|
- [Environment Configuration](environment.md)
|
||||||
|
- [Background Tasks Architecture](../architecture/background-tasks.md)
|
||||||
|
|||||||
@@ -81,7 +81,7 @@ StandardError=journal
|
|||||||
WantedBy=multi-user.target
|
WantedBy=multi-user.target
|
||||||
```
|
```
|
||||||
|
|
||||||
### Celery Worker (when implemented)
|
### Celery Worker
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo nano /etc/systemd/system/wizamart-celery.service
|
sudo nano /etc/systemd/system/wizamart-celery.service
|
||||||
@@ -98,7 +98,7 @@ Group=wizamart
|
|||||||
WorkingDirectory=/home/wizamart/app
|
WorkingDirectory=/home/wizamart/app
|
||||||
Environment="PATH=/home/wizamart/app/.venv/bin"
|
Environment="PATH=/home/wizamart/app/.venv/bin"
|
||||||
EnvironmentFile=/home/wizamart/app/.env
|
EnvironmentFile=/home/wizamart/app/.env
|
||||||
ExecStart=/home/wizamart/app/.venv/bin/celery -A app.celery worker --loglevel=info --concurrency=4
|
ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config worker --loglevel=info -Q default,long_running,scheduled --concurrency=4
|
||||||
Restart=always
|
Restart=always
|
||||||
RestartSec=3
|
RestartSec=3
|
||||||
StandardOutput=journal
|
StandardOutput=journal
|
||||||
@@ -125,7 +125,32 @@ Group=wizamart
|
|||||||
WorkingDirectory=/home/wizamart/app
|
WorkingDirectory=/home/wizamart/app
|
||||||
Environment="PATH=/home/wizamart/app/.venv/bin"
|
Environment="PATH=/home/wizamart/app/.venv/bin"
|
||||||
EnvironmentFile=/home/wizamart/app/.env
|
EnvironmentFile=/home/wizamart/app/.env
|
||||||
ExecStart=/home/wizamart/app/.venv/bin/celery -A app.celery beat --loglevel=info
|
ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config beat --loglevel=info
|
||||||
|
Restart=always
|
||||||
|
RestartSec=3
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
|
```
|
||||||
|
|
||||||
|
### Flower (Task Monitoring)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
sudo nano /etc/systemd/system/wizamart-flower.service
|
||||||
|
```
|
||||||
|
|
||||||
|
```ini
|
||||||
|
[Unit]
|
||||||
|
Description=Wizamart Flower Task Monitor
|
||||||
|
After=network.target redis.service
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
User=wizamart
|
||||||
|
Group=wizamart
|
||||||
|
WorkingDirectory=/home/wizamart/app
|
||||||
|
Environment="PATH=/home/wizamart/app/.venv/bin"
|
||||||
|
EnvironmentFile=/home/wizamart/app/.env
|
||||||
|
ExecStart=/home/wizamart/app/.venv/bin/celery -A app.core.celery_config flower --port=5555
|
||||||
Restart=always
|
Restart=always
|
||||||
RestartSec=3
|
RestartSec=3
|
||||||
|
|
||||||
@@ -137,8 +162,8 @@ WantedBy=multi-user.target
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
sudo systemctl daemon-reload
|
sudo systemctl daemon-reload
|
||||||
sudo systemctl enable wizamart wizamart-celery wizamart-celery-beat
|
sudo systemctl enable wizamart wizamart-celery wizamart-celery-beat wizamart-flower
|
||||||
sudo systemctl start wizamart wizamart-celery wizamart-celery-beat
|
sudo systemctl start wizamart wizamart-celery wizamart-celery-beat wizamart-flower
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|||||||
Reference in New Issue
Block a user