docs: update documentation with Celery/Redis and Makefile changes
- Update Makefile help sections with correct target names (platform-install) - Add Celery infrastructure section to background-tasks.md - Update installation.md with Celery setup instructions and environment vars - Update index.md with comprehensive development commands including Celery - Add Flower monitoring and task queue commands to docs Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -4,6 +4,77 @@
|
||||
|
||||
This document defines the harmonized architecture for all background tasks in the application. Background tasks are long-running operations that execute asynchronously, allowing users to continue browsing while the task completes.
|
||||
|
||||
## Task Queue Infrastructure
|
||||
|
||||
Wizamart uses **Celery with Redis** for production-grade background task processing:
|
||||
|
||||
| Component | Purpose | Port |
|
||||
|-----------|---------|------|
|
||||
| **Celery Worker** | Executes background tasks | - |
|
||||
| **Celery Beat** | Scheduled task scheduler | - |
|
||||
| **Redis** | Message broker & result backend | 6379 |
|
||||
| **Flower** | Web-based task monitoring | 5555 |
|
||||
|
||||
### Configuration
|
||||
|
||||
```bash
|
||||
# Environment variables (.env)
|
||||
REDIS_URL=redis://localhost:6379/0
|
||||
USE_CELERY=true # false = use FastAPI BackgroundTasks
|
||||
FLOWER_URL=http://localhost:5555
|
||||
FLOWER_PASSWORD=changeme
|
||||
```
|
||||
|
||||
### Running Celery
|
||||
|
||||
```bash
|
||||
# Start Redis
|
||||
docker compose up -d redis
|
||||
|
||||
# Start Celery worker (processes tasks)
|
||||
make celery-worker
|
||||
|
||||
# Start Celery beat (scheduled tasks)
|
||||
make celery-beat
|
||||
|
||||
# Start both together (development)
|
||||
make celery-dev
|
||||
|
||||
# Start Flower monitoring
|
||||
make flower
|
||||
```
|
||||
|
||||
### Feature Flag: USE_CELERY
|
||||
|
||||
The system supports gradual migration via the `USE_CELERY` feature flag:
|
||||
|
||||
- **`USE_CELERY=false` (default)**: Uses FastAPI's `BackgroundTasks` for immediate execution without Redis dependency. Suitable for development.
|
||||
- **`USE_CELERY=true`**: Routes tasks through Celery for persistent queuing, retries, and scheduling. Required for production.
|
||||
|
||||
### Task Routing
|
||||
|
||||
Tasks are routed to queues based on their characteristics:
|
||||
|
||||
| Queue | Tasks | Characteristics |
|
||||
|-------|-------|-----------------|
|
||||
| `default` | Product exports | Fast, non-blocking |
|
||||
| `long_running` | Imports, code quality scans, tests | May take 10+ minutes |
|
||||
| `scheduled` | Subscription maintenance | Triggered by Celery Beat |
|
||||
|
||||
### Celery Task ID Tracking
|
||||
|
||||
All background job models include a `celery_task_id` field for cross-referencing with Flower:
|
||||
|
||||
```python
|
||||
class MarketplaceImportJob(Base):
|
||||
celery_task_id = Column(String(255), nullable=True)
|
||||
```
|
||||
|
||||
This enables:
|
||||
- Deep linking to Flower for task details
|
||||
- Task cancellation via Flower API
|
||||
- Correlation between database records and Celery events
|
||||
|
||||
## Current State Analysis
|
||||
|
||||
| Task Type | Database Model | Status Values | Tracked in BG Tasks Page |
|
||||
|
||||
Reference in New Issue
Block a user