docs: update documentation with Celery/Redis and Makefile changes
- Update Makefile help sections with correct target names (platform-install) - Add Celery infrastructure section to background-tasks.md - Update installation.md with Celery setup instructions and environment vars - Update index.md with comprehensive development commands including Celery - Add Flower monitoring and task queue commands to docs Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
8
Makefile
8
Makefile
@@ -520,7 +520,7 @@ help:
|
|||||||
@echo " migrate-up - Apply pending migrations"
|
@echo " migrate-up - Apply pending migrations"
|
||||||
@echo " migrate-down - Rollback last migration"
|
@echo " migrate-down - Rollback last migration"
|
||||||
@echo " migrate-status - Show migration status"
|
@echo " migrate-status - Show migration status"
|
||||||
@echo " install - First-time setup (validates config + migrate + init)"
|
@echo " platform-install - First-time setup (validates config + migrate + init)"
|
||||||
@echo " init-prod - Initialize platform (admin, CMS, pages, emails)"
|
@echo " init-prod - Initialize platform (admin, CMS, pages, emails)"
|
||||||
@echo " seed-demo - Seed demo data (3 companies + vendors)"
|
@echo " seed-demo - Seed demo data (3 companies + vendors)"
|
||||||
@echo " seed-demo-minimal - Seed minimal demo (1 company + vendor)"
|
@echo " seed-demo-minimal - Seed minimal demo (1 company + vendor)"
|
||||||
@@ -594,7 +594,7 @@ help-db:
|
|||||||
@echo ""
|
@echo ""
|
||||||
@echo "FIRST-TIME INSTALLATION:"
|
@echo "FIRST-TIME INSTALLATION:"
|
||||||
@echo "──────────────────────────────────────────────────────────"
|
@echo "──────────────────────────────────────────────────────────"
|
||||||
@echo " install - Complete installation wizard:"
|
@echo " platform-install - Complete installation wizard:"
|
||||||
@echo " - Validates .env configuration"
|
@echo " - Validates .env configuration"
|
||||||
@echo " - Checks Stripe, Email, Security settings"
|
@echo " - Checks Stripe, Email, Security settings"
|
||||||
@echo " - Runs database migrations"
|
@echo " - Runs database migrations"
|
||||||
@@ -630,7 +630,7 @@ help-db:
|
|||||||
@echo "TYPICAL FIRST-TIME SETUP (Development):"
|
@echo "TYPICAL FIRST-TIME SETUP (Development):"
|
||||||
@echo "──────────────────────────────────────────────────────────"
|
@echo "──────────────────────────────────────────────────────────"
|
||||||
@echo " 1. cp .env.example .env # Configure environment"
|
@echo " 1. cp .env.example .env # Configure environment"
|
||||||
@echo " 2. make install # Validates config + initializes platform"
|
@echo " 2. make platform-install # Validates config + initializes platform"
|
||||||
@echo " 3. make seed-demo # Add demo data (optional)"
|
@echo " 3. make seed-demo # Add demo data (optional)"
|
||||||
@echo " 4. make dev # Start development server"
|
@echo " 4. make dev # Start development server"
|
||||||
@echo ""
|
@echo ""
|
||||||
@@ -642,5 +642,5 @@ help-db:
|
|||||||
@echo " - STRIPE_SECRET_KEY, STRIPE_PUBLISHABLE_KEY"
|
@echo " - STRIPE_SECRET_KEY, STRIPE_PUBLISHABLE_KEY"
|
||||||
@echo " - Email provider settings (SMTP/SendGrid/Mailgun/SES)"
|
@echo " - Email provider settings (SMTP/SendGrid/Mailgun/SES)"
|
||||||
@echo " - ADMIN_PASSWORD (strong password)"
|
@echo " - ADMIN_PASSWORD (strong password)"
|
||||||
@echo " 2. make install # Validates + initializes"
|
@echo " 2. make platform-install # Validates + initializes"
|
||||||
@echo " 3. DO NOT run seed-demo in production!"
|
@echo " 3. DO NOT run seed-demo in production!"
|
||||||
@@ -4,6 +4,77 @@
|
|||||||
|
|
||||||
This document defines the harmonized architecture for all background tasks in the application. Background tasks are long-running operations that execute asynchronously, allowing users to continue browsing while the task completes.
|
This document defines the harmonized architecture for all background tasks in the application. Background tasks are long-running operations that execute asynchronously, allowing users to continue browsing while the task completes.
|
||||||
|
|
||||||
|
## Task Queue Infrastructure
|
||||||
|
|
||||||
|
Wizamart uses **Celery with Redis** for production-grade background task processing:
|
||||||
|
|
||||||
|
| Component | Purpose | Port |
|
||||||
|
|-----------|---------|------|
|
||||||
|
| **Celery Worker** | Executes background tasks | - |
|
||||||
|
| **Celery Beat** | Scheduled task scheduler | - |
|
||||||
|
| **Redis** | Message broker & result backend | 6379 |
|
||||||
|
| **Flower** | Web-based task monitoring | 5555 |
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Environment variables (.env)
|
||||||
|
REDIS_URL=redis://localhost:6379/0
|
||||||
|
USE_CELERY=true # false = use FastAPI BackgroundTasks
|
||||||
|
FLOWER_URL=http://localhost:5555
|
||||||
|
FLOWER_PASSWORD=changeme
|
||||||
|
```
|
||||||
|
|
||||||
|
### Running Celery
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start Redis
|
||||||
|
docker compose up -d redis
|
||||||
|
|
||||||
|
# Start Celery worker (processes tasks)
|
||||||
|
make celery-worker
|
||||||
|
|
||||||
|
# Start Celery beat (scheduled tasks)
|
||||||
|
make celery-beat
|
||||||
|
|
||||||
|
# Start both together (development)
|
||||||
|
make celery-dev
|
||||||
|
|
||||||
|
# Start Flower monitoring
|
||||||
|
make flower
|
||||||
|
```
|
||||||
|
|
||||||
|
### Feature Flag: USE_CELERY
|
||||||
|
|
||||||
|
The system supports gradual migration via the `USE_CELERY` feature flag:
|
||||||
|
|
||||||
|
- **`USE_CELERY=false` (default)**: Uses FastAPI's `BackgroundTasks` for immediate execution without Redis dependency. Suitable for development.
|
||||||
|
- **`USE_CELERY=true`**: Routes tasks through Celery for persistent queuing, retries, and scheduling. Required for production.
|
||||||
|
|
||||||
|
### Task Routing
|
||||||
|
|
||||||
|
Tasks are routed to queues based on their characteristics:
|
||||||
|
|
||||||
|
| Queue | Tasks | Characteristics |
|
||||||
|
|-------|-------|-----------------|
|
||||||
|
| `default` | Product exports | Fast, non-blocking |
|
||||||
|
| `long_running` | Imports, code quality scans, tests | May take 10+ minutes |
|
||||||
|
| `scheduled` | Subscription maintenance | Triggered by Celery Beat |
|
||||||
|
|
||||||
|
### Celery Task ID Tracking
|
||||||
|
|
||||||
|
All background job models include a `celery_task_id` field for cross-referencing with Flower:
|
||||||
|
|
||||||
|
```python
|
||||||
|
class MarketplaceImportJob(Base):
|
||||||
|
celery_task_id = Column(String(255), nullable=True)
|
||||||
|
```
|
||||||
|
|
||||||
|
This enables:
|
||||||
|
- Deep linking to Flower for task details
|
||||||
|
- Task cancellation via Flower API
|
||||||
|
- Correlation between database records and Celery events
|
||||||
|
|
||||||
## Current State Analysis
|
## Current State Analysis
|
||||||
|
|
||||||
| Task Type | Database Model | Status Values | Tracked in BG Tasks Page |
|
| Task Type | Database Model | Status Values | Tracked in BG Tasks Page |
|
||||||
|
|||||||
@@ -6,8 +6,8 @@ This guide will help you set up the Wizamart Platform for development or product
|
|||||||
|
|
||||||
Before you begin, ensure you have the following installed:
|
Before you begin, ensure you have the following installed:
|
||||||
|
|
||||||
- **Python 3.10 or higher**
|
- **Python 3.11 or higher**
|
||||||
- **Docker and Docker Compose** (required for database)
|
- **Docker and Docker Compose** (required for database and Redis)
|
||||||
- **Git**
|
- **Git**
|
||||||
- **PostgreSQL client tools** (optional, for debugging)
|
- **PostgreSQL client tools** (optional, for debugging)
|
||||||
|
|
||||||
@@ -97,27 +97,53 @@ docker run --name wizamart-postgres \
|
|||||||
-d postgres:15
|
-d postgres:15
|
||||||
```
|
```
|
||||||
|
|
||||||
### 5. Initialize Database
|
### 5. Initialize Platform
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Run database migrations
|
# Option A: Full installation wizard (recommended)
|
||||||
python -m alembic upgrade head
|
make platform-install
|
||||||
|
|
||||||
# Create initial admin user (optional)
|
# Option B: Manual steps
|
||||||
python scripts/create_admin.py
|
make migrate-up # Run database migrations
|
||||||
|
make init-prod # Initialize admin, CMS, email templates
|
||||||
```
|
```
|
||||||
|
|
||||||
### 6. Run the Application
|
### 6. Start Background Task Queue (Optional)
|
||||||
|
|
||||||
|
For production or to test background tasks:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Start Redis (required for Celery)
|
||||||
|
docker compose up -d redis
|
||||||
|
|
||||||
|
# Start Celery worker (in separate terminal)
|
||||||
|
make celery-worker
|
||||||
|
|
||||||
|
# Start Celery beat scheduler (in separate terminal, for scheduled tasks)
|
||||||
|
make celery-beat
|
||||||
|
|
||||||
|
# Or start both together for development
|
||||||
|
make celery-dev
|
||||||
|
```
|
||||||
|
|
||||||
|
!!! tip "Development without Celery"
|
||||||
|
Set `USE_CELERY=false` in `.env` to use FastAPI BackgroundTasks instead.
|
||||||
|
This is the default and doesn't require Redis.
|
||||||
|
|
||||||
|
### 7. Run the Application
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Start development server
|
# Start development server
|
||||||
uvicorn main:app --reload --host 0.0.0.0 --port 8000 # or make dev
|
make dev
|
||||||
```
|
```
|
||||||
|
|
||||||
The application will be available at:
|
The application will be available at:
|
||||||
|
|
||||||
- **API**: http://localhost:8000
|
- **API**: http://localhost:8000
|
||||||
- **Interactive API Docs**: http://localhost:8000/docs
|
- **Interactive API Docs**: http://localhost:8000/docs
|
||||||
- **Alternative API Docs**: http://localhost:8000/redoc
|
- **Alternative API Docs**: http://localhost:8000/redoc
|
||||||
|
- **Admin Panel**: http://localhost:8000/admin
|
||||||
|
- **Flower** (if Celery enabled): http://localhost:5555
|
||||||
|
|
||||||
## Docker Setup
|
## Docker Setup
|
||||||
|
|
||||||
@@ -192,15 +218,31 @@ pytest -m integration
|
|||||||
|
|
||||||
## Environment Variables
|
## Environment Variables
|
||||||
|
|
||||||
|
### Core Settings
|
||||||
|
|
||||||
| Variable | Description | Default | Required |
|
| Variable | Description | Default | Required |
|
||||||
|----------|-------------|---------|----------|
|
|----------|-------------|---------|----------|
|
||||||
| `DATABASE_URL` | PostgreSQL connection string | - | ✅ |
|
| `DATABASE_URL` | PostgreSQL connection string | - | ✅ |
|
||||||
| `SECRET_KEY` | Application secret key | - | ✅ |
|
|
||||||
| `JWT_SECRET_KEY` | JWT token secret | - | ✅ |
|
| `JWT_SECRET_KEY` | JWT token secret | - | ✅ |
|
||||||
| `ENVIRONMENT` | Environment (dev/staging/prod) | `development` | ❌ |
|
|
||||||
| `DEBUG` | Enable debug mode | `False` | ❌ |
|
| `DEBUG` | Enable debug mode | `False` | ❌ |
|
||||||
| `LOG_LEVEL` | Logging level | `INFO` | ❌ |
|
| `LOG_LEVEL` | Logging level | `INFO` | ❌ |
|
||||||
| `CORS_ORIGINS` | Allowed CORS origins | `["*"]` | ❌ |
|
|
||||||
|
### Celery / Redis
|
||||||
|
|
||||||
|
| Variable | Description | Default | Required |
|
||||||
|
|----------|-------------|---------|----------|
|
||||||
|
| `REDIS_URL` | Redis connection string | `redis://localhost:6379/0` | ❌ |
|
||||||
|
| `USE_CELERY` | Enable Celery task queue | `false` | ❌ |
|
||||||
|
| `FLOWER_URL` | Flower dashboard URL | `http://localhost:5555` | ❌ |
|
||||||
|
| `FLOWER_PASSWORD` | Flower authentication password | `changeme` | ❌ |
|
||||||
|
|
||||||
|
### Stripe Billing
|
||||||
|
|
||||||
|
| Variable | Description | Default | Required |
|
||||||
|
|----------|-------------|---------|----------|
|
||||||
|
| `STRIPE_SECRET_KEY` | Stripe secret key | - | ❌ |
|
||||||
|
| `STRIPE_PUBLISHABLE_KEY` | Stripe publishable key | - | ❌ |
|
||||||
|
| `STRIPE_WEBHOOK_SECRET` | Stripe webhook secret | - | ❌ |
|
||||||
|
|
||||||
## Troubleshooting
|
## Troubleshooting
|
||||||
|
|
||||||
|
|||||||
@@ -354,26 +354,37 @@ Quick reference for common development tasks:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Setup
|
# Setup
|
||||||
make install-all # Install all dependencies
|
make install # Install production dependencies
|
||||||
|
make install-all # Install all dependencies (dev + test + docs)
|
||||||
|
make platform-install # First-time setup wizard (validates config)
|
||||||
make setup # Complete setup (install + migrate + seed)
|
make setup # Complete setup (install + migrate + seed)
|
||||||
|
|
||||||
# Development
|
# Development
|
||||||
make dev # Start development server
|
make dev # Start development server (port 8000)
|
||||||
make docs-serve # Start documentation server
|
make docs-serve # Start documentation server (port 9991)
|
||||||
|
|
||||||
# Database
|
# Database
|
||||||
make migrate-up # Run migrations
|
make migrate-up # Run migrations
|
||||||
|
make migrate-create message="description" # Create new migration
|
||||||
make seed-demo # Seed demo data
|
make seed-demo # Seed demo data
|
||||||
make db-setup # Complete database setup
|
make db-setup # Complete database setup
|
||||||
|
|
||||||
|
# Background Tasks (Celery)
|
||||||
|
make celery-worker # Start Celery worker
|
||||||
|
make celery-beat # Start Celery scheduler
|
||||||
|
make celery-dev # Start worker + beat together (dev)
|
||||||
|
make flower # Start Flower monitoring (port 5555)
|
||||||
|
|
||||||
# Testing
|
# Testing
|
||||||
make test # Run all tests
|
make test # Run all tests
|
||||||
|
make test-unit # Run unit tests only
|
||||||
make test-coverage # Run with coverage report
|
make test-coverage # Run with coverage report
|
||||||
|
|
||||||
# Code Quality
|
# Code Quality
|
||||||
make format # Format code (black + isort)
|
make format # Format code (ruff)
|
||||||
make lint # Run linters
|
make lint # Run linters (ruff + mypy)
|
||||||
make check # Format + lint
|
make arch-check # Validate architecture patterns
|
||||||
|
make check # Format + lint + verify imports
|
||||||
|
|
||||||
# Documentation
|
# Documentation
|
||||||
make docs-build # Build documentation
|
make docs-build # Build documentation
|
||||||
|
|||||||
Reference in New Issue
Block a user