docs: update documentation with Celery/Redis and Makefile changes

- Update Makefile help sections with correct target names (platform-install)
- Add Celery infrastructure section to background-tasks.md
- Update installation.md with Celery setup instructions and environment vars
- Update index.md with comprehensive development commands including Celery
- Add Flower monitoring and task queue commands to docs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
2026-01-11 18:56:46 +01:00
parent 63fd891f36
commit 0d204f39a8
4 changed files with 146 additions and 22 deletions

View File

@@ -520,7 +520,7 @@ help:
@echo " migrate-up - Apply pending migrations" @echo " migrate-up - Apply pending migrations"
@echo " migrate-down - Rollback last migration" @echo " migrate-down - Rollback last migration"
@echo " migrate-status - Show migration status" @echo " migrate-status - Show migration status"
@echo " install - First-time setup (validates config + migrate + init)" @echo " platform-install - First-time setup (validates config + migrate + init)"
@echo " init-prod - Initialize platform (admin, CMS, pages, emails)" @echo " init-prod - Initialize platform (admin, CMS, pages, emails)"
@echo " seed-demo - Seed demo data (3 companies + vendors)" @echo " seed-demo - Seed demo data (3 companies + vendors)"
@echo " seed-demo-minimal - Seed minimal demo (1 company + vendor)" @echo " seed-demo-minimal - Seed minimal demo (1 company + vendor)"
@@ -594,7 +594,7 @@ help-db:
@echo "" @echo ""
@echo "FIRST-TIME INSTALLATION:" @echo "FIRST-TIME INSTALLATION:"
@echo "──────────────────────────────────────────────────────────" @echo "──────────────────────────────────────────────────────────"
@echo " install - Complete installation wizard:" @echo " platform-install - Complete installation wizard:"
@echo " - Validates .env configuration" @echo " - Validates .env configuration"
@echo " - Checks Stripe, Email, Security settings" @echo " - Checks Stripe, Email, Security settings"
@echo " - Runs database migrations" @echo " - Runs database migrations"
@@ -630,7 +630,7 @@ help-db:
@echo "TYPICAL FIRST-TIME SETUP (Development):" @echo "TYPICAL FIRST-TIME SETUP (Development):"
@echo "──────────────────────────────────────────────────────────" @echo "──────────────────────────────────────────────────────────"
@echo " 1. cp .env.example .env # Configure environment" @echo " 1. cp .env.example .env # Configure environment"
@echo " 2. make install # Validates config + initializes platform" @echo " 2. make platform-install # Validates config + initializes platform"
@echo " 3. make seed-demo # Add demo data (optional)" @echo " 3. make seed-demo # Add demo data (optional)"
@echo " 4. make dev # Start development server" @echo " 4. make dev # Start development server"
@echo "" @echo ""
@@ -642,5 +642,5 @@ help-db:
@echo " - STRIPE_SECRET_KEY, STRIPE_PUBLISHABLE_KEY" @echo " - STRIPE_SECRET_KEY, STRIPE_PUBLISHABLE_KEY"
@echo " - Email provider settings (SMTP/SendGrid/Mailgun/SES)" @echo " - Email provider settings (SMTP/SendGrid/Mailgun/SES)"
@echo " - ADMIN_PASSWORD (strong password)" @echo " - ADMIN_PASSWORD (strong password)"
@echo " 2. make install # Validates + initializes" @echo " 2. make platform-install # Validates + initializes"
@echo " 3. DO NOT run seed-demo in production!" @echo " 3. DO NOT run seed-demo in production!"

View File

@@ -4,6 +4,77 @@
This document defines the harmonized architecture for all background tasks in the application. Background tasks are long-running operations that execute asynchronously, allowing users to continue browsing while the task completes. This document defines the harmonized architecture for all background tasks in the application. Background tasks are long-running operations that execute asynchronously, allowing users to continue browsing while the task completes.
## Task Queue Infrastructure
Wizamart uses **Celery with Redis** for production-grade background task processing:
| Component | Purpose | Port |
|-----------|---------|------|
| **Celery Worker** | Executes background tasks | - |
| **Celery Beat** | Scheduled task scheduler | - |
| **Redis** | Message broker & result backend | 6379 |
| **Flower** | Web-based task monitoring | 5555 |
### Configuration
```bash
# Environment variables (.env)
REDIS_URL=redis://localhost:6379/0
USE_CELERY=true # false = use FastAPI BackgroundTasks
FLOWER_URL=http://localhost:5555
FLOWER_PASSWORD=changeme
```
### Running Celery
```bash
# Start Redis
docker compose up -d redis
# Start Celery worker (processes tasks)
make celery-worker
# Start Celery beat (scheduled tasks)
make celery-beat
# Start both together (development)
make celery-dev
# Start Flower monitoring
make flower
```
### Feature Flag: USE_CELERY
The system supports gradual migration via the `USE_CELERY` feature flag:
- **`USE_CELERY=false` (default)**: Uses FastAPI's `BackgroundTasks` for immediate execution without Redis dependency. Suitable for development.
- **`USE_CELERY=true`**: Routes tasks through Celery for persistent queuing, retries, and scheduling. Required for production.
### Task Routing
Tasks are routed to queues based on their characteristics:
| Queue | Tasks | Characteristics |
|-------|-------|-----------------|
| `default` | Product exports | Fast, non-blocking |
| `long_running` | Imports, code quality scans, tests | May take 10+ minutes |
| `scheduled` | Subscription maintenance | Triggered by Celery Beat |
### Celery Task ID Tracking
All background job models include a `celery_task_id` field for cross-referencing with Flower:
```python
class MarketplaceImportJob(Base):
celery_task_id = Column(String(255), nullable=True)
```
This enables:
- Deep linking to Flower for task details
- Task cancellation via Flower API
- Correlation between database records and Celery events
## Current State Analysis ## Current State Analysis
| Task Type | Database Model | Status Values | Tracked in BG Tasks Page | | Task Type | Database Model | Status Values | Tracked in BG Tasks Page |

View File

@@ -6,8 +6,8 @@ This guide will help you set up the Wizamart Platform for development or product
Before you begin, ensure you have the following installed: Before you begin, ensure you have the following installed:
- **Python 3.10 or higher** - **Python 3.11 or higher**
- **Docker and Docker Compose** (required for database) - **Docker and Docker Compose** (required for database and Redis)
- **Git** - **Git**
- **PostgreSQL client tools** (optional, for debugging) - **PostgreSQL client tools** (optional, for debugging)
@@ -97,27 +97,53 @@ docker run --name wizamart-postgres \
-d postgres:15 -d postgres:15
``` ```
### 5. Initialize Database ### 5. Initialize Platform
```bash ```bash
# Run database migrations # Option A: Full installation wizard (recommended)
python -m alembic upgrade head make platform-install
# Create initial admin user (optional) # Option B: Manual steps
python scripts/create_admin.py make migrate-up # Run database migrations
make init-prod # Initialize admin, CMS, email templates
``` ```
### 6. Run the Application ### 6. Start Background Task Queue (Optional)
For production or to test background tasks:
```bash
# Start Redis (required for Celery)
docker compose up -d redis
# Start Celery worker (in separate terminal)
make celery-worker
# Start Celery beat scheduler (in separate terminal, for scheduled tasks)
make celery-beat
# Or start both together for development
make celery-dev
```
!!! tip "Development without Celery"
Set `USE_CELERY=false` in `.env` to use FastAPI BackgroundTasks instead.
This is the default and doesn't require Redis.
### 7. Run the Application
```bash ```bash
# Start development server # Start development server
uvicorn main:app --reload --host 0.0.0.0 --port 8000 # or make dev make dev
``` ```
The application will be available at: The application will be available at:
- **API**: http://localhost:8000 - **API**: http://localhost:8000
- **Interactive API Docs**: http://localhost:8000/docs - **Interactive API Docs**: http://localhost:8000/docs
- **Alternative API Docs**: http://localhost:8000/redoc - **Alternative API Docs**: http://localhost:8000/redoc
- **Admin Panel**: http://localhost:8000/admin
- **Flower** (if Celery enabled): http://localhost:5555
## Docker Setup ## Docker Setup
@@ -192,15 +218,31 @@ pytest -m integration
## Environment Variables ## Environment Variables
### Core Settings
| Variable | Description | Default | Required | | Variable | Description | Default | Required |
|----------|-------------|---------|----------| |----------|-------------|---------|----------|
| `DATABASE_URL` | PostgreSQL connection string | - | ✅ | | `DATABASE_URL` | PostgreSQL connection string | - | ✅ |
| `SECRET_KEY` | Application secret key | - | ✅ |
| `JWT_SECRET_KEY` | JWT token secret | - | ✅ | | `JWT_SECRET_KEY` | JWT token secret | - | ✅ |
| `ENVIRONMENT` | Environment (dev/staging/prod) | `development` | ❌ |
| `DEBUG` | Enable debug mode | `False` | ❌ | | `DEBUG` | Enable debug mode | `False` | ❌ |
| `LOG_LEVEL` | Logging level | `INFO` | ❌ | | `LOG_LEVEL` | Logging level | `INFO` | ❌ |
| `CORS_ORIGINS` | Allowed CORS origins | `["*"]` | ❌ |
### Celery / Redis
| Variable | Description | Default | Required |
|----------|-------------|---------|----------|
| `REDIS_URL` | Redis connection string | `redis://localhost:6379/0` | ❌ |
| `USE_CELERY` | Enable Celery task queue | `false` | ❌ |
| `FLOWER_URL` | Flower dashboard URL | `http://localhost:5555` | ❌ |
| `FLOWER_PASSWORD` | Flower authentication password | `changeme` | ❌ |
### Stripe Billing
| Variable | Description | Default | Required |
|----------|-------------|---------|----------|
| `STRIPE_SECRET_KEY` | Stripe secret key | - | ❌ |
| `STRIPE_PUBLISHABLE_KEY` | Stripe publishable key | - | ❌ |
| `STRIPE_WEBHOOK_SECRET` | Stripe webhook secret | - | ❌ |
## Troubleshooting ## Troubleshooting

View File

@@ -354,26 +354,37 @@ Quick reference for common development tasks:
```bash ```bash
# Setup # Setup
make install-all # Install all dependencies make install # Install production dependencies
make install-all # Install all dependencies (dev + test + docs)
make platform-install # First-time setup wizard (validates config)
make setup # Complete setup (install + migrate + seed) make setup # Complete setup (install + migrate + seed)
# Development # Development
make dev # Start development server make dev # Start development server (port 8000)
make docs-serve # Start documentation server make docs-serve # Start documentation server (port 9991)
# Database # Database
make migrate-up # Run migrations make migrate-up # Run migrations
make migrate-create message="description" # Create new migration
make seed-demo # Seed demo data make seed-demo # Seed demo data
make db-setup # Complete database setup make db-setup # Complete database setup
# Background Tasks (Celery)
make celery-worker # Start Celery worker
make celery-beat # Start Celery scheduler
make celery-dev # Start worker + beat together (dev)
make flower # Start Flower monitoring (port 5555)
# Testing # Testing
make test # Run all tests make test # Run all tests
make test-unit # Run unit tests only
make test-coverage # Run with coverage report make test-coverage # Run with coverage report
# Code Quality # Code Quality
make format # Format code (black + isort) make format # Format code (ruff)
make lint # Run linters make lint # Run linters (ruff + mypy)
make check # Format + lint make arch-check # Validate architecture patterns
make check # Format + lint + verify imports
# Documentation # Documentation
make docs-build # Build documentation make docs-build # Build documentation