diff --git a/Makefile b/Makefile index b9575053..0de560cb 100644 --- a/Makefile +++ b/Makefile @@ -520,7 +520,7 @@ help: @echo " migrate-up - Apply pending migrations" @echo " migrate-down - Rollback last migration" @echo " migrate-status - Show migration status" - @echo " install - First-time setup (validates config + migrate + init)" + @echo " platform-install - First-time setup (validates config + migrate + init)" @echo " init-prod - Initialize platform (admin, CMS, pages, emails)" @echo " seed-demo - Seed demo data (3 companies + vendors)" @echo " seed-demo-minimal - Seed minimal demo (1 company + vendor)" @@ -594,7 +594,7 @@ help-db: @echo "" @echo "FIRST-TIME INSTALLATION:" @echo "──────────────────────────────────────────────────────────" - @echo " install - Complete installation wizard:" + @echo " platform-install - Complete installation wizard:" @echo " - Validates .env configuration" @echo " - Checks Stripe, Email, Security settings" @echo " - Runs database migrations" @@ -630,7 +630,7 @@ help-db: @echo "TYPICAL FIRST-TIME SETUP (Development):" @echo "──────────────────────────────────────────────────────────" @echo " 1. cp .env.example .env # Configure environment" - @echo " 2. make install # Validates config + initializes platform" + @echo " 2. make platform-install # Validates config + initializes platform" @echo " 3. make seed-demo # Add demo data (optional)" @echo " 4. make dev # Start development server" @echo "" @@ -642,5 +642,5 @@ help-db: @echo " - STRIPE_SECRET_KEY, STRIPE_PUBLISHABLE_KEY" @echo " - Email provider settings (SMTP/SendGrid/Mailgun/SES)" @echo " - ADMIN_PASSWORD (strong password)" - @echo " 2. make install # Validates + initializes" + @echo " 2. make platform-install # Validates + initializes" @echo " 3. DO NOT run seed-demo in production!" \ No newline at end of file diff --git a/docs/architecture/background-tasks.md b/docs/architecture/background-tasks.md index f9aca384..a45e64a4 100644 --- a/docs/architecture/background-tasks.md +++ b/docs/architecture/background-tasks.md @@ -4,6 +4,77 @@ This document defines the harmonized architecture for all background tasks in the application. Background tasks are long-running operations that execute asynchronously, allowing users to continue browsing while the task completes. +## Task Queue Infrastructure + +Wizamart uses **Celery with Redis** for production-grade background task processing: + +| Component | Purpose | Port | +|-----------|---------|------| +| **Celery Worker** | Executes background tasks | - | +| **Celery Beat** | Scheduled task scheduler | - | +| **Redis** | Message broker & result backend | 6379 | +| **Flower** | Web-based task monitoring | 5555 | + +### Configuration + +```bash +# Environment variables (.env) +REDIS_URL=redis://localhost:6379/0 +USE_CELERY=true # false = use FastAPI BackgroundTasks +FLOWER_URL=http://localhost:5555 +FLOWER_PASSWORD=changeme +``` + +### Running Celery + +```bash +# Start Redis +docker compose up -d redis + +# Start Celery worker (processes tasks) +make celery-worker + +# Start Celery beat (scheduled tasks) +make celery-beat + +# Start both together (development) +make celery-dev + +# Start Flower monitoring +make flower +``` + +### Feature Flag: USE_CELERY + +The system supports gradual migration via the `USE_CELERY` feature flag: + +- **`USE_CELERY=false` (default)**: Uses FastAPI's `BackgroundTasks` for immediate execution without Redis dependency. Suitable for development. +- **`USE_CELERY=true`**: Routes tasks through Celery for persistent queuing, retries, and scheduling. Required for production. + +### Task Routing + +Tasks are routed to queues based on their characteristics: + +| Queue | Tasks | Characteristics | +|-------|-------|-----------------| +| `default` | Product exports | Fast, non-blocking | +| `long_running` | Imports, code quality scans, tests | May take 10+ minutes | +| `scheduled` | Subscription maintenance | Triggered by Celery Beat | + +### Celery Task ID Tracking + +All background job models include a `celery_task_id` field for cross-referencing with Flower: + +```python +class MarketplaceImportJob(Base): + celery_task_id = Column(String(255), nullable=True) +``` + +This enables: +- Deep linking to Flower for task details +- Task cancellation via Flower API +- Correlation between database records and Celery events + ## Current State Analysis | Task Type | Database Model | Status Values | Tracked in BG Tasks Page | diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md index 226a3ba5..934d0374 100644 --- a/docs/getting-started/installation.md +++ b/docs/getting-started/installation.md @@ -6,8 +6,8 @@ This guide will help you set up the Wizamart Platform for development or product Before you begin, ensure you have the following installed: -- **Python 3.10 or higher** -- **Docker and Docker Compose** (required for database) +- **Python 3.11 or higher** +- **Docker and Docker Compose** (required for database and Redis) - **Git** - **PostgreSQL client tools** (optional, for debugging) @@ -97,27 +97,53 @@ docker run --name wizamart-postgres \ -d postgres:15 ``` -### 5. Initialize Database +### 5. Initialize Platform ```bash -# Run database migrations -python -m alembic upgrade head +# Option A: Full installation wizard (recommended) +make platform-install -# Create initial admin user (optional) -python scripts/create_admin.py +# Option B: Manual steps +make migrate-up # Run database migrations +make init-prod # Initialize admin, CMS, email templates ``` -### 6. Run the Application +### 6. Start Background Task Queue (Optional) + +For production or to test background tasks: + +```bash +# Start Redis (required for Celery) +docker compose up -d redis + +# Start Celery worker (in separate terminal) +make celery-worker + +# Start Celery beat scheduler (in separate terminal, for scheduled tasks) +make celery-beat + +# Or start both together for development +make celery-dev +``` + +!!! tip "Development without Celery" + Set `USE_CELERY=false` in `.env` to use FastAPI BackgroundTasks instead. + This is the default and doesn't require Redis. + +### 7. Run the Application ```bash # Start development server -uvicorn main:app --reload --host 0.0.0.0 --port 8000 # or make dev +make dev ``` The application will be available at: + - **API**: http://localhost:8000 - **Interactive API Docs**: http://localhost:8000/docs - **Alternative API Docs**: http://localhost:8000/redoc +- **Admin Panel**: http://localhost:8000/admin +- **Flower** (if Celery enabled): http://localhost:5555 ## Docker Setup @@ -192,15 +218,31 @@ pytest -m integration ## Environment Variables +### Core Settings + | Variable | Description | Default | Required | |----------|-------------|---------|----------| | `DATABASE_URL` | PostgreSQL connection string | - | ✅ | -| `SECRET_KEY` | Application secret key | - | ✅ | | `JWT_SECRET_KEY` | JWT token secret | - | ✅ | -| `ENVIRONMENT` | Environment (dev/staging/prod) | `development` | ❌ | | `DEBUG` | Enable debug mode | `False` | ❌ | | `LOG_LEVEL` | Logging level | `INFO` | ❌ | -| `CORS_ORIGINS` | Allowed CORS origins | `["*"]` | ❌ | + +### Celery / Redis + +| Variable | Description | Default | Required | +|----------|-------------|---------|----------| +| `REDIS_URL` | Redis connection string | `redis://localhost:6379/0` | ❌ | +| `USE_CELERY` | Enable Celery task queue | `false` | ❌ | +| `FLOWER_URL` | Flower dashboard URL | `http://localhost:5555` | ❌ | +| `FLOWER_PASSWORD` | Flower authentication password | `changeme` | ❌ | + +### Stripe Billing + +| Variable | Description | Default | Required | +|----------|-------------|---------|----------| +| `STRIPE_SECRET_KEY` | Stripe secret key | - | ❌ | +| `STRIPE_PUBLISHABLE_KEY` | Stripe publishable key | - | ❌ | +| `STRIPE_WEBHOOK_SECRET` | Stripe webhook secret | - | ❌ | ## Troubleshooting diff --git a/docs/index.md b/docs/index.md index 4e9be7de..47aeaaae 100644 --- a/docs/index.md +++ b/docs/index.md @@ -354,26 +354,37 @@ Quick reference for common development tasks: ```bash # Setup -make install-all # Install all dependencies +make install # Install production dependencies +make install-all # Install all dependencies (dev + test + docs) +make platform-install # First-time setup wizard (validates config) make setup # Complete setup (install + migrate + seed) # Development -make dev # Start development server -make docs-serve # Start documentation server +make dev # Start development server (port 8000) +make docs-serve # Start documentation server (port 9991) # Database make migrate-up # Run migrations +make migrate-create message="description" # Create new migration make seed-demo # Seed demo data make db-setup # Complete database setup +# Background Tasks (Celery) +make celery-worker # Start Celery worker +make celery-beat # Start Celery scheduler +make celery-dev # Start worker + beat together (dev) +make flower # Start Flower monitoring (port 5555) + # Testing make test # Run all tests +make test-unit # Run unit tests only make test-coverage # Run with coverage report # Code Quality -make format # Format code (black + isort) -make lint # Run linters -make check # Format + lint +make format # Format code (ruff) +make lint # Run linters (ruff + mypy) +make arch-check # Validate architecture patterns +make check # Format + lint + verify imports # Documentation make docs-build # Build documentation