From 7dbdbd4c7e1efcf2f1c2dd45a707e47c12dec936 Mon Sep 17 00:00:00 2001 From: Samir Boulahtit Date: Tue, 27 Jan 2026 22:41:19 +0100 Subject: [PATCH] docs: add observability, creating modules guide, and unified migration plan - Add observability framework documentation (health checks, metrics, Sentry) - Add developer guide for creating modules - Add comprehensive module migration plan with Celery task integration - Update architecture overview with module system and observability sections - Update module-system.md with links to new docs Co-Authored-By: Claude Opus 4.5 --- docs/architecture/module-system.md | 2 + docs/architecture/observability.md | 429 +++++++++++++++++ docs/architecture/overview.md | 78 +++- docs/development/creating-modules.md | 483 +++++++++++++++++++ docs/proposals/module-migration-plan.md | 596 ++++++++++++++++++++++++ mkdocs.yml | 3 + 6 files changed, 1583 insertions(+), 8 deletions(-) create mode 100644 docs/architecture/observability.md create mode 100644 docs/development/creating-modules.md create mode 100644 docs/proposals/module-migration-plan.md diff --git a/docs/architecture/module-system.md b/docs/architecture/module-system.md index d35bd76f..f2afd8a6 100644 --- a/docs/architecture/module-system.md +++ b/docs/architecture/module-system.md @@ -417,5 +417,7 @@ register_module_health_checks() ## Related Documentation - [Menu Management](menu-management.md) - Sidebar and menu configuration +- [Creating Modules](../development/creating-modules.md) - Developer guide for building modules +- [Observability](observability.md) - Health checks and module health integration - [Multi-Tenant System](multi-tenant.md) - Platform isolation - [Feature Gating](../implementation/feature-gating-system.md) - Tier-based access diff --git a/docs/architecture/observability.md b/docs/architecture/observability.md new file mode 100644 index 00000000..de521299 --- /dev/null +++ b/docs/architecture/observability.md @@ -0,0 +1,429 @@ +# Observability Framework + +The Wizamart platform includes a comprehensive observability framework for monitoring application health, collecting metrics, and tracking errors. This is part of the Framework Layer - infrastructure that modules depend on. + +## Overview + +``` +┌─────────────────────────────────────────────────────────────────────────┐ +│ OBSERVABILITY FRAMEWORK │ +│ app/core/observability.py │ +│ │ +│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │ +│ │ Health Checks │ │ Prometheus │ │ Sentry │ │ +│ │ Registry │ │ Metrics │ │ Integration │ │ +│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │ +│ │ │ │ │ +│ ▼ ▼ ▼ │ +│ ┌─────────────────────────────────────────────────────────────────┐ │ +│ │ API Endpoints │ │ +│ │ /health │ /health/live │ /health/ready │ /metrics │ │ +│ └─────────────────────────────────────────────────────────────────┘ │ +└─────────────────────────────────────────────────────────────────────────┘ + │ + ▼ + ┌───────────────────────────────┐ + │ External Tools │ + │ Flower │ Grafana │ Prometheus│ + └───────────────────────────────┘ +``` + +## Health Checks + +### Health Check Registry + +Components register health checks that are aggregated into a single endpoint. + +```python +from app.core.observability import ( + health_registry, + HealthCheckResult, + HealthStatus, +) + +# Register using decorator +@health_registry.register("database") +def check_database() -> HealthCheckResult: + try: + db.execute("SELECT 1") + return HealthCheckResult( + name="database", + status=HealthStatus.HEALTHY, + message="Database connection OK" + ) + except Exception as e: + return HealthCheckResult( + name="database", + status=HealthStatus.UNHEALTHY, + message=str(e) + ) + +# Or register directly +health_registry.register_check("redis", check_redis_connection) +``` + +### Health Status Levels + +| Status | Description | HTTP Response | +|--------|-------------|---------------| +| `HEALTHY` | All systems operational | 200 | +| `DEGRADED` | Partial functionality available | 200 | +| `UNHEALTHY` | Critical failure | 200 (check response body) | + +### HealthCheckResult Fields + +| Field | Type | Description | +|-------|------|-------------| +| `name` | str | Check identifier | +| `status` | HealthStatus | Health status level | +| `message` | str | Optional status message | +| `latency_ms` | float | Check execution time | +| `details` | dict | Additional diagnostic data | +| `checked_at` | datetime | Timestamp of check | + +## API Endpoints + +### GET /health + +Aggregated health check endpoint. Returns combined status from all registered checks. + +**Response:** +```json +{ + "status": "healthy", + "timestamp": "2026-01-27T10:30:00Z", + "checks": [ + { + "name": "database", + "status": "healthy", + "message": "Connection OK", + "latency_ms": 2.5, + "details": {} + }, + { + "name": "module:billing", + "status": "healthy", + "message": "", + "latency_ms": 0.1, + "details": {} + } + ] +} +``` + +**Overall Status Logic:** +- If any check is `UNHEALTHY` → overall is `UNHEALTHY` +- If any check is `DEGRADED` and none `UNHEALTHY` → overall is `DEGRADED` +- Otherwise → `HEALTHY` + +### GET /health/live + +Kubernetes liveness probe. Returns 200 if application is running. + +**Response:** +```json +{"status": "alive"} +``` + +### GET /health/ready + +Kubernetes readiness probe. Returns ready status based on health checks. + +**Response:** +```json +{ + "status": "ready", + "health": "healthy" +} +``` + +Or if unhealthy: +```json +{ + "status": "not_ready", + "health": "unhealthy" +} +``` + +### GET /metrics + +Prometheus metrics endpoint. Returns metrics in Prometheus text format. + +**Response:** +``` +# HELP http_requests_total Total HTTP requests +# TYPE http_requests_total counter +http_requests_total{method="GET",endpoint="/api/products",status="200"} 1234 +... +``` + +### GET /health/tools + +Returns URLs to external monitoring tools. + +**Response:** +```json +{ + "flower": "http://flower.example.com:5555", + "grafana": "http://grafana.example.com:3000", + "prometheus": null +} +``` + +## Prometheus Metrics + +### MetricsRegistry + +The metrics registry provides a wrapper around `prometheus_client` with fallback when the library isn't installed. + +```python +from app.core.observability import metrics_registry + +# Counter - tracks cumulative values +request_counter = metrics_registry.counter( + "http_requests_total", + "Total HTTP requests", + ["method", "endpoint", "status"] +) +request_counter.labels(method="GET", endpoint="/api/products", status="200").inc() + +# Histogram - tracks distributions (latency, sizes) +request_latency = metrics_registry.histogram( + "http_request_duration_seconds", + "HTTP request latency", + ["endpoint"], + buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0] +) +request_latency.labels(endpoint="/api/products").observe(0.045) + +# Gauge - tracks current values +active_connections = metrics_registry.gauge( + "active_connections", + "Number of active connections", + ["pool"] +) +active_connections.labels(pool="database").set(42) +``` + +### Enabling Metrics + +Metrics are disabled by default. Enable during initialization: + +```python +from app.core.observability import init_observability + +init_observability( + enable_metrics=True, + # ... other options +) +``` + +### Dummy Metrics + +When `prometheus_client` isn't installed or metrics are disabled, the registry returns dummy metrics that silently ignore all operations. This allows code to use metrics without checking if they're enabled. + +## Sentry Integration + +### Configuration + +```python +from app.core.observability import sentry, init_observability + +# Initialize via init_observability +init_observability( + sentry_dsn="https://key@sentry.io/project", + environment="production", +) + +# Or initialize directly +sentry.init( + dsn="https://key@sentry.io/project", + environment="production", + release="1.0.0" +) +``` + +### Capturing Errors + +```python +from app.core.observability import sentry + +try: + risky_operation() +except Exception as e: + event_id = sentry.capture_exception(e) + logger.error(f"Operation failed, Sentry event: {event_id}") + +# Capture messages +sentry.capture_message("User reached rate limit", level="warning") +``` + +### Without Sentry + +If `sentry_sdk` isn't installed or DSN isn't provided, all capture calls silently return `None`. + +## Module Health Checks + +Modules can provide health checks that are automatically registered. + +### Defining Module Health Check + +```python +# In module definition +from app.modules.base import ModuleDefinition + +def check_billing_health() -> dict: + """Check billing service health.""" + try: + # Check Stripe connection + stripe.Account.retrieve() + return {"status": "healthy", "message": "Stripe connected"} + except Exception as e: + return {"status": "unhealthy", "message": str(e)} + +billing_module = ModuleDefinition( + code="billing", + name="Billing", + health_check=check_billing_health, + # ... +) +``` + +### Registering Module Health Checks + +```python +from app.core.observability import register_module_health_checks + +# Call after modules are loaded (e.g., in app lifespan) +register_module_health_checks() +``` + +This registers health checks as `module:{code}` (e.g., `module:billing`). + +## External Tools + +### Flower (Celery Monitoring) + +Configure Flower URL to include in `/health/tools`: + +```python +init_observability( + flower_url="http://flower:5555", +) +``` + +### Grafana + +Configure Grafana URL: + +```python +init_observability( + grafana_url="http://grafana:3000", +) +``` + +## Initialization + +### Application Lifespan + +```python +# main.py +from contextlib import asynccontextmanager +from fastapi import FastAPI +from app.core.observability import ( + init_observability, + shutdown_observability, + health_router, + register_module_health_checks, +) + +@asynccontextmanager +async def lifespan(app: FastAPI): + # Startup + init_observability( + enable_metrics=True, + sentry_dsn=settings.SENTRY_DSN, + environment=settings.ENVIRONMENT, + flower_url=settings.FLOWER_URL, + grafana_url=settings.GRAFANA_URL, + ) + register_module_health_checks() + + yield + + # Shutdown + shutdown_observability() + +app = FastAPI(lifespan=lifespan) +app.include_router(health_router) +``` + +### Environment Variables + +| Variable | Description | Default | +|----------|-------------|---------| +| `SENTRY_DSN` | Sentry DSN for error tracking | None (disabled) | +| `ENVIRONMENT` | Environment name | "development" | +| `ENABLE_METRICS` | Enable Prometheus metrics | False | +| `FLOWER_URL` | Flower dashboard URL | None | +| `GRAFANA_URL` | Grafana dashboard URL | None | + +## Kubernetes Integration + +### Deployment Configuration + +```yaml +apiVersion: apps/v1 +kind: Deployment +spec: + template: + spec: + containers: + - name: app + livenessProbe: + httpGet: + path: /health/live + port: 8000 + initialDelaySeconds: 10 + periodSeconds: 30 + readinessProbe: + httpGet: + path: /health/ready + port: 8000 + initialDelaySeconds: 5 + periodSeconds: 10 +``` + +### Prometheus ServiceMonitor + +```yaml +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +spec: + endpoints: + - port: http + path: /metrics + interval: 15s +``` + +## Best Practices + +### Do + +- Register health checks for critical dependencies (database, cache, external APIs) +- Use appropriate metric types (counter for counts, histogram for latency) +- Include meaningful labels but avoid high cardinality +- Set up alerts based on health status changes + +### Don't + +- Create health checks that are slow or have side effects +- Add high-cardinality labels to metrics (e.g., user IDs) +- Ignore Sentry errors in production +- Skip readiness probes in Kubernetes deployments + +## Related Documentation + +- [Module System](module-system.md) - Module health check integration +- [Background Tasks](background-tasks.md) - Celery monitoring with Flower +- [Deployment](../deployment/index.md) - Production deployment with monitoring diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 4c24021e..d61170c4 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -76,7 +76,42 @@ Custom middleware handles: **See:** [Authentication & RBAC](auth-rbac.md) for details -### 4. Request Flow +### 4. Module System + +The platform uses a modular architecture with three-tier classification: + +``` +┌─────────────────────────────────────────────────────────────┐ +│ FRAMEWORK LAYER │ +│ (Infrastructure - always available, not modules) │ +│ Config │ Database │ Auth │ Permissions │ Observability │ +└─────────────────────────────────────────────────────────────┘ + │ + ▼ +┌─────────────────────────────────────────────────────────────┐ +│ MODULE LAYER │ +│ │ +│ CORE (Always Enabled) OPTIONAL (Per-Platform) │ +│ ├── core ├── payments │ +│ ├── tenancy ├── billing │ +│ ├── cms ├── inventory │ +│ └── customers ├── orders │ +│ ├── marketplace │ +│ INTERNAL (Admin-Only) ├── analytics │ +│ ├── dev-tools └── messaging │ +│ └── monitoring │ +└─────────────────────────────────────────────────────────────┘ +``` + +**Module Features:** +- Enable/disable modules per platform +- Module dependencies (billing requires payments) +- Health checks and lifecycle hooks +- Self-contained modules with own services, models, migrations + +**See:** [Module System](module-system.md) for complete documentation + +### 5. Request Flow ```mermaid graph TB @@ -252,7 +287,15 @@ project/ │ ├── api/ # API routes │ ├── routes/ # Page routes (HTML) │ ├── services/ # Business logic -│ ├── core/ # Core functionality +│ ├── core/ # Core functionality (config, db, observability) +│ ├── modules/ # Module definitions and self-contained modules +│ │ ├── base.py # ModuleDefinition class +│ │ ├── registry.py # Module registry (CORE, OPTIONAL, INTERNAL) +│ │ ├── service.py # Module enablement service +│ │ ├── events.py # Module event bus +│ │ ├── cms/ # Self-contained CMS module +│ │ ├── payments/ # Self-contained payments module +│ │ └── ... # Other modules │ └── exceptions/ # Custom exceptions │ ├── middleware/ # Custom middleware @@ -288,6 +331,27 @@ project/ ## Monitoring & Observability +The platform includes a comprehensive observability framework: + +### Health Checks + +- Aggregated health endpoint (`/health`) +- Kubernetes probes (`/health/live`, `/health/ready`) +- Module health check integration +- External tool links (`/health/tools`) + +### Metrics + +- Prometheus metrics endpoint (`/metrics`) +- Request latency histograms +- Counter and gauge support + +### Error Tracking + +- Sentry integration for production +- Exception capture with context +- Environment-aware configuration + ### Logging - Structured logging with Python logging module @@ -295,12 +359,7 @@ project/ - Error tracking with stack traces - Audit logging for admin actions -### Performance Monitoring - -- Request timing headers (`X-Process-Time`) -- Database query monitoring (SQLAlchemy echo) -- Slow query identification -- Memory usage tracking +**See:** [Observability](observability.md) for complete documentation ## Deployment Architecture @@ -332,6 +391,9 @@ Internet ───────────│ Load Balancer│ ## Related Documentation - [Multi-Tenant System](multi-tenant.md) - Detailed multi-tenancy implementation +- [Module System](module-system.md) - Module architecture and classification +- [Menu Management](menu-management.md) - Sidebar and menu configuration +- [Observability](observability.md) - Health checks, metrics, error tracking - [Middleware Stack](middleware.md) - Complete middleware documentation - [Authentication & RBAC](auth-rbac.md) - Security and access control - [Request Flow](request-flow.md) - Detailed request processing diff --git a/docs/development/creating-modules.md b/docs/development/creating-modules.md new file mode 100644 index 00000000..02e81a42 --- /dev/null +++ b/docs/development/creating-modules.md @@ -0,0 +1,483 @@ +# Creating Modules + +This guide explains how to create new modules for the Wizamart platform, including both simple wrapper modules and self-contained modules with their own services, models, and migrations. + +## Module Types + +| Type | Complexity | Use Case | +|------|------------|----------| +| **Wrapper Module** | Simple | Groups existing routes and features under a toggleable module | +| **Self-Contained Module** | Complex | Full isolation with own services, models, templates, migrations | + +## Quick Start: Wrapper Module + +For a simple module that wraps existing functionality: + +```python +# app/modules/analytics/definition.py +from app.modules.base import ModuleDefinition +from models.database.admin_menu_config import FrontendType + +analytics_module = ModuleDefinition( + code="analytics", + name="Analytics & Reports", + description="Business analytics, reports, and dashboards", + version="1.0.0", + features=[ + "analytics_dashboard", + "sales_reports", + "customer_insights", + ], + menu_items={ + FrontendType.ADMIN: ["analytics-dashboard", "reports"], + FrontendType.VENDOR: ["analytics", "sales-reports"], + }, + is_core=False, + is_internal=False, +) +``` + +## Module Definition Fields + +### Required Fields + +| Field | Type | Description | +|-------|------|-------------| +| `code` | str | Unique identifier (e.g., "billing", "analytics") | +| `name` | str | Display name (e.g., "Billing & Subscriptions") | + +### Optional Fields + +| Field | Type | Default | Description | +|-------|------|---------|-------------| +| `description` | str | "" | What the module provides | +| `version` | str | "1.0.0" | Semantic version | +| `requires` | list[str] | [] | Module codes this depends on | +| `features` | list[str] | [] | Feature codes for tier gating | +| `menu_items` | dict | {} | Menu items per frontend type | +| `permissions` | list[str] | [] | Permission codes defined | +| `is_core` | bool | False | Cannot be disabled if True | +| `is_internal` | bool | False | Admin-only if True | + +### Configuration Fields + +| Field | Type | Description | +|-------|------|-------------| +| `config_schema` | type[BaseModel] | Pydantic model for validation | +| `default_config` | dict | Default configuration values | + +### Lifecycle Hooks + +| Field | Signature | Description | +|-------|-----------|-------------| +| `on_enable` | (platform_id: int) -> None | Called when enabled | +| `on_disable` | (platform_id: int) -> None | Called when disabled | +| `on_startup` | () -> None | Called on app startup | +| `health_check` | () -> dict | Returns health status | + +## Module Classification + +### Core Modules + +Cannot be disabled. Use for essential platform functionality. + +```python +core_module = ModuleDefinition( + code="tenancy", + name="Platform Tenancy", + is_core=True, # Cannot be disabled + # ... +) +``` + +### Optional Modules + +Can be enabled/disabled per platform. + +```python +billing_module = ModuleDefinition( + code="billing", + name="Billing", + is_core=False, # Can be toggled + is_internal=False, + # ... +) +``` + +### Internal Modules + +Admin-only tools not visible to customers/vendors. + +```python +devtools_module = ModuleDefinition( + code="dev-tools", + name="Developer Tools", + is_internal=True, # Admin-only + # ... +) +``` + +## Module Dependencies + +Declare dependencies using the `requires` field: + +```python +# orders module requires payments +orders_module = ModuleDefinition( + code="orders", + name="Orders", + requires=["payments"], # Must have payments enabled + # ... +) +``` + +**Dependency Rules:** +1. Core modules cannot depend on optional modules +2. Enabling a module auto-enables its dependencies +3. Disabling a module auto-disables modules that depend on it +4. Circular dependencies are not allowed + +## Self-Contained Module Structure + +For modules with their own services, models, and templates: + +``` +app/modules/cms/ +├── __init__.py +├── definition.py # ModuleDefinition +├── config.py # Configuration schema (optional) +├── exceptions.py # Module-specific exceptions +├── routes/ +│ ├── __init__.py +│ ├── admin.py # Admin API routes +│ └── vendor.py # Vendor API routes +├── services/ +│ ├── __init__.py +│ └── content_service.py +├── models/ +│ ├── __init__.py +│ └── content_page.py +├── schemas/ +│ ├── __init__.py +│ └── content.py +├── templates/ +│ ├── admin/ +│ └── vendor/ +├── migrations/ +│ └── versions/ +│ ├── cms_001_create_content_pages.py +│ └── cms_002_add_seo_fields.py +└── locales/ + ├── en.json + └── fr.json +``` + +### Self-Contained Definition + +```python +# app/modules/cms/definition.py +from pydantic import BaseModel, Field +from app.modules.base import ModuleDefinition +from models.database.admin_menu_config import FrontendType + +class CMSConfig(BaseModel): + """CMS module configuration.""" + max_pages_per_vendor: int = Field(default=100, ge=1, le=1000) + enable_seo: bool = True + default_language: str = "en" + +cms_module = ModuleDefinition( + # Identity + code="cms", + name="Content Management", + description="Content pages, media library, and themes", + version="1.0.0", + + # Classification + is_core=True, + is_self_contained=True, + + # Features + features=[ + "content_pages", + "media_library", + "vendor_themes", + ], + + # Menu items + menu_items={ + FrontendType.ADMIN: ["content-pages", "vendor-themes"], + FrontendType.VENDOR: ["content-pages", "media"], + }, + + # Configuration + config_schema=CMSConfig, + default_config={ + "max_pages_per_vendor": 100, + "enable_seo": True, + }, + + # Self-contained paths + services_path="app.modules.cms.services", + models_path="app.modules.cms.models", + schemas_path="app.modules.cms.schemas", + templates_path="templates", + exceptions_path="app.modules.cms.exceptions", + locales_path="locales", + migrations_path="migrations", + + # Lifecycle + health_check=lambda: {"status": "healthy"}, +) +``` + +## Creating Module Routes + +### Admin Routes + +```python +# app/modules/payments/routes/admin.py +from fastapi import APIRouter, Depends +from sqlalchemy.orm import Session +from app.api.deps import get_db, get_current_admin_user + +admin_router = APIRouter(prefix="/api/admin/payments", tags=["Admin - Payments"]) + +@admin_router.get("/gateways") +async def list_gateways( + db: Session = Depends(get_db), + current_user = Depends(get_current_admin_user), +): + """List configured payment gateways.""" + # Implementation + pass +``` + +### Vendor Routes + +```python +# app/modules/payments/routes/vendor.py +from fastapi import APIRouter, Depends +from app.api.deps import get_db, get_current_vendor_user + +vendor_router = APIRouter(prefix="/api/vendor/payments", tags=["Vendor - Payments"]) + +@vendor_router.get("/methods") +async def list_payment_methods( + db: Session = Depends(get_db), + current_user = Depends(get_current_vendor_user), +): + """List vendor's stored payment methods.""" + pass +``` + +### Lazy Router Loading + +To avoid circular imports, use lazy loading: + +```python +# app/modules/payments/definition.py + +def _get_admin_router(): + """Lazy import to avoid circular imports.""" + from app.modules.payments.routes.admin import admin_router + return admin_router + +def _get_vendor_router(): + from app.modules.payments.routes.vendor import vendor_router + return vendor_router + +payments_module = ModuleDefinition( + code="payments", + # ... +) + +def get_payments_module_with_routers(): + """Get module with routers attached.""" + payments_module.admin_router = _get_admin_router() + payments_module.vendor_router = _get_vendor_router() + return payments_module +``` + +## Module Migrations + +### Naming Convention + +``` +{module_code}_{sequence}_{description}.py +``` + +Examples: +- `cms_001_create_content_pages.py` +- `cms_002_add_seo_fields.py` +- `billing_001_create_subscriptions.py` + +### Migration Template + +```python +# app/modules/cms/migrations/versions/cms_001_create_content_pages.py +"""Create content pages table. + +Revision ID: cms_001 +Revises: +Create Date: 2026-01-27 + +""" +from alembic import op +import sqlalchemy as sa + +revision = "cms_001" +down_revision = None # Or previous module migration +branch_labels = ("cms",) +depends_on = None + +def upgrade() -> None: + op.create_table( + "cms_content_pages", + sa.Column("id", sa.Integer(), primary_key=True), + sa.Column("vendor_id", sa.Integer(), sa.ForeignKey("vendors.id")), + sa.Column("title", sa.String(200), nullable=False), + sa.Column("slug", sa.String(200), nullable=False), + sa.Column("content", sa.Text()), + sa.Column("created_at", sa.DateTime(timezone=True)), + sa.Column("updated_at", sa.DateTime(timezone=True)), + ) + +def downgrade() -> None: + op.drop_table("cms_content_pages") +``` + +## Module Configuration + +### Defining Configuration Schema + +```python +from pydantic import BaseModel, Field + +class BillingConfig(BaseModel): + """Billing module configuration.""" + + stripe_mode: Literal["test", "live"] = "test" + trial_days: int = Field(default=14, ge=0, le=365) + enable_invoices: bool = True + invoice_prefix: str = Field(default="INV-", max_length=10) +``` + +### Using Configuration + +```python +from app.modules.service import module_service + +# Get module config for platform +config = module_service.get_module_config(db, platform_id, "billing") +# Returns: {"stripe_mode": "test", "trial_days": 14, ...} + +# Set module config +module_service.set_module_config( + db, platform_id, "billing", + {"trial_days": 30} +) +``` + +## Health Checks + +### Defining Health Check + +```python +def check_billing_health() -> dict: + """Check billing service dependencies.""" + issues = [] + + # Check Stripe + try: + stripe.Account.retrieve() + except stripe.AuthenticationError: + issues.append("Stripe authentication failed") + + if issues: + return { + "status": "unhealthy", + "message": "; ".join(issues), + } + + return { + "status": "healthy", + "details": {"stripe": "connected"}, + } + +billing_module = ModuleDefinition( + code="billing", + health_check=check_billing_health, + # ... +) +``` + +## Registering the Module + +### Add to Registry + +```python +# app/modules/registry.py + +from app.modules.analytics.definition import analytics_module + +OPTIONAL_MODULES: dict[str, ModuleDefinition] = { + # ... existing modules + "analytics": analytics_module, +} + +MODULES = {**CORE_MODULES, **OPTIONAL_MODULES, **INTERNAL_MODULES} +``` + +## Module Events + +Subscribe to module lifecycle events: + +```python +from app.modules.events import module_event_bus, ModuleEvent, ModuleEventData + +@module_event_bus.subscribe(ModuleEvent.ENABLED) +def on_analytics_enabled(data: ModuleEventData): + if data.module_code == "analytics": + # Initialize analytics for this platform + setup_analytics_dashboard(data.platform_id) + +@module_event_bus.subscribe(ModuleEvent.DISABLED) +def on_analytics_disabled(data: ModuleEventData): + if data.module_code == "analytics": + # Cleanup + cleanup_analytics_data(data.platform_id) +``` + +## Checklist + +### New Module Checklist + +- [ ] Create module directory: `app/modules/{code}/` +- [ ] Create `definition.py` with ModuleDefinition +- [ ] Add module to appropriate dict in `registry.py` +- [ ] Create routes if needed (admin.py, vendor.py) +- [ ] Register menu items in menu registry +- [ ] Create migrations if adding database tables +- [ ] Add health check if module has dependencies +- [ ] Document features and configuration options +- [ ] Write tests for module functionality + +### Self-Contained Module Checklist + +- [ ] All items from basic checklist +- [ ] Create `services/` directory with business logic +- [ ] Create `models/` directory with SQLAlchemy models +- [ ] Create `schemas/` directory with Pydantic schemas +- [ ] Create `exceptions.py` for module-specific errors +- [ ] Create `config.py` with Pydantic config model +- [ ] Set up `migrations/versions/` directory +- [ ] Create `templates/` if module has UI +- [ ] Create `locales/` if module needs translations +- [ ] Set `is_self_contained=True` and path attributes + +## Related Documentation + +- [Module System](../architecture/module-system.md) - Architecture overview +- [Menu Management](../architecture/menu-management.md) - Menu item integration +- [Database Migrations](migration/database-migrations.md) - Migration guide diff --git a/docs/proposals/module-migration-plan.md b/docs/proposals/module-migration-plan.md new file mode 100644 index 00000000..d9b94fa6 --- /dev/null +++ b/docs/proposals/module-migration-plan.md @@ -0,0 +1,596 @@ +# Module Migration Plan + +**Status:** In Progress +**Started:** 2026-01-25 +**Last Updated:** 2026-01-27 +**Target:** v1.0.0 + +This is the unified migration plan for transforming Wizamart into a fully modular architecture. + +--- + +## Executive Summary + +Transform the platform from a monolithic structure to self-contained modules where each module owns its: +- Services (business logic) +- Models (database entities) +- Schemas (Pydantic DTOs) +- Tasks (Celery background jobs) +- Templates (UI if applicable) +- Migrations (database changes) + +--- + +## Current State Assessment + +### Module Migration Status + +| Module | Classification | Current State | Services | Models | Tasks | Target | +|--------|---------------|---------------|----------|--------|-------|--------| +| `cms` | Core | ✅ **Complete** | ✅ | ✅ | - | Done | +| `payments` | Optional | 🟡 Partial | ✅ | ✅ | - | Done | +| `billing` | Optional | 🔴 Shell | ❌ | ❌ | ❌ | Full | +| `marketplace` | Optional | 🔴 Shell | ❌ | ❌ | ❌ | Full | +| `orders` | Optional | 🔴 Shell | ❌ | ❌ | - | Full | +| `inventory` | Optional | 🔴 Shell | ❌ | ❌ | - | Full | +| `customers` | Core | 🔴 Shell | ❌ | ❌ | - | Full | +| `analytics` | Optional | 🔴 Shell | ❌ | ❌ | - | Full | +| `messaging` | Optional | 🔴 Shell | ❌ | ❌ | - | Full | +| `monitoring` | Internal | 🔴 Shell | ❌ | ❌ | ❌ | Full | +| `dev-tools` | Internal | 🔴 Shell | ❌ | ❌ | ❌ | Full | +| `tenancy` | Core | 🔴 Shell | ❌ | ❌ | - | Full | +| `core` | Core | 🔴 Shell | ❌ | ❌ | - | Minimal | + +### Current Code Locations (To Be Migrated) + +``` +app/services/ # → Move to respective modules +├── subscription_service.py # → billing/services/ +├── stripe_service.py # → billing/services/ (or payments) +├── customer_service.py # → customers/services/ +├── order_service.py # → orders/services/ +├── inventory_service.py # → inventory/services/ +├── letzshop_service.py # → marketplace/services/ +├── marketplace_import_service.py # → marketplace/services/ +├── messaging_service.py # → messaging/services/ +└── ... + +models/database/ # → Move to respective modules +├── subscription.py # → billing/models/ +├── customer.py # → customers/models/ +├── order.py # → orders/models/ +├── inventory.py # → inventory/models/ +├── letzshop_*.py # → marketplace/models/ +└── ... + +app/tasks/celery_tasks/ # → Move to respective modules +├── subscription.py # → billing/tasks/ +├── marketplace.py # → marketplace/tasks/ +├── letzshop.py # → marketplace/tasks/ +├── export.py # → marketplace/tasks/ +├── code_quality.py # → dev_tools/tasks/ +└── test_runner.py # → dev_tools/tasks/ +``` + +### Celery Task Mapping + +| Current Task | Current Location | Target Module | +|--------------|------------------|---------------| +| `reset_period_counters` | subscription.py | `billing` | +| `check_trial_expirations` | subscription.py | `billing` | +| `sync_stripe_status` | subscription.py | `billing` | +| `cleanup_stale_subscriptions` | subscription.py | `billing` | +| `capture_capacity_snapshot` | subscription.py | `monitoring` | +| `process_marketplace_import` | marketplace.py | `marketplace` | +| `process_historical_import` | letzshop.py | `marketplace` | +| `sync_vendor_directory` | letzshop.py | `marketplace` | +| `export_vendor_products_to_folder` | export.py | `marketplace` | +| `export_marketplace_products` | export.py | `marketplace` | +| `execute_code_quality_scan` | code_quality.py | `dev-tools` | +| `execute_test_run` | test_runner.py | `dev-tools` | + +--- + +## Infrastructure Work (Completed) + +### ✅ Phase 1: Module Classification +- Three-tier system: Core, Optional, Internal +- Renamed `platform-admin` → `tenancy` +- Module registry with CORE_MODULES, OPTIONAL_MODULES, INTERNAL_MODULES + +### ✅ Phase 2: Module Infrastructure +- `ModuleDefinition` with lifecycle hooks +- Module events system (`events.py`) +- Module migration discovery (`migrations.py`) +- Observability framework (`observability.py`) + +### ✅ Phase 3: Documentation +- Module system architecture docs +- Menu management docs +- Observability docs +- Creating modules developer guide + +--- + +## Infrastructure Work (Remaining) + +### Phase 4: Celery Task Infrastructure + +Add task support to module system before migrating modules. + +#### 4.1 Update ModuleDefinition + +```python +# app/modules/base.py - Add fields + +@dataclass +class ScheduledTask: + """Celery Beat scheduled task definition.""" + name: str # e.g., "billing.reset_counters" + task: str # Full task path + schedule: str | dict # Cron string or crontab dict + args: tuple = () + kwargs: dict = field(default_factory=dict) + +@dataclass +class ModuleDefinition: + # ... existing fields ... + + # Task configuration (NEW) + tasks_path: str | None = None + scheduled_tasks: list[ScheduledTask] = field(default_factory=list) +``` + +#### 4.2 Create Task Discovery + +```python +# app/modules/tasks.py (NEW) + +def discover_module_tasks() -> list[str]: + """Discover task modules from all registered modules.""" + ... + +def build_beat_schedule() -> dict: + """Build Celery Beat schedule from module definitions.""" + ... +``` + +#### 4.3 Create Module Task Base + +```python +# app/modules/task_base.py (NEW) + +class ModuleTask(Task): + """Base Celery task with DB session management.""" + + @contextmanager + def get_db(self): + ... +``` + +#### 4.4 Update Celery Config + +```python +# app/core/celery_config.py + +from app.modules.tasks import discover_module_tasks, build_beat_schedule + +# Auto-discover from modules +celery_app.autodiscover_tasks(discover_module_tasks()) + +# Build schedule from modules +celery_app.conf.beat_schedule = build_beat_schedule() +``` + +**Files to create:** +- `app/modules/tasks.py` +- `app/modules/task_base.py` + +**Files to modify:** +- `app/modules/base.py` +- `app/core/celery_config.py` + +--- + +## Module Migration Phases + +Each module migration includes: services, models, schemas, tasks, templates (if any). + +### Phase 5: Billing Module (High Priority) + +**Why first:** Has most Celery tasks, critical business logic. + +#### Components to Move + +| From | To | Type | +|------|-----|------| +| `app/services/subscription_service.py` | `app/modules/billing/services/` | Service | +| `app/services/stripe_service.py` | `app/modules/billing/services/` | Service | +| `models/database/subscription.py` | `app/modules/billing/models/` | Model | +| `models/database/subscription_tier.py` | `app/modules/billing/models/` | Model | +| `models/schema/subscription.py` | `app/modules/billing/schemas/` | Schema | +| `app/tasks/celery_tasks/subscription.py` | `app/modules/billing/tasks/` | Tasks | + +#### Target Structure + +``` +app/modules/billing/ +├── __init__.py +├── definition.py # Update with tasks_path, scheduled_tasks +├── config.py # BillingConfig schema +├── exceptions.py # BillingException, SubscriptionError, etc. +├── services/ +│ ├── __init__.py +│ ├── subscription_service.py +│ └── stripe_service.py +├── models/ +│ ├── __init__.py +│ ├── subscription.py +│ └── subscription_tier.py +├── schemas/ +│ ├── __init__.py +│ └── subscription.py +├── tasks/ +│ ├── __init__.py +│ └── subscription.py # 4 scheduled tasks +├── routes/ +│ ├── admin.py +│ └── vendor.py +└── migrations/ + └── versions/ +``` + +#### Tasks + +| Task | Schedule | Queue | +|------|----------|-------| +| `reset_period_counters` | Daily 00:05 | scheduled | +| `check_trial_expirations` | Daily 01:00 | scheduled | +| `sync_stripe_status` | Hourly :30 | scheduled | +| `cleanup_stale_subscriptions` | Weekly Sun 03:00 | scheduled | + +#### Migration Checklist + +- [ ] Create `billing/services/` directory +- [ ] Move `subscription_service.py` → update imports +- [ ] Move `stripe_service.py` → update imports +- [ ] Create `billing/models/` directory +- [ ] Move subscription models → update imports +- [ ] Create `billing/schemas/` directory +- [ ] Move subscription schemas → update imports +- [ ] Create `billing/tasks/` directory +- [ ] Move subscription tasks → update task paths +- [ ] Create `billing/exceptions.py` +- [ ] Update `definition.py` with `tasks_path`, `scheduled_tasks` +- [ ] Create module migrations +- [ ] Update all imports across codebase +- [ ] Test subscription flows +- [ ] Test scheduled tasks + +--- + +### Phase 6: Marketplace Module (High Priority) + +**Why second:** Has most tasks, complex import logic. + +#### Components to Move + +| From | To | Type | +|------|-----|------| +| `app/services/letzshop_service.py` | `app/modules/marketplace/services/` | Service | +| `app/services/marketplace_import_service.py` | `app/modules/marketplace/services/` | Service | +| `app/services/letzshop_credentials_service.py` | `app/modules/marketplace/services/` | Service | +| `models/database/letzshop_*.py` | `app/modules/marketplace/models/` | Models | +| `models/database/marketplace_import_job.py` | `app/modules/marketplace/models/` | Model | +| `app/tasks/celery_tasks/marketplace.py` | `app/modules/marketplace/tasks/` | Tasks | +| `app/tasks/celery_tasks/letzshop.py` | `app/modules/marketplace/tasks/` | Tasks | +| `app/tasks/celery_tasks/export.py` | `app/modules/marketplace/tasks/` | Tasks | + +#### Target Structure + +``` +app/modules/marketplace/ +├── __init__.py +├── definition.py +├── exceptions.py +├── services/ +│ ├── __init__.py +│ ├── letzshop_service.py +│ ├── letzshop_credentials_service.py +│ ├── import_service.py +│ └── export_service.py +├── models/ +│ ├── __init__.py +│ ├── letzshop_order.py +│ ├── letzshop_credentials.py +│ └── import_job.py +├── schemas/ +│ └── ... +├── tasks/ +│ ├── __init__.py +│ ├── import_tasks.py # process_marketplace_import +│ ├── letzshop.py # process_historical_import, sync_vendor_directory +│ └── export.py # export tasks +└── routes/ +``` + +#### Tasks + +| Task | Type | Queue | +|------|------|-------| +| `process_marketplace_import` | On-demand | long_running | +| `process_historical_import` | On-demand | long_running | +| `sync_vendor_directory` | Scheduled (optional) | long_running | +| `export_vendor_products_to_folder` | On-demand | default | +| `export_marketplace_products` | On-demand | default | + +--- + +### Phase 7: Dev-Tools Module (Medium Priority) + +**Why:** Internal module with code quality and test runner tasks. + +#### Components to Move + +| From | To | Type | +|------|-----|------| +| `app/services/code_quality_service.py` | `app/modules/dev_tools/services/` | Service | +| `app/services/test_runner_service.py` | `app/modules/dev_tools/services/` | Service | +| `models/database/architecture_scan.py` | `app/modules/dev_tools/models/` | Model | +| `models/database/test_run.py` | `app/modules/dev_tools/models/` | Model | +| `app/tasks/celery_tasks/code_quality.py` | `app/modules/dev_tools/tasks/` | Tasks | +| `app/tasks/celery_tasks/test_runner.py` | `app/modules/dev_tools/tasks/` | Tasks | + +#### Target Structure + +``` +app/modules/dev_tools/ +├── __init__.py +├── definition.py +├── services/ +│ ├── code_quality_service.py +│ └── test_runner_service.py +├── models/ +│ ├── architecture_scan.py +│ ├── architecture_violation.py +│ └── test_run.py +├── tasks/ +│ ├── code_quality.py +│ └── test_runner.py +└── routes/ +``` + +--- + +### Phase 8: Monitoring Module (Medium Priority) + +#### Components to Move + +| From | To | Type | +|------|-----|------| +| `app/services/background_tasks_service.py` | `app/modules/monitoring/services/` | Service | +| `capture_capacity_snapshot` task | `app/modules/monitoring/tasks/` | Task | +| `models/database/capacity_snapshot.py` | `app/modules/monitoring/models/` | Model | + +#### Target Structure + +``` +app/modules/monitoring/ +├── __init__.py +├── definition.py +├── services/ +│ └── background_tasks_service.py +├── models/ +│ └── capacity_snapshot.py +├── tasks/ +│ └── capacity.py # capture_capacity_snapshot +└── routes/ +``` + +--- + +### Phase 9: Customers Module (Core) + +#### Components to Move + +| From | To | +|------|-----| +| `app/services/customer_service.py` | `app/modules/customers/services/` | +| `models/database/customer.py` | `app/modules/customers/models/` | +| Customer-related schemas | `app/modules/customers/schemas/` | + +--- + +### Phase 10: Orders Module + +#### Components to Move + +| From | To | +|------|-----| +| `app/services/order_service.py` | `app/modules/orders/services/` | +| `models/database/order.py` | `app/modules/orders/models/` | +| Order-related schemas | `app/modules/orders/schemas/` | + +--- + +### Phase 11: Inventory Module + +#### Components to Move + +| From | To | +|------|-----| +| `app/services/inventory_service.py` | `app/modules/inventory/services/` | +| `models/database/inventory*.py` | `app/modules/inventory/models/` | + +--- + +### Phase 12: Remaining Modules + +- **Analytics** - Move analytics services +- **Messaging** - Move messaging service, notification models +- **Tenancy** - Move platform, company, vendor services + +--- + +## Phase 13: Cleanup & Full Celery Migration + +After all modules are migrated: + +### 13.1 Remove Fallback Tasks + +Delete: +``` +app/tasks/background_tasks.py +app/tasks/letzshop_tasks.py +app/tasks/subscription_tasks.py +app/tasks/code_quality_tasks.py +app/tasks/test_runner_tasks.py +``` + +### 13.2 Remove USE_CELERY Flag + +```python +# app/core/config.py +# DELETE: USE_CELERY: bool = False +``` + +### 13.3 Simplify Dispatcher + +Rewrite `app/tasks/dispatcher.py` to Celery-only. + +### 13.4 Delete Old Task Directory + +``` +app/tasks/celery_tasks/ # DELETE - moved to modules +``` + +### 13.5 Clean Up Old Services/Models + +Remove files from: +- `app/services/` (moved to modules) +- `models/database/` (moved to modules) + +Leave re-exports for backward compatibility if needed. + +--- + +## Queue Configuration + +### Final Queue Structure + +| Queue | Purpose | Modules | +|-------|---------|---------| +| `default` | Fast tasks | marketplace (exports) | +| `long_running` | CPU-intensive | marketplace (imports), dev-tools | +| `scheduled` | Celery Beat | billing, monitoring | + +### Task Routing + +```python +task_routes = { + "app.modules.billing.tasks.*": {"queue": "scheduled"}, + "app.modules.marketplace.tasks.import_tasks.*": {"queue": "long_running"}, + "app.modules.marketplace.tasks.letzshop.*": {"queue": "long_running"}, + "app.modules.marketplace.tasks.export.*": {"queue": "default"}, + "app.modules.monitoring.tasks.*": {"queue": "scheduled"}, + "app.modules.dev_tools.tasks.*": {"queue": "long_running"}, +} +``` + +--- + +## Migration Strategy + +### For Each Module + +1. **Create directory structure** +2. **Move models first** (fewest dependencies) +3. **Move services** (depend on models) +4. **Move schemas** (depend on models) +5. **Move tasks** (depend on services) +6. **Update all imports** across codebase +7. **Create re-exports** in old locations (temporary) +8. **Test thoroughly** +9. **Remove re-exports** in next phase + +### Import Update Strategy + +Use IDE refactoring or script: +```bash +# Example: Update subscription_service imports +find . -name "*.py" -exec sed -i 's/from app.services.subscription_service/from app.modules.billing.services.subscription_service/g' {} \; +``` + +--- + +## Verification Checklist + +### Per-Module Verification + +- [ ] Module directory has all components +- [ ] All imports updated +- [ ] Services work +- [ ] API routes work +- [ ] Tasks execute correctly +- [ ] Scheduled tasks run (Celery Beat) +- [ ] No circular imports +- [ ] Tests pass + +### Final Verification + +- [ ] All 12 tasks in modules +- [ ] `USE_CELERY` flag removed +- [ ] Fallback tasks deleted +- [ ] Old service files removed +- [ ] Old model files removed +- [ ] All tests pass +- [ ] Production smoke test + +--- + +## Priority Order + +| Priority | Phase | Module | Reason | +|----------|-------|--------|--------| +| 1 | 4 | Infrastructure | Required for task migration | +| 2 | 5 | `billing` | Most tasks, critical business logic | +| 3 | 6 | `marketplace` | Most complex, many tasks | +| 4 | 7 | `dev-tools` | Internal, low risk | +| 5 | 8 | `monitoring` | Internal, low risk | +| 6 | 9-12 | Others | Lower complexity | +| 7 | 13 | Cleanup | Final step | + +--- + +## Estimated Effort + +| Phase | Sessions | Notes | +|-------|----------|-------| +| 4 (Infrastructure) | 1 | Task base class, discovery | +| 5 (Billing) | 2-3 | Complex, many components | +| 6 (Marketplace) | 3-4 | Most complex module | +| 7-8 (Internal) | 1-2 | Simpler modules | +| 9-12 (Others) | 4-6 | Medium complexity each | +| 13 (Cleanup) | 1 | Removing old code | + +**Total:** ~12-17 sessions + +--- + +## Rollback Plan + +Each phase can be rolled back independently: +1. Restore moved files to original locations +2. Revert import changes +3. Restore celery_config.py + +No database schema changes during code migration = no DB rollback needed. + +--- + +## Related Documents + +- [Module System Architecture](../architecture/module-system.md) +- [Creating Modules](../development/creating-modules.md) +- [Observability](../architecture/observability.md) +- [SESSION_NOTE_2026-01-27](SESSION_NOTE_2026-01-27_module-reclassification.md) diff --git a/mkdocs.yml b/mkdocs.yml index ec3c88c7..d4609f62 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -41,6 +41,7 @@ nav: - Middleware Stack: architecture/middleware.md - Module System: architecture/module-system.md - Menu Management: architecture/menu-management.md + - Observability: architecture/observability.md - Request Flow: architecture/request-flow.md - Authentication & RBAC: architecture/auth-rbac.md - Frontend Structure: architecture/frontend-structure.md @@ -115,6 +116,7 @@ nav: # --- Development --- - Development: - Contributing Guide: development/contributing.md + - Creating Modules: development/creating-modules.md - Code Quality: development/code-quality.md - Architecture Rules: development/architecture-rules.md - Security Rules: development/security-rules.md @@ -157,6 +159,7 @@ nav: - Vendor Operations Expansion: development/migration/vendor-operations-expansion.md - Implementation Plans: + - Module Migration Plan: proposals/module-migration-plan.md - Admin Inventory Management: implementation/inventory-admin-migration.md - Admin Notification System: implementation/admin-notification-system.md - Letzshop Order Import: implementation/letzshop-order-import-improvements.md