sboulahtit/orion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: add observability, creating modules guide, and unified migration plan

Browse Source 
- Add observability framework documentation (health checks, metrics, Sentry)
- Add developer guide for creating modules
- Add comprehensive module migration plan with Celery task integration
- Update architecture overview with module system and observability sections
- Update module-system.md with links to new docs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Samir Boulahtit
2026-01-27 22:41:19 +01:00
parent e3cab29c1a
commit 7dbdbd4c7e
6 changed files with 1583 additions and 8 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/architecture/module-system.md
View File

@@ -417,5 +417,7 @@ register_module_health_checks()
## Related Documentation ## Related Documentation
- [Menu Management](menu-management.md) - Sidebar and menu configuration - [Menu Management](menu-management.md) - Sidebar and menu configuration
- [Creating Modules](../development/creating-modules.md) - Developer guide for building modules
- [Observability](observability.md) - Health checks and module health integration
- [Multi-Tenant System](multi-tenant.md) - Platform isolation - [Multi-Tenant System](multi-tenant.md) - Platform isolation
- [Feature Gating](../implementation/feature-gating-system.md) - Tier-based access - [Feature Gating](../implementation/feature-gating-system.md) - Tier-based access

429
docs/architecture/observability.md Normal file
View File

@@ -0,0 +1,429 @@
# Observability Framework
The Wizamart platform includes a comprehensive observability framework for monitoring application health, collecting metrics, and tracking errors. This is part of the Framework Layer - infrastructure that modules depend on.
## Overview
```
┌─────────────────────────────────────────────────────────────────────────┐
│ OBSERVABILITY FRAMEWORK │
│ app/core/observability.py │
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
│ │ Health Checks │ │ Prometheus │ │ Sentry │ │
│ │ Registry │ │ Metrics │ │ Integration │ │
│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ API Endpoints │ │
│ │ /health │ /health/live │ /health/ready │ /metrics │ │
│ └─────────────────────────────────────────────────────────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘
│
▼
┌───────────────────────────────┐
│ External Tools │
│ Flower │ Grafana │ Prometheus│
└───────────────────────────────┘
```
## Health Checks
### Health Check Registry
Components register health checks that are aggregated into a single endpoint.
```python
from app.core.observability import (
health_registry,
HealthCheckResult,
HealthStatus,
)
# Register using decorator
@health_registry.register("database")
def check_database() -> HealthCheckResult:
try:
db.execute("SELECT 1")
return HealthCheckResult(
name="database",
status=HealthStatus.HEALTHY,
message="Database connection OK"
)
except Exception as e:
return HealthCheckResult(
name="database",
status=HealthStatus.UNHEALTHY,
message=str(e)
)
# Or register directly
health_registry.register_check("redis", check_redis_connection)
```
### Health Status Levels
| Status | Description | HTTP Response |
|--------|-------------|---------------|
| `HEALTHY` | All systems operational | 200 |
| `DEGRADED` | Partial functionality available | 200 |
| `UNHEALTHY` | Critical failure | 200 (check response body) |
### HealthCheckResult Fields
| Field | Type | Description |
|-------|------|-------------|
| `name` | str | Check identifier |
| `status` | HealthStatus | Health status level |
| `message` | str | Optional status message |
| `latency_ms` | float | Check execution time |
| `details` | dict | Additional diagnostic data |
| `checked_at` | datetime | Timestamp of check |
## API Endpoints
### GET /health
Aggregated health check endpoint. Returns combined status from all registered checks.
**Response:**
```json
{
"status": "healthy",
"timestamp": "2026-01-27T10:30:00Z",
"checks": [
{
"name": "database",
"status": "healthy",
"message": "Connection OK",
"latency_ms": 2.5,
"details": {}
},
{
"name": "module:billing",
"status": "healthy",
"message": "",
"latency_ms": 0.1,
"details": {}
}
]
}
```
**Overall Status Logic:**
- If any check is `UNHEALTHY` → overall is `UNHEALTHY`
- If any check is `DEGRADED` and none `UNHEALTHY` → overall is `DEGRADED`
- Otherwise → `HEALTHY`
### GET /health/live
Kubernetes liveness probe. Returns 200 if application is running.
**Response:**
```json
{"status": "alive"}
```
### GET /health/ready
Kubernetes readiness probe. Returns ready status based on health checks.
**Response:**
```json
{
"status": "ready",
"health": "healthy"
}
```
Or if unhealthy:
```json
{
"status": "not_ready",
"health": "unhealthy"
}
```
### GET /metrics
Prometheus metrics endpoint. Returns metrics in Prometheus text format.
**Response:**
```
# HELP http_requests_total Total HTTP requests
# TYPE http_requests_total counter
http_requests_total{method="GET",endpoint="/api/products",status="200"} 1234
...
```
### GET /health/tools
Returns URLs to external monitoring tools.
**Response:**
```json
{
"flower": "http://flower.example.com:5555",
"grafana": "http://grafana.example.com:3000",
"prometheus": null
}
```
## Prometheus Metrics
### MetricsRegistry
The metrics registry provides a wrapper around `prometheus_client` with fallback when the library isn't installed.
```python
from app.core.observability import metrics_registry
# Counter - tracks cumulative values
request_counter = metrics_registry.counter(
"http_requests_total",
"Total HTTP requests",
["method", "endpoint", "status"]
)
request_counter.labels(method="GET", endpoint="/api/products", status="200").inc()
# Histogram - tracks distributions (latency, sizes)
request_latency = metrics_registry.histogram(
"http_request_duration_seconds",
"HTTP request latency",
["endpoint"],
buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
)
request_latency.labels(endpoint="/api/products").observe(0.045)
# Gauge - tracks current values
active_connections = metrics_registry.gauge(
"active_connections",
"Number of active connections",
["pool"]
)
active_connections.labels(pool="database").set(42)
```
### Enabling Metrics
Metrics are disabled by default. Enable during initialization:
```python
from app.core.observability import init_observability
init_observability(
enable_metrics=True,
# ... other options
)
```
### Dummy Metrics
When `prometheus_client` isn't installed or metrics are disabled, the registry returns dummy metrics that silently ignore all operations. This allows code to use metrics without checking if they're enabled.
## Sentry Integration
### Configuration
```python
from app.core.observability import sentry, init_observability
# Initialize via init_observability
init_observability(
sentry_dsn="https://key@sentry.io/project",
environment="production",
)
# Or initialize directly
sentry.init(
dsn="https://key@sentry.io/project",
environment="production",
release="1.0.0"
)
```
### Capturing Errors
```python
from app.core.observability import sentry
try:
risky_operation()
except Exception as e:
event_id = sentry.capture_exception(e)
logger.error(f"Operation failed, Sentry event: {event_id}")
# Capture messages
sentry.capture_message("User reached rate limit", level="warning")
```
### Without Sentry
If `sentry_sdk` isn't installed or DSN isn't provided, all capture calls silently return `None`.
## Module Health Checks
Modules can provide health checks that are automatically registered.
### Defining Module Health Check
```python
# In module definition
from app.modules.base import ModuleDefinition
def check_billing_health() -> dict:
"""Check billing service health."""
try:
# Check Stripe connection
stripe.Account.retrieve()
return {"status": "healthy", "message": "Stripe connected"}
except Exception as e:
return {"status": "unhealthy", "message": str(e)}
billing_module = ModuleDefinition(
code="billing",
name="Billing",
health_check=check_billing_health,
# ...
)
```
### Registering Module Health Checks
```python
from app.core.observability import register_module_health_checks
# Call after modules are loaded (e.g., in app lifespan)
register_module_health_checks()
```
This registers health checks as `module:{code}` (e.g., `module:billing`).
## External Tools
### Flower (Celery Monitoring)
Configure Flower URL to include in `/health/tools`:
```python
init_observability(
flower_url="http://flower:5555",
)
```
### Grafana
Configure Grafana URL:
```python
init_observability(
grafana_url="http://grafana:3000",
)
```
## Initialization
### Application Lifespan
```python
# main.py
from contextlib import asynccontextmanager
from fastapi import FastAPI
from app.core.observability import (
init_observability,
shutdown_observability,
health_router,
register_module_health_checks,
)
@asynccontextmanager
async def lifespan(app: FastAPI):
# Startup
init_observability(
enable_metrics=True,
sentry_dsn=settings.SENTRY_DSN,
environment=settings.ENVIRONMENT,
flower_url=settings.FLOWER_URL,
grafana_url=settings.GRAFANA_URL,
)
register_module_health_checks()
yield
# Shutdown
shutdown_observability()
app = FastAPI(lifespan=lifespan)
app.include_router(health_router)
```
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `SENTRY_DSN` | Sentry DSN for error tracking | None (disabled) |
| `ENVIRONMENT` | Environment name | "development" |
| `ENABLE_METRICS` | Enable Prometheus metrics | False |
| `FLOWER_URL` | Flower dashboard URL | None |
| `GRAFANA_URL` | Grafana dashboard URL | None |
## Kubernetes Integration
### Deployment Configuration
```yaml
apiVersion: apps/v1
kind: Deployment
spec:
template:
spec:
containers:
- name: app
livenessProbe:
httpGet:
path: /health/live
port: 8000
initialDelaySeconds: 10
periodSeconds: 30
readinessProbe:
httpGet:
path: /health/ready
port: 8000
initialDelaySeconds: 5
periodSeconds: 10
```
### Prometheus ServiceMonitor
```yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
spec:
endpoints:
- port: http
path: /metrics
interval: 15s
```
## Best Practices
### Do
- Register health checks for critical dependencies (database, cache, external APIs)
- Use appropriate metric types (counter for counts, histogram for latency)
- Include meaningful labels but avoid high cardinality
- Set up alerts based on health status changes
### Don't
- Create health checks that are slow or have side effects
- Add high-cardinality labels to metrics (e.g., user IDs)
- Ignore Sentry errors in production
- Skip readiness probes in Kubernetes deployments
## Related Documentation
- [Module System](module-system.md) - Module health check integration
- [Background Tasks](background-tasks.md) - Celery monitoring with Flower
- [Deployment](../deployment/index.md) - Production deployment with monitoring

78
docs/architecture/overview.md
View File

@@ -76,7 +76,42 @@ Custom middleware handles:
**See:** [Authentication & RBAC](auth-rbac.md) for details **See:** [Authentication & RBAC](auth-rbac.md) for details
### 4. Request Flow ### 4. Module System
The platform uses a modular architecture with three-tier classification:
```
┌─────────────────────────────────────────────────────────────┐
│ FRAMEWORK LAYER │
│ (Infrastructure - always available, not modules) │
│ Config │ Database │ Auth │ Permissions │ Observability │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ MODULE LAYER │
│ │
│ CORE (Always Enabled) OPTIONAL (Per-Platform) │
│ ├── core ├── payments │
│ ├── tenancy ├── billing │
│ ├── cms ├── inventory │
│ └── customers ├── orders │
│ ├── marketplace │
│ INTERNAL (Admin-Only) ├── analytics │
│ ├── dev-tools └── messaging │
│ └── monitoring │
└─────────────────────────────────────────────────────────────┘
```
**Module Features:**
- Enable/disable modules per platform
- Module dependencies (billing requires payments)
- Health checks and lifecycle hooks
- Self-contained modules with own services, models, migrations
**See:** [Module System](module-system.md) for complete documentation
### 5. Request Flow
```mermaid ```mermaid
graph TB graph TB
@@ -252,7 +287,15 @@ project/
│ ├── api/ # API routes │ ├── api/ # API routes
│ ├── routes/ # Page routes (HTML) │ ├── routes/ # Page routes (HTML)
│ ├── services/ # Business logic │ ├── services/ # Business logic
│ ├── core/ # Core functionality │ ├── core/ # Core functionality (config, db, observability)
│ ├── modules/ # Module definitions and self-contained modules
│ │ ├── base.py # ModuleDefinition class
│ │ ├── registry.py # Module registry (CORE, OPTIONAL, INTERNAL)
│ │ ├── service.py # Module enablement service
│ │ ├── events.py # Module event bus
│ │ ├── cms/ # Self-contained CMS module
│ │ ├── payments/ # Self-contained payments module
│ │ └── ... # Other modules
│ └── exceptions/ # Custom exceptions │ └── exceptions/ # Custom exceptions
│ │
├── middleware/ # Custom middleware ├── middleware/ # Custom middleware
@@ -288,6 +331,27 @@ project/
## Monitoring & Observability ## Monitoring & Observability
The platform includes a comprehensive observability framework:
### Health Checks
- Aggregated health endpoint (`/health`)
- Kubernetes probes (`/health/live`, `/health/ready`)
- Module health check integration
- External tool links (`/health/tools`)
### Metrics
- Prometheus metrics endpoint (`/metrics`)
- Request latency histograms
- Counter and gauge support
### Error Tracking
- Sentry integration for production
- Exception capture with context
- Environment-aware configuration
### Logging ### Logging
- Structured logging with Python logging module - Structured logging with Python logging module
@@ -295,12 +359,7 @@ project/
- Error tracking with stack traces - Error tracking with stack traces
- Audit logging for admin actions - Audit logging for admin actions
### Performance Monitoring **See:** [Observability](observability.md) for complete documentation
- Request timing headers (`X-Process-Time`)
- Database query monitoring (SQLAlchemy echo)
- Slow query identification
- Memory usage tracking
## Deployment Architecture ## Deployment Architecture
@@ -332,6 +391,9 @@ Internet ───────────│ Load Balancer│
## Related Documentation ## Related Documentation
- [Multi-Tenant System](multi-tenant.md) - Detailed multi-tenancy implementation - [Multi-Tenant System](multi-tenant.md) - Detailed multi-tenancy implementation
- [Module System](module-system.md) - Module architecture and classification
- [Menu Management](menu-management.md) - Sidebar and menu configuration
- [Observability](observability.md) - Health checks, metrics, error tracking
- [Middleware Stack](middleware.md) - Complete middleware documentation - [Middleware Stack](middleware.md) - Complete middleware documentation
- [Authentication & RBAC](auth-rbac.md) - Security and access control - [Authentication & RBAC](auth-rbac.md) - Security and access control
- [Request Flow](request-flow.md) - Detailed request processing - [Request Flow](request-flow.md) - Detailed request processing

483
docs/development/creating-modules.md Normal file
View File

@@ -0,0 +1,483 @@
# Creating Modules
This guide explains how to create new modules for the Wizamart platform, including both simple wrapper modules and self-contained modules with their own services, models, and migrations.
## Module Types
| Type | Complexity | Use Case |
|------|------------|----------|
| **Wrapper Module** | Simple | Groups existing routes and features under a toggleable module |
| **Self-Contained Module** | Complex | Full isolation with own services, models, templates, migrations |
## Quick Start: Wrapper Module
For a simple module that wraps existing functionality:
```python
# app/modules/analytics/definition.py
from app.modules.base import ModuleDefinition
from models.database.admin_menu_config import FrontendType
analytics_module = ModuleDefinition(
code="analytics",
name="Analytics & Reports",
description="Business analytics, reports, and dashboards",
version="1.0.0",
features=[
"analytics_dashboard",
"sales_reports",
"customer_insights",
],
menu_items={
FrontendType.ADMIN: ["analytics-dashboard", "reports"],
FrontendType.VENDOR: ["analytics", "sales-reports"],
},
is_core=False,
is_internal=False,
)
```
## Module Definition Fields
### Required Fields
| Field | Type | Description |
|-------|------|-------------|
| `code` | str | Unique identifier (e.g., "billing", "analytics") |
| `name` | str | Display name (e.g., "Billing & Subscriptions") |
### Optional Fields
| Field | Type | Default | Description |
|-------|------|---------|-------------|
| `description` | str | "" | What the module provides |
| `version` | str | "1.0.0" | Semantic version |
| `requires` | list[str] | [] | Module codes this depends on |
| `features` | list[str] | [] | Feature codes for tier gating |
| `menu_items` | dict | {} | Menu items per frontend type |
| `permissions` | list[str] | [] | Permission codes defined |
| `is_core` | bool | False | Cannot be disabled if True |
| `is_internal` | bool | False | Admin-only if True |
### Configuration Fields
| Field | Type | Description |
|-------|------|-------------|
| `config_schema` | type[BaseModel] | Pydantic model for validation |
| `default_config` | dict | Default configuration values |
### Lifecycle Hooks
| Field | Signature | Description |
|-------|-----------|-------------|
| `on_enable` | (platform_id: int) -> None | Called when enabled |
| `on_disable` | (platform_id: int) -> None | Called when disabled |
| `on_startup` | () -> None | Called on app startup |
| `health_check` | () -> dict | Returns health status |
## Module Classification
### Core Modules
Cannot be disabled. Use for essential platform functionality.
```python
core_module = ModuleDefinition(
code="tenancy",
name="Platform Tenancy",
is_core=True, # Cannot be disabled
# ...
)
```
### Optional Modules
Can be enabled/disabled per platform.
```python
billing_module = ModuleDefinition(
code="billing",
name="Billing",
is_core=False, # Can be toggled
is_internal=False,
# ...
)
```
### Internal Modules
Admin-only tools not visible to customers/vendors.
```python
devtools_module = ModuleDefinition(
code="dev-tools",
name="Developer Tools",
is_internal=True, # Admin-only
# ...
)
```
## Module Dependencies
Declare dependencies using the `requires` field:
```python
# orders module requires payments
orders_module = ModuleDefinition(
code="orders",
name="Orders",
requires=["payments"], # Must have payments enabled
# ...
)
```
**Dependency Rules:**
1. Core modules cannot depend on optional modules
2. Enabling a module auto-enables its dependencies
3. Disabling a module auto-disables modules that depend on it
4. Circular dependencies are not allowed
## Self-Contained Module Structure
For modules with their own services, models, and templates:
```
app/modules/cms/
├── __init__.py
├── definition.py # ModuleDefinition
├── config.py # Configuration schema (optional)
├── exceptions.py # Module-specific exceptions
├── routes/
│ ├── __init__.py
│ ├── admin.py # Admin API routes
│ └── vendor.py # Vendor API routes
├── services/
│ ├── __init__.py
│ └── content_service.py
├── models/
│ ├── __init__.py
│ └── content_page.py
├── schemas/
│ ├── __init__.py
│ └── content.py
├── templates/
│ ├── admin/
│ └── vendor/
├── migrations/
│ └── versions/
│ ├── cms_001_create_content_pages.py
│ └── cms_002_add_seo_fields.py
└── locales/
├── en.json
└── fr.json
```
### Self-Contained Definition
```python
# app/modules/cms/definition.py
from pydantic import BaseModel, Field
from app.modules.base import ModuleDefinition
from models.database.admin_menu_config import FrontendType
class CMSConfig(BaseModel):
"""CMS module configuration."""
max_pages_per_vendor: int = Field(default=100, ge=1, le=1000)
enable_seo: bool = True
default_language: str = "en"
cms_module = ModuleDefinition(
# Identity
code="cms",
name="Content Management",
description="Content pages, media library, and themes",
version="1.0.0",
# Classification
is_core=True,
is_self_contained=True,
# Features
features=[
"content_pages",
"media_library",
"vendor_themes",
],
# Menu items
menu_items={
FrontendType.ADMIN: ["content-pages", "vendor-themes"],
FrontendType.VENDOR: ["content-pages", "media"],
},
# Configuration
config_schema=CMSConfig,
default_config={
"max_pages_per_vendor": 100,
"enable_seo": True,
},
# Self-contained paths
services_path="app.modules.cms.services",
models_path="app.modules.cms.models",
schemas_path="app.modules.cms.schemas",
templates_path="templates",
exceptions_path="app.modules.cms.exceptions",
locales_path="locales",
migrations_path="migrations",
# Lifecycle
health_check=lambda: {"status": "healthy"},
)
```
## Creating Module Routes
### Admin Routes
```python
# app/modules/payments/routes/admin.py
from fastapi import APIRouter, Depends
from sqlalchemy.orm import Session
from app.api.deps import get_db, get_current_admin_user
admin_router = APIRouter(prefix="/api/admin/payments", tags=["Admin - Payments"])
@admin_router.get("/gateways")
async def list_gateways(
db: Session = Depends(get_db),
current_user = Depends(get_current_admin_user),
):
"""List configured payment gateways."""
# Implementation
pass
```
### Vendor Routes
```python
# app/modules/payments/routes/vendor.py
from fastapi import APIRouter, Depends
from app.api.deps import get_db, get_current_vendor_user
vendor_router = APIRouter(prefix="/api/vendor/payments", tags=["Vendor - Payments"])
@vendor_router.get("/methods")
async def list_payment_methods(
db: Session = Depends(get_db),
current_user = Depends(get_current_vendor_user),
):
"""List vendor's stored payment methods."""
pass
```
### Lazy Router Loading
To avoid circular imports, use lazy loading:
```python
# app/modules/payments/definition.py
def _get_admin_router():
"""Lazy import to avoid circular imports."""
from app.modules.payments.routes.admin import admin_router
return admin_router
def _get_vendor_router():
from app.modules.payments.routes.vendor import vendor_router
return vendor_router
payments_module = ModuleDefinition(
code="payments",
# ...
)
def get_payments_module_with_routers():
"""Get module with routers attached."""
payments_module.admin_router = _get_admin_router()
payments_module.vendor_router = _get_vendor_router()
return payments_module
```
## Module Migrations
### Naming Convention
```
{module_code}_{sequence}_{description}.py
```
Examples:
- `cms_001_create_content_pages.py`
- `cms_002_add_seo_fields.py`
- `billing_001_create_subscriptions.py`
### Migration Template
```python
# app/modules/cms/migrations/versions/cms_001_create_content_pages.py
"""Create content pages table.
Revision ID: cms_001
Revises:
Create Date: 2026-01-27
"""
from alembic import op
import sqlalchemy as sa
revision = "cms_001"
down_revision = None # Or previous module migration
branch_labels = ("cms",)
depends_on = None
def upgrade() -> None:
op.create_table(
"cms_content_pages",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("vendor_id", sa.Integer(), sa.ForeignKey("vendors.id")),
sa.Column("title", sa.String(200), nullable=False),
sa.Column("slug", sa.String(200), nullable=False),
sa.Column("content", sa.Text()),
sa.Column("created_at", sa.DateTime(timezone=True)),
sa.Column("updated_at", sa.DateTime(timezone=True)),
)
def downgrade() -> None:
op.drop_table("cms_content_pages")
```
## Module Configuration
### Defining Configuration Schema
```python
from pydantic import BaseModel, Field
class BillingConfig(BaseModel):
"""Billing module configuration."""
stripe_mode: Literal["test", "live"] = "test"
trial_days: int = Field(default=14, ge=0, le=365)
enable_invoices: bool = True
invoice_prefix: str = Field(default="INV-", max_length=10)
```
### Using Configuration
```python
from app.modules.service import module_service
# Get module config for platform
config = module_service.get_module_config(db, platform_id, "billing")
# Returns: {"stripe_mode": "test", "trial_days": 14, ...}
# Set module config
module_service.set_module_config(
db, platform_id, "billing",
{"trial_days": 30}
)
```
## Health Checks
### Defining Health Check
```python
def check_billing_health() -> dict:
"""Check billing service dependencies."""
issues = []
# Check Stripe
try:
stripe.Account.retrieve()
except stripe.AuthenticationError:
issues.append("Stripe authentication failed")
if issues:
return {
"status": "unhealthy",
"message": "; ".join(issues),
}
return {
"status": "healthy",
"details": {"stripe": "connected"},
}
billing_module = ModuleDefinition(
code="billing",
health_check=check_billing_health,
# ...
)
```
## Registering the Module
### Add to Registry
```python
# app/modules/registry.py
from app.modules.analytics.definition import analytics_module
OPTIONAL_MODULES: dict[str, ModuleDefinition] = {
# ... existing modules
"analytics": analytics_module,
}
MODULES = {**CORE_MODULES, **OPTIONAL_MODULES, **INTERNAL_MODULES}
```
## Module Events
Subscribe to module lifecycle events:
```python
from app.modules.events import module_event_bus, ModuleEvent, ModuleEventData
@module_event_bus.subscribe(ModuleEvent.ENABLED)
def on_analytics_enabled(data: ModuleEventData):
if data.module_code == "analytics":
# Initialize analytics for this platform
setup_analytics_dashboard(data.platform_id)
@module_event_bus.subscribe(ModuleEvent.DISABLED)
def on_analytics_disabled(data: ModuleEventData):
if data.module_code == "analytics":
# Cleanup
cleanup_analytics_data(data.platform_id)
```
## Checklist
### New Module Checklist
- [ ] Create module directory: `app/modules/{code}/`
- [ ] Create `definition.py` with ModuleDefinition
- [ ] Add module to appropriate dict in `registry.py`
- [ ] Create routes if needed (admin.py, vendor.py)
- [ ] Register menu items in menu registry
- [ ] Create migrations if adding database tables
- [ ] Add health check if module has dependencies
- [ ] Document features and configuration options
- [ ] Write tests for module functionality
### Self-Contained Module Checklist
- [ ] All items from basic checklist
- [ ] Create `services/` directory with business logic
- [ ] Create `models/` directory with SQLAlchemy models
- [ ] Create `schemas/` directory with Pydantic schemas
- [ ] Create `exceptions.py` for module-specific errors
- [ ] Create `config.py` with Pydantic config model
- [ ] Set up `migrations/versions/` directory
- [ ] Create `templates/` if module has UI
- [ ] Create `locales/` if module needs translations
- [ ] Set `is_self_contained=True` and path attributes
## Related Documentation
- [Module System](../architecture/module-system.md) - Architecture overview
- [Menu Management](../architecture/menu-management.md) - Menu item integration
- [Database Migrations](migration/database-migrations.md) - Migration guide

596
docs/proposals/module-migration-plan.md Normal file
View File

@@ -0,0 +1,596 @@
# Module Migration Plan
**Status:** In Progress
**Started:** 2026-01-25
**Last Updated:** 2026-01-27
**Target:** v1.0.0
This is the unified migration plan for transforming Wizamart into a fully modular architecture.
---
## Executive Summary
Transform the platform from a monolithic structure to self-contained modules where each module owns its:
- Services (business logic)
- Models (database entities)
- Schemas (Pydantic DTOs)
- Tasks (Celery background jobs)
- Templates (UI if applicable)
- Migrations (database changes)
---
## Current State Assessment
### Module Migration Status
| Module | Classification | Current State | Services | Models | Tasks | Target |
|--------|---------------|---------------|----------|--------|-------|--------|
| `cms` | Core | ✅ **Complete** | ✅ | ✅ | - | Done |
| `payments` | Optional | 🟡 Partial | ✅ | ✅ | - | Done |
| `billing` | Optional | 🔴 Shell | ❌ | ❌ | ❌ | Full |
| `marketplace` | Optional | 🔴 Shell | ❌ | ❌ | ❌ | Full |
| `orders` | Optional | 🔴 Shell | ❌ | ❌ | - | Full |
| `inventory` | Optional | 🔴 Shell | ❌ | ❌ | - | Full |
| `customers` | Core | 🔴 Shell | ❌ | ❌ | - | Full |
| `analytics` | Optional | 🔴 Shell | ❌ | ❌ | - | Full |
| `messaging` | Optional | 🔴 Shell | ❌ | ❌ | - | Full |
| `monitoring` | Internal | 🔴 Shell | ❌ | ❌ | ❌ | Full |
| `dev-tools` | Internal | 🔴 Shell | ❌ | ❌ | ❌ | Full |
| `tenancy` | Core | 🔴 Shell | ❌ | ❌ | - | Full |
| `core` | Core | 🔴 Shell | ❌ | ❌ | - | Minimal |
### Current Code Locations (To Be Migrated)
```
app/services/ # → Move to respective modules
├── subscription_service.py # → billing/services/
├── stripe_service.py # → billing/services/ (or payments)
├── customer_service.py # → customers/services/
├── order_service.py # → orders/services/
├── inventory_service.py # → inventory/services/
├── letzshop_service.py # → marketplace/services/
├── marketplace_import_service.py # → marketplace/services/
├── messaging_service.py # → messaging/services/
└── ...
models/database/ # → Move to respective modules
├── subscription.py # → billing/models/
├── customer.py # → customers/models/
├── order.py # → orders/models/
├── inventory.py # → inventory/models/
├── letzshop_*.py # → marketplace/models/
└── ...
app/tasks/celery_tasks/ # → Move to respective modules
├── subscription.py # → billing/tasks/
├── marketplace.py # → marketplace/tasks/
├── letzshop.py # → marketplace/tasks/
├── export.py # → marketplace/tasks/
├── code_quality.py # → dev_tools/tasks/
└── test_runner.py # → dev_tools/tasks/
```
### Celery Task Mapping
| Current Task | Current Location | Target Module |
|--------------|------------------|---------------|
| `reset_period_counters` | subscription.py | `billing` |
| `check_trial_expirations` | subscription.py | `billing` |
| `sync_stripe_status` | subscription.py | `billing` |
| `cleanup_stale_subscriptions` | subscription.py | `billing` |
| `capture_capacity_snapshot` | subscription.py | `monitoring` |
| `process_marketplace_import` | marketplace.py | `marketplace` |
| `process_historical_import` | letzshop.py | `marketplace` |
| `sync_vendor_directory` | letzshop.py | `marketplace` |
| `export_vendor_products_to_folder` | export.py | `marketplace` |
| `export_marketplace_products` | export.py | `marketplace` |
| `execute_code_quality_scan` | code_quality.py | `dev-tools` |
| `execute_test_run` | test_runner.py | `dev-tools` |
---
## Infrastructure Work (Completed)
### ✅ Phase 1: Module Classification
- Three-tier system: Core, Optional, Internal
- Renamed `platform-admin` → `tenancy`
- Module registry with CORE_MODULES, OPTIONAL_MODULES, INTERNAL_MODULES
### ✅ Phase 2: Module Infrastructure
- `ModuleDefinition` with lifecycle hooks
- Module events system (`events.py`)
- Module migration discovery (`migrations.py`)
- Observability framework (`observability.py`)
### ✅ Phase 3: Documentation
- Module system architecture docs
- Menu management docs
- Observability docs
- Creating modules developer guide
---
## Infrastructure Work (Remaining)
### Phase 4: Celery Task Infrastructure
Add task support to module system before migrating modules.
#### 4.1 Update ModuleDefinition
```python
# app/modules/base.py - Add fields
@dataclass
class ScheduledTask:
"""Celery Beat scheduled task definition."""
name: str # e.g., "billing.reset_counters"
task: str # Full task path
schedule: str | dict # Cron string or crontab dict
args: tuple = ()
kwargs: dict = field(default_factory=dict)
@dataclass
class ModuleDefinition:
# ... existing fields ...
# Task configuration (NEW)
tasks_path: str | None = None
scheduled_tasks: list[ScheduledTask] = field(default_factory=list)
```
#### 4.2 Create Task Discovery
```python
# app/modules/tasks.py (NEW)
def discover_module_tasks() -> list[str]:
"""Discover task modules from all registered modules."""
...
def build_beat_schedule() -> dict:
"""Build Celery Beat schedule from module definitions."""
...
```
#### 4.3 Create Module Task Base
```python
# app/modules/task_base.py (NEW)
class ModuleTask(Task):
"""Base Celery task with DB session management."""
@contextmanager
def get_db(self):
...
```
#### 4.4 Update Celery Config
```python
# app/core/celery_config.py
from app.modules.tasks import discover_module_tasks, build_beat_schedule
# Auto-discover from modules
celery_app.autodiscover_tasks(discover_module_tasks())
# Build schedule from modules
celery_app.conf.beat_schedule = build_beat_schedule()
```
**Files to create:**
- `app/modules/tasks.py`
- `app/modules/task_base.py`
**Files to modify:**
- `app/modules/base.py`
- `app/core/celery_config.py`
---
## Module Migration Phases
Each module migration includes: services, models, schemas, tasks, templates (if any).
### Phase 5: Billing Module (High Priority)
**Why first:** Has most Celery tasks, critical business logic.
#### Components to Move
| From | To | Type |
|------|-----|------|
| `app/services/subscription_service.py` | `app/modules/billing/services/` | Service |
| `app/services/stripe_service.py` | `app/modules/billing/services/` | Service |
| `models/database/subscription.py` | `app/modules/billing/models/` | Model |
| `models/database/subscription_tier.py` | `app/modules/billing/models/` | Model |
| `models/schema/subscription.py` | `app/modules/billing/schemas/` | Schema |
| `app/tasks/celery_tasks/subscription.py` | `app/modules/billing/tasks/` | Tasks |
#### Target Structure
```
app/modules/billing/
├── __init__.py
├── definition.py # Update with tasks_path, scheduled_tasks
├── config.py # BillingConfig schema
├── exceptions.py # BillingException, SubscriptionError, etc.
├── services/
│ ├── __init__.py
│ ├── subscription_service.py
│ └── stripe_service.py
├── models/
│ ├── __init__.py
│ ├── subscription.py
│ └── subscription_tier.py
├── schemas/
│ ├── __init__.py
│ └── subscription.py
├── tasks/
│ ├── __init__.py
│ └── subscription.py # 4 scheduled tasks
├── routes/
│ ├── admin.py
│ └── vendor.py
└── migrations/
└── versions/
```
#### Tasks
| Task | Schedule | Queue |
|------|----------|-------|
| `reset_period_counters` | Daily 00:05 | scheduled |
| `check_trial_expirations` | Daily 01:00 | scheduled |
| `sync_stripe_status` | Hourly :30 | scheduled |
| `cleanup_stale_subscriptions` | Weekly Sun 03:00 | scheduled |
#### Migration Checklist
- [ ] Create `billing/services/` directory
- [ ] Move `subscription_service.py` → update imports
- [ ] Move `stripe_service.py` → update imports
- [ ] Create `billing/models/` directory
- [ ] Move subscription models → update imports
- [ ] Create `billing/schemas/` directory
- [ ] Move subscription schemas → update imports
- [ ] Create `billing/tasks/` directory
- [ ] Move subscription tasks → update task paths
- [ ] Create `billing/exceptions.py`
- [ ] Update `definition.py` with `tasks_path`, `scheduled_tasks`
- [ ] Create module migrations
- [ ] Update all imports across codebase
- [ ] Test subscription flows
- [ ] Test scheduled tasks
---
### Phase 6: Marketplace Module (High Priority)
**Why second:** Has most tasks, complex import logic.
#### Components to Move
| From | To | Type |
|------|-----|------|
| `app/services/letzshop_service.py` | `app/modules/marketplace/services/` | Service |
| `app/services/marketplace_import_service.py` | `app/modules/marketplace/services/` | Service |
| `app/services/letzshop_credentials_service.py` | `app/modules/marketplace/services/` | Service |
| `models/database/letzshop_*.py` | `app/modules/marketplace/models/` | Models |
| `models/database/marketplace_import_job.py` | `app/modules/marketplace/models/` | Model |
| `app/tasks/celery_tasks/marketplace.py` | `app/modules/marketplace/tasks/` | Tasks |
| `app/tasks/celery_tasks/letzshop.py` | `app/modules/marketplace/tasks/` | Tasks |
| `app/tasks/celery_tasks/export.py` | `app/modules/marketplace/tasks/` | Tasks |
#### Target Structure
```
app/modules/marketplace/
├── __init__.py
├── definition.py
├── exceptions.py
├── services/
│ ├── __init__.py
│ ├── letzshop_service.py
│ ├── letzshop_credentials_service.py
│ ├── import_service.py
│ └── export_service.py
├── models/
│ ├── __init__.py
│ ├── letzshop_order.py
│ ├── letzshop_credentials.py
│ └── import_job.py
├── schemas/
│ └── ...
├── tasks/
│ ├── __init__.py
│ ├── import_tasks.py # process_marketplace_import
│ ├── letzshop.py # process_historical_import, sync_vendor_directory
│ └── export.py # export tasks
└── routes/
```
#### Tasks
| Task | Type | Queue |
|------|------|-------|
| `process_marketplace_import` | On-demand | long_running |
| `process_historical_import` | On-demand | long_running |
| `sync_vendor_directory` | Scheduled (optional) | long_running |
| `export_vendor_products_to_folder` | On-demand | default |
| `export_marketplace_products` | On-demand | default |
---
### Phase 7: Dev-Tools Module (Medium Priority)
**Why:** Internal module with code quality and test runner tasks.
#### Components to Move
| From | To | Type |
|------|-----|------|
| `app/services/code_quality_service.py` | `app/modules/dev_tools/services/` | Service |
| `app/services/test_runner_service.py` | `app/modules/dev_tools/services/` | Service |
| `models/database/architecture_scan.py` | `app/modules/dev_tools/models/` | Model |
| `models/database/test_run.py` | `app/modules/dev_tools/models/` | Model |
| `app/tasks/celery_tasks/code_quality.py` | `app/modules/dev_tools/tasks/` | Tasks |
| `app/tasks/celery_tasks/test_runner.py` | `app/modules/dev_tools/tasks/` | Tasks |
#### Target Structure
```
app/modules/dev_tools/
├── __init__.py
├── definition.py
├── services/
│ ├── code_quality_service.py
│ └── test_runner_service.py
├── models/
│ ├── architecture_scan.py
│ ├── architecture_violation.py
│ └── test_run.py
├── tasks/
│ ├── code_quality.py
│ └── test_runner.py
└── routes/
```
---
### Phase 8: Monitoring Module (Medium Priority)
#### Components to Move
| From | To | Type |
|------|-----|------|
| `app/services/background_tasks_service.py` | `app/modules/monitoring/services/` | Service |
| `capture_capacity_snapshot` task | `app/modules/monitoring/tasks/` | Task |
| `models/database/capacity_snapshot.py` | `app/modules/monitoring/models/` | Model |
#### Target Structure
```
app/modules/monitoring/
├── __init__.py
├── definition.py
├── services/
│ └── background_tasks_service.py
├── models/
│ └── capacity_snapshot.py
├── tasks/
│ └── capacity.py # capture_capacity_snapshot
└── routes/
```
---
### Phase 9: Customers Module (Core)
#### Components to Move
| From | To |
|------|-----|
| `app/services/customer_service.py` | `app/modules/customers/services/` |
| `models/database/customer.py` | `app/modules/customers/models/` |
| Customer-related schemas | `app/modules/customers/schemas/` |
---
### Phase 10: Orders Module
#### Components to Move
| From | To |
|------|-----|
| `app/services/order_service.py` | `app/modules/orders/services/` |
| `models/database/order.py` | `app/modules/orders/models/` |
| Order-related schemas | `app/modules/orders/schemas/` |
---
### Phase 11: Inventory Module
#### Components to Move
| From | To |
|------|-----|
| `app/services/inventory_service.py` | `app/modules/inventory/services/` |
| `models/database/inventory*.py` | `app/modules/inventory/models/` |
---
### Phase 12: Remaining Modules
- **Analytics** - Move analytics services
- **Messaging** - Move messaging service, notification models
- **Tenancy** - Move platform, company, vendor services
---
## Phase 13: Cleanup & Full Celery Migration
After all modules are migrated:
### 13.1 Remove Fallback Tasks
Delete:
```
app/tasks/background_tasks.py
app/tasks/letzshop_tasks.py
app/tasks/subscription_tasks.py
app/tasks/code_quality_tasks.py
app/tasks/test_runner_tasks.py
```
### 13.2 Remove USE_CELERY Flag
```python
# app/core/config.py
# DELETE: USE_CELERY: bool = False
```
### 13.3 Simplify Dispatcher
Rewrite `app/tasks/dispatcher.py` to Celery-only.
### 13.4 Delete Old Task Directory
```
app/tasks/celery_tasks/ # DELETE - moved to modules
```
### 13.5 Clean Up Old Services/Models
Remove files from:
- `app/services/` (moved to modules)
- `models/database/` (moved to modules)
Leave re-exports for backward compatibility if needed.
---
## Queue Configuration
### Final Queue Structure
| Queue | Purpose | Modules |
|-------|---------|---------|
| `default` | Fast tasks | marketplace (exports) |
| `long_running` | CPU-intensive | marketplace (imports), dev-tools |
| `scheduled` | Celery Beat | billing, monitoring |
### Task Routing
```python
task_routes = {
"app.modules.billing.tasks.*": {"queue": "scheduled"},
"app.modules.marketplace.tasks.import_tasks.*": {"queue": "long_running"},
"app.modules.marketplace.tasks.letzshop.*": {"queue": "long_running"},
"app.modules.marketplace.tasks.export.*": {"queue": "default"},
"app.modules.monitoring.tasks.*": {"queue": "scheduled"},
"app.modules.dev_tools.tasks.*": {"queue": "long_running"},
}
```
---
## Migration Strategy
### For Each Module
1. **Create directory structure**
2. **Move models first** (fewest dependencies)
3. **Move services** (depend on models)
4. **Move schemas** (depend on models)
5. **Move tasks** (depend on services)
6. **Update all imports** across codebase
7. **Create re-exports** in old locations (temporary)
8. **Test thoroughly**
9. **Remove re-exports** in next phase
### Import Update Strategy
Use IDE refactoring or script:
```bash
# Example: Update subscription_service imports
find . -name "*.py" -exec sed -i 's/from app.services.subscription_service/from app.modules.billing.services.subscription_service/g' {} \;
```
---
## Verification Checklist
### Per-Module Verification
- [ ] Module directory has all components
- [ ] All imports updated
- [ ] Services work
- [ ] API routes work
- [ ] Tasks execute correctly
- [ ] Scheduled tasks run (Celery Beat)
- [ ] No circular imports
- [ ] Tests pass
### Final Verification
- [ ] All 12 tasks in modules
- [ ] `USE_CELERY` flag removed
- [ ] Fallback tasks deleted
- [ ] Old service files removed
- [ ] Old model files removed
- [ ] All tests pass
- [ ] Production smoke test
---
## Priority Order
| Priority | Phase | Module | Reason |
|----------|-------|--------|--------|
| 1 | 4 | Infrastructure | Required for task migration |
| 2 | 5 | `billing` | Most tasks, critical business logic |
| 3 | 6 | `marketplace` | Most complex, many tasks |
| 4 | 7 | `dev-tools` | Internal, low risk |
| 5 | 8 | `monitoring` | Internal, low risk |
| 6 | 9-12 | Others | Lower complexity |
| 7 | 13 | Cleanup | Final step |
---
## Estimated Effort
| Phase | Sessions | Notes |
|-------|----------|-------|
| 4 (Infrastructure) | 1 | Task base class, discovery |
| 5 (Billing) | 2-3 | Complex, many components |
| 6 (Marketplace) | 3-4 | Most complex module |
| 7-8 (Internal) | 1-2 | Simpler modules |
| 9-12 (Others) | 4-6 | Medium complexity each |
| 13 (Cleanup) | 1 | Removing old code |
**Total:** ~12-17 sessions
---
## Rollback Plan
Each phase can be rolled back independently:
1. Restore moved files to original locations
2. Revert import changes
3. Restore celery_config.py
No database schema changes during code migration = no DB rollback needed.
---
## Related Documents
- [Module System Architecture](../architecture/module-system.md)
- [Creating Modules](../development/creating-modules.md)
- [Observability](../architecture/observability.md)
- [SESSION_NOTE_2026-01-27](SESSION_NOTE_2026-01-27_module-reclassification.md)

3
mkdocs.yml
View File

@@ -41,6 +41,7 @@ nav:
- Middleware Stack: architecture/middleware.md - Middleware Stack: architecture/middleware.md
- Module System: architecture/module-system.md - Module System: architecture/module-system.md
- Menu Management: architecture/menu-management.md - Menu Management: architecture/menu-management.md
- Observability: architecture/observability.md
- Request Flow: architecture/request-flow.md - Request Flow: architecture/request-flow.md
- Authentication & RBAC: architecture/auth-rbac.md - Authentication & RBAC: architecture/auth-rbac.md
- Frontend Structure: architecture/frontend-structure.md - Frontend Structure: architecture/frontend-structure.md
@@ -115,6 +116,7 @@ nav:
# --- Development --- # --- Development ---
- Development: - Development:
- Contributing Guide: development/contributing.md - Contributing Guide: development/contributing.md
- Creating Modules: development/creating-modules.md
- Code Quality: development/code-quality.md - Code Quality: development/code-quality.md
- Architecture Rules: development/architecture-rules.md - Architecture Rules: development/architecture-rules.md
- Security Rules: development/security-rules.md - Security Rules: development/security-rules.md
@@ -157,6 +159,7 @@ nav:
- Vendor Operations Expansion: development/migration/vendor-operations-expansion.md - Vendor Operations Expansion: development/migration/vendor-operations-expansion.md
- Implementation Plans: - Implementation Plans:
- Module Migration Plan: proposals/module-migration-plan.md
- Admin Inventory Management: implementation/inventory-admin-migration.md - Admin Inventory Management: implementation/inventory-admin-migration.md
- Admin Notification System: implementation/admin-notification-system.md - Admin Notification System: implementation/admin-notification-system.md
- Letzshop Order Import: implementation/letzshop-order-import-improvements.md - Letzshop Order Import: implementation/letzshop-order-import-improvements.md