docs: add observability, creating modules guide, and unified migration plan
- Add observability framework documentation (health checks, metrics, Sentry) - Add developer guide for creating modules - Add comprehensive module migration plan with Celery task integration - Update architecture overview with module system and observability sections - Update module-system.md with links to new docs Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -417,5 +417,7 @@ register_module_health_checks()
|
||||
## Related Documentation
|
||||
|
||||
- [Menu Management](menu-management.md) - Sidebar and menu configuration
|
||||
- [Creating Modules](../development/creating-modules.md) - Developer guide for building modules
|
||||
- [Observability](observability.md) - Health checks and module health integration
|
||||
- [Multi-Tenant System](multi-tenant.md) - Platform isolation
|
||||
- [Feature Gating](../implementation/feature-gating-system.md) - Tier-based access
|
||||
|
||||
429
docs/architecture/observability.md
Normal file
429
docs/architecture/observability.md
Normal file
@@ -0,0 +1,429 @@
|
||||
# Observability Framework
|
||||
|
||||
The Wizamart platform includes a comprehensive observability framework for monitoring application health, collecting metrics, and tracking errors. This is part of the Framework Layer - infrastructure that modules depend on.
|
||||
|
||||
## Overview
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ OBSERVABILITY FRAMEWORK │
|
||||
│ app/core/observability.py │
|
||||
│ │
|
||||
│ ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ │
|
||||
│ │ Health Checks │ │ Prometheus │ │ Sentry │ │
|
||||
│ │ Registry │ │ Metrics │ │ Integration │ │
|
||||
│ └────────┬────────┘ └────────┬────────┘ └────────┬────────┘ │
|
||||
│ │ │ │ │
|
||||
│ ▼ ▼ ▼ │
|
||||
│ ┌─────────────────────────────────────────────────────────────────┐ │
|
||||
│ │ API Endpoints │ │
|
||||
│ │ /health │ /health/live │ /health/ready │ /metrics │ │
|
||||
│ └─────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌───────────────────────────────┐
|
||||
│ External Tools │
|
||||
│ Flower │ Grafana │ Prometheus│
|
||||
└───────────────────────────────┘
|
||||
```
|
||||
|
||||
## Health Checks
|
||||
|
||||
### Health Check Registry
|
||||
|
||||
Components register health checks that are aggregated into a single endpoint.
|
||||
|
||||
```python
|
||||
from app.core.observability import (
|
||||
health_registry,
|
||||
HealthCheckResult,
|
||||
HealthStatus,
|
||||
)
|
||||
|
||||
# Register using decorator
|
||||
@health_registry.register("database")
|
||||
def check_database() -> HealthCheckResult:
|
||||
try:
|
||||
db.execute("SELECT 1")
|
||||
return HealthCheckResult(
|
||||
name="database",
|
||||
status=HealthStatus.HEALTHY,
|
||||
message="Database connection OK"
|
||||
)
|
||||
except Exception as e:
|
||||
return HealthCheckResult(
|
||||
name="database",
|
||||
status=HealthStatus.UNHEALTHY,
|
||||
message=str(e)
|
||||
)
|
||||
|
||||
# Or register directly
|
||||
health_registry.register_check("redis", check_redis_connection)
|
||||
```
|
||||
|
||||
### Health Status Levels
|
||||
|
||||
| Status | Description | HTTP Response |
|
||||
|--------|-------------|---------------|
|
||||
| `HEALTHY` | All systems operational | 200 |
|
||||
| `DEGRADED` | Partial functionality available | 200 |
|
||||
| `UNHEALTHY` | Critical failure | 200 (check response body) |
|
||||
|
||||
### HealthCheckResult Fields
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `name` | str | Check identifier |
|
||||
| `status` | HealthStatus | Health status level |
|
||||
| `message` | str | Optional status message |
|
||||
| `latency_ms` | float | Check execution time |
|
||||
| `details` | dict | Additional diagnostic data |
|
||||
| `checked_at` | datetime | Timestamp of check |
|
||||
|
||||
## API Endpoints
|
||||
|
||||
### GET /health
|
||||
|
||||
Aggregated health check endpoint. Returns combined status from all registered checks.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"timestamp": "2026-01-27T10:30:00Z",
|
||||
"checks": [
|
||||
{
|
||||
"name": "database",
|
||||
"status": "healthy",
|
||||
"message": "Connection OK",
|
||||
"latency_ms": 2.5,
|
||||
"details": {}
|
||||
},
|
||||
{
|
||||
"name": "module:billing",
|
||||
"status": "healthy",
|
||||
"message": "",
|
||||
"latency_ms": 0.1,
|
||||
"details": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Overall Status Logic:**
|
||||
- If any check is `UNHEALTHY` → overall is `UNHEALTHY`
|
||||
- If any check is `DEGRADED` and none `UNHEALTHY` → overall is `DEGRADED`
|
||||
- Otherwise → `HEALTHY`
|
||||
|
||||
### GET /health/live
|
||||
|
||||
Kubernetes liveness probe. Returns 200 if application is running.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{"status": "alive"}
|
||||
```
|
||||
|
||||
### GET /health/ready
|
||||
|
||||
Kubernetes readiness probe. Returns ready status based on health checks.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "ready",
|
||||
"health": "healthy"
|
||||
}
|
||||
```
|
||||
|
||||
Or if unhealthy:
|
||||
```json
|
||||
{
|
||||
"status": "not_ready",
|
||||
"health": "unhealthy"
|
||||
}
|
||||
```
|
||||
|
||||
### GET /metrics
|
||||
|
||||
Prometheus metrics endpoint. Returns metrics in Prometheus text format.
|
||||
|
||||
**Response:**
|
||||
```
|
||||
# HELP http_requests_total Total HTTP requests
|
||||
# TYPE http_requests_total counter
|
||||
http_requests_total{method="GET",endpoint="/api/products",status="200"} 1234
|
||||
...
|
||||
```
|
||||
|
||||
### GET /health/tools
|
||||
|
||||
Returns URLs to external monitoring tools.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"flower": "http://flower.example.com:5555",
|
||||
"grafana": "http://grafana.example.com:3000",
|
||||
"prometheus": null
|
||||
}
|
||||
```
|
||||
|
||||
## Prometheus Metrics
|
||||
|
||||
### MetricsRegistry
|
||||
|
||||
The metrics registry provides a wrapper around `prometheus_client` with fallback when the library isn't installed.
|
||||
|
||||
```python
|
||||
from app.core.observability import metrics_registry
|
||||
|
||||
# Counter - tracks cumulative values
|
||||
request_counter = metrics_registry.counter(
|
||||
"http_requests_total",
|
||||
"Total HTTP requests",
|
||||
["method", "endpoint", "status"]
|
||||
)
|
||||
request_counter.labels(method="GET", endpoint="/api/products", status="200").inc()
|
||||
|
||||
# Histogram - tracks distributions (latency, sizes)
|
||||
request_latency = metrics_registry.histogram(
|
||||
"http_request_duration_seconds",
|
||||
"HTTP request latency",
|
||||
["endpoint"],
|
||||
buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 5.0]
|
||||
)
|
||||
request_latency.labels(endpoint="/api/products").observe(0.045)
|
||||
|
||||
# Gauge - tracks current values
|
||||
active_connections = metrics_registry.gauge(
|
||||
"active_connections",
|
||||
"Number of active connections",
|
||||
["pool"]
|
||||
)
|
||||
active_connections.labels(pool="database").set(42)
|
||||
```
|
||||
|
||||
### Enabling Metrics
|
||||
|
||||
Metrics are disabled by default. Enable during initialization:
|
||||
|
||||
```python
|
||||
from app.core.observability import init_observability
|
||||
|
||||
init_observability(
|
||||
enable_metrics=True,
|
||||
# ... other options
|
||||
)
|
||||
```
|
||||
|
||||
### Dummy Metrics
|
||||
|
||||
When `prometheus_client` isn't installed or metrics are disabled, the registry returns dummy metrics that silently ignore all operations. This allows code to use metrics without checking if they're enabled.
|
||||
|
||||
## Sentry Integration
|
||||
|
||||
### Configuration
|
||||
|
||||
```python
|
||||
from app.core.observability import sentry, init_observability
|
||||
|
||||
# Initialize via init_observability
|
||||
init_observability(
|
||||
sentry_dsn="https://key@sentry.io/project",
|
||||
environment="production",
|
||||
)
|
||||
|
||||
# Or initialize directly
|
||||
sentry.init(
|
||||
dsn="https://key@sentry.io/project",
|
||||
environment="production",
|
||||
release="1.0.0"
|
||||
)
|
||||
```
|
||||
|
||||
### Capturing Errors
|
||||
|
||||
```python
|
||||
from app.core.observability import sentry
|
||||
|
||||
try:
|
||||
risky_operation()
|
||||
except Exception as e:
|
||||
event_id = sentry.capture_exception(e)
|
||||
logger.error(f"Operation failed, Sentry event: {event_id}")
|
||||
|
||||
# Capture messages
|
||||
sentry.capture_message("User reached rate limit", level="warning")
|
||||
```
|
||||
|
||||
### Without Sentry
|
||||
|
||||
If `sentry_sdk` isn't installed or DSN isn't provided, all capture calls silently return `None`.
|
||||
|
||||
## Module Health Checks
|
||||
|
||||
Modules can provide health checks that are automatically registered.
|
||||
|
||||
### Defining Module Health Check
|
||||
|
||||
```python
|
||||
# In module definition
|
||||
from app.modules.base import ModuleDefinition
|
||||
|
||||
def check_billing_health() -> dict:
|
||||
"""Check billing service health."""
|
||||
try:
|
||||
# Check Stripe connection
|
||||
stripe.Account.retrieve()
|
||||
return {"status": "healthy", "message": "Stripe connected"}
|
||||
except Exception as e:
|
||||
return {"status": "unhealthy", "message": str(e)}
|
||||
|
||||
billing_module = ModuleDefinition(
|
||||
code="billing",
|
||||
name="Billing",
|
||||
health_check=check_billing_health,
|
||||
# ...
|
||||
)
|
||||
```
|
||||
|
||||
### Registering Module Health Checks
|
||||
|
||||
```python
|
||||
from app.core.observability import register_module_health_checks
|
||||
|
||||
# Call after modules are loaded (e.g., in app lifespan)
|
||||
register_module_health_checks()
|
||||
```
|
||||
|
||||
This registers health checks as `module:{code}` (e.g., `module:billing`).
|
||||
|
||||
## External Tools
|
||||
|
||||
### Flower (Celery Monitoring)
|
||||
|
||||
Configure Flower URL to include in `/health/tools`:
|
||||
|
||||
```python
|
||||
init_observability(
|
||||
flower_url="http://flower:5555",
|
||||
)
|
||||
```
|
||||
|
||||
### Grafana
|
||||
|
||||
Configure Grafana URL:
|
||||
|
||||
```python
|
||||
init_observability(
|
||||
grafana_url="http://grafana:3000",
|
||||
)
|
||||
```
|
||||
|
||||
## Initialization
|
||||
|
||||
### Application Lifespan
|
||||
|
||||
```python
|
||||
# main.py
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import FastAPI
|
||||
from app.core.observability import (
|
||||
init_observability,
|
||||
shutdown_observability,
|
||||
health_router,
|
||||
register_module_health_checks,
|
||||
)
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
# Startup
|
||||
init_observability(
|
||||
enable_metrics=True,
|
||||
sentry_dsn=settings.SENTRY_DSN,
|
||||
environment=settings.ENVIRONMENT,
|
||||
flower_url=settings.FLOWER_URL,
|
||||
grafana_url=settings.GRAFANA_URL,
|
||||
)
|
||||
register_module_health_checks()
|
||||
|
||||
yield
|
||||
|
||||
# Shutdown
|
||||
shutdown_observability()
|
||||
|
||||
app = FastAPI(lifespan=lifespan)
|
||||
app.include_router(health_router)
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
| Variable | Description | Default |
|
||||
|----------|-------------|---------|
|
||||
| `SENTRY_DSN` | Sentry DSN for error tracking | None (disabled) |
|
||||
| `ENVIRONMENT` | Environment name | "development" |
|
||||
| `ENABLE_METRICS` | Enable Prometheus metrics | False |
|
||||
| `FLOWER_URL` | Flower dashboard URL | None |
|
||||
| `GRAFANA_URL` | Grafana dashboard URL | None |
|
||||
|
||||
## Kubernetes Integration
|
||||
|
||||
### Deployment Configuration
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
spec:
|
||||
template:
|
||||
spec:
|
||||
containers:
|
||||
- name: app
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /health/live
|
||||
port: 8000
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 30
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /health/ready
|
||||
port: 8000
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
```
|
||||
|
||||
### Prometheus ServiceMonitor
|
||||
|
||||
```yaml
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: ServiceMonitor
|
||||
spec:
|
||||
endpoints:
|
||||
- port: http
|
||||
path: /metrics
|
||||
interval: 15s
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Do
|
||||
|
||||
- Register health checks for critical dependencies (database, cache, external APIs)
|
||||
- Use appropriate metric types (counter for counts, histogram for latency)
|
||||
- Include meaningful labels but avoid high cardinality
|
||||
- Set up alerts based on health status changes
|
||||
|
||||
### Don't
|
||||
|
||||
- Create health checks that are slow or have side effects
|
||||
- Add high-cardinality labels to metrics (e.g., user IDs)
|
||||
- Ignore Sentry errors in production
|
||||
- Skip readiness probes in Kubernetes deployments
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Module System](module-system.md) - Module health check integration
|
||||
- [Background Tasks](background-tasks.md) - Celery monitoring with Flower
|
||||
- [Deployment](../deployment/index.md) - Production deployment with monitoring
|
||||
@@ -76,7 +76,42 @@ Custom middleware handles:
|
||||
|
||||
**See:** [Authentication & RBAC](auth-rbac.md) for details
|
||||
|
||||
### 4. Request Flow
|
||||
### 4. Module System
|
||||
|
||||
The platform uses a modular architecture with three-tier classification:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ FRAMEWORK LAYER │
|
||||
│ (Infrastructure - always available, not modules) │
|
||||
│ Config │ Database │ Auth │ Permissions │ Observability │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────────────┐
|
||||
│ MODULE LAYER │
|
||||
│ │
|
||||
│ CORE (Always Enabled) OPTIONAL (Per-Platform) │
|
||||
│ ├── core ├── payments │
|
||||
│ ├── tenancy ├── billing │
|
||||
│ ├── cms ├── inventory │
|
||||
│ └── customers ├── orders │
|
||||
│ ├── marketplace │
|
||||
│ INTERNAL (Admin-Only) ├── analytics │
|
||||
│ ├── dev-tools └── messaging │
|
||||
│ └── monitoring │
|
||||
└─────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Module Features:**
|
||||
- Enable/disable modules per platform
|
||||
- Module dependencies (billing requires payments)
|
||||
- Health checks and lifecycle hooks
|
||||
- Self-contained modules with own services, models, migrations
|
||||
|
||||
**See:** [Module System](module-system.md) for complete documentation
|
||||
|
||||
### 5. Request Flow
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
@@ -252,7 +287,15 @@ project/
|
||||
│ ├── api/ # API routes
|
||||
│ ├── routes/ # Page routes (HTML)
|
||||
│ ├── services/ # Business logic
|
||||
│ ├── core/ # Core functionality
|
||||
│ ├── core/ # Core functionality (config, db, observability)
|
||||
│ ├── modules/ # Module definitions and self-contained modules
|
||||
│ │ ├── base.py # ModuleDefinition class
|
||||
│ │ ├── registry.py # Module registry (CORE, OPTIONAL, INTERNAL)
|
||||
│ │ ├── service.py # Module enablement service
|
||||
│ │ ├── events.py # Module event bus
|
||||
│ │ ├── cms/ # Self-contained CMS module
|
||||
│ │ ├── payments/ # Self-contained payments module
|
||||
│ │ └── ... # Other modules
|
||||
│ └── exceptions/ # Custom exceptions
|
||||
│
|
||||
├── middleware/ # Custom middleware
|
||||
@@ -288,6 +331,27 @@ project/
|
||||
|
||||
## Monitoring & Observability
|
||||
|
||||
The platform includes a comprehensive observability framework:
|
||||
|
||||
### Health Checks
|
||||
|
||||
- Aggregated health endpoint (`/health`)
|
||||
- Kubernetes probes (`/health/live`, `/health/ready`)
|
||||
- Module health check integration
|
||||
- External tool links (`/health/tools`)
|
||||
|
||||
### Metrics
|
||||
|
||||
- Prometheus metrics endpoint (`/metrics`)
|
||||
- Request latency histograms
|
||||
- Counter and gauge support
|
||||
|
||||
### Error Tracking
|
||||
|
||||
- Sentry integration for production
|
||||
- Exception capture with context
|
||||
- Environment-aware configuration
|
||||
|
||||
### Logging
|
||||
|
||||
- Structured logging with Python logging module
|
||||
@@ -295,12 +359,7 @@ project/
|
||||
- Error tracking with stack traces
|
||||
- Audit logging for admin actions
|
||||
|
||||
### Performance Monitoring
|
||||
|
||||
- Request timing headers (`X-Process-Time`)
|
||||
- Database query monitoring (SQLAlchemy echo)
|
||||
- Slow query identification
|
||||
- Memory usage tracking
|
||||
**See:** [Observability](observability.md) for complete documentation
|
||||
|
||||
## Deployment Architecture
|
||||
|
||||
@@ -332,6 +391,9 @@ Internet ───────────│ Load Balancer│
|
||||
## Related Documentation
|
||||
|
||||
- [Multi-Tenant System](multi-tenant.md) - Detailed multi-tenancy implementation
|
||||
- [Module System](module-system.md) - Module architecture and classification
|
||||
- [Menu Management](menu-management.md) - Sidebar and menu configuration
|
||||
- [Observability](observability.md) - Health checks, metrics, error tracking
|
||||
- [Middleware Stack](middleware.md) - Complete middleware documentation
|
||||
- [Authentication & RBAC](auth-rbac.md) - Security and access control
|
||||
- [Request Flow](request-flow.md) - Detailed request processing
|
||||
|
||||
483
docs/development/creating-modules.md
Normal file
483
docs/development/creating-modules.md
Normal file
@@ -0,0 +1,483 @@
|
||||
# Creating Modules
|
||||
|
||||
This guide explains how to create new modules for the Wizamart platform, including both simple wrapper modules and self-contained modules with their own services, models, and migrations.
|
||||
|
||||
## Module Types
|
||||
|
||||
| Type | Complexity | Use Case |
|
||||
|------|------------|----------|
|
||||
| **Wrapper Module** | Simple | Groups existing routes and features under a toggleable module |
|
||||
| **Self-Contained Module** | Complex | Full isolation with own services, models, templates, migrations |
|
||||
|
||||
## Quick Start: Wrapper Module
|
||||
|
||||
For a simple module that wraps existing functionality:
|
||||
|
||||
```python
|
||||
# app/modules/analytics/definition.py
|
||||
from app.modules.base import ModuleDefinition
|
||||
from models.database.admin_menu_config import FrontendType
|
||||
|
||||
analytics_module = ModuleDefinition(
|
||||
code="analytics",
|
||||
name="Analytics & Reports",
|
||||
description="Business analytics, reports, and dashboards",
|
||||
version="1.0.0",
|
||||
features=[
|
||||
"analytics_dashboard",
|
||||
"sales_reports",
|
||||
"customer_insights",
|
||||
],
|
||||
menu_items={
|
||||
FrontendType.ADMIN: ["analytics-dashboard", "reports"],
|
||||
FrontendType.VENDOR: ["analytics", "sales-reports"],
|
||||
},
|
||||
is_core=False,
|
||||
is_internal=False,
|
||||
)
|
||||
```
|
||||
|
||||
## Module Definition Fields
|
||||
|
||||
### Required Fields
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `code` | str | Unique identifier (e.g., "billing", "analytics") |
|
||||
| `name` | str | Display name (e.g., "Billing & Subscriptions") |
|
||||
|
||||
### Optional Fields
|
||||
|
||||
| Field | Type | Default | Description |
|
||||
|-------|------|---------|-------------|
|
||||
| `description` | str | "" | What the module provides |
|
||||
| `version` | str | "1.0.0" | Semantic version |
|
||||
| `requires` | list[str] | [] | Module codes this depends on |
|
||||
| `features` | list[str] | [] | Feature codes for tier gating |
|
||||
| `menu_items` | dict | {} | Menu items per frontend type |
|
||||
| `permissions` | list[str] | [] | Permission codes defined |
|
||||
| `is_core` | bool | False | Cannot be disabled if True |
|
||||
| `is_internal` | bool | False | Admin-only if True |
|
||||
|
||||
### Configuration Fields
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `config_schema` | type[BaseModel] | Pydantic model for validation |
|
||||
| `default_config` | dict | Default configuration values |
|
||||
|
||||
### Lifecycle Hooks
|
||||
|
||||
| Field | Signature | Description |
|
||||
|-------|-----------|-------------|
|
||||
| `on_enable` | (platform_id: int) -> None | Called when enabled |
|
||||
| `on_disable` | (platform_id: int) -> None | Called when disabled |
|
||||
| `on_startup` | () -> None | Called on app startup |
|
||||
| `health_check` | () -> dict | Returns health status |
|
||||
|
||||
## Module Classification
|
||||
|
||||
### Core Modules
|
||||
|
||||
Cannot be disabled. Use for essential platform functionality.
|
||||
|
||||
```python
|
||||
core_module = ModuleDefinition(
|
||||
code="tenancy",
|
||||
name="Platform Tenancy",
|
||||
is_core=True, # Cannot be disabled
|
||||
# ...
|
||||
)
|
||||
```
|
||||
|
||||
### Optional Modules
|
||||
|
||||
Can be enabled/disabled per platform.
|
||||
|
||||
```python
|
||||
billing_module = ModuleDefinition(
|
||||
code="billing",
|
||||
name="Billing",
|
||||
is_core=False, # Can be toggled
|
||||
is_internal=False,
|
||||
# ...
|
||||
)
|
||||
```
|
||||
|
||||
### Internal Modules
|
||||
|
||||
Admin-only tools not visible to customers/vendors.
|
||||
|
||||
```python
|
||||
devtools_module = ModuleDefinition(
|
||||
code="dev-tools",
|
||||
name="Developer Tools",
|
||||
is_internal=True, # Admin-only
|
||||
# ...
|
||||
)
|
||||
```
|
||||
|
||||
## Module Dependencies
|
||||
|
||||
Declare dependencies using the `requires` field:
|
||||
|
||||
```python
|
||||
# orders module requires payments
|
||||
orders_module = ModuleDefinition(
|
||||
code="orders",
|
||||
name="Orders",
|
||||
requires=["payments"], # Must have payments enabled
|
||||
# ...
|
||||
)
|
||||
```
|
||||
|
||||
**Dependency Rules:**
|
||||
1. Core modules cannot depend on optional modules
|
||||
2. Enabling a module auto-enables its dependencies
|
||||
3. Disabling a module auto-disables modules that depend on it
|
||||
4. Circular dependencies are not allowed
|
||||
|
||||
## Self-Contained Module Structure
|
||||
|
||||
For modules with their own services, models, and templates:
|
||||
|
||||
```
|
||||
app/modules/cms/
|
||||
├── __init__.py
|
||||
├── definition.py # ModuleDefinition
|
||||
├── config.py # Configuration schema (optional)
|
||||
├── exceptions.py # Module-specific exceptions
|
||||
├── routes/
|
||||
│ ├── __init__.py
|
||||
│ ├── admin.py # Admin API routes
|
||||
│ └── vendor.py # Vendor API routes
|
||||
├── services/
|
||||
│ ├── __init__.py
|
||||
│ └── content_service.py
|
||||
├── models/
|
||||
│ ├── __init__.py
|
||||
│ └── content_page.py
|
||||
├── schemas/
|
||||
│ ├── __init__.py
|
||||
│ └── content.py
|
||||
├── templates/
|
||||
│ ├── admin/
|
||||
│ └── vendor/
|
||||
├── migrations/
|
||||
│ └── versions/
|
||||
│ ├── cms_001_create_content_pages.py
|
||||
│ └── cms_002_add_seo_fields.py
|
||||
└── locales/
|
||||
├── en.json
|
||||
└── fr.json
|
||||
```
|
||||
|
||||
### Self-Contained Definition
|
||||
|
||||
```python
|
||||
# app/modules/cms/definition.py
|
||||
from pydantic import BaseModel, Field
|
||||
from app.modules.base import ModuleDefinition
|
||||
from models.database.admin_menu_config import FrontendType
|
||||
|
||||
class CMSConfig(BaseModel):
|
||||
"""CMS module configuration."""
|
||||
max_pages_per_vendor: int = Field(default=100, ge=1, le=1000)
|
||||
enable_seo: bool = True
|
||||
default_language: str = "en"
|
||||
|
||||
cms_module = ModuleDefinition(
|
||||
# Identity
|
||||
code="cms",
|
||||
name="Content Management",
|
||||
description="Content pages, media library, and themes",
|
||||
version="1.0.0",
|
||||
|
||||
# Classification
|
||||
is_core=True,
|
||||
is_self_contained=True,
|
||||
|
||||
# Features
|
||||
features=[
|
||||
"content_pages",
|
||||
"media_library",
|
||||
"vendor_themes",
|
||||
],
|
||||
|
||||
# Menu items
|
||||
menu_items={
|
||||
FrontendType.ADMIN: ["content-pages", "vendor-themes"],
|
||||
FrontendType.VENDOR: ["content-pages", "media"],
|
||||
},
|
||||
|
||||
# Configuration
|
||||
config_schema=CMSConfig,
|
||||
default_config={
|
||||
"max_pages_per_vendor": 100,
|
||||
"enable_seo": True,
|
||||
},
|
||||
|
||||
# Self-contained paths
|
||||
services_path="app.modules.cms.services",
|
||||
models_path="app.modules.cms.models",
|
||||
schemas_path="app.modules.cms.schemas",
|
||||
templates_path="templates",
|
||||
exceptions_path="app.modules.cms.exceptions",
|
||||
locales_path="locales",
|
||||
migrations_path="migrations",
|
||||
|
||||
# Lifecycle
|
||||
health_check=lambda: {"status": "healthy"},
|
||||
)
|
||||
```
|
||||
|
||||
## Creating Module Routes
|
||||
|
||||
### Admin Routes
|
||||
|
||||
```python
|
||||
# app/modules/payments/routes/admin.py
|
||||
from fastapi import APIRouter, Depends
|
||||
from sqlalchemy.orm import Session
|
||||
from app.api.deps import get_db, get_current_admin_user
|
||||
|
||||
admin_router = APIRouter(prefix="/api/admin/payments", tags=["Admin - Payments"])
|
||||
|
||||
@admin_router.get("/gateways")
|
||||
async def list_gateways(
|
||||
db: Session = Depends(get_db),
|
||||
current_user = Depends(get_current_admin_user),
|
||||
):
|
||||
"""List configured payment gateways."""
|
||||
# Implementation
|
||||
pass
|
||||
```
|
||||
|
||||
### Vendor Routes
|
||||
|
||||
```python
|
||||
# app/modules/payments/routes/vendor.py
|
||||
from fastapi import APIRouter, Depends
|
||||
from app.api.deps import get_db, get_current_vendor_user
|
||||
|
||||
vendor_router = APIRouter(prefix="/api/vendor/payments", tags=["Vendor - Payments"])
|
||||
|
||||
@vendor_router.get("/methods")
|
||||
async def list_payment_methods(
|
||||
db: Session = Depends(get_db),
|
||||
current_user = Depends(get_current_vendor_user),
|
||||
):
|
||||
"""List vendor's stored payment methods."""
|
||||
pass
|
||||
```
|
||||
|
||||
### Lazy Router Loading
|
||||
|
||||
To avoid circular imports, use lazy loading:
|
||||
|
||||
```python
|
||||
# app/modules/payments/definition.py
|
||||
|
||||
def _get_admin_router():
|
||||
"""Lazy import to avoid circular imports."""
|
||||
from app.modules.payments.routes.admin import admin_router
|
||||
return admin_router
|
||||
|
||||
def _get_vendor_router():
|
||||
from app.modules.payments.routes.vendor import vendor_router
|
||||
return vendor_router
|
||||
|
||||
payments_module = ModuleDefinition(
|
||||
code="payments",
|
||||
# ...
|
||||
)
|
||||
|
||||
def get_payments_module_with_routers():
|
||||
"""Get module with routers attached."""
|
||||
payments_module.admin_router = _get_admin_router()
|
||||
payments_module.vendor_router = _get_vendor_router()
|
||||
return payments_module
|
||||
```
|
||||
|
||||
## Module Migrations
|
||||
|
||||
### Naming Convention
|
||||
|
||||
```
|
||||
{module_code}_{sequence}_{description}.py
|
||||
```
|
||||
|
||||
Examples:
|
||||
- `cms_001_create_content_pages.py`
|
||||
- `cms_002_add_seo_fields.py`
|
||||
- `billing_001_create_subscriptions.py`
|
||||
|
||||
### Migration Template
|
||||
|
||||
```python
|
||||
# app/modules/cms/migrations/versions/cms_001_create_content_pages.py
|
||||
"""Create content pages table.
|
||||
|
||||
Revision ID: cms_001
|
||||
Revises:
|
||||
Create Date: 2026-01-27
|
||||
|
||||
"""
|
||||
from alembic import op
|
||||
import sqlalchemy as sa
|
||||
|
||||
revision = "cms_001"
|
||||
down_revision = None # Or previous module migration
|
||||
branch_labels = ("cms",)
|
||||
depends_on = None
|
||||
|
||||
def upgrade() -> None:
|
||||
op.create_table(
|
||||
"cms_content_pages",
|
||||
sa.Column("id", sa.Integer(), primary_key=True),
|
||||
sa.Column("vendor_id", sa.Integer(), sa.ForeignKey("vendors.id")),
|
||||
sa.Column("title", sa.String(200), nullable=False),
|
||||
sa.Column("slug", sa.String(200), nullable=False),
|
||||
sa.Column("content", sa.Text()),
|
||||
sa.Column("created_at", sa.DateTime(timezone=True)),
|
||||
sa.Column("updated_at", sa.DateTime(timezone=True)),
|
||||
)
|
||||
|
||||
def downgrade() -> None:
|
||||
op.drop_table("cms_content_pages")
|
||||
```
|
||||
|
||||
## Module Configuration
|
||||
|
||||
### Defining Configuration Schema
|
||||
|
||||
```python
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
class BillingConfig(BaseModel):
|
||||
"""Billing module configuration."""
|
||||
|
||||
stripe_mode: Literal["test", "live"] = "test"
|
||||
trial_days: int = Field(default=14, ge=0, le=365)
|
||||
enable_invoices: bool = True
|
||||
invoice_prefix: str = Field(default="INV-", max_length=10)
|
||||
```
|
||||
|
||||
### Using Configuration
|
||||
|
||||
```python
|
||||
from app.modules.service import module_service
|
||||
|
||||
# Get module config for platform
|
||||
config = module_service.get_module_config(db, platform_id, "billing")
|
||||
# Returns: {"stripe_mode": "test", "trial_days": 14, ...}
|
||||
|
||||
# Set module config
|
||||
module_service.set_module_config(
|
||||
db, platform_id, "billing",
|
||||
{"trial_days": 30}
|
||||
)
|
||||
```
|
||||
|
||||
## Health Checks
|
||||
|
||||
### Defining Health Check
|
||||
|
||||
```python
|
||||
def check_billing_health() -> dict:
|
||||
"""Check billing service dependencies."""
|
||||
issues = []
|
||||
|
||||
# Check Stripe
|
||||
try:
|
||||
stripe.Account.retrieve()
|
||||
except stripe.AuthenticationError:
|
||||
issues.append("Stripe authentication failed")
|
||||
|
||||
if issues:
|
||||
return {
|
||||
"status": "unhealthy",
|
||||
"message": "; ".join(issues),
|
||||
}
|
||||
|
||||
return {
|
||||
"status": "healthy",
|
||||
"details": {"stripe": "connected"},
|
||||
}
|
||||
|
||||
billing_module = ModuleDefinition(
|
||||
code="billing",
|
||||
health_check=check_billing_health,
|
||||
# ...
|
||||
)
|
||||
```
|
||||
|
||||
## Registering the Module
|
||||
|
||||
### Add to Registry
|
||||
|
||||
```python
|
||||
# app/modules/registry.py
|
||||
|
||||
from app.modules.analytics.definition import analytics_module
|
||||
|
||||
OPTIONAL_MODULES: dict[str, ModuleDefinition] = {
|
||||
# ... existing modules
|
||||
"analytics": analytics_module,
|
||||
}
|
||||
|
||||
MODULES = {**CORE_MODULES, **OPTIONAL_MODULES, **INTERNAL_MODULES}
|
||||
```
|
||||
|
||||
## Module Events
|
||||
|
||||
Subscribe to module lifecycle events:
|
||||
|
||||
```python
|
||||
from app.modules.events import module_event_bus, ModuleEvent, ModuleEventData
|
||||
|
||||
@module_event_bus.subscribe(ModuleEvent.ENABLED)
|
||||
def on_analytics_enabled(data: ModuleEventData):
|
||||
if data.module_code == "analytics":
|
||||
# Initialize analytics for this platform
|
||||
setup_analytics_dashboard(data.platform_id)
|
||||
|
||||
@module_event_bus.subscribe(ModuleEvent.DISABLED)
|
||||
def on_analytics_disabled(data: ModuleEventData):
|
||||
if data.module_code == "analytics":
|
||||
# Cleanup
|
||||
cleanup_analytics_data(data.platform_id)
|
||||
```
|
||||
|
||||
## Checklist
|
||||
|
||||
### New Module Checklist
|
||||
|
||||
- [ ] Create module directory: `app/modules/{code}/`
|
||||
- [ ] Create `definition.py` with ModuleDefinition
|
||||
- [ ] Add module to appropriate dict in `registry.py`
|
||||
- [ ] Create routes if needed (admin.py, vendor.py)
|
||||
- [ ] Register menu items in menu registry
|
||||
- [ ] Create migrations if adding database tables
|
||||
- [ ] Add health check if module has dependencies
|
||||
- [ ] Document features and configuration options
|
||||
- [ ] Write tests for module functionality
|
||||
|
||||
### Self-Contained Module Checklist
|
||||
|
||||
- [ ] All items from basic checklist
|
||||
- [ ] Create `services/` directory with business logic
|
||||
- [ ] Create `models/` directory with SQLAlchemy models
|
||||
- [ ] Create `schemas/` directory with Pydantic schemas
|
||||
- [ ] Create `exceptions.py` for module-specific errors
|
||||
- [ ] Create `config.py` with Pydantic config model
|
||||
- [ ] Set up `migrations/versions/` directory
|
||||
- [ ] Create `templates/` if module has UI
|
||||
- [ ] Create `locales/` if module needs translations
|
||||
- [ ] Set `is_self_contained=True` and path attributes
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Module System](../architecture/module-system.md) - Architecture overview
|
||||
- [Menu Management](../architecture/menu-management.md) - Menu item integration
|
||||
- [Database Migrations](migration/database-migrations.md) - Migration guide
|
||||
596
docs/proposals/module-migration-plan.md
Normal file
596
docs/proposals/module-migration-plan.md
Normal file
@@ -0,0 +1,596 @@
|
||||
# Module Migration Plan
|
||||
|
||||
**Status:** In Progress
|
||||
**Started:** 2026-01-25
|
||||
**Last Updated:** 2026-01-27
|
||||
**Target:** v1.0.0
|
||||
|
||||
This is the unified migration plan for transforming Wizamart into a fully modular architecture.
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
Transform the platform from a monolithic structure to self-contained modules where each module owns its:
|
||||
- Services (business logic)
|
||||
- Models (database entities)
|
||||
- Schemas (Pydantic DTOs)
|
||||
- Tasks (Celery background jobs)
|
||||
- Templates (UI if applicable)
|
||||
- Migrations (database changes)
|
||||
|
||||
---
|
||||
|
||||
## Current State Assessment
|
||||
|
||||
### Module Migration Status
|
||||
|
||||
| Module | Classification | Current State | Services | Models | Tasks | Target |
|
||||
|--------|---------------|---------------|----------|--------|-------|--------|
|
||||
| `cms` | Core | ✅ **Complete** | ✅ | ✅ | - | Done |
|
||||
| `payments` | Optional | 🟡 Partial | ✅ | ✅ | - | Done |
|
||||
| `billing` | Optional | 🔴 Shell | ❌ | ❌ | ❌ | Full |
|
||||
| `marketplace` | Optional | 🔴 Shell | ❌ | ❌ | ❌ | Full |
|
||||
| `orders` | Optional | 🔴 Shell | ❌ | ❌ | - | Full |
|
||||
| `inventory` | Optional | 🔴 Shell | ❌ | ❌ | - | Full |
|
||||
| `customers` | Core | 🔴 Shell | ❌ | ❌ | - | Full |
|
||||
| `analytics` | Optional | 🔴 Shell | ❌ | ❌ | - | Full |
|
||||
| `messaging` | Optional | 🔴 Shell | ❌ | ❌ | - | Full |
|
||||
| `monitoring` | Internal | 🔴 Shell | ❌ | ❌ | ❌ | Full |
|
||||
| `dev-tools` | Internal | 🔴 Shell | ❌ | ❌ | ❌ | Full |
|
||||
| `tenancy` | Core | 🔴 Shell | ❌ | ❌ | - | Full |
|
||||
| `core` | Core | 🔴 Shell | ❌ | ❌ | - | Minimal |
|
||||
|
||||
### Current Code Locations (To Be Migrated)
|
||||
|
||||
```
|
||||
app/services/ # → Move to respective modules
|
||||
├── subscription_service.py # → billing/services/
|
||||
├── stripe_service.py # → billing/services/ (or payments)
|
||||
├── customer_service.py # → customers/services/
|
||||
├── order_service.py # → orders/services/
|
||||
├── inventory_service.py # → inventory/services/
|
||||
├── letzshop_service.py # → marketplace/services/
|
||||
├── marketplace_import_service.py # → marketplace/services/
|
||||
├── messaging_service.py # → messaging/services/
|
||||
└── ...
|
||||
|
||||
models/database/ # → Move to respective modules
|
||||
├── subscription.py # → billing/models/
|
||||
├── customer.py # → customers/models/
|
||||
├── order.py # → orders/models/
|
||||
├── inventory.py # → inventory/models/
|
||||
├── letzshop_*.py # → marketplace/models/
|
||||
└── ...
|
||||
|
||||
app/tasks/celery_tasks/ # → Move to respective modules
|
||||
├── subscription.py # → billing/tasks/
|
||||
├── marketplace.py # → marketplace/tasks/
|
||||
├── letzshop.py # → marketplace/tasks/
|
||||
├── export.py # → marketplace/tasks/
|
||||
├── code_quality.py # → dev_tools/tasks/
|
||||
└── test_runner.py # → dev_tools/tasks/
|
||||
```
|
||||
|
||||
### Celery Task Mapping
|
||||
|
||||
| Current Task | Current Location | Target Module |
|
||||
|--------------|------------------|---------------|
|
||||
| `reset_period_counters` | subscription.py | `billing` |
|
||||
| `check_trial_expirations` | subscription.py | `billing` |
|
||||
| `sync_stripe_status` | subscription.py | `billing` |
|
||||
| `cleanup_stale_subscriptions` | subscription.py | `billing` |
|
||||
| `capture_capacity_snapshot` | subscription.py | `monitoring` |
|
||||
| `process_marketplace_import` | marketplace.py | `marketplace` |
|
||||
| `process_historical_import` | letzshop.py | `marketplace` |
|
||||
| `sync_vendor_directory` | letzshop.py | `marketplace` |
|
||||
| `export_vendor_products_to_folder` | export.py | `marketplace` |
|
||||
| `export_marketplace_products` | export.py | `marketplace` |
|
||||
| `execute_code_quality_scan` | code_quality.py | `dev-tools` |
|
||||
| `execute_test_run` | test_runner.py | `dev-tools` |
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure Work (Completed)
|
||||
|
||||
### ✅ Phase 1: Module Classification
|
||||
- Three-tier system: Core, Optional, Internal
|
||||
- Renamed `platform-admin` → `tenancy`
|
||||
- Module registry with CORE_MODULES, OPTIONAL_MODULES, INTERNAL_MODULES
|
||||
|
||||
### ✅ Phase 2: Module Infrastructure
|
||||
- `ModuleDefinition` with lifecycle hooks
|
||||
- Module events system (`events.py`)
|
||||
- Module migration discovery (`migrations.py`)
|
||||
- Observability framework (`observability.py`)
|
||||
|
||||
### ✅ Phase 3: Documentation
|
||||
- Module system architecture docs
|
||||
- Menu management docs
|
||||
- Observability docs
|
||||
- Creating modules developer guide
|
||||
|
||||
---
|
||||
|
||||
## Infrastructure Work (Remaining)
|
||||
|
||||
### Phase 4: Celery Task Infrastructure
|
||||
|
||||
Add task support to module system before migrating modules.
|
||||
|
||||
#### 4.1 Update ModuleDefinition
|
||||
|
||||
```python
|
||||
# app/modules/base.py - Add fields
|
||||
|
||||
@dataclass
|
||||
class ScheduledTask:
|
||||
"""Celery Beat scheduled task definition."""
|
||||
name: str # e.g., "billing.reset_counters"
|
||||
task: str # Full task path
|
||||
schedule: str | dict # Cron string or crontab dict
|
||||
args: tuple = ()
|
||||
kwargs: dict = field(default_factory=dict)
|
||||
|
||||
@dataclass
|
||||
class ModuleDefinition:
|
||||
# ... existing fields ...
|
||||
|
||||
# Task configuration (NEW)
|
||||
tasks_path: str | None = None
|
||||
scheduled_tasks: list[ScheduledTask] = field(default_factory=list)
|
||||
```
|
||||
|
||||
#### 4.2 Create Task Discovery
|
||||
|
||||
```python
|
||||
# app/modules/tasks.py (NEW)
|
||||
|
||||
def discover_module_tasks() -> list[str]:
|
||||
"""Discover task modules from all registered modules."""
|
||||
...
|
||||
|
||||
def build_beat_schedule() -> dict:
|
||||
"""Build Celery Beat schedule from module definitions."""
|
||||
...
|
||||
```
|
||||
|
||||
#### 4.3 Create Module Task Base
|
||||
|
||||
```python
|
||||
# app/modules/task_base.py (NEW)
|
||||
|
||||
class ModuleTask(Task):
|
||||
"""Base Celery task with DB session management."""
|
||||
|
||||
@contextmanager
|
||||
def get_db(self):
|
||||
...
|
||||
```
|
||||
|
||||
#### 4.4 Update Celery Config
|
||||
|
||||
```python
|
||||
# app/core/celery_config.py
|
||||
|
||||
from app.modules.tasks import discover_module_tasks, build_beat_schedule
|
||||
|
||||
# Auto-discover from modules
|
||||
celery_app.autodiscover_tasks(discover_module_tasks())
|
||||
|
||||
# Build schedule from modules
|
||||
celery_app.conf.beat_schedule = build_beat_schedule()
|
||||
```
|
||||
|
||||
**Files to create:**
|
||||
- `app/modules/tasks.py`
|
||||
- `app/modules/task_base.py`
|
||||
|
||||
**Files to modify:**
|
||||
- `app/modules/base.py`
|
||||
- `app/core/celery_config.py`
|
||||
|
||||
---
|
||||
|
||||
## Module Migration Phases
|
||||
|
||||
Each module migration includes: services, models, schemas, tasks, templates (if any).
|
||||
|
||||
### Phase 5: Billing Module (High Priority)
|
||||
|
||||
**Why first:** Has most Celery tasks, critical business logic.
|
||||
|
||||
#### Components to Move
|
||||
|
||||
| From | To | Type |
|
||||
|------|-----|------|
|
||||
| `app/services/subscription_service.py` | `app/modules/billing/services/` | Service |
|
||||
| `app/services/stripe_service.py` | `app/modules/billing/services/` | Service |
|
||||
| `models/database/subscription.py` | `app/modules/billing/models/` | Model |
|
||||
| `models/database/subscription_tier.py` | `app/modules/billing/models/` | Model |
|
||||
| `models/schema/subscription.py` | `app/modules/billing/schemas/` | Schema |
|
||||
| `app/tasks/celery_tasks/subscription.py` | `app/modules/billing/tasks/` | Tasks |
|
||||
|
||||
#### Target Structure
|
||||
|
||||
```
|
||||
app/modules/billing/
|
||||
├── __init__.py
|
||||
├── definition.py # Update with tasks_path, scheduled_tasks
|
||||
├── config.py # BillingConfig schema
|
||||
├── exceptions.py # BillingException, SubscriptionError, etc.
|
||||
├── services/
|
||||
│ ├── __init__.py
|
||||
│ ├── subscription_service.py
|
||||
│ └── stripe_service.py
|
||||
├── models/
|
||||
│ ├── __init__.py
|
||||
│ ├── subscription.py
|
||||
│ └── subscription_tier.py
|
||||
├── schemas/
|
||||
│ ├── __init__.py
|
||||
│ └── subscription.py
|
||||
├── tasks/
|
||||
│ ├── __init__.py
|
||||
│ └── subscription.py # 4 scheduled tasks
|
||||
├── routes/
|
||||
│ ├── admin.py
|
||||
│ └── vendor.py
|
||||
└── migrations/
|
||||
└── versions/
|
||||
```
|
||||
|
||||
#### Tasks
|
||||
|
||||
| Task | Schedule | Queue |
|
||||
|------|----------|-------|
|
||||
| `reset_period_counters` | Daily 00:05 | scheduled |
|
||||
| `check_trial_expirations` | Daily 01:00 | scheduled |
|
||||
| `sync_stripe_status` | Hourly :30 | scheduled |
|
||||
| `cleanup_stale_subscriptions` | Weekly Sun 03:00 | scheduled |
|
||||
|
||||
#### Migration Checklist
|
||||
|
||||
- [ ] Create `billing/services/` directory
|
||||
- [ ] Move `subscription_service.py` → update imports
|
||||
- [ ] Move `stripe_service.py` → update imports
|
||||
- [ ] Create `billing/models/` directory
|
||||
- [ ] Move subscription models → update imports
|
||||
- [ ] Create `billing/schemas/` directory
|
||||
- [ ] Move subscription schemas → update imports
|
||||
- [ ] Create `billing/tasks/` directory
|
||||
- [ ] Move subscription tasks → update task paths
|
||||
- [ ] Create `billing/exceptions.py`
|
||||
- [ ] Update `definition.py` with `tasks_path`, `scheduled_tasks`
|
||||
- [ ] Create module migrations
|
||||
- [ ] Update all imports across codebase
|
||||
- [ ] Test subscription flows
|
||||
- [ ] Test scheduled tasks
|
||||
|
||||
---
|
||||
|
||||
### Phase 6: Marketplace Module (High Priority)
|
||||
|
||||
**Why second:** Has most tasks, complex import logic.
|
||||
|
||||
#### Components to Move
|
||||
|
||||
| From | To | Type |
|
||||
|------|-----|------|
|
||||
| `app/services/letzshop_service.py` | `app/modules/marketplace/services/` | Service |
|
||||
| `app/services/marketplace_import_service.py` | `app/modules/marketplace/services/` | Service |
|
||||
| `app/services/letzshop_credentials_service.py` | `app/modules/marketplace/services/` | Service |
|
||||
| `models/database/letzshop_*.py` | `app/modules/marketplace/models/` | Models |
|
||||
| `models/database/marketplace_import_job.py` | `app/modules/marketplace/models/` | Model |
|
||||
| `app/tasks/celery_tasks/marketplace.py` | `app/modules/marketplace/tasks/` | Tasks |
|
||||
| `app/tasks/celery_tasks/letzshop.py` | `app/modules/marketplace/tasks/` | Tasks |
|
||||
| `app/tasks/celery_tasks/export.py` | `app/modules/marketplace/tasks/` | Tasks |
|
||||
|
||||
#### Target Structure
|
||||
|
||||
```
|
||||
app/modules/marketplace/
|
||||
├── __init__.py
|
||||
├── definition.py
|
||||
├── exceptions.py
|
||||
├── services/
|
||||
│ ├── __init__.py
|
||||
│ ├── letzshop_service.py
|
||||
│ ├── letzshop_credentials_service.py
|
||||
│ ├── import_service.py
|
||||
│ └── export_service.py
|
||||
├── models/
|
||||
│ ├── __init__.py
|
||||
│ ├── letzshop_order.py
|
||||
│ ├── letzshop_credentials.py
|
||||
│ └── import_job.py
|
||||
├── schemas/
|
||||
│ └── ...
|
||||
├── tasks/
|
||||
│ ├── __init__.py
|
||||
│ ├── import_tasks.py # process_marketplace_import
|
||||
│ ├── letzshop.py # process_historical_import, sync_vendor_directory
|
||||
│ └── export.py # export tasks
|
||||
└── routes/
|
||||
```
|
||||
|
||||
#### Tasks
|
||||
|
||||
| Task | Type | Queue |
|
||||
|------|------|-------|
|
||||
| `process_marketplace_import` | On-demand | long_running |
|
||||
| `process_historical_import` | On-demand | long_running |
|
||||
| `sync_vendor_directory` | Scheduled (optional) | long_running |
|
||||
| `export_vendor_products_to_folder` | On-demand | default |
|
||||
| `export_marketplace_products` | On-demand | default |
|
||||
|
||||
---
|
||||
|
||||
### Phase 7: Dev-Tools Module (Medium Priority)
|
||||
|
||||
**Why:** Internal module with code quality and test runner tasks.
|
||||
|
||||
#### Components to Move
|
||||
|
||||
| From | To | Type |
|
||||
|------|-----|------|
|
||||
| `app/services/code_quality_service.py` | `app/modules/dev_tools/services/` | Service |
|
||||
| `app/services/test_runner_service.py` | `app/modules/dev_tools/services/` | Service |
|
||||
| `models/database/architecture_scan.py` | `app/modules/dev_tools/models/` | Model |
|
||||
| `models/database/test_run.py` | `app/modules/dev_tools/models/` | Model |
|
||||
| `app/tasks/celery_tasks/code_quality.py` | `app/modules/dev_tools/tasks/` | Tasks |
|
||||
| `app/tasks/celery_tasks/test_runner.py` | `app/modules/dev_tools/tasks/` | Tasks |
|
||||
|
||||
#### Target Structure
|
||||
|
||||
```
|
||||
app/modules/dev_tools/
|
||||
├── __init__.py
|
||||
├── definition.py
|
||||
├── services/
|
||||
│ ├── code_quality_service.py
|
||||
│ └── test_runner_service.py
|
||||
├── models/
|
||||
│ ├── architecture_scan.py
|
||||
│ ├── architecture_violation.py
|
||||
│ └── test_run.py
|
||||
├── tasks/
|
||||
│ ├── code_quality.py
|
||||
│ └── test_runner.py
|
||||
└── routes/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 8: Monitoring Module (Medium Priority)
|
||||
|
||||
#### Components to Move
|
||||
|
||||
| From | To | Type |
|
||||
|------|-----|------|
|
||||
| `app/services/background_tasks_service.py` | `app/modules/monitoring/services/` | Service |
|
||||
| `capture_capacity_snapshot` task | `app/modules/monitoring/tasks/` | Task |
|
||||
| `models/database/capacity_snapshot.py` | `app/modules/monitoring/models/` | Model |
|
||||
|
||||
#### Target Structure
|
||||
|
||||
```
|
||||
app/modules/monitoring/
|
||||
├── __init__.py
|
||||
├── definition.py
|
||||
├── services/
|
||||
│ └── background_tasks_service.py
|
||||
├── models/
|
||||
│ └── capacity_snapshot.py
|
||||
├── tasks/
|
||||
│ └── capacity.py # capture_capacity_snapshot
|
||||
└── routes/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Phase 9: Customers Module (Core)
|
||||
|
||||
#### Components to Move
|
||||
|
||||
| From | To |
|
||||
|------|-----|
|
||||
| `app/services/customer_service.py` | `app/modules/customers/services/` |
|
||||
| `models/database/customer.py` | `app/modules/customers/models/` |
|
||||
| Customer-related schemas | `app/modules/customers/schemas/` |
|
||||
|
||||
---
|
||||
|
||||
### Phase 10: Orders Module
|
||||
|
||||
#### Components to Move
|
||||
|
||||
| From | To |
|
||||
|------|-----|
|
||||
| `app/services/order_service.py` | `app/modules/orders/services/` |
|
||||
| `models/database/order.py` | `app/modules/orders/models/` |
|
||||
| Order-related schemas | `app/modules/orders/schemas/` |
|
||||
|
||||
---
|
||||
|
||||
### Phase 11: Inventory Module
|
||||
|
||||
#### Components to Move
|
||||
|
||||
| From | To |
|
||||
|------|-----|
|
||||
| `app/services/inventory_service.py` | `app/modules/inventory/services/` |
|
||||
| `models/database/inventory*.py` | `app/modules/inventory/models/` |
|
||||
|
||||
---
|
||||
|
||||
### Phase 12: Remaining Modules
|
||||
|
||||
- **Analytics** - Move analytics services
|
||||
- **Messaging** - Move messaging service, notification models
|
||||
- **Tenancy** - Move platform, company, vendor services
|
||||
|
||||
---
|
||||
|
||||
## Phase 13: Cleanup & Full Celery Migration
|
||||
|
||||
After all modules are migrated:
|
||||
|
||||
### 13.1 Remove Fallback Tasks
|
||||
|
||||
Delete:
|
||||
```
|
||||
app/tasks/background_tasks.py
|
||||
app/tasks/letzshop_tasks.py
|
||||
app/tasks/subscription_tasks.py
|
||||
app/tasks/code_quality_tasks.py
|
||||
app/tasks/test_runner_tasks.py
|
||||
```
|
||||
|
||||
### 13.2 Remove USE_CELERY Flag
|
||||
|
||||
```python
|
||||
# app/core/config.py
|
||||
# DELETE: USE_CELERY: bool = False
|
||||
```
|
||||
|
||||
### 13.3 Simplify Dispatcher
|
||||
|
||||
Rewrite `app/tasks/dispatcher.py` to Celery-only.
|
||||
|
||||
### 13.4 Delete Old Task Directory
|
||||
|
||||
```
|
||||
app/tasks/celery_tasks/ # DELETE - moved to modules
|
||||
```
|
||||
|
||||
### 13.5 Clean Up Old Services/Models
|
||||
|
||||
Remove files from:
|
||||
- `app/services/` (moved to modules)
|
||||
- `models/database/` (moved to modules)
|
||||
|
||||
Leave re-exports for backward compatibility if needed.
|
||||
|
||||
---
|
||||
|
||||
## Queue Configuration
|
||||
|
||||
### Final Queue Structure
|
||||
|
||||
| Queue | Purpose | Modules |
|
||||
|-------|---------|---------|
|
||||
| `default` | Fast tasks | marketplace (exports) |
|
||||
| `long_running` | CPU-intensive | marketplace (imports), dev-tools |
|
||||
| `scheduled` | Celery Beat | billing, monitoring |
|
||||
|
||||
### Task Routing
|
||||
|
||||
```python
|
||||
task_routes = {
|
||||
"app.modules.billing.tasks.*": {"queue": "scheduled"},
|
||||
"app.modules.marketplace.tasks.import_tasks.*": {"queue": "long_running"},
|
||||
"app.modules.marketplace.tasks.letzshop.*": {"queue": "long_running"},
|
||||
"app.modules.marketplace.tasks.export.*": {"queue": "default"},
|
||||
"app.modules.monitoring.tasks.*": {"queue": "scheduled"},
|
||||
"app.modules.dev_tools.tasks.*": {"queue": "long_running"},
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### For Each Module
|
||||
|
||||
1. **Create directory structure**
|
||||
2. **Move models first** (fewest dependencies)
|
||||
3. **Move services** (depend on models)
|
||||
4. **Move schemas** (depend on models)
|
||||
5. **Move tasks** (depend on services)
|
||||
6. **Update all imports** across codebase
|
||||
7. **Create re-exports** in old locations (temporary)
|
||||
8. **Test thoroughly**
|
||||
9. **Remove re-exports** in next phase
|
||||
|
||||
### Import Update Strategy
|
||||
|
||||
Use IDE refactoring or script:
|
||||
```bash
|
||||
# Example: Update subscription_service imports
|
||||
find . -name "*.py" -exec sed -i 's/from app.services.subscription_service/from app.modules.billing.services.subscription_service/g' {} \;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
### Per-Module Verification
|
||||
|
||||
- [ ] Module directory has all components
|
||||
- [ ] All imports updated
|
||||
- [ ] Services work
|
||||
- [ ] API routes work
|
||||
- [ ] Tasks execute correctly
|
||||
- [ ] Scheduled tasks run (Celery Beat)
|
||||
- [ ] No circular imports
|
||||
- [ ] Tests pass
|
||||
|
||||
### Final Verification
|
||||
|
||||
- [ ] All 12 tasks in modules
|
||||
- [ ] `USE_CELERY` flag removed
|
||||
- [ ] Fallback tasks deleted
|
||||
- [ ] Old service files removed
|
||||
- [ ] Old model files removed
|
||||
- [ ] All tests pass
|
||||
- [ ] Production smoke test
|
||||
|
||||
---
|
||||
|
||||
## Priority Order
|
||||
|
||||
| Priority | Phase | Module | Reason |
|
||||
|----------|-------|--------|--------|
|
||||
| 1 | 4 | Infrastructure | Required for task migration |
|
||||
| 2 | 5 | `billing` | Most tasks, critical business logic |
|
||||
| 3 | 6 | `marketplace` | Most complex, many tasks |
|
||||
| 4 | 7 | `dev-tools` | Internal, low risk |
|
||||
| 5 | 8 | `monitoring` | Internal, low risk |
|
||||
| 6 | 9-12 | Others | Lower complexity |
|
||||
| 7 | 13 | Cleanup | Final step |
|
||||
|
||||
---
|
||||
|
||||
## Estimated Effort
|
||||
|
||||
| Phase | Sessions | Notes |
|
||||
|-------|----------|-------|
|
||||
| 4 (Infrastructure) | 1 | Task base class, discovery |
|
||||
| 5 (Billing) | 2-3 | Complex, many components |
|
||||
| 6 (Marketplace) | 3-4 | Most complex module |
|
||||
| 7-8 (Internal) | 1-2 | Simpler modules |
|
||||
| 9-12 (Others) | 4-6 | Medium complexity each |
|
||||
| 13 (Cleanup) | 1 | Removing old code |
|
||||
|
||||
**Total:** ~12-17 sessions
|
||||
|
||||
---
|
||||
|
||||
## Rollback Plan
|
||||
|
||||
Each phase can be rolled back independently:
|
||||
1. Restore moved files to original locations
|
||||
2. Revert import changes
|
||||
3. Restore celery_config.py
|
||||
|
||||
No database schema changes during code migration = no DB rollback needed.
|
||||
|
||||
---
|
||||
|
||||
## Related Documents
|
||||
|
||||
- [Module System Architecture](../architecture/module-system.md)
|
||||
- [Creating Modules](../development/creating-modules.md)
|
||||
- [Observability](../architecture/observability.md)
|
||||
- [SESSION_NOTE_2026-01-27](SESSION_NOTE_2026-01-27_module-reclassification.md)
|
||||
Reference in New Issue
Block a user