sboulahtit/orion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add module config and migrations auto-discovery infrastructure

Browse Source 
Add self-contained configuration and migrations support for modules:

Config auto-discovery (app/modules/config.py):
- Modules can have config.py with Pydantic Settings
- Environment variables prefixed with MODULE_NAME_
- Auto-discovered via get_module_config()

Migrations auto-discovery:
- Each module has migrations/versions/ directory
- Alembic discovers module migrations automatically
- Naming convention: {module}_{seq}_{description}.py

New architecture rules (MOD-013 to MOD-015):
- MOD-013: config.py should export config/config_class
- MOD-014: Migrations must follow naming convention
- MOD-015: Migrations directory must have __init__.py

Created for all 11 self-contained modules:
- config.py placeholder files
- migrations/ directories with __init__.py files

Added core and tenancy module definitions for completeness.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Samir Boulahtit
2026-01-28 22:19:41 +01:00
parent eb47daec8b
commit 2466dfd7ed
43 changed files with 2338 additions and 581 deletions
Download Patch File Download Diff File Expand all files Collapse all files

527
docs/architecture/module-system.md
View File

@@ -1,6 +1,13 @@
# Module System Architecture
The Wizamart platform uses a modular architecture that allows features to be enabled or disabled per platform. This document explains the module system, its classification tiers, and how modules interact with the rest of the application.
The Wizamart platform uses a **plug-and-play modular architecture** where modules are fully self-contained and automatically discovered. Simply create a module directory with the required structure, and the framework handles registration, routing, and resource loading automatically.
## Key Features
- **Auto-Discovery**: Modules are automatically discovered from `app/modules/*/definition.py`
- **Zero Configuration**: No changes to `main.py`, `registry.py`, or other framework files needed
- **Self-Contained**: Each module owns its routes, services, models, templates, and translations
- **Hot-Pluggable**: Add or remove modules by simply adding/removing directories
## Overview
@@ -14,7 +21,7 @@ The Wizamart platform uses a modular architecture that allows features to be ena
│
▼
┌─────────────────────────────────────────────────────────────────────┐
│ MODULE LAYER │
│ AUTO-DISCOVERED MODULE LAYER │
│ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ CORE MODULES (Always Enabled) │ │
@@ -33,6 +40,36 @@ The Wizamart platform uses a modular architecture that allows features to be ena
└─────────────────────────────────────────────────────────────────────┘
```
## Auto-Discovery System
All module components are automatically discovered by the framework:
| Component | Discovery Location | Auto-Loaded By |
|-----------|-------------------|----------------|
| **Registry** | `*/definition.py` | `app/modules/discovery.py` |
| **Configuration** | `*/config.py` | `app/modules/config.py` |
| **API Routes** | `*/routes/api/*.py` | `app/modules/routes.py` |
| **Page Routes** | `*/routes/pages/*.py` | `app/modules/routes.py` |
| **Tasks** | `*/tasks/__init__.py` | `app/modules/tasks.py` |
| **Templates** | `*/templates/` | `app/templates_config.py` |
| **Static Files** | `*/static/` | `main.py` |
| **Locales** | `*/locales/*.json` | `app/utils/i18n.py` |
| **Migrations** | `*/migrations/versions/` | `app/modules/migrations.py` |
### Creating a New Module (Zero Framework Changes)
```bash
# 1. Create module directory
mkdir -p app/modules/mymodule/{routes/{api,pages},services,models,schemas,templates/mymodule/vendor,static/vendor/js,locales,tasks}
# 2. Create required files
touch app/modules/mymodule/__init__.py
touch app/modules/mymodule/definition.py
touch app/modules/mymodule/exceptions.py
# 3. That's it! The framework auto-discovers and registers everything.
```
## Three-Tier Classification
### Core Modules (4)
@@ -69,6 +106,160 @@ Internal modules are **admin-only tools** not exposed to customers or vendors.
| `dev-tools` | Component library, icon browser |
| `monitoring` | Logs, background tasks, Flower, Grafana integration |
## Self-Contained Module Structure
Every module follows this standardized structure:
```
app/modules/analytics/
├── __init__.py # Module package marker
├── definition.py # ModuleDefinition (REQUIRED for auto-discovery)
├── config.py # Environment config (auto-discovered)
├── exceptions.py # Module-specific exceptions
├── routes/
│ ├── __init__.py
│ ├── api/ # API endpoints (auto-discovered)
│ │ ├── __init__.py
│ │ ├── admin.py # Must export: router = APIRouter()
│ │ └── vendor.py # Must export: router = APIRouter()
│ └── pages/ # HTML page routes (auto-discovered)
│ ├── __init__.py
│ └── vendor.py # Must export: router = APIRouter()
├── services/
│ ├── __init__.py
│ └── stats_service.py
├── models/
│ ├── __init__.py
│ └── report.py
├── schemas/
│ ├── __init__.py
│ └── stats.py
├── templates/ # Auto-discovered by Jinja2
│ └── analytics/
│ └── vendor/
│ └── analytics.html
├── static/ # Auto-mounted at /static/modules/analytics/
│ └── vendor/
│ └── js/
│ └── analytics.js
├── locales/ # Auto-loaded translations
│ ├── en.json
│ ├── de.json
│ ├── fr.json
│ └── lu.json
├── tasks/ # Auto-discovered by Celery
│ ├── __init__.py # REQUIRED for Celery discovery
│ └── reports.py
└── migrations/ # Auto-discovered by Alembic
├── __init__.py # REQUIRED for discovery
└── versions/
├── __init__.py # REQUIRED for discovery
└── analytics_001_create_reports.py
```
## Module Definition
Each module must have a `definition.py` with a `ModuleDefinition` instance:
```python
# app/modules/analytics/definition.py
from app.modules.base import ModuleDefinition
from models.database.admin_menu_config import FrontendType
analytics_module = ModuleDefinition(
# Identity
code="analytics",
name="Analytics & Reporting",
description="Dashboard analytics, custom reports, and data exports.",
version="1.0.0",
# Classification (determines tier)
is_core=False, # Set True for core modules
is_internal=False, # Set True for admin-only modules
# Dependencies
requires=[], # List other module codes this depends on
# Features (for tier-based gating)
features=[
"basic_reports",
"analytics_dashboard",
"custom_reports",
],
# Menu items per frontend
menu_items={
FrontendType.ADMIN: [], # Analytics uses dashboard
FrontendType.VENDOR: ["analytics"],
},
# Self-contained module configuration
is_self_contained=True,
services_path="app.modules.analytics.services",
models_path="app.modules.analytics.models",
schemas_path="app.modules.analytics.schemas",
exceptions_path="app.modules.analytics.exceptions",
templates_path="templates",
locales_path="locales",
)
```
### ModuleDefinition Fields
| Field | Type | Description |
|-------|------|-------------|
| `code` | `str` | Unique identifier (e.g., "billing") |
| `name` | `str` | Display name |
| `description` | `str` | What the module provides |
| `version` | `str` | Semantic version (default: "1.0.0") |
| `requires` | `list[str]` | Module codes this depends on |
| `features` | `list[str]` | Feature codes for tier gating |
| `menu_items` | `dict` | Menu items per frontend type |
| `is_core` | `bool` | Cannot be disabled if True |
| `is_internal` | `bool` | Admin-only if True |
| `is_self_contained` | `bool` | Uses self-contained structure |
## Route Auto-Discovery
Routes in `routes/api/` and `routes/pages/` are automatically discovered and registered.
### API Routes (`routes/api/vendor.py`)
```python
# app/modules/analytics/routes/api/vendor.py
from fastapi import APIRouter, Depends
from app.api.deps import get_current_vendor_api, get_db
router = APIRouter() # MUST be named 'router' for auto-discovery
@router.get("")
def get_analytics(
current_user = Depends(get_current_vendor_api),
db = Depends(get_db),
):
"""Get vendor analytics."""
pass
```
**Auto-registered at:** `/api/v1/vendor/analytics`
### Page Routes (`routes/pages/vendor.py`)
```python
# app/modules/analytics/routes/pages/vendor.py
from fastapi import APIRouter, Depends, Request
from fastapi.responses import HTMLResponse
router = APIRouter() # MUST be named 'router' for auto-discovery
@router.get("/{vendor_code}/analytics", response_class=HTMLResponse)
async def analytics_page(request: Request, vendor_code: str):
"""Render analytics page."""
pass
```
**Auto-registered at:** `/vendor/{vendor_code}/analytics`
## Framework Layer
The Framework Layer provides infrastructure that modules depend on. These are **not modules** - they're always available and cannot be disabled.
@@ -87,74 +278,6 @@ The Framework Layer provides infrastructure that modules depend on. These are **
| Dependencies | `app/api/deps.py` | FastAPI DI |
| Base Exceptions | `app/exceptions/base.py` | Exception hierarchy |
## Module Definition
Each module is defined using the `ModuleDefinition` dataclass:
```python
from app.modules.base import ModuleDefinition
from models.database.admin_menu_config import FrontendType
billing_module = ModuleDefinition(
# Identity
code="billing",
name="Billing & Subscriptions",
description="Platform subscriptions and vendor invoices",
version="1.0.0",
# Dependencies
requires=["payments"], # Must have payments enabled
# Features (for tier-based gating)
features=[
"subscription_management",
"billing_history",
"invoice_generation",
],
# Menu items per frontend
menu_items={
FrontendType.ADMIN: ["subscription-tiers", "subscriptions"],
FrontendType.VENDOR: ["billing", "invoices"],
},
# Classification
is_core=False,
is_internal=False,
# Configuration schema (optional)
config_schema=BillingConfig,
default_config={"trial_days": 14},
# Lifecycle hooks (optional)
on_enable=lambda platform_id: setup_billing(platform_id),
on_disable=lambda platform_id: cleanup_billing(platform_id),
health_check=lambda: {"status": "healthy"},
)
```
### ModuleDefinition Fields
| Field | Type | Description |
|-------|------|-------------|
| `code` | `str` | Unique identifier (e.g., "billing") |
| `name` | `str` | Display name |
| `description` | `str` | What the module provides |
| `version` | `str` | Semantic version (default: "1.0.0") |
| `requires` | `list[str]` | Module codes this depends on |
| `features` | `list[str]` | Feature codes for tier gating |
| `menu_items` | `dict` | Menu items per frontend type |
| `permissions` | `list[str]` | Permission codes defined by module |
| `is_core` | `bool` | Cannot be disabled if True |
| `is_internal` | `bool` | Admin-only if True |
| `config_schema` | `type[BaseModel]` | Pydantic model for configuration |
| `default_config` | `dict` | Default configuration values |
| `on_enable` | `Callable` | Called when module is enabled |
| `on_disable` | `Callable` | Called when module is disabled |
| `on_startup` | `Callable` | Called on application startup |
| `health_check` | `Callable` | Returns health status dict |
| `migrations_path` | `str` | Path to module migrations (relative) |
## Module Dependencies
Modules can depend on other modules. When enabling a module, its dependencies are automatically enabled.
@@ -178,18 +301,17 @@ Modules can depend on other modules. When enabling a module, its dependencies ar
## Module Registry
All modules are registered in `app/modules/registry.py`:
The registry auto-discovers all modules:
```python
from app.modules.registry import (
MODULES, # All modules
MODULES, # All modules (auto-discovered)
CORE_MODULES, # Core only
OPTIONAL_MODULES, # Optional only
INTERNAL_MODULES, # Internal only
get_module,
get_core_module_codes,
get_module_tier,
is_core_module,
)
# Get a specific module
@@ -211,7 +333,6 @@ from app.modules.service import module_service
# Check if module is enabled
if module_service.is_module_enabled(db, platform_id, "billing"):
# Module is enabled for this platform
pass
# Get all enabled modules for a platform
@@ -222,178 +343,144 @@ module_service.enable_module(db, platform_id, "billing", user_id=current_user.id
# Disable a module (auto-disables dependents)
module_service.disable_module(db, platform_id, "billing", user_id=current_user.id)
# Get module configuration
config = module_service.get_module_config(db, platform_id, "billing")
# Set module configuration
module_service.set_module_config(db, platform_id, "billing", {"trial_days": 30})
```
## Module Events
## Module Configuration
The event system allows components to react to module lifecycle changes:
Modules can have environment-based configuration using Pydantic Settings. The `config.py` file is auto-discovered by `app/modules/config.py`.
```python
from app.modules.events import module_event_bus, ModuleEvent, ModuleEventData
# app/modules/marketplace/config.py
from pydantic import Field
from pydantic_settings import BaseSettings
# Subscribe to events
@module_event_bus.subscribe(ModuleEvent.ENABLED)
def on_module_enabled(data: ModuleEventData):
print(f"Module {data.module_code} enabled for platform {data.platform_id}")
class MarketplaceConfig(BaseSettings):
"""Configuration for marketplace module."""
@module_event_bus.subscribe(ModuleEvent.DISABLED)
def on_module_disabled(data: ModuleEventData):
clear_module_cache(data.platform_id, data.module_code)
# Settings loaded from env vars with MARKETPLACE_ prefix
api_timeout: int = Field(default=30, description="API timeout in seconds")
batch_size: int = Field(default=100, description="Import batch size")
max_retries: int = Field(default=3, description="Max retry attempts")
@module_event_bus.subscribe(ModuleEvent.CONFIG_CHANGED)
def on_config_changed(data: ModuleEventData):
print(f"Config changed: {data.config}")
model_config = {"env_prefix": "MARKETPLACE_"}
# Events are emitted by ModuleService automatically
# You can also emit manually:
module_event_bus.emit_enabled("billing", platform_id=1, user_id=42)
# Export for auto-discovery
config_class = MarketplaceConfig
config = MarketplaceConfig()
```
### Event Types
**Usage:**
| Event | When Fired | Data Available |
|-------|------------|----------------|
| `ENABLED` | Module enabled for platform | `module_code`, `platform_id`, `user_id` |
| `DISABLED` | Module disabled for platform | `module_code`, `platform_id`, `user_id` |
| `STARTUP` | Application starting | `module_code` |
| `SHUTDOWN` | Application shutting down | `module_code` |
| `CONFIG_CHANGED` | Module config updated | `module_code`, `platform_id`, `config` |
```python
# Direct import
from app.modules.marketplace.config import config
timeout = config.api_timeout
## Module-Specific Migrations
Self-contained modules can have their own database migrations:
```
app/modules/cms/
├── migrations/
│ └── versions/
│ ├── cms_001_create_content_pages.py
│ └── cms_002_add_seo_fields.py
├── models/
├── services/
└── ...
# Via discovery
from app.modules.config import get_module_config
config = get_module_config("marketplace")
```
**Migration Naming Convention:**
**Environment variables:**
```bash
MARKETPLACE_API_TIMEOUT=60
MARKETPLACE_BATCH_SIZE=500
```
## Module Migrations
Each module owns its database migrations in the `migrations/versions/` directory. Alembic auto-discovers these via `app/modules/migrations.py`.
### Migration Structure
```
app/modules/cms/migrations/
├── __init__.py # REQUIRED for discovery
└── versions/
├── __init__.py # REQUIRED for discovery
├── cms_001_create_content_pages.py
├── cms_002_add_sections.py
└── cms_003_add_media_library.py
```
### Naming Convention
```
{module_code}_{sequence}_{description}.py
```
Alembic automatically discovers module migrations:
### Migration Template
```python
# In alembic/env.py
from app.modules.migrations import get_all_migration_paths
# app/modules/cms/migrations/versions/cms_001_create_content_pages.py
"""Create content_pages table.
version_locations = [str(p) for p in get_all_migration_paths()]
Revision ID: cms_001
Create Date: 2026-01-28
"""
from alembic import op
import sqlalchemy as sa
revision = "cms_001"
down_revision = None
branch_labels = ("cms",) # Module-specific branch
def upgrade() -> None:
op.create_table(
"content_pages",
sa.Column("id", sa.Integer(), primary_key=True),
sa.Column("vendor_id", sa.Integer(), sa.ForeignKey("vendors.id")),
sa.Column("slug", sa.String(100), nullable=False),
sa.Column("title", sa.String(200), nullable=False),
)
def downgrade() -> None:
op.drop_table("content_pages")
```
## Self-Contained Module Structure
### Running Migrations
Modules can be self-contained with their own services, models, and templates:
Module migrations are automatically discovered:
```
app/modules/cms/
├── __init__.py
├── definition.py # ModuleDefinition
├── config.py # Configuration schema (optional)
├── exceptions.py # Module-specific exceptions
├── routes/
│ ├── admin.py # Admin API routes
│ └── vendor.py # Vendor API routes
├── services/
│ └── content_service.py
├── models/
│ └── content_page.py
├── schemas/
│ └── content.py # Pydantic schemas
├── templates/
│ ├── admin/
│ └── vendor/
├── migrations/
│ └── versions/
└── locales/
├── en.json
└── fr.json
```bash
# Run all migrations (core + modules)
alembic upgrade head
# View migration history
alembic history
```
Configure paths in the definition:
### Current State
```python
cms_module = ModuleDefinition(
code="cms",
name="Content Management",
is_self_contained=True,
services_path="app.modules.cms.services",
models_path="app.modules.cms.models",
schemas_path="app.modules.cms.schemas",
templates_path="templates",
exceptions_path="app.modules.cms.exceptions",
locales_path="locales",
migrations_path="migrations",
)
```
Currently, all migrations reside in central `alembic/versions/`. The module-specific directories are in place for:
- **New modules**: Should create migrations in their own `migrations/versions/`
- **Future reorganization**: Existing migrations will be moved to modules pre-production
## Database Storage
## Architecture Validation Rules
Module enablement is stored in the `platform_modules` table:
The architecture validator (`scripts/validate_architecture.py`) enforces module structure:
| Column | Type | Description |
|--------|------|-------------|
| `platform_id` | FK | Platform reference |
| `module_code` | string | Module identifier |
| `is_enabled` | boolean | Whether enabled |
| `enabled_at` | timestamp | When enabled |
| `enabled_by_user_id` | FK | Who enabled it |
| `disabled_at` | timestamp | When disabled |
| `disabled_by_user_id` | FK | Who disabled it |
| `config` | JSON | Module-specific configuration |
| Rule | Severity | Description |
|------|----------|-------------|
| MOD-001 | ERROR | Self-contained modules must have required directories |
| MOD-002 | WARNING | Services must contain actual code, not re-exports |
| MOD-003 | WARNING | Schemas must contain actual code, not re-exports |
| MOD-004 | WARNING | Routes must import from module, not legacy locations |
| MOD-005 | WARNING | Modules with UI must have templates and static |
| MOD-006 | INFO | Modules should have locales for i18n |
| MOD-007 | ERROR | Definition paths must match directory structure |
| MOD-008 | WARNING | Self-contained modules must have exceptions.py |
| MOD-009 | ERROR | Modules must have definition.py for auto-discovery |
| MOD-010 | WARNING | Route files must export `router` variable |
| MOD-011 | WARNING | Tasks directory must have `__init__.py` |
| MOD-012 | INFO | Locales should have all language files |
| MOD-013 | INFO | config.py should export `config` or `config_class` |
| MOD-014 | WARNING | Migrations must follow naming convention |
| MOD-015 | WARNING | Migrations directory must have `__init__.py` files |
## Menu Item Filtering
Menu items are filtered based on enabled modules:
```python
from app.modules.service import module_service
from models.database.admin_menu_config import FrontendType
# Get available menu items for platform
menu_items = module_service.get_module_menu_items(
db, platform_id, FrontendType.ADMIN
)
# Check if specific menu item's module is enabled
is_available = module_service.is_menu_item_module_enabled(
db, platform_id, "subscription-tiers", FrontendType.ADMIN
)
```
## Health Checks
Modules can provide health checks that are aggregated:
```python
from app.core.observability import health_registry, register_module_health_checks
# Register all module health checks on startup
register_module_health_checks()
# Health endpoint aggregates results
# GET /health returns:
{
"status": "healthy",
"checks": [
{"name": "module:billing", "status": "healthy"},
{"name": "module:payments", "status": "healthy"},
...
]
}
Run validation:
```bash
python scripts/validate_architecture.py
```
## Best Practices
@@ -404,7 +491,9 @@ register_module_health_checks()
- Use `requires` for hard dependencies
- Provide `health_check` for critical modules
- Use events for cross-module communication
- Document module features and menu items
- Follow the standard directory structure
- Export `router` variable in route files
- Include all supported languages in locales
### Don't
@@ -412,12 +501,12 @@ register_module_health_checks()
- Make core modules depend on optional modules
- Put framework-level code in modules
- Skip migration naming conventions
- Forget to register menu items
- Forget `__init__.py` in tasks directory
- Manually register modules in registry.py (use auto-discovery)
## Related Documentation
- [Menu Management](menu-management.md) - Sidebar and menu configuration
- [Creating Modules](../development/creating-modules.md) - Developer guide for building modules
- [Observability](observability.md) - Health checks and module health integration
- [Multi-Tenant System](multi-tenant.md) - Platform isolation
- [Creating Modules](../development/creating-modules.md) - Step-by-step guide
- [Menu Management](menu-management.md) - Sidebar configuration
- [Observability](observability.md) - Health checks integration
- [Feature Gating](../implementation/feature-gating-system.md) - Tier-based access