sboulahtit/orion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
dad2ebc677f4151edace2a99a3fb8ef7bfbef086
orion/docs/proposals/module-migration-plan.md
Samir Boulahtit fbcf07914e chore: update API routers, validation, and docs for module system 
- app/api/v1/admin/__init__.py: Updated router imports
- app/api/v1/vendor/__init__.py: Updated router imports
- app/exceptions/code_quality.py: Added module exception imports
- scripts/validate_architecture.py: Added module validation rules
- .architecture-rules/_main.yaml: Include module.yaml rules
- docs/proposals/module-migration-plan.md: Updated migration status

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-28 22:22:43 +01:00

610 lines
21 KiB
Markdown
Raw Blame History

# Module Migration Plan
**Status:** In Progress
**Started:** 2026-01-25
**Last Updated:** 2026-01-28
**Target:** v1.0.0
This is the unified migration plan for transforming Wizamart into a fully modular architecture.
---
## Executive Summary
Transform the platform from a monolithic structure to self-contained modules where each module owns its:
- Services (business logic)
- Models (database entities)
- Schemas (Pydantic DTOs)
- Tasks (Celery background jobs)
- Templates (UI if applicable)
- Migrations (database changes)
---
## Current State Assessment
### Module Migration Status
| Module | Classification | Current State | Services | Models | Tasks | Target |
|--------|---------------|---------------|----------|--------|-------|--------|
| `cms` | Core | ✅ **Complete** | ✅ | ✅ | - | Done |
| `payments` | Optional | 🟡 Partial | ✅ | ✅ | - | Done |
| `billing` | Optional | ✅ **Complete** | ✅ | ✅ | ✅ | Done |
| `marketplace` | Optional | ✅ **Complete** | ✅ | ✅ | ✅ | Done |
| `orders` | Optional | ✅ **Complete** | ✅ | ✅ | - | Done |
| `inventory` | Optional | ✅ **Complete** | ✅ | ✅ | - | Done |
| `customers` | Core | ✅ **Complete** | ✅ | ✅ | - | Done |
| `analytics` | Optional | ✅ **Complete** | ✅ | - | - | Done |
| `messaging` | Optional | ✅ **Complete** | ✅ | ✅ | - | Done |
| `monitoring` | Internal | ✅ **Complete** | ✅ | ✅ | - | Done |
| `dev-tools` | Internal | ✅ **Complete** | ✅ | ✅ | ✅ | Done |
| `tenancy` | Core | 🔴 Shell | ❌ | ❌ | - | Full |
| `core` | Core | 🔴 Shell | ❌ | ❌ | - | Minimal |
### Current Code Locations (To Be Migrated)
```
app/services/ # → Move to respective modules
├── subscription_service.py # → billing/services/
├── stripe_service.py # → billing/services/ (or payments)
├── customer_service.py # → customers/services/
├── order_service.py # → orders/services/
├── inventory_service.py # → inventory/services/
├── letzshop_service.py # → marketplace/services/
├── marketplace_import_service.py # → marketplace/services/
├── messaging_service.py # → messaging/services/
└── ...
models/database/ # → Move to respective modules
├── subscription.py # → billing/models/
├── customer.py # → customers/models/
├── order.py # → orders/models/
├── inventory.py # → inventory/models/
├── letzshop_*.py # → marketplace/models/
└── ...
app/tasks/celery_tasks/ # → Move to respective modules
├── subscription.py # → billing/tasks/
├── marketplace.py # → marketplace/tasks/
├── letzshop.py # → marketplace/tasks/
├── export.py # → marketplace/tasks/
├── code_quality.py # → dev_tools/tasks/
└── test_runner.py # → dev_tools/tasks/
```
### Celery Task Mapping
| Current Task | Current Location | Target Module |
|--------------|------------------|---------------|
| `reset_period_counters` | subscription.py | `billing` |
| `check_trial_expirations` | subscription.py | `billing` |
| `sync_stripe_status` | subscription.py | `billing` |
| `cleanup_stale_subscriptions` | subscription.py | `billing` |
| `capture_capacity_snapshot` | subscription.py | `monitoring` |
| `process_marketplace_import` | marketplace.py | `marketplace` |
| `process_historical_import` | letzshop.py | `marketplace` |
| `sync_vendor_directory` | letzshop.py | `marketplace` |
| `export_vendor_products_to_folder` | export.py | `marketplace` |
| `export_marketplace_products` | export.py | `marketplace` |
| `execute_code_quality_scan` | code_quality.py | `dev-tools` |
| `execute_test_run` | test_runner.py | `dev-tools` |
---
## Infrastructure Work (Completed)
### ✅ Phase 1: Module Classification
- Three-tier system: Core, Optional, Internal
- Renamed `platform-admin` → `tenancy`
- Module registry with CORE_MODULES, OPTIONAL_MODULES, INTERNAL_MODULES
### ✅ Phase 2: Module Infrastructure
- `ModuleDefinition` with lifecycle hooks
- Module events system (`events.py`)
- Module migration discovery (`migrations.py`)
- Observability framework (`observability.py`)
### ✅ Phase 3: Documentation
- Module system architecture docs
- Menu management docs
- Observability docs
- Creating modules developer guide
### ✅ Phase 4: Celery Task Infrastructure
- Added `ScheduledTask` dataclass to `ModuleDefinition`
- Added `tasks_path` and `scheduled_tasks` fields
- Created `app/modules/task_base.py` with `ModuleTask` base class
- Created `app/modules/tasks.py` with discovery utilities
- Updated `celery_config.py` to use module discovery
### ✅ Phase 5: Billing Module Migration
- Created `app/modules/billing/services/` with subscription, stripe, admin services
- Created `app/modules/billing/models/` re-exporting from central location
- Created `app/modules/billing/schemas/` re-exporting from central location
- Created `app/modules/billing/tasks/` with 4 scheduled tasks
- Created `app/modules/billing/exceptions.py`
- Updated `definition.py` with self-contained configuration
- Created backward-compatible re-exports in `app/services/`
- Updated legacy celery_config.py to not duplicate scheduled tasks
### ✅ Phase 6: Marketplace Module Migration
- Created `app/modules/marketplace/services/` re-exporting from existing locations
- Created `app/modules/marketplace/models/` re-exporting marketplace & letzshop models
- Created `app/modules/marketplace/schemas/` re-exporting marketplace schemas
- Created `app/modules/marketplace/tasks/` with:
- `import_tasks.py` - process_marketplace_import, process_historical_import
- `sync_tasks.py` - sync_vendor_directory (scheduled daily)
- `export_tasks.py` - export_vendor_products_to_folder, export_marketplace_products
- Created `app/modules/marketplace/exceptions.py`
- Updated `definition.py` with self-contained configuration
- Updated legacy task files to re-export from new location
- Removed marketplace/letzshop/export from LEGACY_TASK_MODULES
### ✅ Phase 7: Dev-Tools Module Migration
- Created `app/modules/dev_tools/models/` with:
- `architecture_scan.py` - ArchitectureScan, ArchitectureViolation, ArchitectureRule, etc.
- `test_run.py` - TestRun, TestResult, TestCollection
- Created `app/modules/dev_tools/services/` re-exporting code_quality_service, test_runner_service
- Created `app/modules/dev_tools/schemas/` (placeholder for future schemas)
- Created `app/modules/dev_tools/tasks/` with:
- `code_quality.py` - execute_code_quality_scan
- `test_runner.py` - execute_test_run
- Created `app/modules/dev_tools/exceptions.py` with test runner exceptions
- Created `app/modules/dev_tools/routes/api/` with admin router
- Updated `definition.py` with self-contained configuration
- Updated legacy model files to re-export from module location
- Updated legacy task files to re-export from module location
- Used lazy imports in `__init__.py` to avoid circular import issues
### ✅ Phase 8: Monitoring Module Migration
- Created `app/modules/monitoring/models/` re-exporting CapacitySnapshot, AdminNotification
- Created `app/modules/monitoring/services/` re-exporting BackgroundTasksService
- Created `app/modules/monitoring/schemas/` (placeholder)
- Created `app/modules/monitoring/exceptions.py` with monitoring-specific exceptions
- Updated `definition.py` with self-contained configuration
- Used lazy imports in `__init__.py` to avoid circular import issues
### ✅ Phase 9: Customers Module Migration
- Created `app/modules/customers/models/` re-exporting Customer, CustomerAddress, PasswordResetToken
- Created `app/modules/customers/services/` re-exporting customer services
- Created `app/modules/customers/schemas/` re-exporting customer schemas
- Created `app/modules/customers/exceptions.py` re-exporting customer exceptions
- Updated `definition.py` with self-contained configuration
- Used lazy imports in `__init__.py` to avoid circular import issues
### ✅ Phase 10: Orders Module Migration
- Created `app/modules/orders/models/` re-exporting Order, OrderItem, Invoice, etc.
- Created `app/modules/orders/services/` re-exporting order and invoice services
- Created `app/modules/orders/schemas/` re-exporting order and invoice schemas
- Created `app/modules/orders/exceptions.py` re-exporting order exceptions
- Updated `definition.py` with self-contained configuration
- Used lazy imports in `__init__.py` to avoid circular import issues
### ✅ Phase 11: Inventory Module Migration
- Created `app/modules/inventory/models/` re-exporting Inventory, InventoryTransaction
- Created `app/modules/inventory/services/` re-exporting inventory services
- Created `app/modules/inventory/schemas/` re-exporting inventory schemas
- Created `app/modules/inventory/exceptions.py` re-exporting inventory exceptions
- Updated `definition.py` with self-contained configuration
- Used lazy imports in `__init__.py` to avoid circular import issues
### ✅ Phase 12: Analytics & Messaging Module Migration
- **Analytics Module:**
- Created `app/modules/analytics/models/` (empty - uses data from other modules)
- Created `app/modules/analytics/services/` re-exporting StatsService, UsageService
- Created `app/modules/analytics/schemas/` re-exporting stats schemas
- Created `app/modules/analytics/exceptions.py` with reporting exceptions
- Updated `definition.py` with self-contained configuration
- **Messaging Module:**
- Created `app/modules/messaging/models/` re-exporting Conversation, Message, etc.
- Created `app/modules/messaging/services/` re-exporting messaging services
- Created `app/modules/messaging/schemas/` re-exporting messaging schemas
- Created `app/modules/messaging/exceptions.py` re-exporting message exceptions
- Updated `definition.py` with self-contained configuration
---
## Module Migration Phases
Each module migration includes: services, models, schemas, tasks, templates (if any).
### Phase 5: Billing Module (High Priority)
**Why first:** Has most Celery tasks, critical business logic.
#### Components to Move
| From | To | Type |
|------|-----|------|
| `app/services/subscription_service.py` | `app/modules/billing/services/` | Service |
| `app/services/stripe_service.py` | `app/modules/billing/services/` | Service |
| `models/database/subscription.py` | `app/modules/billing/models/` | Model |
| `models/database/subscription_tier.py` | `app/modules/billing/models/` | Model |
| `models/schema/subscription.py` | `app/modules/billing/schemas/` | Schema |
| `app/tasks/celery_tasks/subscription.py` | `app/modules/billing/tasks/` | Tasks |
#### Target Structure
```
app/modules/billing/
├── __init__.py
├── definition.py # Update with tasks_path, scheduled_tasks
├── config.py # BillingConfig schema
├── exceptions.py # BillingException, SubscriptionError, etc.
├── services/
│ ├── __init__.py
│ ├── subscription_service.py
│ └── stripe_service.py
├── models/
│ ├── __init__.py
│ ├── subscription.py
│ └── subscription_tier.py
├── schemas/
│ ├── __init__.py
│ └── subscription.py
├── tasks/
│ ├── __init__.py
│ └── subscription.py # 4 scheduled tasks
├── routes/
│ ├── admin.py
│ └── vendor.py
└── migrations/
└── versions/
```
#### Tasks
| Task | Schedule | Queue |
|------|----------|-------|
| `reset_period_counters` | Daily 00:05 | scheduled |
| `check_trial_expirations` | Daily 01:00 | scheduled |
| `sync_stripe_status` | Hourly :30 | scheduled |
| `cleanup_stale_subscriptions` | Weekly Sun 03:00 | scheduled |
#### Migration Checklist
- [ ] Create `billing/services/` directory
- [ ] Move `subscription_service.py` → update imports
- [ ] Move `stripe_service.py` → update imports
- [ ] Create `billing/models/` directory
- [ ] Move subscription models → update imports
- [ ] Create `billing/schemas/` directory
- [ ] Move subscription schemas → update imports
- [ ] Create `billing/tasks/` directory
- [ ] Move subscription tasks → update task paths
- [ ] Create `billing/exceptions.py`
- [ ] Update `definition.py` with `tasks_path`, `scheduled_tasks`
- [ ] Create module migrations
- [ ] Update all imports across codebase
- [ ] Test subscription flows
- [ ] Test scheduled tasks
---
### Phase 6: Marketplace Module (High Priority)
**Why second:** Has most tasks, complex import logic.
#### Components to Move
| From | To | Type |
|------|-----|------|
| `app/services/letzshop_service.py` | `app/modules/marketplace/services/` | Service |
| `app/services/marketplace_import_service.py` | `app/modules/marketplace/services/` | Service |
| `app/services/letzshop_credentials_service.py` | `app/modules/marketplace/services/` | Service |
| `models/database/letzshop_*.py` | `app/modules/marketplace/models/` | Models |
| `models/database/marketplace_import_job.py` | `app/modules/marketplace/models/` | Model |
| `app/tasks/celery_tasks/marketplace.py` | `app/modules/marketplace/tasks/` | Tasks |
| `app/tasks/celery_tasks/letzshop.py` | `app/modules/marketplace/tasks/` | Tasks |
| `app/tasks/celery_tasks/export.py` | `app/modules/marketplace/tasks/` | Tasks |
#### Target Structure
```
app/modules/marketplace/
├── __init__.py
├── definition.py
├── exceptions.py
├── services/
│ ├── __init__.py
│ ├── letzshop_service.py
│ ├── letzshop_credentials_service.py
│ ├── import_service.py
│ └── export_service.py
├── models/
│ ├── __init__.py
│ ├── letzshop_order.py
│ ├── letzshop_credentials.py
│ └── import_job.py
├── schemas/
│ └── ...
├── tasks/
│ ├── __init__.py
│ ├── import_tasks.py # process_marketplace_import
│ ├── letzshop.py # process_historical_import, sync_vendor_directory
│ └── export.py # export tasks
└── routes/
```
#### Tasks
| Task | Type | Queue |
|------|------|-------|
| `process_marketplace_import` | On-demand | long_running |
| `process_historical_import` | On-demand | long_running |
| `sync_vendor_directory` | Scheduled (optional) | long_running |
| `export_vendor_products_to_folder` | On-demand | default |
| `export_marketplace_products` | On-demand | default |
---
### Phase 7: Dev-Tools Module (Medium Priority)
**Why:** Internal module with code quality and test runner tasks.
#### Components to Move
| From | To | Type |
|------|-----|------|
| `app/services/code_quality_service.py` | `app/modules/dev_tools/services/` | Service |
| `app/services/test_runner_service.py` | `app/modules/dev_tools/services/` | Service |
| `models/database/architecture_scan.py` | `app/modules/dev_tools/models/` | Model |
| `models/database/test_run.py` | `app/modules/dev_tools/models/` | Model |
| `app/tasks/celery_tasks/code_quality.py` | `app/modules/dev_tools/tasks/` | Tasks |
| `app/tasks/celery_tasks/test_runner.py` | `app/modules/dev_tools/tasks/` | Tasks |
#### Target Structure
```
app/modules/dev_tools/
├── __init__.py
├── definition.py
├── services/
│ ├── code_quality_service.py
│ └── test_runner_service.py
├── models/
│ ├── architecture_scan.py
│ ├── architecture_violation.py
│ └── test_run.py
├── tasks/
│ ├── code_quality.py
│ └── test_runner.py
└── routes/
```
---
### Phase 8: Monitoring Module (Medium Priority)
#### Components to Move
| From | To | Type |
|------|-----|------|
| `app/services/background_tasks_service.py` | `app/modules/monitoring/services/` | Service |
| `capture_capacity_snapshot` task | `app/modules/monitoring/tasks/` | Task |
| `models/database/capacity_snapshot.py` | `app/modules/monitoring/models/` | Model |
#### Target Structure
```
app/modules/monitoring/
├── __init__.py
├── definition.py
├── services/
│ └── background_tasks_service.py
├── models/
│ └── capacity_snapshot.py
├── tasks/
│ └── capacity.py # capture_capacity_snapshot
└── routes/
```
---
### Phase 9: Customers Module (Core)
#### Components to Move
| From | To |
|------|-----|
| `app/services/customer_service.py` | `app/modules/customers/services/` |
| `models/database/customer.py` | `app/modules/customers/models/` |
| Customer-related schemas | `app/modules/customers/schemas/` |
---
### Phase 10: Orders Module
#### Components to Move
| From | To |
|------|-----|
| `app/services/order_service.py` | `app/modules/orders/services/` |
| `models/database/order.py` | `app/modules/orders/models/` |
| Order-related schemas | `app/modules/orders/schemas/` |
---
### Phase 11: Inventory Module
#### Components to Move
| From | To |
|------|-----|
| `app/services/inventory_service.py` | `app/modules/inventory/services/` |
| `models/database/inventory*.py` | `app/modules/inventory/models/` |
---
### Phase 12: Remaining Modules
- **Analytics** - Move analytics services
- **Messaging** - Move messaging service, notification models
- **Tenancy** - Move platform, company, vendor services
---
## Phase 13: Cleanup & Full Celery Migration
After all modules are migrated:
### 13.1 Remove Fallback Tasks
Delete:
```
app/tasks/background_tasks.py
app/tasks/letzshop_tasks.py
app/tasks/subscription_tasks.py
app/tasks/code_quality_tasks.py
app/tasks/test_runner_tasks.py
```
### 13.2 Remove USE_CELERY Flag
```python
# app/core/config.py
# DELETE: USE_CELERY: bool = False
```
### 13.3 Simplify Dispatcher
Rewrite `app/tasks/dispatcher.py` to Celery-only.
### 13.4 Delete Old Task Directory
```
app/tasks/celery_tasks/ # DELETE - moved to modules
```
### 13.5 Clean Up Old Services/Models
Remove files from:
- `app/services/` (moved to modules)
- `models/database/` (moved to modules)
Leave re-exports for backward compatibility if needed.
---
## Queue Configuration
### Final Queue Structure
| Queue | Purpose | Modules |
|-------|---------|---------|
| `default` | Fast tasks | marketplace (exports) |
| `long_running` | CPU-intensive | marketplace (imports), dev-tools |
| `scheduled` | Celery Beat | billing, monitoring |
### Task Routing
```python
task_routes = {
"app.modules.billing.tasks.*": {"queue": "scheduled"},
"app.modules.marketplace.tasks.import_tasks.*": {"queue": "long_running"},
"app.modules.marketplace.tasks.letzshop.*": {"queue": "long_running"},
"app.modules.marketplace.tasks.export.*": {"queue": "default"},
"app.modules.monitoring.tasks.*": {"queue": "scheduled"},
"app.modules.dev_tools.tasks.*": {"queue": "long_running"},
}
```
---
## Migration Strategy
### For Each Module
1. **Create directory structure**
2. **Move models first** (fewest dependencies)
3. **Move services** (depend on models)
4. **Move schemas** (depend on models)
5. **Move tasks** (depend on services)
6. **Update all imports** across codebase
7. **Create re-exports** in old locations (temporary)
8. **Test thoroughly**
9. **Remove re-exports** in next phase
### Import Update Strategy
Use IDE refactoring or script:
```bash
# Example: Update subscription_service imports
find . -name "*.py" -exec sed -i 's/from app.services.subscription_service/from app.modules.billing.services.subscription_service/g' {} \;
```
---
## Verification Checklist
### Per-Module Verification
- [ ] Module directory has all components
- [ ] All imports updated
- [ ] Services work
- [ ] API routes work
- [ ] Tasks execute correctly
- [ ] Scheduled tasks run (Celery Beat)
- [ ] No circular imports
- [ ] Tests pass
### Final Verification
- [ ] All 12 tasks in modules
- [ ] `USE_CELERY` flag removed
- [ ] Fallback tasks deleted
- [ ] Old service files removed
- [ ] Old model files removed
- [ ] All tests pass
- [ ] Production smoke test
---
## Priority Order
| Priority | Phase | Module | Reason |
|----------|-------|--------|--------|
| 1 | 4 | Infrastructure | Required for task migration |
| 2 | 5 | `billing` | Most tasks, critical business logic |
| 3 | 6 | `marketplace` | Most complex, many tasks |
| 4 | 7 | `dev-tools` | Internal, low risk |
| 5 | 8 | `monitoring` | Internal, low risk |
| 6 | 9-12 | Others | Lower complexity |
| 7 | 13 | Cleanup | Final step |
---
## Estimated Effort
| Phase | Sessions | Notes |
|-------|----------|-------|
| 4 (Infrastructure) | 1 | Task base class, discovery |
| 5 (Billing) | 2-3 | Complex, many components |
| 6 (Marketplace) | 3-4 | Most complex module |
| 7-8 (Internal) | 1-2 | Simpler modules |
| 9-12 (Others) | 4-6 | Medium complexity each |
| 13 (Cleanup) | 1 | Removing old code |
**Total:** ~12-17 sessions
---
## Rollback Plan
Each phase can be rolled back independently:
1. Restore moved files to original locations
2. Revert import changes
3. Restore celery_config.py
No database schema changes during code migration = no DB rollback needed.
---
## Related Documents
- [Module System Architecture](../architecture/module-system.md)
- [Creating Modules](../development/creating-modules.md)
- [Observability](../architecture/observability.md)
- [SESSION_NOTE_2026-01-27](SESSION_NOTE_2026-01-27_module-reclassification.md)
Reference in New Issue View Git Blame Copy Permalink