feat: add logging, marketplace, and admin enhancements
Database & Migrations: - Add application_logs table migration for hybrid cloud logging - Add companies table migration and restructure vendor relationships Logging System: - Implement hybrid logging system (database + file) - Add log_service for centralized log management - Create admin logs page with filtering and viewing capabilities - Add init_log_settings.py script for log configuration - Enhance core logging with database integration Marketplace Integration: - Add marketplace admin page with product management - Create marketplace vendor page with product listings - Implement marketplace.js for both admin and vendor interfaces - Add marketplace integration documentation Admin Enhancements: - Add imports management page and functionality - Create settings page for admin configuration - Add vendor themes management page - Enhance vendor detail and edit pages - Improve code quality dashboard and violation details - Add logs viewing and management - Update icons guide and shared icon system Architecture & Documentation: - Document frontend structure and component architecture - Document models structure and relationships - Add vendor-in-token architecture documentation - Add vendor RBAC (role-based access control) documentation - Document marketplace integration patterns - Update architecture patterns documentation Infrastructure: - Add platform static files structure (css, img, js) - Move architecture_scan.py to proper models location - Update model imports and registrations - Enhance exception handling - Update dependency injection patterns UI/UX: - Improve vendor edit interface - Update admin user interface - Enhance page templates documentation - Add vendor marketplace interface
This commit is contained in:
469
docs/architecture/models-structure.md
Normal file
469
docs/architecture/models-structure.md
Normal file
@@ -0,0 +1,469 @@
|
||||
# Models Structure
|
||||
|
||||
## Overview
|
||||
|
||||
This project follows a **standardized models structure** at the root level, separating database models from Pydantic schemas.
|
||||
|
||||
## Directory Structure
|
||||
|
||||
```
|
||||
models/
|
||||
├── database/ # SQLAlchemy database models (ORM)
|
||||
│ ├── __init__.py
|
||||
│ ├── user.py
|
||||
│ ├── vendor.py
|
||||
│ ├── product.py
|
||||
│ ├── order.py
|
||||
│ ├── admin.py
|
||||
│ ├── architecture_scan.py
|
||||
│ └── ...
|
||||
│
|
||||
└── schema/ # Pydantic schemas (API validation)
|
||||
├── __init__.py
|
||||
├── auth.py
|
||||
├── admin.py
|
||||
├── product.py
|
||||
├── order.py
|
||||
└── ...
|
||||
```
|
||||
|
||||
## Important Rules
|
||||
|
||||
### ✅ DO: Use Root-Level Models
|
||||
|
||||
**ALL models must be in the root `models/` directory:**
|
||||
- Database models → `models/database/`
|
||||
- Pydantic schemas → `models/schema/`
|
||||
|
||||
### ❌ DON'T: Create `app/models/`
|
||||
|
||||
**NEVER create or use `app/models/` directory.**
|
||||
|
||||
The application structure is:
|
||||
```
|
||||
app/ # Application code (routes, services, core)
|
||||
models/ # Models (database & schemas)
|
||||
```
|
||||
|
||||
NOT:
|
||||
```
|
||||
app/
|
||||
models/ # ❌ WRONG - Don't create this!
|
||||
models/ # ✓ Correct location
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Database Models (`models/database/`)
|
||||
|
||||
### Purpose
|
||||
SQLAlchemy ORM models that represent database tables.
|
||||
|
||||
### Naming Convention
|
||||
- Singular class names: `User`, `Product`, `Order`
|
||||
- File names match class: `user.py`, `product.py`, `order.py`
|
||||
|
||||
### Example Structure
|
||||
|
||||
**File:** `models/database/product.py`
|
||||
```python
|
||||
"""Product database model"""
|
||||
|
||||
from sqlalchemy import Column, Integer, String, Float, ForeignKey
|
||||
from sqlalchemy.orm import relationship
|
||||
|
||||
from .base import Base
|
||||
|
||||
|
||||
class Product(Base):
|
||||
"""Product database model"""
|
||||
|
||||
__tablename__ = "products"
|
||||
|
||||
id = Column(Integer, primary_key=True, index=True)
|
||||
name = Column(String(255), nullable=False)
|
||||
price = Column(Float, nullable=False)
|
||||
vendor_id = Column(Integer, ForeignKey("vendors.id"))
|
||||
|
||||
# Relationships
|
||||
vendor = relationship("Vendor", back_populates="products")
|
||||
```
|
||||
|
||||
### Exporting Models
|
||||
|
||||
All database models must be exported in `models/database/__init__.py`:
|
||||
|
||||
```python
|
||||
# models/database/__init__.py
|
||||
from .user import User
|
||||
from .vendor import Vendor
|
||||
from .product import Product
|
||||
from .order import Order, OrderItem
|
||||
|
||||
__all__ = [
|
||||
"User",
|
||||
"Vendor",
|
||||
"Product",
|
||||
"Order",
|
||||
"OrderItem",
|
||||
]
|
||||
```
|
||||
|
||||
### Importing Database Models
|
||||
|
||||
```python
|
||||
# ✅ CORRECT - Import from models.database
|
||||
from models.database import User, Product
|
||||
from models.database.vendor import Vendor
|
||||
|
||||
# ❌ WRONG - Don't import from app.models
|
||||
from app.models.user import User # This path doesn't exist!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Pydantic Schemas (`models/schema/`)
|
||||
|
||||
### Purpose
|
||||
Pydantic models for API request/response validation and serialization.
|
||||
|
||||
### Naming Convention
|
||||
- Use descriptive suffixes: `Create`, `Update`, `Response`, `InDB`
|
||||
- Group related schemas in same file
|
||||
- File names match domain: `auth.py`, `product.py`, `order.py`
|
||||
|
||||
### Example Structure
|
||||
|
||||
**File:** `models/schema/product.py`
|
||||
```python
|
||||
"""Product Pydantic schemas"""
|
||||
|
||||
from typing import Optional
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
|
||||
class ProductBase(BaseModel):
|
||||
"""Base product schema"""
|
||||
name: str = Field(..., min_length=1, max_length=255)
|
||||
description: Optional[str] = None
|
||||
price: float = Field(..., gt=0)
|
||||
|
||||
|
||||
class ProductCreate(ProductBase):
|
||||
"""Schema for creating a product"""
|
||||
vendor_id: int
|
||||
|
||||
|
||||
class ProductUpdate(BaseModel):
|
||||
"""Schema for updating a product"""
|
||||
name: Optional[str] = Field(None, min_length=1, max_length=255)
|
||||
description: Optional[str] = None
|
||||
price: Optional[float] = Field(None, gt=0)
|
||||
|
||||
|
||||
class ProductResponse(ProductBase):
|
||||
"""Schema for product API response"""
|
||||
id: int
|
||||
vendor_id: int
|
||||
|
||||
class Config:
|
||||
from_attributes = True # Pydantic v2
|
||||
# orm_mode = True # Pydantic v1
|
||||
```
|
||||
|
||||
### Exporting Schemas
|
||||
|
||||
Export schemas in `models/schema/__init__.py`:
|
||||
|
||||
```python
|
||||
# models/schema/__init__.py
|
||||
from .auth import LoginRequest, TokenResponse
|
||||
from .product import ProductCreate, ProductUpdate, ProductResponse
|
||||
|
||||
__all__ = [
|
||||
"LoginRequest",
|
||||
"TokenResponse",
|
||||
"ProductCreate",
|
||||
"ProductUpdate",
|
||||
"ProductResponse",
|
||||
]
|
||||
```
|
||||
|
||||
### Importing Schemas
|
||||
|
||||
```python
|
||||
# ✅ CORRECT
|
||||
from models.schema import ProductCreate, ProductResponse
|
||||
from models.schema.auth import LoginRequest
|
||||
|
||||
# ❌ WRONG
|
||||
from app.models.schema.product import ProductCreate
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Pattern 1: Database Model with Schema
|
||||
|
||||
**Database Model:** `models/database/vendor.py`
|
||||
```python
|
||||
from sqlalchemy import Column, Integer, String, Boolean
|
||||
from .base import Base
|
||||
|
||||
class Vendor(Base):
|
||||
__tablename__ = "vendors"
|
||||
|
||||
id = Column(Integer, primary_key=True)
|
||||
name = Column(String(255), nullable=False)
|
||||
code = Column(String(50), unique=True, nullable=False)
|
||||
is_active = Column(Boolean, default=True)
|
||||
```
|
||||
|
||||
**Pydantic Schema:** `models/schema/vendor.py`
|
||||
```python
|
||||
from pydantic import BaseModel
|
||||
|
||||
class VendorBase(BaseModel):
|
||||
name: str
|
||||
code: str
|
||||
|
||||
class VendorCreate(VendorBase):
|
||||
pass
|
||||
|
||||
class VendorResponse(VendorBase):
|
||||
id: int
|
||||
is_active: bool
|
||||
|
||||
class Config:
|
||||
from_attributes = True
|
||||
```
|
||||
|
||||
**Usage in API:**
|
||||
```python
|
||||
from fastapi import APIRouter
|
||||
from sqlalchemy.orm import Session
|
||||
|
||||
from models.database import Vendor
|
||||
from models.schema import VendorCreate, VendorResponse
|
||||
|
||||
router = APIRouter()
|
||||
|
||||
@router.post("/vendors", response_model=VendorResponse)
|
||||
def create_vendor(vendor_data: VendorCreate, db: Session):
|
||||
# VendorCreate validates input
|
||||
db_vendor = Vendor(**vendor_data.dict())
|
||||
db.add(db_vendor)
|
||||
db.commit()
|
||||
db.refresh(db_vendor)
|
||||
# VendorResponse serializes output
|
||||
return db_vendor
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Pattern 2: Complex Schemas
|
||||
|
||||
For complex domains, organize schemas by purpose:
|
||||
|
||||
```python
|
||||
# models/schema/order.py
|
||||
class OrderBase(BaseModel):
|
||||
"""Base order fields"""
|
||||
pass
|
||||
|
||||
class OrderCreate(OrderBase):
|
||||
"""Create order from customer"""
|
||||
items: List[OrderItemCreate]
|
||||
|
||||
class OrderUpdate(BaseModel):
|
||||
"""Admin order update"""
|
||||
status: Optional[OrderStatus]
|
||||
|
||||
class OrderResponse(OrderBase):
|
||||
"""Order API response"""
|
||||
id: int
|
||||
items: List[OrderItemResponse]
|
||||
|
||||
class OrderAdminResponse(OrderResponse):
|
||||
"""Extended response for admin"""
|
||||
internal_notes: Optional[str]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Migration Guide
|
||||
|
||||
If you accidentally created models in the wrong location:
|
||||
|
||||
### Moving Database Models
|
||||
|
||||
```bash
|
||||
# If you created app/models/my_model.py (WRONG)
|
||||
# Move to correct location:
|
||||
mv app/models/my_model.py models/database/my_model.py
|
||||
|
||||
# Update imports in all files
|
||||
# FROM: from app.models.my_model import MyModel
|
||||
# TO: from models.database.my_model import MyModel
|
||||
|
||||
# Add to models/database/__init__.py
|
||||
# Remove app/models/ directory
|
||||
rm -rf app/models/
|
||||
```
|
||||
|
||||
### Moving Pydantic Schemas
|
||||
|
||||
```bash
|
||||
# If you created app/schemas/my_schema.py (WRONG)
|
||||
# Move to correct location:
|
||||
mv app/schemas/my_schema.py models/schema/my_schema.py
|
||||
|
||||
# Update imports
|
||||
# FROM: from app.schemas.my_schema import MySchema
|
||||
# TO: from models.schema.my_schema import MySchema
|
||||
|
||||
# Add to models/schema/__init__.py
|
||||
# Remove app/schemas/ directory
|
||||
rm -rf app/schemas/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Why This Structure?
|
||||
|
||||
### ✅ Benefits
|
||||
|
||||
1. **Clear Separation**
|
||||
- Database layer separate from application layer
|
||||
- Easy to understand where models live
|
||||
|
||||
2. **Import Consistency**
|
||||
- `from models.database import ...`
|
||||
- `from models.schema import ...`
|
||||
- No confusion about import paths
|
||||
|
||||
3. **Testing**
|
||||
- Easy to mock database models
|
||||
- Easy to test schema validation
|
||||
|
||||
4. **Scalability**
|
||||
- Models can be used by multiple apps
|
||||
- Clean separation of concerns
|
||||
|
||||
5. **Tool Compatibility**
|
||||
- Alembic migrations find models easily
|
||||
- IDE autocomplete works better
|
||||
- Linters understand structure
|
||||
|
||||
### ❌ Problems with `app/models/`
|
||||
|
||||
1. **Confusion**: Is it database or schema?
|
||||
2. **Import Issues**: Circular dependencies
|
||||
3. **Migration Problems**: Alembic can't find models
|
||||
4. **Inconsistency**: Different parts of codebase use different paths
|
||||
|
||||
---
|
||||
|
||||
## Verification Checklist
|
||||
|
||||
Use this checklist when adding new models:
|
||||
|
||||
### Database Model Checklist
|
||||
- [ ] File in `models/database/{name}.py`
|
||||
- [ ] Inherits from `Base`
|
||||
- [ ] Has `__tablename__` defined
|
||||
- [ ] Exported in `models/database/__init__.py`
|
||||
- [ ] Imported using `from models.database import ...`
|
||||
- [ ] NO file in `app/models/`
|
||||
|
||||
### Pydantic Schema Checklist
|
||||
- [ ] File in `models/schema/{name}.py`
|
||||
- [ ] Inherits from `BaseModel`
|
||||
- [ ] Has descriptive suffix (`Create`, `Update`, `Response`)
|
||||
- [ ] Exported in `models/schema/__init__.py`
|
||||
- [ ] Imported using `from models.schema import ...`
|
||||
- [ ] NO file in `app/schemas/`
|
||||
|
||||
---
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
project/
|
||||
├── app/
|
||||
│ ├── api/ # API routes
|
||||
│ ├── core/ # Core functionality (config, database, auth)
|
||||
│ ├── services/ # Business logic
|
||||
│ ├── templates/ # Jinja2 templates
|
||||
│ └── routes/ # Page routes
|
||||
│
|
||||
├── models/ # ✓ Models live here!
|
||||
│ ├── database/ # ✓ SQLAlchemy models
|
||||
│ └── schema/ # ✓ Pydantic schemas
|
||||
│
|
||||
├── static/ # Frontend assets
|
||||
├── docs/ # Documentation
|
||||
├── tests/ # Tests
|
||||
└── scripts/ # Utility scripts
|
||||
```
|
||||
|
||||
**NOT:**
|
||||
```
|
||||
app/
|
||||
models/ # ❌ Don't create this
|
||||
schemas/ # ❌ Don't create this
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Examples from the Codebase
|
||||
|
||||
### ✅ Correct Examples
|
||||
|
||||
**Database Model:**
|
||||
```python
|
||||
# models/database/architecture_scan.py
|
||||
from sqlalchemy import Column, Integer, String
|
||||
from .base import Base
|
||||
|
||||
class ArchitectureScan(Base):
|
||||
__tablename__ = "architecture_scans"
|
||||
id = Column(Integer, primary_key=True)
|
||||
```
|
||||
|
||||
**Import in Service:**
|
||||
```python
|
||||
# app/services/code_quality_service.py
|
||||
from models.database.architecture_scan import ArchitectureScan
|
||||
```
|
||||
|
||||
**Pydantic Schema:**
|
||||
```python
|
||||
# models/schema/admin.py
|
||||
from pydantic import BaseModel
|
||||
|
||||
class AdminDashboardStats(BaseModel):
|
||||
total_vendors: int
|
||||
total_users: int
|
||||
```
|
||||
|
||||
**Import in API:**
|
||||
```python
|
||||
# app/api/v1/admin/dashboard.py
|
||||
from models.schema.admin import AdminDashboardStats
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
**Golden Rule:** All models in `models/`, never in `app/models/` or `app/schemas/`.
|
||||
|
||||
**Quick Reference:**
|
||||
- Database models → `models/database/`
|
||||
- Pydantic schemas → `models/schema/`
|
||||
- Import pattern → `from models.{type} import ...`
|
||||
- No models in `app/` directory
|
||||
|
||||
This standard ensures consistency, clarity, and maintainability across the entire project.
|
||||
Reference in New Issue
Block a user