docs: add multi-marketplace product architecture documentation

- Add marketplace-integration.md: Full architecture for bidirectional
  marketplace integration (products, orders, fulfillment, inventory sync)
- Add multi-marketplace-product-architecture.md: Detailed product data
  model with translations and vendor override pattern
- Add product-migration-database-changes.md: Database migration plan
  with SQL scripts and model updates
- Update mkdocs.yml with new documentation entries

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
2025-12-11 05:52:50 +01:00
parent e98a3efe57
commit 2d6410164e
4 changed files with 3708 additions and 0 deletions

File diff suppressed because it is too large Load Diff

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,783 @@
# Product Migration - Database Changes Plan
## Overview
This document details the database schema changes required for Phase 1 of the Multi-Marketplace Product Architecture. It serves as the implementation guide for the Alembic migrations.
**Related Documents:**
- [Multi-Marketplace Product Architecture](./multi-marketplace-product-architecture.md) - Full architecture design
- [Marketplace Integration Architecture](../../architecture/marketplace-integration.md) - System-wide integration plan
---
## Current State Analysis
### Existing Tables
#### `marketplace_products` (Current)
| Column | Type | Constraints | Notes |
|--------|------|-------------|-------|
| id | Integer | PK, Index | |
| marketplace_product_id | String | Unique, Index, NOT NULL | Feed product ID |
| title | String | NOT NULL | Single language only |
| description | String | | Single language only |
| link | String | | |
| image_link | String | | |
| availability | String | Index | |
| price | String | | Raw price string "19.99 EUR" |
| sale_price | String | | Raw price string |
| brand | String | Index | |
| gtin | String | Index | |
| mpn | String | | |
| condition | String | | |
| adult | String | | |
| multipack | Integer | | |
| is_bundle | String | | |
| age_group | String | | |
| color | String | | |
| gender | String | | |
| material | String | | |
| pattern | String | | |
| size | String | | |
| size_type | String | | |
| size_system | String | | |
| item_group_id | String | | |
| google_product_category | String | Index | |
| product_type | String | | Raw feed value |
| custom_label_0-4 | String | | |
| additional_image_link | String | | Single string, not array |
| unit_pricing_measure | String | | |
| unit_pricing_base_measure | String | | |
| identifier_exists | String | | |
| shipping | String | | |
| currency | String | | |
| marketplace | String | Index, Default='Letzshop' | |
| vendor_name | String | Index | |
| created_at | DateTime | | TimestampMixin |
| updated_at | DateTime | | TimestampMixin |
**Indexes:**
- `idx_marketplace_vendor` (marketplace, vendor_name)
- `idx_marketplace_brand` (marketplace, brand)
#### `products` (Current)
| Column | Type | Constraints | Notes |
|--------|------|-------------|-------|
| id | Integer | PK, Index | |
| vendor_id | Integer | FK → vendors.id, NOT NULL | |
| marketplace_product_id | Integer | FK → marketplace_products.id, NOT NULL | |
| product_id | String | | Vendor's internal SKU |
| price | Float | | Override |
| sale_price | Float | | Override |
| currency | String | | Override |
| availability | String | | Override |
| condition | String | | Override |
| is_featured | Boolean | Default=False | |
| is_active | Boolean | Default=True | |
| display_order | Integer | Default=0 | |
| min_quantity | Integer | Default=1 | |
| max_quantity | Integer | | |
| created_at | DateTime | | TimestampMixin |
| updated_at | DateTime | | TimestampMixin |
**Constraints:**
- `uq_product` UNIQUE (vendor_id, marketplace_product_id)
**Indexes:**
- `idx_product_active` (vendor_id, is_active)
- `idx_product_featured` (vendor_id, is_featured)
### Issues with Current Schema
| Issue | Impact | Solution |
|-------|--------|----------|
| No translation support | Cannot support multi-language feeds | Add translation tables |
| No product type distinction | Cannot differentiate physical/digital | Add product_type enum |
| No digital product fields | Cannot support game keys, downloads | Add digital-specific columns |
| Price as String | Harder to filter/sort by price | Add parsed numeric price |
| Single additional_image_link | Can't store multiple images properly | Add JSON array column |
| No override pattern properties | No `effective_*` helpers | Add to model layer |
| One-to-one relationship | Same product can't exist for multiple vendors | Fix to one-to-many |
---
## Target Schema
### Visual Diagram
```
┌─────────────────────────────────┐
│ marketplace_products │
├─────────────────────────────────┤
│ id (PK) │
│ marketplace_product_id (UNIQUE) │
│ marketplace │
│ vendor_name │
│ │
│ # Product Type (NEW) │
│ product_type (ENUM) │
│ is_digital │
│ digital_delivery_method (ENUM) │
│ platform │
│ region_restrictions (JSON) │
│ license_type │
│ │
│ # Pricing (ENHANCED) │
│ price (String) [raw] │
│ price_numeric (Float) [NEW] │
│ sale_price (String) [raw] │
│ sale_price_numeric (Float) [NEW]│
│ currency │
│ │
│ # Media (ENHANCED) │
│ image_link │
│ additional_image_link [legacy] │
│ additional_images (JSON) [NEW] │
│ │
│ # Attributes (NEW) │
│ attributes (JSON) │
│ │
│ # Status (NEW) │
│ is_active │
│ │
│ # Renamed │
│ product_type_raw [was product_type] │
│ │
│ # Preserved Google Shopping │
│ brand, gtin, mpn, condition... │
│ google_product_category... │
│ custom_label_0-4... │
└─────────────────────────────────┘
│ 1:N
┌─────────────────────────────────┐
│ marketplace_product_translations │
├─────────────────────────────────┤
│ id (PK) │
│ marketplace_product_id (FK) │
│ language ('en','fr','de','lb') │
│ │
│ # Localized Content │
│ title (NOT NULL) │
│ description │
│ short_description │
│ │
│ # SEO │
│ meta_title │
│ meta_description │
│ url_slug │
│ │
│ # Source Tracking │
│ source_import_id │
│ source_file │
│ │
│ created_at, updated_at │
├─────────────────────────────────┤
│ UNIQUE(marketplace_product_id, │
│ language) │
└─────────────────────────────────┘
┌─────────────────────────────────┐
│ products │
├─────────────────────────────────┤
│ id (PK) │
│ vendor_id (FK) │
│ marketplace_product_id (FK) │
│ │
│ # Renamed │
│ vendor_sku [was product_id] │
│ │
│ # Existing Overrides │
│ price │
│ sale_price │
│ currency │
│ availability │
│ condition │
│ │
│ # New Overrides │
│ brand (NEW) │
│ primary_image_url (NEW) │
│ additional_images (JSON) (NEW) │
│ download_url (NEW) │
│ license_type (NEW) │
│ fulfillment_email_template (NEW)│
│ │
│ # Vendor-Specific │
│ is_featured │
│ is_active │
│ display_order │
│ min_quantity │
│ max_quantity │
│ │
│ created_at, updated_at │
├─────────────────────────────────┤
│ UNIQUE(vendor_id, │
│ marketplace_product_id) │
└─────────────────────────────────┘
│ 1:N
┌─────────────────────────────────┐
│ product_translations │
├─────────────────────────────────┤
│ id (PK) │
│ product_id (FK) │
│ language │
│ │
│ # Overridable (NULL = inherit) │
│ title │
│ description │
│ short_description │
│ meta_title │
│ meta_description │
│ url_slug │
│ │
│ created_at, updated_at │
├─────────────────────────────────┤
│ UNIQUE(product_id, language) │
└─────────────────────────────────┘
```
---
## Migration Plan
### Migration 1: Add Product Type and Digital Fields
**File:** `alembic/versions/xxxx_add_product_type_digital_fields.py`
**Changes:**
```sql
-- Create ENUMs
CREATE TYPE product_type_enum AS ENUM ('physical', 'digital', 'service', 'subscription');
CREATE TYPE digital_delivery_enum AS ENUM ('download', 'email', 'in_app', 'streaming', 'license_key');
-- Add columns to marketplace_products
ALTER TABLE marketplace_products ADD COLUMN product_type product_type_enum NOT NULL DEFAULT 'physical';
ALTER TABLE marketplace_products ADD COLUMN is_digital BOOLEAN NOT NULL DEFAULT false;
ALTER TABLE marketplace_products ADD COLUMN digital_delivery_method digital_delivery_enum;
ALTER TABLE marketplace_products ADD COLUMN platform VARCHAR;
ALTER TABLE marketplace_products ADD COLUMN region_restrictions JSON;
ALTER TABLE marketplace_products ADD COLUMN license_type VARCHAR;
ALTER TABLE marketplace_products ADD COLUMN source_url VARCHAR;
ALTER TABLE marketplace_products ADD COLUMN attributes JSON;
ALTER TABLE marketplace_products ADD COLUMN additional_images JSON;
ALTER TABLE marketplace_products ADD COLUMN is_active BOOLEAN NOT NULL DEFAULT true;
ALTER TABLE marketplace_products ADD COLUMN price_numeric FLOAT;
ALTER TABLE marketplace_products ADD COLUMN sale_price_numeric FLOAT;
-- Rename product_type to product_type_raw (keep original feed value)
ALTER TABLE marketplace_products RENAME COLUMN product_type TO product_type_raw;
-- Add index
CREATE INDEX idx_mp_product_type ON marketplace_products (product_type, is_digital);
```
**Rollback:**
```sql
DROP INDEX idx_mp_product_type;
ALTER TABLE marketplace_products RENAME COLUMN product_type_raw TO product_type;
ALTER TABLE marketplace_products DROP COLUMN sale_price_numeric;
ALTER TABLE marketplace_products DROP COLUMN price_numeric;
ALTER TABLE marketplace_products DROP COLUMN is_active;
ALTER TABLE marketplace_products DROP COLUMN additional_images;
ALTER TABLE marketplace_products DROP COLUMN attributes;
ALTER TABLE marketplace_products DROP COLUMN source_url;
ALTER TABLE marketplace_products DROP COLUMN license_type;
ALTER TABLE marketplace_products DROP COLUMN region_restrictions;
ALTER TABLE marketplace_products DROP COLUMN platform;
ALTER TABLE marketplace_products DROP COLUMN digital_delivery_method;
ALTER TABLE marketplace_products DROP COLUMN is_digital;
ALTER TABLE marketplace_products DROP COLUMN product_type;
DROP TYPE digital_delivery_enum;
DROP TYPE product_type_enum;
```
---
### Migration 2: Create Translation Tables
**File:** `alembic/versions/xxxx_create_translation_tables.py`
**Changes:**
```sql
-- Create marketplace_product_translations
CREATE TABLE marketplace_product_translations (
id SERIAL PRIMARY KEY,
marketplace_product_id INTEGER NOT NULL REFERENCES marketplace_products(id) ON DELETE CASCADE,
language VARCHAR(5) NOT NULL,
title VARCHAR NOT NULL,
description TEXT,
short_description VARCHAR(500),
meta_title VARCHAR(70),
meta_description VARCHAR(160),
url_slug VARCHAR(255),
source_import_id INTEGER,
source_file VARCHAR,
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uq_marketplace_product_translation UNIQUE (marketplace_product_id, language)
);
CREATE INDEX idx_mpt_language ON marketplace_product_translations (language);
CREATE INDEX idx_mpt_mp_id ON marketplace_product_translations (marketplace_product_id);
-- Create product_translations
CREATE TABLE product_translations (
id SERIAL PRIMARY KEY,
product_id INTEGER NOT NULL REFERENCES products(id) ON DELETE CASCADE,
language VARCHAR(5) NOT NULL,
title VARCHAR,
description TEXT,
short_description VARCHAR(500),
meta_title VARCHAR(70),
meta_description VARCHAR(160),
url_slug VARCHAR(255),
created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
CONSTRAINT uq_product_translation UNIQUE (product_id, language)
);
CREATE INDEX idx_pt_product_language ON product_translations (product_id, language);
```
**Rollback:**
```sql
DROP TABLE product_translations;
DROP TABLE marketplace_product_translations;
```
---
### Migration 3: Add Override Fields to Products
**File:** `alembic/versions/xxxx_add_product_override_fields.py`
**Changes:**
```sql
-- Rename product_id to vendor_sku
ALTER TABLE products RENAME COLUMN product_id TO vendor_sku;
-- Add new override columns
ALTER TABLE products ADD COLUMN brand VARCHAR;
ALTER TABLE products ADD COLUMN primary_image_url VARCHAR;
ALTER TABLE products ADD COLUMN additional_images JSON;
ALTER TABLE products ADD COLUMN download_url VARCHAR;
ALTER TABLE products ADD COLUMN license_type VARCHAR;
ALTER TABLE products ADD COLUMN fulfillment_email_template VARCHAR;
-- Add index for vendor_sku
CREATE INDEX idx_product_vendor_sku ON products (vendor_id, vendor_sku);
```
**Rollback:**
```sql
DROP INDEX idx_product_vendor_sku;
ALTER TABLE products DROP COLUMN fulfillment_email_template;
ALTER TABLE products DROP COLUMN license_type;
ALTER TABLE products DROP COLUMN download_url;
ALTER TABLE products DROP COLUMN additional_images;
ALTER TABLE products DROP COLUMN primary_image_url;
ALTER TABLE products DROP COLUMN brand;
ALTER TABLE products RENAME COLUMN vendor_sku TO product_id;
```
---
### Migration 4: Data Migration
**File:** `alembic/versions/xxxx_migrate_product_data.py`
**Changes:**
```sql
-- Migrate existing title/description to translations table (default language: 'en')
INSERT INTO marketplace_product_translations (
marketplace_product_id,
language,
title,
description,
created_at,
updated_at
)
SELECT
id,
'en',
title,
description,
created_at,
updated_at
FROM marketplace_products
WHERE title IS NOT NULL
ON CONFLICT (marketplace_product_id, language) DO NOTHING;
-- Parse prices to numeric (handled in Python for complex parsing)
-- This will be done via a Python data migration function
```
**Python Migration Function:**
```python
def parse_and_update_prices(connection):
"""Parse price strings to numeric values."""
import re
# Get all marketplace products
result = connection.execute(
text("SELECT id, price, sale_price FROM marketplace_products")
)
for row in result:
price_numeric = parse_price(row.price)
sale_price_numeric = parse_price(row.sale_price)
connection.execute(
text("""
UPDATE marketplace_products
SET price_numeric = :price, sale_price_numeric = :sale_price
WHERE id = :id
"""),
{"id": row.id, "price": price_numeric, "sale_price": sale_price_numeric}
)
def parse_price(price_str: str) -> float | None:
"""Parse price string like '19.99 EUR' to float."""
if not price_str:
return None
# Extract numeric value
numbers = re.findall(r'[\d.,]+', str(price_str))
if numbers:
num_str = numbers[0].replace(',', '.')
try:
return float(num_str)
except ValueError:
pass
return None
```
**Rollback:**
```sql
-- Data migration is one-way, original columns preserved
-- No rollback needed for data
```
---
## Migration Execution Order
| Order | Migration | Risk Level | Notes |
|-------|-----------|------------|-------|
| 1 | Add product type & digital fields | Low | All columns nullable or have defaults |
| 2 | Create translation tables | Low | New tables, no existing data affected |
| 3 | Add product override fields | Low | All columns nullable |
| 4 | Data migration | Medium | Copies data, original preserved |
---
## Model Layer Updates Required
### MarketplaceProduct Model
```python
# Add to models/database/marketplace_product.py
class ProductType(str, Enum):
PHYSICAL = "physical"
DIGITAL = "digital"
SERVICE = "service"
SUBSCRIPTION = "subscription"
class DigitalDeliveryMethod(str, Enum):
DOWNLOAD = "download"
EMAIL = "email"
IN_APP = "in_app"
STREAMING = "streaming"
LICENSE_KEY = "license_key"
class MarketplaceProduct(Base, TimestampMixin):
# ... existing fields ...
# NEW FIELDS
product_type = Column(SQLEnum(ProductType), default=ProductType.PHYSICAL, nullable=False)
is_digital = Column(Boolean, default=False, nullable=False)
digital_delivery_method = Column(SQLEnum(DigitalDeliveryMethod), nullable=True)
platform = Column(String, nullable=True)
region_restrictions = Column(JSON, nullable=True)
license_type = Column(String, nullable=True)
source_url = Column(String, nullable=True)
attributes = Column(JSON, nullable=True)
additional_images = Column(JSON, nullable=True)
is_active = Column(Boolean, default=True, nullable=False)
price_numeric = Column(Float, nullable=True)
sale_price_numeric = Column(Float, nullable=True)
product_type_raw = Column(String) # Renamed from product_type
# NEW RELATIONSHIP
translations = relationship(
"MarketplaceProductTranslation",
back_populates="marketplace_product",
cascade="all, delete-orphan"
)
# Change to one-to-many
vendor_products = relationship("Product", back_populates="marketplace_product")
```
### MarketplaceProductTranslation Model (NEW)
```python
# models/database/marketplace_product_translation.py
class MarketplaceProductTranslation(Base, TimestampMixin):
__tablename__ = "marketplace_product_translations"
id = Column(Integer, primary_key=True)
marketplace_product_id = Column(
Integer,
ForeignKey("marketplace_products.id", ondelete="CASCADE"),
nullable=False
)
language = Column(String(5), nullable=False)
title = Column(String, nullable=False)
description = Column(Text, nullable=True)
short_description = Column(String(500), nullable=True)
meta_title = Column(String(70), nullable=True)
meta_description = Column(String(160), nullable=True)
url_slug = Column(String(255), nullable=True)
source_import_id = Column(Integer, nullable=True)
source_file = Column(String, nullable=True)
marketplace_product = relationship(
"MarketplaceProduct",
back_populates="translations"
)
__table_args__ = (
UniqueConstraint("marketplace_product_id", "language", name="uq_marketplace_product_translation"),
Index("idx_mpt_language", "language"),
)
```
### Product Model Updates
```python
# Update models/database/product.py
class Product(Base, TimestampMixin):
# ... existing fields ...
# RENAMED
vendor_sku = Column(String) # Was: product_id
# NEW OVERRIDE FIELDS
brand = Column(String, nullable=True)
primary_image_url = Column(String, nullable=True)
additional_images = Column(JSON, nullable=True)
download_url = Column(String, nullable=True)
license_type = Column(String, nullable=True)
fulfillment_email_template = Column(String, nullable=True)
# NEW RELATIONSHIP
translations = relationship(
"ProductTranslation",
back_populates="product",
cascade="all, delete-orphan"
)
# OVERRIDABLE FIELDS LIST
OVERRIDABLE_FIELDS = [
"price", "sale_price", "currency", "brand", "condition",
"availability", "primary_image_url", "additional_images",
"download_url", "license_type"
]
# EFFECTIVE PROPERTIES
@property
def effective_price(self) -> float | None:
if self.price is not None:
return self.price
return self.marketplace_product.price_numeric if self.marketplace_product else None
@property
def effective_brand(self) -> str | None:
if self.brand is not None:
return self.brand
return self.marketplace_product.brand if self.marketplace_product else None
# ... other effective_* properties ...
def get_override_info(self) -> dict:
"""Get all fields with inheritance flags."""
mp = self.marketplace_product
return {
"price": self.effective_price,
"price_overridden": self.price is not None,
"price_source": mp.price_numeric if mp else None,
# ... other fields ...
}
def reset_field_to_source(self, field_name: str) -> bool:
"""Reset a field to inherit from marketplace product."""
if field_name in self.OVERRIDABLE_FIELDS:
setattr(self, field_name, None)
return True
return False
```
### ProductTranslation Model (NEW)
```python
# models/database/product_translation.py
class ProductTranslation(Base, TimestampMixin):
__tablename__ = "product_translations"
id = Column(Integer, primary_key=True)
product_id = Column(
Integer,
ForeignKey("products.id", ondelete="CASCADE"),
nullable=False
)
language = Column(String(5), nullable=False)
title = Column(String, nullable=True) # NULL = inherit
description = Column(Text, nullable=True)
short_description = Column(String(500), nullable=True)
meta_title = Column(String(70), nullable=True)
meta_description = Column(String(160), nullable=True)
url_slug = Column(String(255), nullable=True)
product = relationship("Product", back_populates="translations")
OVERRIDABLE_FIELDS = [
"title", "description", "short_description",
"meta_title", "meta_description", "url_slug"
]
def get_effective_title(self) -> str | None:
if self.title is not None:
return self.title
return self._get_marketplace_translation_field("title")
def _get_marketplace_translation_field(self, field: str) -> str | None:
mp = self.product.marketplace_product
if mp:
for t in mp.translations:
if t.language == self.language:
return getattr(t, field, None)
return None
__table_args__ = (
UniqueConstraint("product_id", "language", name="uq_product_translation"),
Index("idx_pt_product_language", "product_id", "language"),
)
```
---
## Code Updates Required
### Files to Update
| File | Changes |
|------|---------|
| `models/database/__init__.py` | Export new models |
| `models/database/marketplace_product.py` | Add new fields, enums, relationship |
| `models/database/product.py` | Rename column, add override fields/properties |
| `models/schema/product.py` | Update Pydantic schemas |
| `app/services/product_service.py` | Add reset logic, translation support |
| `app/utils/csv_processor.py` | Update to use translations |
| `tests/` | Update all product-related tests |
### Import Service Updates
The CSV processor needs to:
1. Accept language parameter
2. Create/update translations instead of direct title/description
3. Parse prices to numeric
4. Detect digital products
---
## Open Questions (Requires Decision)
### 1. Keep Original Title/Description Columns?
**Option A: Keep as cache/fallback**
- Pros: Simpler migration, backward compatible
- Cons: Data duplication
**Option B: Remove after migration**
- Pros: Cleaner schema
- Cons: Breaking change, requires code updates
**Recommendation:** Option A for Phase 1, deprecate later
---
### 2. Default Language for Existing Data?
Current data appears to be English. Confirm before migration.
**Default:** `'en'`
---
### 3. Price Parsing Strategy?
**Option A: Parse during migration**
- Run Python migration to parse all existing prices
**Option B: Parse on next import only**
- Leave existing data, parse going forward
**Recommendation:** Option A - parse during migration for consistency
---
## Testing Checklist
Before running migrations in production:
- [ ] Run migrations on local database
- [ ] Run migrations on staging/test database with production data copy
- [ ] Verify data integrity after migration
- [ ] Run full test suite
- [ ] Test import with multi-language CSV
- [ ] Test override pattern (set override, reset to source)
- [ ] Test translation inheritance
- [ ] Performance test with large dataset
---
## Rollback Plan
If issues occur:
1. Migrations are designed to be reversible
2. Each migration has explicit downgrade function
3. Original data is preserved (title/description columns kept)
4. Run `alembic downgrade -1` for each migration in reverse order
---
## Next Steps
1. [ ] Confirm open questions above
2. [ ] Create Alembic migration files
3. [ ] Update SQLAlchemy models
4. [ ] Update Pydantic schemas
5. [ ] Update services
6. [ ] Update tests
7. [ ] Run on staging environment
8. [ ] Deploy to production

View File

@@ -29,6 +29,7 @@ nav:
# ============================================
- Architecture:
- Overview: architecture/overview.md
- Marketplace Integration: architecture/marketplace-integration.md
- Architecture Patterns: architecture/architecture-patterns.md
- Company-Vendor Management: architecture/company-vendor-management.md
- Multi-Tenant System: architecture/multi-tenant.md
@@ -127,6 +128,8 @@ nav:
- Makefile Refactoring: development/migration/makefile-refactoring-complete.md
- SVC-006 Migration Plan: development/migration/svc-006-migration-plan.md
- Vendor Contact Inheritance: development/migration/vendor-contact-inheritance.md
- Multi-Marketplace Product Architecture: development/migration/multi-marketplace-product-architecture.md
- Product Migration Database Changes: development/migration/product-migration-database-changes.md
- Seed Scripts Audit: development/seed-scripts-audit.md
- Database Seeder:
- Documentation: development/database-seeder/database-seeder-documentation.md