orion/docs/guides/marketplace-integration.md

Marketplace Integration Guide

Complete guide for importing products from Letzshop marketplace into the platform.

Table of Contents

• Overview
• Architecture
• Store Import Interface
• Admin Import Interfaces
• Backend APIs
• Import Process Flow
• Configuration
• Troubleshooting

---

Overview

The Letzshop Product Import platform provides a complete marketplace integration system for importing products from Letzshop CSV feeds. The system supports:

• Multi-language imports (French, English, German)
• Background processing with status tracking
• Batch processing for large CSV files
• Google Shopping Feed format support
• Real-time monitoring and error reporting
• Role-based access control (Stores, Admins)

---

Architecture

System Components


                    Frontend Interfaces                       
$
  Store Portal             Admin Self-Service    Platform  
  /store/{code}/           /admin/marketplace    Monitoring
  marketplace                                      /admin/   
                                                   imports   
$
                       API Layer                              
$
  /api/v1/store/           /api/v1/admin/                   
  marketplace/*             marketplace-import-jobs/*        
$
                   Background Processing                      
$
  CSV Processor    Import Job Manager    Auto-refresh     
$
                      Data Layer                              
$
  MarketplaceProduct    Product    MarketplaceImportJob    


Data Flow

1. Import Trigger: User initiates import via UI
2. Job Creation: System creates MarketplaceImportJob record
3. Background Processing: CSV downloaded and processed in batches
4. Product Creation: MarketplaceProduct records created/updated
5. Status Updates: Job status updated in real-time
6. Store Publishing: Stores publish products to their catalog

---

Store Import Interface

URL: /store/{store_code}/marketplace

Purpose: Self-service interface for stores to import their own products

Features

Import Form

• CSV URL Input: Enter Letzshop CSV feed URL
• Language Selection: Choose French, English, or German
• Marketplace: Auto-set to "Letzshop"
• Batch Size: Configure products per batch (100-5000)

Quick Fill

• Pre-configured CSV URLs from store settings
• One-click population for each language
• Stored in store settings:
  • letzshop_csv_url_fr
  • letzshop_csv_url_en
  • letzshop_csv_url_de

Import History

• Paginated list of store's import jobs (newest first)
• Real-time status tracking
• Progress metrics (imported, updated, errors)
• Duration calculations
• Job details modal with:
  • Job configuration (marketplace, language, source URL)
  • Processing metrics and timestamps
  • Error viewer: View detailed import errors with row data
  • Pagination for large error lists

Auto-Refresh

• Refreshes every 10 seconds when active jobs exist
• Automatic status updates
• No page reload required

Usage

1. Navigate to Marketplace Import in store sidebar
2. Enter CSV URL or click Quick Fill button
3. Select language and batch size (optional)
4. Click Start Import
5. Monitor progress in Import History table

File Reference

Template: app/templates/store/marketplace.html JavaScript: static/store/js/marketplace.js Route: app/routes/store_pages.py - store_marketplace_page()

---

Admin Import Interfaces

Admins have two separate interfaces for different purposes:

1. Self-Service Import (/admin/marketplace)

Purpose: Quick import tool for triggering imports for any store

Features

• Store Selection: Dropdown to select any store
• CSV URL Input: Manual entry or quick-fill from store settings
• Import History: Shows only jobs triggered by current admin user
• Quick Access: Link to view all system imports

Use Cases

• Import products for a specific store
• Test import functionality
• One-off manual imports
• Quick store assistance

File Reference

Template: app/templates/admin/marketplace.html JavaScript: static/admin/js/marketplace.js Route: app/routes/admin_pages.py - admin_marketplace_page()

---

2. Platform Monitoring (/admin/imports)

Purpose: System-wide monitoring and oversight of all imports

Location: Platform Monitoring section in admin sidebar

Features

Statistics Dashboard

• Total Jobs: All import jobs in system
• Active Jobs: Currently pending or processing
• Completed: Successfully completed imports
• Failed: Failed import jobs

Advanced Filtering

• Filter by Store: See imports for specific store
• Filter by Status: pending, processing, completed, failed, completed_with_errors
• Filter by Marketplace: Currently "Letzshop"
• Filter by Creator: All Users or My Jobs Only

Complete Job Table

• All standard job information
• Created By column shows who triggered the import
• Store name display
• Real-time status updates
• Detailed progress metrics

Auto-Refresh

• Refreshes every 15 seconds when active jobs exist
• Automatic statistics updates

Use Cases

• Monitor platform-wide import activity
• Identify problematic stores or patterns
• Analyze import performance
• System health monitoring
• Troubleshooting import issues

File Reference

Template: app/templates/admin/imports.html JavaScript: static/admin/js/imports.js Route: app/routes/admin_pages.py - admin_imports_page()

---

Backend APIs

Store APIs

Base Path: /api/v1/store/marketplace

Endpoints

# Start new import
POST /import
Body: {
    "source_url": "https://example.com/products.csv",
    "marketplace": "Letzshop",
    "batch_size": 1000
}
Response: { "job_id": 123, "status": "pending" }

# Get job status
GET /imports/{job_id}
Response: {
    "id": 123,
    "status": "processing",
    "imported_count": 150,
    "updated_count": 25,
    "error_count": 2,
    "total_processed": 177
}

# List all store's jobs
GET /imports?page=1&limit=10
Response: {
    "items": [...],
    "total": 50,
    "page": 1,
    "limit": 10
}

Authentication: Store JWT token required File: app/api/v1/store/marketplace.py

---

Admin APIs

Base Path: /api/v1/admin/marketplace-import-jobs

Endpoints

# Start import for any store
POST /
Body: {
    "store_id": 5,
    "source_url": "https://example.com/products.csv",
    "marketplace": "Letzshop",
    "batch_size": 1000
}
Response: { "job_id": 123, "status": "pending" }

# Get job details
GET /{job_id}
Response: { ...job details... }

# List all jobs with filters
GET /?store_id=5&status=completed&created_by_me=true
Response: {
    "items": [...],
    "total": 100
}

# Get statistics
GET /stats
Response: {
    "total": 500,
    "pending": 5,
    "processing": 3,
    "completed": 480,
    "failed": 12
}

Authentication: Admin JWT token required File: app/api/v1/admin/marketplace.py

---

Import Process Flow

1. Import Initiation

User Interface
    "
API Endpoint (/store/marketplace/import or /admin/marketplace-import-jobs)
    "
Create MarketplaceImportJob (status: pending)
    "
Trigger Background Task (process_marketplace_import)
    "
Return job_id to user

2. Background Processing

Download CSV from source_url
    "
Detect encoding (utf-8, latin-1, iso-8859-1, cp1252)
    "
Detect delimiter (comma, semicolon, tab)
    "
Map columns (support Google Shopping Feed format)
    "
Process in batches (default: 1000 rows)
    "
For each row:
    "
    Parse product data
    "
    Normalize GTIN, price, availability
    "
    Create or update MarketplaceProduct
    "
    Update job counts (imported_count, updated_count, error_count)
    "
Update job status (completed, failed, completed_with_errors)

3. Status Tracking

Frontend Auto-Refresh (10-15 seconds)
    "
API Call: GET /imports/{job_id}
    "
Fetch latest job status
    "
Update UI with real-time progress

---

Configuration

Store Settings

Configure Letzshop CSV URLs in store settings:

# Database fields (models/database/store.py)
letzshop_csv_url_fr: str  # French CSV URL
letzshop_csv_url_en: str  # English CSV URL
letzshop_csv_url_de: str  # German CSV URL

Access: /store/{store_code}/settings or /admin/stores/{store_code}/edit

CSV Format Support

The system automatically detects and supports:

Encodings

• UTF-8
• Latin-1 (ISO-8859-1)
• Windows-1252 (CP1252)

Delimiters

• Comma (,)
• Semicolon (;)
• Tab (\t)

Column Formats

Standard Format:

id,title,description,price,image_link,brand,gtin
123,Product Name,Description,29.99,http://...,BrandName,1234567890123

Google Shopping Feed Format:

g:id,g:title,g:description,g:price,g:image_link,g:brand,g:gtin
123,Product Name,Description,29.99 EUR,http://...,BrandName,1234567890123

Both formats are automatically detected and mapped.

---

Database Models

MarketplaceImportJob

Table: marketplace_import_jobs

Purpose: Track import job lifecycle and metrics

class MarketplaceImportJob(Base):
    id: int
    store_id: int
    user_id: int  # Who triggered the import
    marketplace: str = "Letzshop"
    source_url: str
    language: str = "en"  # Language for translations (en, fr, de)
    status: str  # pending, processing, completed, failed, completed_with_errors

    # Metrics
    imported_count: int = 0
    updated_count: int = 0
    error_count: int = 0
    total_processed: int = 0

    # Timestamps
    started_at: datetime
    completed_at: datetime
    created_at: datetime
    updated_at: datetime

    # Relationships
    store: Store
    user: User
    errors: List[MarketplaceImportError]  # Detailed error records

File: models/database/marketplace_import_job.py

---

MarketplaceImportError

Table: marketplace_import_errors

Purpose: Store detailed information about individual import errors for review

class MarketplaceImportError(Base):
    id: int
    import_job_id: int  # FK to MarketplaceImportJob
    row_number: int  # Row in source CSV
    identifier: str  # Product ID (marketplace_product_id, gtin, etc.)
    error_type: str  # missing_title, missing_id, parse_error, etc.
    error_message: str  # Human-readable description
    row_data: dict  # JSON snapshot of key fields from failing row

    # Relationships
    import_job: MarketplaceImportJob

Error Types:

• missing_title - Row has no title field
• missing_id - Row has no product ID
• parse_error - Data parsing failed
• validation_error - Data validation failed

File: models/database/marketplace_import_job.py

---

MarketplaceProduct

Table: marketplace_products

Purpose: Staging area for imported products (store-agnostic catalog)

class MarketplaceProduct(Base):
    id: int
    marketplace: str = "Letzshop"
    marketplace_product_id: str  # Unique ID from marketplace
    store_name: str

    # Product Information (Google Shopping Feed fields)
    title: str
    description: str
    link: str
    image_link: str
    price: Decimal
    currency: str = "EUR"
    availability: str
    condition: str
    brand: str
    gtin: str
    mpn: str

    # Additional fields (40+ attributes)
    # See full model for complete list

    # Timestamps
    created_at: datetime
    updated_at: datetime

Indexes:

• marketplace, marketplace_product_id (unique)
• marketplace
• store_name
• brand
• gtin
• availability

File: models/database/marketplace_product.py

---

Product

Table: products

Purpose: Store's published product catalog (public-facing)

class Product(Base):
    id: int
    store_id: int
    marketplace_product_id: int  # FK to MarketplaceProduct

    # Store overrides
    price: Decimal  # Can override marketplace price
    sale_price: Decimal
    availability: str
    condition: str

    # Store settings
    is_featured: bool = False
    is_active: bool = True
    display_order: int

    # Inventory
    min_quantity: int
    max_quantity: int

    # Relationships
    store: Store
    marketplace_product: MarketplaceProduct

Unique Constraint: store_id, marketplace_product_id

File: models/database/product.py

---

Services Layer

CSV Processor

File: app/utils/csv_processor.py

Class: CSVProcessor

Key Methods:

def process_csv(
    url: str,
    marketplace: str,
    store_id: int,
    batch_size: int = 1000
) -> Dict[str, int]:
    """
    Download and process CSV file

    Returns:
        {
            "imported": 150,
            "updated": 25,
            "errors": 2,
            "total": 177
        }
    """

Features:

• Auto-encoding detection
• Auto-delimiter detection
• Column mapping (standard & Google Shopping Feed)
• GTIN normalization
• Price parsing with currency detection
• Batch processing
• Error collection

---

Import Job Service

File: app/services/marketplace_import_job_service.py

Class: MarketplaceImportJobService

Key Methods:

def create_import_job(
    store_id: int,
    user_id: int,
    source_url: str,
    marketplace: str = "Letzshop"
) -> MarketplaceImportJob:
    """Create new import job"""

def get_import_job_by_id(job_id: int, store_id: int) -> MarketplaceImportJob:
    """Get job with access control"""

def get_import_jobs(
    store_id: Optional[int] = None,
    marketplace: Optional[str] = None,
    status: Optional[str] = None,
    skip: int = 0,
    limit: int = 10
) -> List[MarketplaceImportJob]:
    """List jobs with filters"""

---

Background Task

File: app/tasks/background_tasks.py

Function: process_marketplace_import()

async def process_marketplace_import(
    job_id: int,
    url: str,
    marketplace: str,
    store_id: int,
    batch_size: int = 1000
):
    """
    Background task for processing marketplace imports

    Updates job status throughout:
    - pending ' processing ' completed/failed/completed_with_errors
    """

Flow:

1. Update job status to "processing"
2. Call CSVProcessor.process_csv()
3. Update job with results
4. Set final status based on errors
5. Record completion timestamp

---

Navigation Structure

Store Portal

=ñ Store Sidebar
 <à Dashboard
 =æ Products
    All Products
     Marketplace Import     Import products
    Inventory
 =Ò Sales
 ™ Settings

Admin Portal

=ñ Admin Sidebar
 <à Dashboard
 =e Users
 <ê Stores
 =Ò Marketplace Import     Self-service import (my jobs)
 =' Developer Tools
 =Ê Platform Monitoring
    =Ë Import Jobs         System-wide monitoring (all jobs)
    =Ä Application Logs
 ™ Settings

---

Troubleshooting

Import Job Stuck in "Pending"

Possible Causes:

• Background task not running
• CSV URL not accessible
• Network issues

Solutions:

1. Check background task is running
2. Verify CSV URL is accessible
3. Check server logs: tail -f logs/app.log
4. Restart background workers if needed

---

CSV Not Parsing

Possible Causes:

• Unsupported encoding
• Unsupported delimiter
• Invalid CSV format

Solutions:

1. Check CSV encoding (should be UTF-8, Latin-1, or Windows-1252)
2. Verify delimiter (comma, semicolon, or tab)
3. Check error_details in job record
4. Test with sample CSV first

---

Products Not Appearing

Possible Causes:

• Import completed but products not published
• Store filter applied
• Products inactive

Solutions:

1. Check MarketplaceProduct table for imported products
2. Verify store published products to catalog
3. Check is_active flag on Product records
4. Use admin view to see all products

---

High Error Count

Possible Causes:

• Missing required fields in CSV
• Invalid data format (price, GTIN, etc.)
• Duplicate product IDs

Solutions:

1. View error_details in job modal
2. Validate CSV format matches expected structure
3. Check for required fields: id, title, price
4. Ensure GTINs are valid (13 digits)

---

Best Practices

For Stores

1. Configure CSV URLs in settings for quick access
2. Test with small batches first (set batch_size=100)
3. Monitor import progress until completion
4. Review error details if import has errors
5. Publish products to catalog after successful import

For Admins

1. Use self-service import (/admin/marketplace) for quick one-off imports
2. Use platform monitoring (/admin/imports) for system oversight
3. Filter by store to identify store-specific issues
4. Monitor statistics for system health
5. Review failed jobs and assist stores as needed

---

API Quick Reference

Store Endpoints

Method	Endpoint	Description
POST	/api/v1/store/marketplace/import	Start import
GET	/api/v1/store/marketplace/imports/{id}	Get job status
GET	/api/v1/store/marketplace/imports	List jobs

Admin Endpoints

Method	Endpoint	Description
POST	/api/v1/admin/marketplace-import-jobs	Start import for store
GET	/api/v1/admin/marketplace-import-jobs/{id}	Get job details
GET	/api/v1/admin/marketplace-import-jobs	List all jobs
GET	/api/v1/admin/marketplace-import-jobs/stats	Get statistics

---

Related Documentation

• CSV Import Guide
• Product Management
• Store RBAC
• Admin Integration Guide

---

Version History

• v1.1 (2025-12-13): Import error tracking and improvements
  • Added MarketplaceImportError model for detailed error tracking
  • Error viewer modal with row data and pagination
  • Fixed language parameter propagation from UI to database
  • Import history tables now sorted by newest first
  • Added id DESC tiebreaker for consistent ordering
• v1.0 (2025-01-30): Initial marketplace import system
  • Store self-service import
  • Admin self-service import
  • Platform monitoring dashboard
  • Multi-language support (FR, EN, DE)
  • Background processing
  • Real-time status tracking