orion/docs/guides/marketplace-integration.md

Marketplace Integration Guide

Complete guide for importing products from Letzshop marketplace into the platform.

Table of Contents

• Overview
• Architecture
• Vendor Import Interface
• Admin Import Interfaces
• Backend APIs
• Import Process Flow
• Configuration
• Troubleshooting

---

Overview

The Letzshop Product Import platform provides a complete marketplace integration system for importing products from Letzshop CSV feeds. The system supports:

• Multi-language imports (French, English, German)
• Background processing with status tracking
• Batch processing for large CSV files
• Google Shopping Feed format support
• Real-time monitoring and error reporting
• Role-based access control (Vendors, Admins)

---

Architecture

System Components


                    Frontend Interfaces                       
$
  Vendor Portal             Admin Self-Service    Platform  
  /vendor/{code}/           /admin/marketplace    Monitoring
  marketplace                                      /admin/   
                                                   imports   
$
                       API Layer                              
$
  /api/v1/vendor/           /api/v1/admin/                   
  marketplace/*             marketplace-import-jobs/*        
$
                   Background Processing                      
$
  CSV Processor    Import Job Manager    Auto-refresh     
$
                      Data Layer                              
$
  MarketplaceProduct    Product    MarketplaceImportJob    


Data Flow

1. Import Trigger: User initiates import via UI
2. Job Creation: System creates MarketplaceImportJob record
3. Background Processing: CSV downloaded and processed in batches
4. Product Creation: MarketplaceProduct records created/updated
5. Status Updates: Job status updated in real-time
6. Vendor Publishing: Vendors publish products to their catalog

---

Vendor Import Interface

URL: /vendor/{vendor_code}/marketplace

Purpose: Self-service interface for vendors to import their own products

Features

Import Form

• CSV URL Input: Enter Letzshop CSV feed URL
• Language Selection: Choose French, English, or German
• Marketplace: Auto-set to "Letzshop"
• Batch Size: Configure products per batch (100-5000)

Quick Fill

• Pre-configured CSV URLs from vendor settings
• One-click population for each language
• Stored in vendor settings:
  • letzshop_csv_url_fr
  • letzshop_csv_url_en
  • letzshop_csv_url_de

Import History

• Paginated list of vendor's import jobs (newest first)
• Real-time status tracking
• Progress metrics (imported, updated, errors)
• Duration calculations
• Job details modal with:
  • Job configuration (marketplace, language, source URL)
  • Processing metrics and timestamps
  • Error viewer: View detailed import errors with row data
  • Pagination for large error lists

Auto-Refresh

• Refreshes every 10 seconds when active jobs exist
• Automatic status updates
• No page reload required

Usage

1. Navigate to Marketplace Import in vendor sidebar
2. Enter CSV URL or click Quick Fill button
3. Select language and batch size (optional)
4. Click Start Import
5. Monitor progress in Import History table

File Reference

Template: app/templates/vendor/marketplace.html JavaScript: static/vendor/js/marketplace.js Route: app/routes/vendor_pages.py - vendor_marketplace_page()

---

Admin Import Interfaces

Admins have two separate interfaces for different purposes:

1. Self-Service Import (/admin/marketplace)

Purpose: Quick import tool for triggering imports for any vendor

Features

• Vendor Selection: Dropdown to select any vendor
• CSV URL Input: Manual entry or quick-fill from vendor settings
• Import History: Shows only jobs triggered by current admin user
• Quick Access: Link to view all system imports

Use Cases

• Import products for a specific vendor
• Test import functionality
• One-off manual imports
• Quick vendor assistance

File Reference

Template: app/templates/admin/marketplace.html JavaScript: static/admin/js/marketplace.js Route: app/routes/admin_pages.py - admin_marketplace_page()

---

2. Platform Monitoring (/admin/imports)

Purpose: System-wide monitoring and oversight of all imports

Location: Platform Monitoring section in admin sidebar

Features

Statistics Dashboard

• Total Jobs: All import jobs in system
• Active Jobs: Currently pending or processing
• Completed: Successfully completed imports
• Failed: Failed import jobs

Advanced Filtering

• Filter by Vendor: See imports for specific vendor
• Filter by Status: pending, processing, completed, failed, completed_with_errors
• Filter by Marketplace: Currently "Letzshop"
• Filter by Creator: All Users or My Jobs Only

Complete Job Table

• All standard job information
• Created By column shows who triggered the import
• Vendor name display
• Real-time status updates
• Detailed progress metrics

Auto-Refresh

• Refreshes every 15 seconds when active jobs exist
• Automatic statistics updates

Use Cases

• Monitor platform-wide import activity
• Identify problematic vendors or patterns
• Analyze import performance
• System health monitoring
• Troubleshooting import issues

File Reference

Template: app/templates/admin/imports.html JavaScript: static/admin/js/imports.js Route: app/routes/admin_pages.py - admin_imports_page()

---

Backend APIs

Vendor APIs

Base Path: /api/v1/vendor/marketplace

Endpoints

# Start new import
POST /import
Body: {
    "source_url": "https://example.com/products.csv",
    "marketplace": "Letzshop",
    "batch_size": 1000
}
Response: { "job_id": 123, "status": "pending" }

# Get job status
GET /imports/{job_id}
Response: {
    "id": 123,
    "status": "processing",
    "imported_count": 150,
    "updated_count": 25,
    "error_count": 2,
    "total_processed": 177
}

# List all vendor's jobs
GET /imports?page=1&limit=10
Response: {
    "items": [...],
    "total": 50,
    "page": 1,
    "limit": 10
}

Authentication: Vendor JWT token required File: app/api/v1/vendor/marketplace.py

---

Admin APIs

Base Path: /api/v1/admin/marketplace-import-jobs

Endpoints

# Start import for any vendor
POST /
Body: {
    "vendor_id": 5,
    "source_url": "https://example.com/products.csv",
    "marketplace": "Letzshop",
    "batch_size": 1000
}
Response: { "job_id": 123, "status": "pending" }

# Get job details
GET /{job_id}
Response: { ...job details... }

# List all jobs with filters
GET /?vendor_id=5&status=completed&created_by_me=true
Response: {
    "items": [...],
    "total": 100
}

# Get statistics
GET /stats
Response: {
    "total": 500,
    "pending": 5,
    "processing": 3,
    "completed": 480,
    "failed": 12
}

Authentication: Admin JWT token required File: app/api/v1/admin/marketplace.py

---

Import Process Flow

1. Import Initiation

User Interface
    "
API Endpoint (/vendor/marketplace/import or /admin/marketplace-import-jobs)
    "
Create MarketplaceImportJob (status: pending)
    "
Trigger Background Task (process_marketplace_import)
    "
Return job_id to user

2. Background Processing

Download CSV from source_url
    "
Detect encoding (utf-8, latin-1, iso-8859-1, cp1252)
    "
Detect delimiter (comma, semicolon, tab)
    "
Map columns (support Google Shopping Feed format)
    "
Process in batches (default: 1000 rows)
    "
For each row:
    "
    Parse product data
    "
    Normalize GTIN, price, availability
    "
    Create or update MarketplaceProduct
    "
    Update job counts (imported_count, updated_count, error_count)
    "
Update job status (completed, failed, completed_with_errors)

3. Status Tracking

Frontend Auto-Refresh (10-15 seconds)
    "
API Call: GET /imports/{job_id}
    "
Fetch latest job status
    "
Update UI with real-time progress

---

Configuration

Vendor Settings

Configure Letzshop CSV URLs in vendor settings:

# Database fields (models/database/vendor.py)
letzshop_csv_url_fr: str  # French CSV URL
letzshop_csv_url_en: str  # English CSV URL
letzshop_csv_url_de: str  # German CSV URL

Access: /vendor/{vendor_code}/settings or /admin/vendors/{vendor_code}/edit

CSV Format Support

The system automatically detects and supports:

Encodings

• UTF-8
• Latin-1 (ISO-8859-1)
• Windows-1252 (CP1252)

Delimiters

• Comma (,)
• Semicolon (;)
• Tab (\t)

Column Formats

Standard Format:

id,title,description,price,image_link,brand,gtin
123,Product Name,Description,29.99,http://...,BrandName,1234567890123

Google Shopping Feed Format:

g:id,g:title,g:description,g:price,g:image_link,g:brand,g:gtin
123,Product Name,Description,29.99 EUR,http://...,BrandName,1234567890123

Both formats are automatically detected and mapped.

---

Database Models

MarketplaceImportJob

Table: marketplace_import_jobs

Purpose: Track import job lifecycle and metrics

class MarketplaceImportJob(Base):
    id: int
    vendor_id: int
    user_id: int  # Who triggered the import
    marketplace: str = "Letzshop"
    source_url: str
    language: str = "en"  # Language for translations (en, fr, de)
    status: str  # pending, processing, completed, failed, completed_with_errors

    # Metrics
    imported_count: int = 0
    updated_count: int = 0
    error_count: int = 0
    total_processed: int = 0

    # Timestamps
    started_at: datetime
    completed_at: datetime
    created_at: datetime
    updated_at: datetime

    # Relationships
    vendor: Vendor
    user: User
    errors: List[MarketplaceImportError]  # Detailed error records

File: models/database/marketplace_import_job.py

---

MarketplaceImportError

Table: marketplace_import_errors

Purpose: Store detailed information about individual import errors for review

class MarketplaceImportError(Base):
    id: int
    import_job_id: int  # FK to MarketplaceImportJob
    row_number: int  # Row in source CSV
    identifier: str  # Product ID (marketplace_product_id, gtin, etc.)
    error_type: str  # missing_title, missing_id, parse_error, etc.
    error_message: str  # Human-readable description
    row_data: dict  # JSON snapshot of key fields from failing row

    # Relationships
    import_job: MarketplaceImportJob

Error Types:

• missing_title - Row has no title field
• missing_id - Row has no product ID
• parse_error - Data parsing failed
• validation_error - Data validation failed

File: models/database/marketplace_import_job.py

---

MarketplaceProduct

Table: marketplace_products

Purpose: Staging area for imported products (vendor-agnostic catalog)

class MarketplaceProduct(Base):
    id: int
    marketplace: str = "Letzshop"
    marketplace_product_id: str  # Unique ID from marketplace
    vendor_name: str

    # Product Information (Google Shopping Feed fields)
    title: str
    description: str
    link: str
    image_link: str
    price: Decimal
    currency: str = "EUR"
    availability: str
    condition: str
    brand: str
    gtin: str
    mpn: str

    # Additional fields (40+ attributes)
    # See full model for complete list

    # Timestamps
    created_at: datetime
    updated_at: datetime

Indexes:

• marketplace, marketplace_product_id (unique)
• marketplace
• vendor_name
• brand
• gtin
• availability

File: models/database/marketplace_product.py

---

Product

Table: products

Purpose: Vendor's published product catalog (public-facing)

class Product(Base):
    id: int
    vendor_id: int
    marketplace_product_id: int  # FK to MarketplaceProduct

    # Vendor overrides
    price: Decimal  # Can override marketplace price
    sale_price: Decimal
    availability: str
    condition: str

    # Vendor settings
    is_featured: bool = False
    is_active: bool = True
    display_order: int

    # Inventory
    min_quantity: int
    max_quantity: int

    # Relationships
    vendor: Vendor
    marketplace_product: MarketplaceProduct

Unique Constraint: vendor_id, marketplace_product_id

File: models/database/product.py

---

Services Layer

CSV Processor

File: app/utils/csv_processor.py

Class: CSVProcessor

Key Methods:

def process_csv(
    url: str,
    marketplace: str,
    vendor_id: int,
    batch_size: int = 1000
) -> Dict[str, int]:
    """
    Download and process CSV file

    Returns:
        {
            "imported": 150,
            "updated": 25,
            "errors": 2,
            "total": 177
        }
    """

Features:

• Auto-encoding detection
• Auto-delimiter detection
• Column mapping (standard & Google Shopping Feed)
• GTIN normalization
• Price parsing with currency detection
• Batch processing
• Error collection

---

Import Job Service

File: app/services/marketplace_import_job_service.py

Class: MarketplaceImportJobService

Key Methods:

def create_import_job(
    vendor_id: int,
    user_id: int,
    source_url: str,
    marketplace: str = "Letzshop"
) -> MarketplaceImportJob:
    """Create new import job"""

def get_import_job_by_id(job_id: int, vendor_id: int) -> MarketplaceImportJob:
    """Get job with access control"""

def get_import_jobs(
    vendor_id: Optional[int] = None,
    marketplace: Optional[str] = None,
    status: Optional[str] = None,
    skip: int = 0,
    limit: int = 10
) -> List[MarketplaceImportJob]:
    """List jobs with filters"""

---

Background Task

File: app/tasks/background_tasks.py

Function: process_marketplace_import()

async def process_marketplace_import(
    job_id: int,
    url: str,
    marketplace: str,
    vendor_id: int,
    batch_size: int = 1000
):
    """
    Background task for processing marketplace imports

    Updates job status throughout:
    - pending ' processing ' completed/failed/completed_with_errors
    """

Flow:

1. Update job status to "processing"
2. Call CSVProcessor.process_csv()
3. Update job with results
4. Set final status based on errors
5. Record completion timestamp

---

Navigation Structure

Vendor Portal

=ñ Vendor Sidebar
 <à Dashboard
 =æ Products
    All Products
     Marketplace Import     Import products
    Inventory
 =Ò Sales
 ™ Settings

Admin Portal

=ñ Admin Sidebar
 <à Dashboard
 =e Users
 <ê Vendors
 =Ò Marketplace Import     Self-service import (my jobs)
 =' Developer Tools
 =Ê Platform Monitoring
    =Ë Import Jobs         System-wide monitoring (all jobs)
    =Ä Application Logs
 ™ Settings

---

Troubleshooting

Import Job Stuck in "Pending"

Possible Causes:

• Background task not running
• CSV URL not accessible
• Network issues

Solutions:

1. Check background task is running
2. Verify CSV URL is accessible
3. Check server logs: tail -f logs/app.log
4. Restart background workers if needed

---

CSV Not Parsing

Possible Causes:

• Unsupported encoding
• Unsupported delimiter
• Invalid CSV format

Solutions:

1. Check CSV encoding (should be UTF-8, Latin-1, or Windows-1252)
2. Verify delimiter (comma, semicolon, or tab)
3. Check error_details in job record
4. Test with sample CSV first

---

Products Not Appearing

Possible Causes:

• Import completed but products not published
• Vendor filter applied
• Products inactive

Solutions:

1. Check MarketplaceProduct table for imported products
2. Verify vendor published products to catalog
3. Check is_active flag on Product records
4. Use admin view to see all products

---

High Error Count

Possible Causes:

• Missing required fields in CSV
• Invalid data format (price, GTIN, etc.)
• Duplicate product IDs

Solutions:

1. View error_details in job modal
2. Validate CSV format matches expected structure
3. Check for required fields: id, title, price
4. Ensure GTINs are valid (13 digits)

---

Best Practices

For Vendors

1. Configure CSV URLs in settings for quick access
2. Test with small batches first (set batch_size=100)
3. Monitor import progress until completion
4. Review error details if import has errors
5. Publish products to catalog after successful import

For Admins

1. Use self-service import (/admin/marketplace) for quick one-off imports
2. Use platform monitoring (/admin/imports) for system oversight
3. Filter by vendor to identify vendor-specific issues
4. Monitor statistics for system health
5. Review failed jobs and assist vendors as needed

---

API Quick Reference

Vendor Endpoints

Method	Endpoint	Description
POST	/api/v1/vendor/marketplace/import	Start import
GET	/api/v1/vendor/marketplace/imports/{id}	Get job status
GET	/api/v1/vendor/marketplace/imports	List jobs

Admin Endpoints

Method	Endpoint	Description
POST	/api/v1/admin/marketplace-import-jobs	Start import for vendor
GET	/api/v1/admin/marketplace-import-jobs/{id}	Get job details
GET	/api/v1/admin/marketplace-import-jobs	List all jobs
GET	/api/v1/admin/marketplace-import-jobs/stats	Get statistics

---

Related Documentation

• CSV Import Guide
• Product Management
• Vendor RBAC
• Admin Integration Guide

---

Version History

• v1.1 (2025-12-13): Import error tracking and improvements
  • Added MarketplaceImportError model for detailed error tracking
  • Error viewer modal with row data and pagination
  • Fixed language parameter propagation from UI to database
  • Import history tables now sorted by newest first
  • Added id DESC tiebreaker for consistent ordering
• v1.0 (2025-01-30): Initial marketplace import system
  • Vendor self-service import
  • Admin self-service import
  • Platform monitoring dashboard
  • Multi-language support (FR, EN, DE)
  • Background processing
  • Real-time status tracking