orion/docs/guides/marketplace-integration.md

# Marketplace Integration Guide

Complete guide for importing products from Letzshop marketplace into the platform.

## Table of Contents

- [Overview](#overview)
- [Architecture](#architecture)
- [Store Import Interface](#store-import-interface)
- [Admin Import Interfaces](#admin-import-interfaces)
- [Backend APIs](#backend-apis)
- [Import Process Flow](#import-process-flow)
- [Configuration](#configuration)
- [Troubleshooting](#troubleshooting)

---

## Overview

The Letzshop Product Import platform provides a complete marketplace integration system for importing products from Letzshop CSV feeds. The system supports:

- **Multi-language imports** (French, English, German)
- **Background processing** with status tracking
- **Batch processing** for large CSV files
- **Google Shopping Feed format** support
- **Real-time monitoring** and error reporting
- **Role-based access control** (Stores, Admins)

---

## Architecture

### System Components

```

                    Frontend Interfaces                       
$
  Store Portal             Admin Self-Service    Platform  
  /store/{code}/           /admin/marketplace    Monitoring
  marketplace                                      /admin/   
                                                   imports   
$
                       API Layer                              
$
  /api/v1/store/           /api/v1/admin/                   
  marketplace/*             marketplace-import-jobs/*        
$
                   Background Processing                      
$
  CSV Processor    Import Job Manager    Auto-refresh     
$
                      Data Layer                              
$
  MarketplaceProduct    Product    MarketplaceImportJob    

```

### Data Flow

1. **Import Trigger**: User initiates import via UI
2. **Job Creation**: System creates `MarketplaceImportJob` record
3. **Background Processing**: CSV downloaded and processed in batches
4. **Product Creation**: `MarketplaceProduct` records created/updated
5. **Status Updates**: Job status updated in real-time
6. **Store Publishing**: Stores publish products to their catalog

---

## Store Import Interface

**URL**: `/store/{store_code}/marketplace`

**Purpose**: Self-service interface for stores to import their own products

### Features

#### Import Form
- **CSV URL Input**: Enter Letzshop CSV feed URL
- **Language Selection**: Choose French, English, or German
- **Marketplace**: Auto-set to "Letzshop"
- **Batch Size**: Configure products per batch (100-5000)

#### Quick Fill
- Pre-configured CSV URLs from store settings
- One-click population for each language
- Stored in store settings:
  - `letzshop_csv_url_fr`
  - `letzshop_csv_url_en`
  - `letzshop_csv_url_de`

#### Import History
- Paginated list of store's import jobs (newest first)
- Real-time status tracking
- Progress metrics (imported, updated, errors)
- Duration calculations
- Job details modal with:
  - Job configuration (marketplace, language, source URL)
  - Processing metrics and timestamps
  - **Error viewer**: View detailed import errors with row data
  - Pagination for large error lists

#### Auto-Refresh
- Refreshes every 10 seconds when active jobs exist
- Automatic status updates
- No page reload required

### Usage

1. Navigate to **Marketplace Import** in store sidebar
2. Enter CSV URL or click **Quick Fill** button
3. Select language and batch size (optional)
4. Click **Start Import**
5. Monitor progress in Import History table

### File Reference

**Template**: `app/templates/store/marketplace.html`
**JavaScript**: `static/store/js/marketplace.js`
**Route**: `app/routes/store_pages.py` - `store_marketplace_page()`

---

## Admin Import Interfaces

Admins have **two separate interfaces** for different purposes:

### 1. Self-Service Import (`/admin/marketplace`)

**Purpose**: Quick import tool for triggering imports for any store

#### Features

- **Store Selection**: Dropdown to select any store
- **CSV URL Input**: Manual entry or quick-fill from store settings
- **Import History**: Shows only jobs triggered by current admin user
- **Quick Access**: Link to view all system imports

#### Use Cases

- Import products for a specific store
- Test import functionality
- One-off manual imports
- Quick store assistance

#### File Reference

**Template**: `app/templates/admin/marketplace.html`
**JavaScript**: `static/admin/js/marketplace.js`
**Route**: `app/routes/admin_pages.py` - `admin_marketplace_page()`

---

### 2. Platform Monitoring (`/admin/imports`)

**Purpose**: System-wide monitoring and oversight of all imports

**Location**: Platform Monitoring section in admin sidebar

#### Features

##### Statistics Dashboard
- **Total Jobs**: All import jobs in system
- **Active Jobs**: Currently pending or processing
- **Completed**: Successfully completed imports
- **Failed**: Failed import jobs

##### Advanced Filtering
- **Filter by Store**: See imports for specific store
- **Filter by Status**: pending, processing, completed, failed, completed_with_errors
- **Filter by Marketplace**: Currently "Letzshop"
- **Filter by Creator**: All Users or My Jobs Only

##### Complete Job Table
- All standard job information
- **Created By** column shows who triggered the import
- Store name display
- Real-time status updates
- Detailed progress metrics

##### Auto-Refresh
- Refreshes every 15 seconds when active jobs exist
- Automatic statistics updates

#### Use Cases

- Monitor platform-wide import activity
- Identify problematic stores or patterns
- Analyze import performance
- System health monitoring
- Troubleshooting import issues

#### File Reference

**Template**: `app/templates/admin/imports.html`
**JavaScript**: `static/admin/js/imports.js`
**Route**: `app/routes/admin_pages.py` - `admin_imports_page()`

---

## Backend APIs

### Store APIs

**Base Path**: `/api/v1/store/marketplace`

#### Endpoints

```python
# Start new import
POST /import
Body: {
    "source_url": "https://example.com/products.csv",
    "marketplace": "Letzshop",
    "batch_size": 1000
}
Response: { "job_id": 123, "status": "pending" }

# Get job status
GET /imports/{job_id}
Response: {
    "id": 123,
    "status": "processing",
    "imported_count": 150,
    "updated_count": 25,
    "error_count": 2,
    "total_processed": 177
}

# List all store's jobs
GET /imports?page=1&limit=10
Response: {
    "items": [...],
    "total": 50,
    "page": 1,
    "limit": 10
}
```

**Authentication**: Store JWT token required
**File**: `app/api/v1/store/marketplace.py`

---

### Admin APIs

**Base Path**: `/api/v1/admin/marketplace-import-jobs`

#### Endpoints

```python
# Start import for any store
POST /
Body: {
    "store_id": 5,
    "source_url": "https://example.com/products.csv",
    "marketplace": "Letzshop",
    "batch_size": 1000
}
Response: { "job_id": 123, "status": "pending" }

# Get job details
GET /{job_id}
Response: { ...job details... }

# List all jobs with filters
GET /?store_id=5&status=completed&created_by_me=true
Response: {
    "items": [...],
    "total": 100
}

# Get statistics
GET /stats
Response: {
    "total": 500,
    "pending": 5,
    "processing": 3,
    "completed": 480,
    "failed": 12
}
```

**Authentication**: Admin JWT token required
**File**: `app/api/v1/admin/marketplace.py`

---

## Import Process Flow

### 1. Import Initiation

```
User Interface
    "
API Endpoint (/store/marketplace/import or /admin/marketplace-import-jobs)
    "
Create MarketplaceImportJob (status: pending)
    "
Trigger Background Task (process_marketplace_import)
    "
Return job_id to user
```

### 2. Background Processing

```
Download CSV from source_url
    "
Detect encoding (utf-8, latin-1, iso-8859-1, cp1252)
    "
Detect delimiter (comma, semicolon, tab)
    "
Map columns (support Google Shopping Feed format)
    "
Process in batches (default: 1000 rows)
    "
For each row:
    "
    Parse product data
    "
    Normalize GTIN, price, availability
    "
    Create or update MarketplaceProduct
    "
    Update job counts (imported_count, updated_count, error_count)
    "
Update job status (completed, failed, completed_with_errors)
```

### 3. Status Tracking

```
Frontend Auto-Refresh (10-15 seconds)
    "
API Call: GET /imports/{job_id}
    "
Fetch latest job status
    "
Update UI with real-time progress
```

---

## Configuration

### Store Settings

Configure Letzshop CSV URLs in store settings:

```python
# Database fields (models/database/store.py)
letzshop_csv_url_fr: str  # French CSV URL
letzshop_csv_url_en: str  # English CSV URL
letzshop_csv_url_de: str  # German CSV URL
```

**Access**: `/store/{store_code}/settings` or `/admin/stores/{store_code}/edit`

### CSV Format Support

The system automatically detects and supports:

#### Encodings
- UTF-8
- Latin-1 (ISO-8859-1)
- Windows-1252 (CP1252)

#### Delimiters
- Comma (`,`)
- Semicolon (`;`)
- Tab (`\t`)

#### Column Formats

**Standard Format**:
```csv
id,title,description,price,image_link,brand,gtin
123,Product Name,Description,29.99,http://...,BrandName,1234567890123
```

**Google Shopping Feed Format**:
```csv
g:id,g:title,g:description,g:price,g:image_link,g:brand,g:gtin
123,Product Name,Description,29.99 EUR,http://...,BrandName,1234567890123
```

Both formats are automatically detected and mapped.

---

## Database Models

### MarketplaceImportJob

**Table**: `marketplace_import_jobs`

**Purpose**: Track import job lifecycle and metrics

```python
class MarketplaceImportJob(Base):
    id: int
    store_id: int
    user_id: int  # Who triggered the import
    marketplace: str = "Letzshop"
    source_url: str
    language: str = "en"  # Language for translations (en, fr, de)
    status: str  # pending, processing, completed, failed, completed_with_errors

    # Metrics
    imported_count: int = 0
    updated_count: int = 0
    error_count: int = 0
    total_processed: int = 0

    # Timestamps
    started_at: datetime
    completed_at: datetime
    created_at: datetime
    updated_at: datetime

    # Relationships
    store: Store
    user: User
    errors: List[MarketplaceImportError]  # Detailed error records
```

**File**: `models/database/marketplace_import_job.py`

---

### MarketplaceImportError

**Table**: `marketplace_import_errors`

**Purpose**: Store detailed information about individual import errors for review

```python
class MarketplaceImportError(Base):
    id: int
    import_job_id: int  # FK to MarketplaceImportJob
    row_number: int  # Row in source CSV
    identifier: str  # Product ID (marketplace_product_id, gtin, etc.)
    error_type: str  # missing_title, missing_id, parse_error, etc.
    error_message: str  # Human-readable description
    row_data: dict  # JSON snapshot of key fields from failing row

    # Relationships
    import_job: MarketplaceImportJob
```

**Error Types**:
- `missing_title` - Row has no title field
- `missing_id` - Row has no product ID
- `parse_error` - Data parsing failed
- `validation_error` - Data validation failed

**File**: `models/database/marketplace_import_job.py`

---

### MarketplaceProduct

**Table**: `marketplace_products`

**Purpose**: Staging area for imported products (store-agnostic catalog)

```python
class MarketplaceProduct(Base):
    id: int
    marketplace: str = "Letzshop"
    marketplace_product_id: str  # Unique ID from marketplace
    store_name: str

    # Product Information (Google Shopping Feed fields)
    title: str
    description: str
    link: str
    image_link: str
    price: Decimal
    currency: str = "EUR"
    availability: str
    condition: str
    brand: str
    gtin: str
    mpn: str

    # Additional fields (40+ attributes)
    # See full model for complete list

    # Timestamps
    created_at: datetime
    updated_at: datetime
```

**Indexes**:
- `marketplace, marketplace_product_id` (unique)
- `marketplace`
- `store_name`
- `brand`
- `gtin`
- `availability`

**File**: `models/database/marketplace_product.py`

---

### Product

**Table**: `products`

**Purpose**: Store's published product catalog (public-facing)

```python
class Product(Base):
    id: int
    store_id: int
    marketplace_product_id: int  # FK to MarketplaceProduct

    # Store overrides
    price: Decimal  # Can override marketplace price
    sale_price: Decimal
    availability: str
    condition: str

    # Store settings
    is_featured: bool = False
    is_active: bool = True
    display_order: int

    # Inventory
    min_quantity: int
    max_quantity: int

    # Relationships
    store: Store
    marketplace_product: MarketplaceProduct
```

**Unique Constraint**: `store_id, marketplace_product_id`

**File**: `models/database/product.py`

---

## Services Layer

### CSV Processor

**File**: `app/utils/csv_processor.py`

**Class**: `CSVProcessor`

**Key Methods**:

```python
def process_csv(
    url: str,
    marketplace: str,
    store_id: int,
    batch_size: int = 1000
) -> Dict[str, int]:
    """
    Download and process CSV file

    Returns:
        {
            "imported": 150,
            "updated": 25,
            "errors": 2,
            "total": 177
        }
    """
```

**Features**:
- Auto-encoding detection
- Auto-delimiter detection
- Column mapping (standard & Google Shopping Feed)
- GTIN normalization
- Price parsing with currency detection
- Batch processing
- Error collection

---

### Import Job Service

**File**: `app/services/marketplace_import_job_service.py`

**Class**: `MarketplaceImportJobService`

**Key Methods**:

```python
def create_import_job(
    store_id: int,
    user_id: int,
    source_url: str,
    marketplace: str = "Letzshop"
) -> MarketplaceImportJob:
    """Create new import job"""

def get_import_job_by_id(job_id: int, store_id: int) -> MarketplaceImportJob:
    """Get job with access control"""

def get_import_jobs(
    store_id: Optional[int] = None,
    marketplace: Optional[str] = None,
    status: Optional[str] = None,
    skip: int = 0,
    limit: int = 10
) -> List[MarketplaceImportJob]:
    """List jobs with filters"""
```

---

### Background Task

**File**: `app/tasks/background_tasks.py`

**Function**: `process_marketplace_import()`

```python
async def process_marketplace_import(
    job_id: int,
    url: str,
    marketplace: str,
    store_id: int,
    batch_size: int = 1000
):
    """
    Background task for processing marketplace imports

    Updates job status throughout:
    - pending ' processing ' completed/failed/completed_with_errors
    """
```

**Flow**:
1. Update job status to "processing"
2. Call CSVProcessor.process_csv()
3. Update job with results
4. Set final status based on errors
5. Record completion timestamp

---

## Navigation Structure

### Store Portal

```
=ñ Store Sidebar
 <à Dashboard
 =æ Products
    All Products
     Marketplace Import     Import products
    Inventory
 =Ò Sales
 ™ Settings
```

### Admin Portal

```
=ñ Admin Sidebar
 <à Dashboard
 =e Users
 <ê Stores
 =Ò Marketplace Import     Self-service import (my jobs)
 =' Developer Tools
 =Ê Platform Monitoring
    =Ë Import Jobs         System-wide monitoring (all jobs)
    =Ä Application Logs
 ™ Settings
```

---

## Troubleshooting

### Import Job Stuck in "Pending"

**Possible Causes**:
- Background task not running
- CSV URL not accessible
- Network issues

**Solutions**:
1. Check background task is running
2. Verify CSV URL is accessible
3. Check server logs: `tail -f logs/app.log`
4. Restart background workers if needed

---

### CSV Not Parsing

**Possible Causes**:
- Unsupported encoding
- Unsupported delimiter
- Invalid CSV format

**Solutions**:
1. Check CSV encoding (should be UTF-8, Latin-1, or Windows-1252)
2. Verify delimiter (comma, semicolon, or tab)
3. Check error_details in job record
4. Test with sample CSV first

---

### Products Not Appearing

**Possible Causes**:
- Import completed but products not published
- Store filter applied
- Products inactive

**Solutions**:
1. Check MarketplaceProduct table for imported products
2. Verify store published products to catalog
3. Check `is_active` flag on Product records
4. Use admin view to see all products

---

### High Error Count

**Possible Causes**:
- Missing required fields in CSV
- Invalid data format (price, GTIN, etc.)
- Duplicate product IDs

**Solutions**:
1. View error_details in job modal
2. Validate CSV format matches expected structure
3. Check for required fields: id, title, price
4. Ensure GTINs are valid (13 digits)

---

## Best Practices

### For Stores

1. **Configure CSV URLs** in settings for quick access
2. **Test with small batches** first (set batch_size=100)
3. **Monitor import progress** until completion
4. **Review error details** if import has errors
5. **Publish products** to catalog after successful import

### For Admins

1. **Use self-service import** (`/admin/marketplace`) for quick one-off imports
2. **Use platform monitoring** (`/admin/imports`) for system oversight
3. **Filter by store** to identify store-specific issues
4. **Monitor statistics** for system health
5. **Review failed jobs** and assist stores as needed

---

## API Quick Reference

### Store Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/api/v1/store/marketplace/import` | Start import |
| GET | `/api/v1/store/marketplace/imports/{id}` | Get job status |
| GET | `/api/v1/store/marketplace/imports` | List jobs |

### Admin Endpoints

| Method | Endpoint | Description |
|--------|----------|-------------|
| POST | `/api/v1/admin/marketplace-import-jobs` | Start import for store |
| GET | `/api/v1/admin/marketplace-import-jobs/{id}` | Get job details |
| GET | `/api/v1/admin/marketplace-import-jobs` | List all jobs |
| GET | `/api/v1/admin/marketplace-import-jobs/stats` | Get statistics |

---

## Related Documentation

- [CSV Import Guide](csv-import.md)
- [Product Management](product-management.md)
- [Store RBAC](../backend/store-rbac.md)
- [Admin Integration Guide](../backend/admin-integration-guide.md)

---

## Version History

- **v1.1** (2025-12-13): Import error tracking and improvements
  - Added `MarketplaceImportError` model for detailed error tracking
  - Error viewer modal with row data and pagination
  - Fixed language parameter propagation from UI to database
  - Import history tables now sorted by newest first
  - Added `id DESC` tiebreaker for consistent ordering
- **v1.0** (2025-01-30): Initial marketplace import system
  - Store self-service import
  - Admin self-service import
  - Platform monitoring dashboard
  - Multi-language support (FR, EN, DE)
  - Background processing
  - Real-time status tracking