compiling project documentation

docs/__temp/__PROJECT_ROADMAP/slice2_doc.md

@@ -0,0 +1,808 @@
+# Slice 2: Marketplace Product Import
+## Vendor Imports Products from Letzshop
+
+**Status**: 📋 NOT STARTED  
+**Timeline**: Week 2 (5 days)  
+**Prerequisites**: Slice 1 complete
+
+## 🎯 Slice Objectives
+
+Enable vendors to import product catalogs from Letzshop marketplace via CSV files.
+
+### User Stories
+- As a Vendor Owner, I can configure my Letzshop CSV URL
+- As a Vendor Owner, I can trigger product imports from Letzshop
+- As a Vendor Owner, I can view import job status and history
+- The system processes CSV data in the background
+- As a Vendor Owner, I can see real-time import progress
+
+### Success Criteria
+- [ ] Vendor can configure Letzshop CSV URL (FR, EN, DE)
+- [ ] Vendor can trigger import jobs manually
+- [ ] System downloads and processes CSV files
+- [ ] Import status updates in real-time (Alpine.js)
+- [ ] Import history is properly tracked
+- [ ] Error handling for failed imports
+- [ ] Products stored in staging area (MarketplaceProduct table)
+- [ ] Large CSV files process without timeout
+
+## 📋 Backend Implementation
+
+### Database Models
+
+#### MarketplaceProduct Model (`models/database/marketplace_product.py`)
+
+```python
+class MarketplaceProduct(Base, TimestampMixin):
+    """
+    Staging table for imported marketplace products
+    Products stay here until vendor publishes them to catalog
+    """
+    __tablename__ = "marketplace_products"
+    
+    id = Column(Integer, primary_key=True, index=True)
+    vendor_id = Column(Integer, ForeignKey("vendors.id"), nullable=False)
+    import_job_id = Column(Integer, ForeignKey("marketplace_import_jobs.id"))
+    
+    # External identifiers
+    external_sku = Column(String, nullable=False, index=True)
+    marketplace = Column(String, default="letzshop")  # Future: other marketplaces
+    
+    # Product information (from CSV)
+    title = Column(String, nullable=False)
+    description = Column(Text)
+    price = Column(Numeric(10, 2))
+    currency = Column(String(3), default="EUR")
+    
+    # Categories and attributes
+    category = Column(String)
+    brand = Column(String)
+    attributes = Column(JSON, default=dict)  # Store all CSV columns
+    
+    # Images
+    image_urls = Column(JSON, default=list)  # List of image URLs
+    
+    # Inventory
+    stock_quantity = Column(Integer)
+    is_in_stock = Column(Boolean, default=True)
+    
+    # Status
+    is_selected = Column(Boolean, default=False)  # Ready to publish?
+    is_published = Column(Boolean, default=False)  # Already in catalog?
+    published_product_id = Column(Integer, ForeignKey("products.id"), nullable=True)
+    
+    # Metadata
+    language = Column(String(2))  # 'fr', 'en', 'de'
+    raw_data = Column(JSON)  # Store complete CSV row
+    
+    # Relationships
+    vendor = relationship("Vendor", back_populates="marketplace_products")
+    import_job = relationship("MarketplaceImportJob", back_populates="products")
+    published_product = relationship("Product", back_populates="marketplace_source")
+    
+    # Indexes
+    __table_args__ = (
+        Index('ix_marketplace_vendor_sku', 'vendor_id', 'external_sku'),
+        Index('ix_marketplace_selected', 'vendor_id', 'is_selected'),
+    )
+```
+
+#### MarketplaceImportJob Model (`models/database/marketplace.py`)
+
+```python
+class MarketplaceImportJob(Base, TimestampMixin):
+    """Track CSV import jobs"""
+    __tablename__ = "marketplace_import_jobs"
+    
+    id = Column(Integer, primary_key=True, index=True)
+    vendor_id = Column(Integer, ForeignKey("vendors.id"), nullable=False)
+    
+    # Job details
+    marketplace = Column(String, default="letzshop")
+    csv_url = Column(String, nullable=False)
+    language = Column(String(2))  # 'fr', 'en', 'de'
+    
+    # Status tracking
+    status = Column(
+        String, 
+        default="pending"
+    )  # pending, processing, completed, failed
+    
+    # Progress
+    total_rows = Column(Integer, default=0)
+    processed_rows = Column(Integer, default=0)
+    imported_count = Column(Integer, default=0)
+    updated_count = Column(Integer, default=0)
+    error_count = Column(Integer, default=0)
+    
+    # Timing
+    started_at = Column(DateTime, nullable=True)
+    completed_at = Column(DateTime, nullable=True)
+    
+    # Error handling
+    error_message = Column(Text, nullable=True)
+    error_details = Column(JSON, nullable=True)
+    
+    # Relationships
+    vendor = relationship("Vendor", back_populates="import_jobs")
+    products = relationship("MarketplaceProduct", back_populates="import_job")
+```
+
+### Pydantic Schemas
+
+#### Import Schemas (`models/schema/marketplace.py`)
+
+```python
+from pydantic import BaseModel, HttpUrl
+from typing import Optional, List
+from datetime import datetime
+
+class MarketplaceImportCreate(BaseModel):
+    """Create new import job"""
+    csv_url: HttpUrl
+    language: str = Field(..., regex="^(fr|en|de)$")
+    marketplace: str = "letzshop"
+
+class MarketplaceImportJobResponse(BaseModel):
+    """Import job details"""
+    id: int
+    vendor_id: int
+    marketplace: str
+    csv_url: str
+    language: str
+    status: str
+    total_rows: int
+    processed_rows: int
+    imported_count: int
+    updated_count: int
+    error_count: int
+    started_at: Optional[datetime]
+    completed_at: Optional[datetime]
+    error_message: Optional[str]
+    created_at: datetime
+    
+    class Config:
+        from_attributes = True
+
+class MarketplaceProductResponse(BaseModel):
+    """Marketplace product in staging"""
+    id: int
+    vendor_id: int
+    external_sku: str
+    title: str
+    description: Optional[str]
+    price: float
+    currency: str
+    category: Optional[str]
+    brand: Optional[str]
+    stock_quantity: Optional[int]
+    is_in_stock: bool
+    is_selected: bool
+    is_published: bool
+    image_urls: List[str]
+    language: str
+    created_at: datetime
+    
+    class Config:
+        from_attributes = True
+```
+
+### Service Layer
+
+#### Marketplace Service (`app/services/marketplace_service.py`)
+
+```python
+from typing import List, Dict, Any
+import csv
+import requests
+from io import StringIO
+from sqlalchemy.orm import Session
+from models.database.marketplace_product import MarketplaceProduct
+from models.database.marketplace import MarketplaceImportJob
+
+class MarketplaceService:
+    """Handle marketplace product imports"""
+    
+    async def create_import_job(
+        self,
+        vendor_id: int,
+        csv_url: str,
+        language: str,
+        db: Session
+    ) -> MarketplaceImportJob:
+        """Create new import job and start processing"""
+        
+        # Create job record
+        job = MarketplaceImportJob(
+            vendor_id=vendor_id,
+            csv_url=csv_url,
+            language=language,
+            marketplace="letzshop",
+            status="pending"
+        )
+        db.add(job)
+        db.commit()
+        db.refresh(job)
+        
+        # Trigger background processing
+        from tasks.marketplace_import import process_csv_import
+        process_csv_import.delay(job.id)
+        
+        return job
+    
+    def process_csv_import(self, job_id: int, db: Session):
+        """
+        Process CSV import (called by Celery task)
+        This is a long-running operation
+        """
+        job = db.query(MarketplaceImportJob).get(job_id)
+        if not job:
+            return
+        
+        try:
+            # Update status
+            job.status = "processing"
+            job.started_at = datetime.utcnow()
+            db.commit()
+            
+            # Download CSV
+            response = requests.get(job.csv_url, timeout=30)
+            response.raise_for_status()
+            
+            # Parse CSV
+            csv_content = StringIO(response.text)
+            reader = csv.DictReader(csv_content)
+            
+            # Count total rows
+            rows = list(reader)
+            job.total_rows = len(rows)
+            db.commit()
+            
+            # Process each row
+            for idx, row in enumerate(rows):
+                try:
+                    self._process_csv_row(job, row, db)
+                    job.processed_rows = idx + 1
+                    
+                    # Commit every 100 rows
+                    if idx % 100 == 0:
+                        db.commit()
+                        
+                except Exception as e:
+                    job.error_count += 1
+                    # Log error but continue
+            
+            # Final commit
+            job.status = "completed"
+            job.completed_at = datetime.utcnow()
+            db.commit()
+            
+        except Exception as e:
+            job.status = "failed"
+            job.error_message = str(e)
+            job.completed_at = datetime.utcnow()
+            db.commit()
+    
+    def _process_csv_row(
+        self,
+        job: MarketplaceImportJob,
+        row: Dict[str, Any],
+        db: Session
+    ):
+        """Process single CSV row"""
+        
+        # Extract fields from CSV
+        external_sku = row.get('SKU') or row.get('sku')
+        if not external_sku:
+            raise ValueError("Missing SKU in CSV row")
+        
+        # Check if product already exists
+        existing = db.query(MarketplaceProduct).filter(
+            MarketplaceProduct.vendor_id == job.vendor_id,
+            MarketplaceProduct.external_sku == external_sku
+        ).first()
+        
+        # Parse image URLs
+        image_urls = []
+        for i in range(1, 6):  # Support up to 5 images
+            img_url = row.get(f'Image{i}') or row.get(f'image_{i}')
+            if img_url:
+                image_urls.append(img_url)
+        
+        if existing:
+            # Update existing product
+            existing.title = row.get('Title') or row.get('title')
+            existing.description = row.get('Description')
+            existing.price = float(row.get('Price', 0))
+            existing.stock_quantity = int(row.get('Stock', 0))
+            existing.is_in_stock = existing.stock_quantity > 0
+            existing.category = row.get('Category')
+            existing.brand = row.get('Brand')
+            existing.image_urls = image_urls
+            existing.raw_data = row
+            existing.import_job_id = job.id
+            
+            job.updated_count += 1
+        else:
+            # Create new product
+            product = MarketplaceProduct(
+                vendor_id=job.vendor_id,
+                import_job_id=job.id,
+                external_sku=external_sku,
+                marketplace="letzshop",
+                title=row.get('Title') or row.get('title'),
+                description=row.get('Description'),
+                price=float(row.get('Price', 0)),
+                currency="EUR",
+                category=row.get('Category'),
+                brand=row.get('Brand'),
+                stock_quantity=int(row.get('Stock', 0)),
+                is_in_stock=int(row.get('Stock', 0)) > 0,
+                image_urls=image_urls,
+                language=job.language,
+                raw_data=row,
+                is_selected=False,
+                is_published=False
+            )
+            db.add(product)
+            job.imported_count += 1
+    
+    def get_import_jobs(
+        self,
+        vendor_id: int,
+        db: Session,
+        skip: int = 0,
+        limit: int = 20
+    ) -> List[MarketplaceImportJob]:
+        """Get import job history for vendor"""
+        return db.query(MarketplaceImportJob).filter(
+            MarketplaceImportJob.vendor_id == vendor_id
+        ).order_by(
+            MarketplaceImportJob.created_at.desc()
+        ).offset(skip).limit(limit).all()
+    
+    def get_marketplace_products(
+        self,
+        vendor_id: int,
+        db: Session,
+        import_job_id: Optional[int] = None,
+        is_selected: Optional[bool] = None,
+        skip: int = 0,
+        limit: int = 100
+    ) -> List[MarketplaceProduct]:
+        """Get marketplace products in staging"""
+        query = db.query(MarketplaceProduct).filter(
+            MarketplaceProduct.vendor_id == vendor_id,
+            MarketplaceProduct.is_published == False  # Only unpublished
+        )
+        
+        if import_job_id:
+            query = query.filter(MarketplaceProduct.import_job_id == import_job_id)
+        
+        if is_selected is not None:
+            query = query.filter(MarketplaceProduct.is_selected == is_selected)
+        
+        return query.order_by(
+            MarketplaceProduct.created_at.desc()
+        ).offset(skip).limit(limit).all()
+```
+
+### API Endpoints
+
+#### Marketplace Endpoints (`app/api/v1/vendor/marketplace.py`)
+
+```python
+from fastapi import APIRouter, Depends, HTTPException
+from sqlalchemy.orm import Session
+from typing import List, Optional
+
+router = APIRouter()
+
+@router.post("/import", response_model=MarketplaceImportJobResponse)
+async def trigger_import(
+    import_data: MarketplaceImportCreate,
+    current_user: User = Depends(get_current_vendor_user),
+    vendor: Vendor = Depends(get_current_vendor),
+    db: Session = Depends(get_db)
+):
+    """Trigger CSV import from marketplace"""
+    service = MarketplaceService()
+    job = await service.create_import_job(
+        vendor_id=vendor.id,
+        csv_url=str(import_data.csv_url),
+        language=import_data.language,
+        db=db
+    )
+    return job
+
+@router.get("/jobs", response_model=List[MarketplaceImportJobResponse])
+async def get_import_jobs(
+    skip: int = 0,
+    limit: int = 20,
+    current_user: User = Depends(get_current_vendor_user),
+    vendor: Vendor = Depends(get_current_vendor),
+    db: Session = Depends(get_db)
+):
+    """Get import job history"""
+    service = MarketplaceService()
+    jobs = service.get_import_jobs(vendor.id, db, skip, limit)
+    return jobs
+
+@router.get("/jobs/{job_id}", response_model=MarketplaceImportJobResponse)
+async def get_import_job(
+    job_id: int,
+    current_user: User = Depends(get_current_vendor_user),
+    vendor: Vendor = Depends(get_current_vendor),
+    db: Session = Depends(get_db)
+):
+    """Get specific import job status"""
+    job = db.query(MarketplaceImportJob).filter(
+        MarketplaceImportJob.id == job_id,
+        MarketplaceImportJob.vendor_id == vendor.id
+    ).first()
+    
+    if not job:
+        raise HTTPException(status_code=404, detail="Import job not found")
+    
+    return job
+
+@router.get("/products", response_model=List[MarketplaceProductResponse])
+async def get_marketplace_products(
+    import_job_id: Optional[int] = None,
+    is_selected: Optional[bool] = None,
+    skip: int = 0,
+    limit: int = 100,
+    current_user: User = Depends(get_current_vendor_user),
+    vendor: Vendor = Depends(get_current_vendor),
+    db: Session = Depends(get_db)
+):
+    """Get products in marketplace staging area"""
+    service = MarketplaceService()
+    products = service.get_marketplace_products(
+        vendor.id, db, import_job_id, is_selected, skip, limit
+    )
+    return products
+```
+
+### Background Tasks
+
+#### Celery Task (`tasks/marketplace_import.py`)
+
+```python
+from celery import shared_task
+from app.core.database import SessionLocal
+from app.services.marketplace_service import MarketplaceService
+
+@shared_task(bind=True, max_retries=3)
+def process_csv_import(self, job_id: int):
+    """
+    Process CSV import in background
+    This can take several minutes for large files
+    """
+    db = SessionLocal()
+    try:
+        service = MarketplaceService()
+        service.process_csv_import(job_id, db)
+    except Exception as e:
+        # Retry on failure
+        raise self.retry(exc=e, countdown=60)
+    finally:
+        db.close()
+```
+
+## 🎨 Frontend Implementation
+
+### Templates
+
+#### Import Dashboard (`templates/vendor/marketplace/imports.html`)
+
+```html
+{% extends "vendor/base_vendor.html" %}
+
+{% block title %}Product Import{% endblock %}
+
+{% block content %}
+<div x-data="marketplaceImport()" x-init="loadJobs()">
+    <!-- Page Header -->
+    <div class="page-header">
+        <h1>Marketplace Import</h1>
+        <button @click="showImportModal = true" class="btn btn-primary">
+            New Import
+        </button>
+    </div>
+    
+    <!-- Import Configuration Card -->
+    <div class="card mb-3">
+        <h3>Letzshop CSV URLs</h3>
+        <div class="config-grid">
+            <div>
+                <label>French (FR)</label>
+                <input 
+                    type="text" 
+                    x-model="vendor.letzshop_csv_url_fr"
+                    class="form-control"
+                    readonly
+                >
+            </div>
+            <div>
+                <label>English (EN)</label>
+                <input 
+                    type="text" 
+                    x-model="vendor.letzshop_csv_url_en"
+                    class="form-control"
+                    readonly
+                >
+            </div>
+            <div>
+                <label>German (DE)</label>
+                <input 
+                    type="text" 
+                    x-model="vendor.letzshop_csv_url_de"
+                    class="form-control"
+                    readonly
+                >
+            </div>
+        </div>
+        <p class="text-muted mt-2">
+            Configure these URLs in vendor settings
+        </p>
+    </div>
+    
+    <!-- Import Jobs List -->
+    <div class="card">
+        <h3>Import History</h3>
+        
+        <template x-if="jobs.length === 0 && !loading">
+            <p class="text-muted">No imports yet. Start your first import!</p>
+        </template>
+        
+        <template x-if="jobs.length > 0">
+            <table class="data-table">
+                <thead>
+                    <tr>
+                        <th>ID</th>
+                        <th>Language</th>
+                        <th>Status</th>
+                        <th>Progress</th>
+                        <th>Results</th>
+                        <th>Started</th>
+                        <th>Actions</th>
+                    </tr>
+                </thead>
+                <tbody>
+                    <template x-for="job in jobs" :key="job.id">
+                        <tr>
+                            <td><strong x-text="`#${job.id}`"></strong></td>
+                            <td x-text="job.language.toUpperCase()"></td>
+                            <td>
+                                <span 
+                                    class="badge"
+                                    :class="{
+                                        'badge-warning': job.status === 'pending' || job.status === 'processing',
+                                        'badge-success': job.status === 'completed',
+                                        'badge-danger': job.status === 'failed'
+                                    }"
+                                    x-text="job.status"
+                                ></span>
+                            </td>
+                            <td>
+                                <template x-if="job.status === 'processing'">
+                                    <div class="progress-bar">
+                                        <div 
+                                            class="progress-fill"
+                                            :style="`width: ${(job.processed_rows / job.total_rows * 100)}%`"
+                                        ></div>
+                                    </div>
+                                    <small x-text="`${job.processed_rows} / ${job.total_rows}`"></small>
+                                </template>
+                                <template x-if="job.status === 'completed'">
+                                    <span x-text="`${job.total_rows} rows`"></span>
+                                </template>
+                            </td>
+                            <td>
+                                <template x-if="job.status === 'completed'">
+                                    <div class="text-sm">
+                                        <div>✓ <span x-text="job.imported_count"></span> imported</div>
+                                        <div>↻ <span x-text="job.updated_count"></span> updated</div>
+                                        <template x-if="job.error_count > 0">
+                                            <div class="text-danger">✗ <span x-text="job.error_count"></span> errors</div>
+                                        </template>
+                                    </div>
+                                </template>
+                            </td>
+                            <td x-text="formatDate(job.started_at)"></td>
+                            <td>
+                                <button 
+                                    @click="viewProducts(job.id)"
+                                    class="btn btn-sm"
+                                    :disabled="job.status !== 'completed'"
+                                >
+                                    View Products
+                                </button>
+                            </td>
+                        </tr>
+                    </template>
+                </tbody>
+            </table>
+        </template>
+    </div>
+    
+    <!-- New Import Modal -->
+    <div x-show="showImportModal" class="modal-overlay" @click.self="showImportModal = false">
+        <div class="modal">
+            <div class="modal-header">
+                <h3>New Import</h3>
+                <button @click="showImportModal = false" class="modal-close">×</button>
+            </div>
+            <div class="modal-body">
+                <form @submit.prevent="triggerImport()">
+                    <div class="form-group">
+                        <label>Language</label>
+                        <select x-model="newImport.language" class="form-control" required>
+                            <option value="">Select language</option>
+                            <option value="fr">French (FR)</option>
+                            <option value="en">English (EN)</option>
+                            <option value="de">German (DE)</option>
+                        </select>
+                    </div>
+                    
+                    <div class="form-group">
+                        <label>CSV URL</label>
+                        <input 
+                            type="url" 
+                            x-model="newImport.csv_url"
+                            class="form-control"
+                            placeholder="https://..."
+                            required
+                        >
+                        <div class="form-help">
+                            Or use configured URL:
+                            <button 
+                                type="button"
+                                @click="newImport.csv_url = vendor.letzshop_csv_url_fr"
+                                class="btn-link"
+                                x-show="newImport.language === 'fr'"
+                            >
+                                Use FR URL
+                            </button>
+                        </div>
+                    </div>
+                    
+                    <div class="modal-footer">
+                        <button type="button" @click="showImportModal = false" class="btn btn-secondary">
+                            Cancel
+                        </button>
+                        <button type="submit" class="btn btn-primary" :disabled="importing">
+                            <span x-show="!importing">Start Import</span>
+                            <span x-show="importing" class="loading-spinner"></span>
+                        </button>
+                    </div>
+                </form>
+            </div>
+        </div>
+    </div>
+</div>
+
+<script>
+    window.vendorData = {{ vendor|tojson }};
+</script>
+{% endblock %}
+
+{% block extra_scripts %}
+<script>
+function marketplaceImport() {
+    return {
+        vendor: window.vendorData,
+        jobs: [],
+        loading: false,
+        importing: false,
+        showImportModal: false,
+        newImport: {
+            language: '',
+            csv_url: ''
+        },
+        
+        async loadJobs() {
+            this.loading = true;
+            try {
+                this.jobs = await apiClient.get('/api/v1/vendor/marketplace/jobs');
+                
+                // Poll for active jobs
+                const activeJobs = this.jobs.filter(j => 
+                    j.status === 'pending' || j.status === 'processing'
+                );
+                
+                if (activeJobs.length > 0) {
+                    setTimeout(() => this.loadJobs(), 3000);  // Poll every 3 seconds
+                }
+            } catch (error) {
+                showNotification('Failed to load imports', 'error');
+            } finally {
+                this.loading = false;
+            }
+        },
+        
+        async triggerImport() {
+            this.importing = true;
+            try {
+                const job = await apiClient.post('/api/v1/vendor/marketplace/import', {
+                    csv_url: this.newImport.csv_url,
+                    language: this.newImport.language
+                });
+                
+                this.jobs.unshift(job);
+                this.showImportModal = false;
+                this.newImport = { language: '', csv_url: '' };
+                
+                showNotification('Import started successfully', 'success');
+                
+                // Start polling
+                setTimeout(() => this.loadJobs(), 3000);
+            } catch (error) {
+                showNotification(error.message || 'Import failed', 'error');
+            } finally {
+                this.importing = false;
+            }
+        },
+        
+        viewProducts(jobId) {
+            window.location.href = `/vendor/marketplace/products?job_id=${jobId}`;
+        },
+        
+        formatDate(dateString) {
+            if (!dateString) return '-';
+            return new Date(dateString).toLocaleString();
+        }
+    }
+}
+</script>
+{% endblock %}
+```
+
+## ✅ Testing Checklist
+
+### Backend Tests
+- [ ] CSV download works with valid URL
+- [ ] CSV parsing handles various formats
+- [ ] Products created in staging table
+- [ ] Duplicate SKUs are updated, not duplicated
+- [ ] Import job status updates correctly
+- [ ] Progress tracking is accurate
+- [ ] Error handling works for invalid CSV
+- [ ] Large CSV files (10,000+ rows) process successfully
+- [ ] Celery tasks execute correctly
+
+### Frontend Tests
+- [ ] Import page loads correctly
+- [ ] New import modal works
+- [ ] Import jobs display in table
+- [ ] Real-time progress updates (polling)
+- [ ] Completed imports show results
+- [ ] Can view products from import
+- [ ] Error states display correctly
+- [ ] Loading states work correctly
+
+### Integration Tests
+- [ ] Complete import workflow works end-to-end
+- [ ] Vendor isolation maintained
+- [ ] API endpoints require authentication
+- [ ] Vendor can only see their own imports
+
+## 🚀 Deployment Checklist
+
+- [ ] Celery worker running
+- [ ] Redis/RabbitMQ configured for Celery
+- [ ] Database migrations applied
+- [ ] CSV download timeout configured
+- [ ] Error logging configured
+- [ ] Background task monitoring set up
+
+## ➡️ Next Steps
+
+After completing Slice 2, move to **Slice 3: Product Catalog Management** to enable vendors to publish imported products to their catalog.
+
+---
+
+**Slice 2 Status**: 📋 Not Started  
+**Dependencies**: Slice 1 must be complete  
+**Estimated Duration**: 5 days