feat: enhance Letzshop order import with EAN matching and stats

- Add historical order import with pagination support - Add customer_locale, shipping_country_iso, billing_country_iso columns - Add gtin/gtin_type columns to Product table for EAN matching - Fix order stats to count all orders server-side (not just visible page) - Add GraphQL introspection script with tracking workaround tests - Enrich inventory units with EAN, MPN, SKU, product name - Add LetzshopOrderStats schema for proper status counts Migrations: - a9a86cef6cca: Add locale and country fields to letzshop_orders - cb88bc9b5f86: Add gtin columns to products table 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2025-12-18 21:04:33 +01:00
parent 6d6c8b44d3
commit 0ab10128ae
17 changed files with 3451 additions and 94 deletions
--- a/docs/implementation/letzshop-order-import-improvements.md
+++ b/docs/implementation/letzshop-order-import-improvements.md
@@ -0,0 +1,517 @@
+# Letzshop Order Import - Improvement Plan
+
+## Current Status (2025-12-17)
+
+### Schema Discovery Complete ✅
+
+After running GraphQL introspection queries, we have identified all available fields.
+
+### Available Fields Summary
+
+| Data | GraphQL Path | Notes |
+|------|-------------|-------|
+| **EAN/GTIN** | `variant.tradeId.number` | The product barcode |
+| **Trade ID Type** | `variant.tradeId.parser` | Format: gtin13, gtin14, gtin12, gtin8, isbn13, isbn10 |
+| **Brand Name** | `product._brand { ... on Brand { name } }` | Union type requires fragment |
+| **MPN** | `variant.mpn` | Manufacturer Part Number |
+| **SKU** | `variant.sku` | Merchant's internal SKU |
+| **Product Name** | `variant.product.name { en, fr, de }` | Translated names |
+| **Price** | `variant.price` | Unit price |
+| **Quantity** | Count of `inventoryUnits` | Each unit = 1 item |
+| **Customer Language** | `order.locale` | Language for invoice (en, fr, de) |
+| **Customer Country** | `order.shipAddress.country` | Country object |
+
+### Key Findings
+
+1. **EAN lives in `tradeId`** - Not a direct field on Variant, but nested in `tradeId.number`
+2. **TradeIdParser enum values**: `gtin14`, `gtin13` (EAN-13), `gtin12` (UPC), `gtin8`, `isbn13`, `isbn10`
+3. **Brand is a Union** - Must use `... on Brand { name }` fragment, also handles `BrandUnknown`
+4. **No quantity field** - Each InventoryUnit represents 1 item; count units to get quantity
+
+## Updated GraphQL Query
+
+```graphql
+query {
+  shipments(state: unconfirmed) {
+    nodes {
+      id
+      number
+      state
+      order {
+        id
+        number
+        email
+        total
+        completedAt
+        locale
+        shipAddress {
+          firstName
+          lastName
+          company
+          streetName
+          streetNumber
+          city
+          zipCode
+          phone
+          country {
+            name
+            iso
+          }
+        }
+        billAddress {
+          firstName
+          lastName
+          company
+          streetName
+          streetNumber
+          city
+          zipCode
+          phone
+          country {
+            name
+            iso
+          }
+        }
+      }
+      inventoryUnits {
+        id
+        state
+        variant {
+          id
+          sku
+          mpn
+          price
+          tradeId {
+            number
+            parser
+          }
+          product {
+            name { en fr de }
+            _brand {
+              ... on Brand { name }
+            }
+          }
+        }
+      }
+      tracking {
+        code
+        provider
+      }
+    }
+  }
+}
+```
+
+## Implementation Steps
+
+### Step 1: Update GraphQL Queries ✅ DONE
+Update in `app/services/letzshop/client_service.py`:
+- `QUERY_SHIPMENTS_UNCONFIRMED` ✅
+- `QUERY_SHIPMENTS_CONFIRMED` ✅
+- `QUERY_SHIPMENT_BY_ID` ✅
+- `QUERY_SHIPMENTS_PAGINATED_TEMPLATE` ✅ (new - for historical import)
+
+### Step 2: Update Order Service ✅ DONE
+Updated `create_order()` and `update_order_from_shipment()` in `app/services/letzshop/order_service.py`:
+- Extract `tradeId.number` as EAN ✅
+- Store MPN if available ✅
+- Store `locale` for invoice language ✅
+- Store shipping/billing country ISO codes ✅
+- Enrich inventory_units with EAN, MPN, SKU, product_name ✅
+
+**Database changes:**
+- Added `customer_locale` column to `LetzshopOrder`
+- Added `shipping_country_iso` column to `LetzshopOrder`
+- Added `billing_country_iso` column to `LetzshopOrder`
+- Migration: `a9a86cef6cca_add_letzshop_order_locale_and_country_.py`
+
+### Step 3: Match Products by EAN ✅ DONE (Basic)
+When importing orders:
+- Use `tradeId.number` (EAN) to find matching local product ✅
+- `_match_eans_to_products()` function added ✅
+- Returns match statistics (products_matched, products_not_found) ✅
+
+**TODO for later:**
+- ⬜ Decrease stock for matched product (needs careful implementation)
+- ⬜ Show match status in order detail view
+
+### Step 4: Update Frontend ✅ DONE (Historical Import)
+- Added "Import History" button to Orders tab ✅
+- Added historical import result display ✅
+- Added `importHistoricalOrders()` JavaScript function ✅
+
+**TODO for later:**
+- ⬜ Show product details in individual order view (EAN, MPN, SKU, match status)
+
+### Step 5: Historical Import Feature ✅ DONE
+Import all confirmed orders for:
+- Sales analytics (how many products sold)
+- Customer records
+- Historical data
+
+**Implementation:**
+- Pagination support with `get_all_shipments_paginated()` ✅
+- Deduplication by `letzshop_order_id` ✅
+- EAN matching during import ✅
+- Progress callback for large imports ✅
+
+**Endpoints Added:**
+- `POST /api/v1/admin/letzshop/vendors/{id}/import-history` - Import historical orders
+- `GET /api/v1/admin/letzshop/vendors/{id}/import-summary` - Get import statistics
+
+**Frontend:**
+- "Import History" button in Orders tab
+- Result display showing imported/updated/skipped counts
+
+**Tests:**
+- Unit tests in `tests/unit/services/test_letzshop_service.py` ✅
+- Manual test script `scripts/test_historical_import.py` ✅
+
+## Test Results (2025-12-17)
+
+### Query Test: PASSED ✅
+
+```
+Example shipment:
+  Shipment #: H43748338602
+  Order #: R702236251
+  Customer: miriana.leal@letzshop.lu
+  Locale: fr <<<< LANGUAGE
+  Total: 32.88 EUR
+
+  Ship to: Miriana Leal Ferreira
+  City: 1468 Luxembourg
+  Country: LU
+
+  Items (1):
+    - Pocket POP! Keychains: Marvel Avengers Infinity War - Iron Spider
+      SKU: 00889698273022
+      MPN: None
+      EAN: 00889698273022 (gtin14) <<<< BARCODE
+      Price: 5.88 EUR
+```
+
+### Known Issues / Letzshop API Bugs
+
+#### Bug 1: `_brand` field causes server error
+- **Error**: `NoMethodError: undefined method 'demodulize' for nil`
+- **Trigger**: Querying `_brand { ... on Brand { name } }` on some products
+- **Workaround**: Removed `_brand` from queries
+- **Status**: To report to Letzshop
+
+#### Bug 2: `tracking` field causes server error (ALL queries)
+- **Error**: `NoMethodError: undefined method 'demodulize' for nil`
+- **Trigger**: Including `tracking { code provider }` in ANY shipment query
+- **Tested and FAILS on**:
+  - Paginated queries: `shipments(state: confirmed, first: 10) { nodes { tracking { code provider } } }`
+  - Non-paginated queries: `shipments(state: confirmed) { nodes { tracking { code provider } } }`
+  - Single shipment queries: Also fails (Letzshop doesn't support `node(id:)` interface)
+- **Impact**: Cannot retrieve tracking numbers and carrier info at all
+- **Workaround**: None - tracking info is currently unavailable via API
+- **Status**: **CRITICAL - Must report to Letzshop**
+- **Date discovered**: 2025-12-17
+
+**Note**: Letzshop automatically creates tracking when orders are confirmed. The carrier picks up parcels. But we cannot retrieve this info due to the API bug.
+
+#### Bug 3: Product table missing `gtin` field ✅ FIXED
+- **Error**: `type object 'Product' has no attribute 'gtin'`
+- **Cause**: `gtin` field only existed on `MarketplaceProduct` (staging table), not on `Product` (operational table)
+- **Date discovered**: 2025-12-17
+- **Date fixed**: 2025-12-18
+- **Fix applied**:
+  1. Migration `cb88bc9b5f86_add_gtin_columns_to_product_table.py` adds:
+     - `gtin` (String(50)) - the barcode number
+     - `gtin_type` (String(20)) - the format type (gtin13, gtin14, etc.)
+     - Indexes: `idx_product_gtin`, `idx_product_vendor_gtin`
+  2. `models/database/product.py` updated with new columns
+  3. `_match_eans_to_products()` now queries `Product.gtin`
+  4. `get_products_by_eans()` now returns products by EAN lookup
+- **Status**: COMPLETE
+
+**GTIN Types Reference:**
+
+| Type | Digits | Common Name | Region/Use |
+|------|--------|-------------|------------|
+| gtin13 | 13 | EAN-13 | Europe (most common) |
+| gtin12 | 12 | UPC-A | North America |
+| gtin14 | 14 | ITF-14 | Logistics/cases |
+| gtin8 | 8 | EAN-8 | Small items |
+| isbn13 | 13 | ISBN-13 | Books |
+| isbn10 | 10 | ISBN-10 | Books (legacy) |
+
+Letzshop API returns:
+- `tradeId.number` → store in `gtin`
+- `tradeId.parser` → store in `gtin_type`
+
+**Letzshop Shipment States (from official docs):**
+
+| Letzshop State | Our sync_status | Description |
+|----------------|-----------------|-------------|
+| `unconfirmed` | `pending` | New order, needs vendor confirmation |
+| `confirmed` | `confirmed` | At least one product confirmed |
+| `declined` | `rejected` | All products rejected |
+
+Note: There is no "shipped" state in Letzshop. Shipping is tracked via the `tracking` field (code + provider), not as a state change.
+
+---
+
+## Historical Confirmed Orders Import
+
+### Purpose
+Import all historical confirmed orders from Letzshop to:
+1. **Sales Analytics** - Track total products sold, revenue by product/category
+2. **Customer Records** - Build customer database with order history
+3. **Inventory Reconciliation** - Understand what was sold to reconcile stock
+
+### Implementation Plan
+
+#### 1. Add "Import Historical Orders" Feature
+- New endpoint: `POST /api/v1/admin/letzshop/vendors/{id}/import-history`
+- Parameters:
+  - `state`: confirmed/shipped/delivered (default: confirmed)
+  - `since`: Optional date filter (import orders after this date)
+  - `dry_run`: Preview without saving
+
+#### 2. Pagination Support
+Letzshop likely returns paginated results. Need to handle:
+```graphql
+query {
+  shipments(state: confirmed, first: 50, after: $cursor) {
+    pageInfo {
+      hasNextPage
+      endCursor
+    }
+    nodes { ... }
+  }
+}
+```
+
+#### 3. Deduplication
+- Check if order already exists by `letzshop_order_id` before inserting
+- Update existing orders if data changed
+
+#### 4. EAN Matching & Stock Adjustment
+When importing historical orders:
+- Match `tradeId.number` (EAN) to local products
+- Calculate total quantity sold per product
+- Option to adjust inventory based on historical sales
+
+#### 5. Customer Database
+Extract and store customer data:
+- Email (unique identifier)
+- Name (from shipping address)
+- Preferred language (from `order.locale`)
+- Order count, total spent
+
+#### 6. UI: Historical Import Page
+Admin interface to:
+- Trigger historical import
+- View import progress
+- See summary: X orders imported, Y customers added, Z products matched
+
+### Data Flow
+
+```
+Letzshop API (confirmed shipments)
+        │
+        ▼
+┌───────────────────────┐
+│  Import Service       │
+│  - Fetch all pages    │
+│  - Deduplicate        │
+│  - Match EAN to SKU   │
+└───────────────────────┘
+        │
+        ▼
+┌───────────────────────┐
+│  Database             │
+│  - letzshop_orders    │
+│  - customers          │
+│  - inventory updates  │
+└───────────────────────┘
+        │
+        ▼
+┌───────────────────────┐
+│  Analytics Dashboard  │
+│  - Sales by product   │
+│  - Revenue over time  │
+│  - Customer insights  │
+└───────────────────────┘
+```
+
+---
+
+## Schema Reference
+
+### Variant Fields
+```
+baseAmount: String
+baseAmountProduct: String
+baseUnit: String
+countOnHand: Int
+id: ID!
+images: [Image]!
+inPresale: Boolean!
+isMaster: Boolean!
+mpn: String
+price: Float!
+priceCrossed: Float
+pricePerUnit: Float
+product: Product!
+properties: [Property]!
+releaseAt: Iso8601Time
+sku: String
+tradeId: TradeId
+uniqueId: String
+url: String!
+```
+
+### TradeId Fields
+```
+isRestricted: Boolean
+number: String!        # <-- THE EAN/GTIN
+parser: TradeIdParser! # <-- Format identifier
+```
+
+### TradeIdParser Enum
+```
+gtin14  - GTIN-14 (14 digits)
+gtin13  - GTIN-13 / EAN-13 (13 digits, most common in Europe)
+gtin12  - GTIN-12 / UPC-A (12 digits, common in North America)
+gtin8   - GTIN-8 / EAN-8 (8 digits)
+isbn13  - ISBN-13 (books)
+isbn10  - ISBN-10 (books)
+```
+
+### Brand (via BrandUnion)
+```
+BrandUnion = Brand | BrandUnknown
+
+Brand fields:
+  id: ID!
+  name: String!
+  identifier: String!
+  descriptor: String
+  logo: Attachment
+  url: String!
+```
+
+### InventoryUnit Fields
+```
+id: ID!
+price: Float!
+state: String!
+taxRate: Float!
+uniqueId: String
+variant: Variant
+```
+
+## Reference: Letzshop Frontend Shows
+
+From the Letzshop merchant interface:
+- Order number: R532332163
+- Shipment number: H74683403433
+- Product: "Pop! Rocks: DJ Khaled - DJ Khaled #237"
+- Brand: Funko
+- Internal merchant number: MH-FU-56757
+- Price: 16,95 €
+- Quantity: 1
+- Shipping: 2,99 €
+- Total: 19,94 €
+
+---
+
+## Completed (2025-12-18)
+
+### Order Stats Fix ✅
+- **Issue**: Order status cards (Pending, Confirmed, etc.) were showing incorrect counts
+- **Cause**: Stats were calculated client-side from only the visible page of orders
+- **Fix**:
+  1. Added `get_order_stats()` method to `LetzshopOrderService`
+  2. Added `LetzshopOrderStats` schema with pending/confirmed/rejected/shipped counts
+  3. API now returns `stats` field with counts for ALL orders
+  4. JavaScript uses server-side stats instead of client-side calculation
+- **Status**: COMPLETE
+
+### Tracking Investigation ✅
+- **Issue**: Letzshop API bug prevents querying tracking field
+- **Added**: `--tracking` option to `letzshop_introspect.py` to investigate workarounds
+- **Findings**: Bug is on Letzshop's side, no client-side workaround possible
+- **Recommendation**: Store tracking info locally after setting via API
+
+---
+
+## Next Steps (TODO)
+
+### Priority 1: Historical Import Progress Bar
+Add real-time progress feedback for historical import (currently no visibility into import progress).
+
+**Requirements:**
+- Show progress indicator while import is running
+- Display current page being fetched (e.g., "Fetching page 3 of 12...")
+- Show running count of orders imported/updated
+- Prevent user from thinking the process is stuck
+
+**Implementation options:**
+1. **Polling approach**: Frontend polls a status endpoint every few seconds
+2. **Server-Sent Events (SSE)**: Real-time updates pushed to frontend
+3. **WebSocket**: Bi-directional real-time communication
+
+**Backend changes needed:**
+- Store import progress in database or cache (Redis)
+- Add endpoint `GET /api/v1/admin/letzshop/vendors/{id}/import-progress`
+- Update `import_historical_shipments()` to report progress
+
+**Frontend changes needed:**
+- Progress bar component in Orders tab
+- Polling/SSE logic to fetch progress updates
+- Disable "Import History" button while import is in progress
+
+### Priority 2: Stock Management
+When an order is confirmed/imported:
+1. Match EAN from order to local product catalog
+2. Decrease stock quantity for matched products
+3. Handle cases where product not found (alert/log)
+
+**Considerations:**
+- Should stock decrease happen on import or only on confirmation?
+- Need rollback mechanism if order is rejected
+- Handle partial matches (some items found, some not)
+
+### Priority 2: Order Detail View Enhancement
+Improve the order detail modal to show:
+- Product details (name, EAN, MPN, SKU)
+- Match status per line item (found/not found in catalog)
+- Link to local product if matched
+
+### Priority 3: Invoice Generation
+Use `customer_locale` to generate invoices in customer's language:
+- Invoice template with multi-language support
+- PDF generation
+
+### Priority 4: Analytics Dashboard
+Build sales analytics based on imported orders:
+- Sales by product
+- Sales by time period
+- Customer statistics
+- Revenue breakdown
+
+---
+
+## Files Modified (2025-12-16 to 2025-12-18)
+
+| File | Changes |
+|------|---------|
+| `app/services/letzshop/client_service.py` | Added paginated query, updated all queries with EAN/locale/country |
+| `app/services/letzshop/order_service.py` | Added historical import, EAN matching, summary endpoint, order stats |
+| `models/database/letzshop.py` | Added locale and country columns |
+| `models/database/product.py` | Added `gtin` and `gtin_type` columns for EAN matching |
+| `models/schema/letzshop.py` | Added `LetzshopOrderStats` schema, stats to order list response |
+| `app/api/v1/admin/letzshop.py` | Added import-history and import-summary endpoints, stats in orders response |
+| `app/templates/admin/partials/letzshop-orders-tab.html` | Added Import History button and result display |
+| `static/admin/js/marketplace-letzshop.js` | Added importHistoricalOrders(), server-side stats |
+| `tests/unit/services/test_letzshop_service.py` | Added tests for new functionality |
+| `scripts/test_historical_import.py` | Manual test script for historical import |
+| `scripts/letzshop_introspect.py` | GraphQL schema introspection tool, tracking workaround tests |
+| `alembic/versions/a9a86cef6cca_*.py` | Migration for locale/country columns |
+| `alembic/versions/cb88bc9b5f86_*.py` | Migration for gtin columns on Product table |