# Letzshop Order Import - Improvement Plan

## Current Status (2025-12-17)

### Schema Discovery Complete ✅

After running GraphQL introspection queries, we have identified all available fields.

### Available Fields Summary

| Data | GraphQL Path | Notes |
|------|-------------|-------|
| **EAN/GTIN** | `variant.tradeId.number` | The product barcode |
| **Trade ID Type** | `variant.tradeId.parser` | Format: gtin13, gtin14, gtin12, gtin8, isbn13, isbn10 |
| **Brand Name** | `product._brand { ... on Brand { name } }` | Union type requires fragment |
| **MPN** | `variant.mpn` | Manufacturer Part Number |
| **SKU** | `variant.sku` | Merchant's internal SKU |
| **Product Name** | `variant.product.name { en, fr, de }` | Translated names |
| **Price** | `variant.price` | Unit price |
| **Quantity** | Count of `inventoryUnits` | Each unit = 1 item |
| **Customer Language** | `order.locale` | Language for invoice (en, fr, de) |
| **Customer Country** | `order.shipAddress.country` | Country object |

### Key Findings

1. **EAN lives in `tradeId`** - Not a direct field on Variant, but nested in `tradeId.number`
2. **TradeIdParser enum values**: `gtin14`, `gtin13` (EAN-13), `gtin12` (UPC), `gtin8`, `isbn13`, `isbn10`
3. **Brand is a Union** - Must use `... on Brand { name }` fragment, also handles `BrandUnknown`
4. **No quantity field** - Each InventoryUnit represents 1 item; count units to get quantity

## Updated GraphQL Query

```graphql
query {
  shipments(state: unconfirmed) {
    nodes {
      id
      number
      state
      order {
        id
        number
        email
        total
        completedAt
        locale
        shipAddress {
          firstName
          lastName
          merchant
          streetName
          streetNumber
          city
          zipCode
          phone
          country {
            name
            iso
          }
        }
        billAddress {
          firstName
          lastName
          merchant
          streetName
          streetNumber
          city
          zipCode
          phone
          country {
            name
            iso
          }
        }
      }
      inventoryUnits {
        id
        state
        variant {
          id
          sku
          mpn
          price
          tradeId {
            number
            parser
          }
          product {
            name { en fr de }
            _brand {
              ... on Brand { name }
            }
          }
        }
      }
      tracking {
        code
        provider
      }
    }
  }
}
```

## Implementation Steps

### Step 1: Update GraphQL Queries ✅ DONE
Update in `app/services/letzshop/client_service.py`:
- `QUERY_SHIPMENTS_UNCONFIRMED` ✅
- `QUERY_SHIPMENTS_CONFIRMED` ✅
- `QUERY_SHIPMENT_BY_ID` ✅
- `QUERY_SHIPMENTS_PAGINATED_TEMPLATE` ✅ (new - for historical import)

### Step 2: Update Order Service ✅ DONE
Updated `create_order()` and `update_order_from_shipment()` in `app/services/letzshop/order_service.py`:
- Extract `tradeId.number` as EAN ✅
- Store MPN if available ✅
- Store `locale` for invoice language ✅
- Store shipping/billing country ISO codes ✅
- Enrich inventory_units with EAN, MPN, SKU, product_name ✅

**Database changes:**
- Added `customer_locale` column to `LetzshopOrder`
- Added `shipping_country_iso` column to `LetzshopOrder`
- Added `billing_country_iso` column to `LetzshopOrder`
- Migration: `a9a86cef6cca_add_letzshop_order_locale_and_country_.py`

### Step 3: Match Products by EAN ✅ DONE (Basic)
When importing orders:
- Use `tradeId.number` (EAN) to find matching local product ✅
- `_match_eans_to_products()` function added ✅
- Returns match statistics (products_matched, products_not_found) ✅

**TODO for later:**
- ⬜ Decrease stock for matched product (needs careful implementation)
- ⬜ Show match status in order detail view

### Step 4: Update Frontend ✅ DONE (Historical Import)
- Added "Import History" button to Orders tab ✅
- Added historical import result display ✅
- Added `importHistoricalOrders()` JavaScript function ✅

**TODO for later:**
- ⬜ Show product details in individual order view (EAN, MPN, SKU, match status)

### Step 5: Historical Import Feature ✅ DONE
Import all confirmed orders for:
- Sales analytics (how many products sold)
- Customer records
- Historical data

**Implementation:**
- Pagination support with `get_all_shipments_paginated()` ✅
- Deduplication by `letzshop_order_id` ✅
- EAN matching during import ✅
- Progress callback for large imports ✅

**Endpoints Added:**
- `POST /api/v1/admin/letzshop/stores/{id}/import-history` - Import historical orders
- `GET /api/v1/admin/letzshop/stores/{id}/import-summary` - Get import statistics

**Frontend:**
- "Import History" button in Orders tab
- Result display showing imported/updated/skipped counts

**Tests:**
- Unit tests in `tests/unit/services/test_letzshop_service.py` ✅
- Manual test script `scripts/test_historical_import.py` ✅

## Test Results (2025-12-17)

### Query Test: PASSED ✅

```
Example shipment:
  Shipment #: H43748338602
  Order #: R702236251
  Customer: miriana.leal@letzshop.lu
  Locale: fr <<<< LANGUAGE
  Total: 32.88 EUR

  Ship to: Miriana Leal Ferreira
  City: 1468 Luxembourg
  Country: LU

  Items (1):
    - Pocket POP! Keychains: Marvel Avengers Infinity War - Iron Spider
      SKU: 00889698273022
      MPN: None
      EAN: 00889698273022 (gtin14) <<<< BARCODE
      Price: 5.88 EUR
```

### Known Issues / Letzshop API Bugs

#### Bug 1: `_brand` field causes server error
- **Error**: `NoMethodError: undefined method 'demodulize' for nil`
- **Trigger**: Querying `_brand { ... on Brand { name } }` on some products
- **Workaround**: Removed `_brand` from queries
- **Status**: To report to Letzshop

#### Bug 2: `tracking` field causes server error (ALL queries)
- **Error**: `NoMethodError: undefined method 'demodulize' for nil`
- **Trigger**: Including `tracking { code provider }` in ANY shipment query
- **Tested and FAILS on**:
  - Paginated queries: `shipments(state: confirmed, first: 10) { nodes { tracking { code provider } } }`
  - Non-paginated queries: `shipments(state: confirmed) { nodes { tracking { code provider } } }`
  - Single shipment queries: Also fails (Letzshop doesn't support `node(id:)` interface)
- **Impact**: Cannot retrieve tracking numbers and carrier info at all
- **Workaround**: None - tracking info is currently unavailable via API
- **Status**: **CRITICAL - Must report to Letzshop**
- **Date discovered**: 2025-12-17

**Note**: Letzshop automatically creates tracking when orders are confirmed. The carrier picks up parcels. But we cannot retrieve this info due to the API bug.

#### Bug 3: Product table missing `gtin` field ✅ FIXED
- **Error**: `type object 'Product' has no attribute 'gtin'`
- **Cause**: `gtin` field only existed on `MarketplaceProduct` (staging table), not on `Product` (operational table)
- **Date discovered**: 2025-12-17
- **Date fixed**: 2025-12-18
- **Fix applied**:
  1. Migration `cb88bc9b5f86_add_gtin_columns_to_product_table.py` adds:
     - `gtin` (String(50)) - the barcode number
     - `gtin_type` (String(20)) - the format type (gtin13, gtin14, etc.)
     - Indexes: `idx_product_gtin`, `idx_product_store_gtin`
  2. `models/database/product.py` updated with new columns
  3. `_match_eans_to_products()` now queries `Product.gtin`
  4. `get_products_by_eans()` now returns products by EAN lookup
- **Status**: COMPLETE

**GTIN Types Reference:**

| Type | Digits | Common Name | Region/Use |
|------|--------|-------------|------------|
| gtin13 | 13 | EAN-13 | Europe (most common) |
| gtin12 | 12 | UPC-A | North America |
| gtin14 | 14 | ITF-14 | Logistics/cases |
| gtin8 | 8 | EAN-8 | Small items |
| isbn13 | 13 | ISBN-13 | Books |
| isbn10 | 10 | ISBN-10 | Books (legacy) |

Letzshop API returns:
- `tradeId.number` → store in `gtin`
- `tradeId.parser` → store in `gtin_type`

**Letzshop Shipment States (from official docs):**

| Letzshop State | Our sync_status | Description |
|----------------|-----------------|-------------|
| `unconfirmed` | `pending` | New order, needs store confirmation |
| `confirmed` | `confirmed` | At least one product confirmed |
| `declined` | `rejected` | All products rejected |

Note: There is no "shipped" state in Letzshop. Shipping is tracked via the `tracking` field (code + provider), not as a state change.

---

## Historical Confirmed Orders Import

### Purpose
Import all historical confirmed orders from Letzshop to:
1. **Sales Analytics** - Track total products sold, revenue by product/category
2. **Customer Records** - Build customer database with order history
3. **Inventory Reconciliation** - Understand what was sold to reconcile stock

### Implementation Plan

#### 1. Add "Import Historical Orders" Feature
- New endpoint: `POST /api/v1/admin/letzshop/stores/{id}/import-history`
- Parameters:
  - `state`: confirmed/shipped/delivered (default: confirmed)
  - `since`: Optional date filter (import orders after this date)
  - `dry_run`: Preview without saving

#### 2. Pagination Support
Letzshop likely returns paginated results. Need to handle:
```graphql
query {
  shipments(state: confirmed, first: 50, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    nodes { ... }
  }
}
```

#### 3. Deduplication
- Check if order already exists by `letzshop_order_id` before inserting
- Update existing orders if data changed

#### 4. EAN Matching & Stock Adjustment
When importing historical orders:
- Match `tradeId.number` (EAN) to local products
- Calculate total quantity sold per product
- Option to adjust inventory based on historical sales

#### 5. Customer Database
Extract and store customer data:
- Email (unique identifier)
- Name (from shipping address)
- Preferred language (from `order.locale`)
- Order count, total spent

#### 6. UI: Historical Import Page
Admin interface to:
- Trigger historical import
- View import progress
- See summary: X orders imported, Y customers added, Z products matched

### Data Flow

```
Letzshop API (confirmed shipments)
        │
        ▼
┌───────────────────────┐
│  Import Service       │
│  - Fetch all pages    │
│  - Deduplicate        │
│  - Match EAN to SKU   │
└───────────────────────┘
        │
        ▼
┌───────────────────────┐
│  Database             │
│  - letzshop_orders    │
│  - customers          │
│  - inventory updates  │
└───────────────────────┘
        │
        ▼
┌───────────────────────┐
│  Analytics Dashboard  │
│  - Sales by product   │
│  - Revenue over time  │
│  - Customer insights  │
└───────────────────────┘
```

---

## Schema Reference

### Variant Fields
```
baseAmount: String
baseAmountProduct: String
baseUnit: String
countOnHand: Int
id: ID!
images: [Image]!
inPresale: Boolean!
isMaster: Boolean!
mpn: String
price: Float!
priceCrossed: Float
pricePerUnit: Float
product: Product!
properties: [Property]!
releaseAt: Iso8601Time
sku: String
tradeId: TradeId
uniqueId: String
url: String!
```

### TradeId Fields
```
isRestricted: Boolean
number: String!        # <-- THE EAN/GTIN
parser: TradeIdParser! # <-- Format identifier
```

### TradeIdParser Enum
```
gtin14  - GTIN-14 (14 digits)
gtin13  - GTIN-13 / EAN-13 (13 digits, most common in Europe)
gtin12  - GTIN-12 / UPC-A (12 digits, common in North America)
gtin8   - GTIN-8 / EAN-8 (8 digits)
isbn13  - ISBN-13 (books)
isbn10  - ISBN-10 (books)
```

### Brand (via BrandUnion)
```
BrandUnion = Brand | BrandUnknown

Brand fields:
  id: ID!
  name: String!
  identifier: String!
  descriptor: String
  logo: Attachment
  url: String!
```

### InventoryUnit Fields
```
id: ID!
price: Float!
state: String!
taxRate: Float!
uniqueId: String
variant: Variant
```

## Reference: Letzshop Frontend Shows

From the Letzshop merchant interface:
- Order number: R532332163
- Shipment number: H74683403433
- Product: "Pop! Rocks: DJ Khaled - DJ Khaled #237"
- Brand: Funko
- Internal merchant number: MH-FU-56757
- Price: 16,95 €
- Quantity: 1
- Shipping: 2,99 €
- Total: 19,94 €

---

## Completed (2025-12-18)

### Order Stats Fix ✅
- **Issue**: Order status cards (Pending, Confirmed, etc.) were showing incorrect counts
- **Cause**: Stats were calculated client-side from only the visible page of orders
- **Fix**:
  1. Added `get_order_stats()` method to `LetzshopOrderService`
  2. Added `LetzshopOrderStats` schema with pending/confirmed/rejected/shipped counts
  3. API now returns `stats` field with counts for ALL orders
  4. JavaScript uses server-side stats instead of client-side calculation
- **Status**: COMPLETE

### Tracking Investigation ✅
- **Issue**: Letzshop API bug prevents querying tracking field
- **Added**: `--tracking` option to `letzshop_introspect.py` to investigate workarounds
- **Findings**: Bug is on Letzshop's side, no client-side workaround possible
- **Recommendation**: Store tracking info locally after setting via API

### Item-Level Confirmation ✅
- **Issue**: Orders were being confirmed/declined at order level, but Letzshop requires item-level actions
- **Letzshop Model**:
  - Each `inventoryUnit` must be confirmed/declined individually via `confirmInventoryUnits` mutation
  - `isAvailable: true` = confirmed, `isAvailable: false` = declined
  - Inventory unit states: `unconfirmed` → `confirmed_available` / `confirmed_unavailable` / `returned`
  - Shipment states derived from items: `unconfirmed` / `confirmed` / `declined`
  - Partial confirmation allowed (some items confirmed, some declined)
- **Fix**:
  1. Order detail modal now shows each item with product details (name, EAN, SKU, MPN, price)
  2. Per-item confirm/decline buttons for pending items
  3. Admin API endpoints for single-item and bulk operations:
     - `POST /stores/{id}/orders/{id}/items/{id}/confirm`
     - `POST /stores/{id}/orders/{id}/items/{id}/decline`
  4. Order status automatically updates based on item states:
     - All items declined → order status = "declined"
     - Any item confirmed → order status = "confirmed"

### Terminology Update ✅
- Changed "Rejected" to "Declined" throughout UI to match Letzshop terminology
- Internal `sync_status` value remains "rejected" for backwards compatibility
- Filter dropdown, status badges, and action buttons now use "Declined"
- Added "Declined" stats card to orders dashboard

### Historical Import: Multiple Phases ✅
- Historical import now fetches both `confirmed` AND `unconfirmed` (pending) shipments
- Note: "declined" is NOT a valid Letzshop shipment state - declined items are tracked at inventory unit level
- Combined stats shown in import result

---

## Completed (2025-12-19)

### Historical Import Progress Bar ✅
Real-time progress feedback for historical import using background tasks with database polling.

**Implementation:**
- Background task (`app/tasks/letzshop_tasks.py`) runs historical import asynchronously
- Progress stored in `LetzshopHistoricalImportJob` database model
- Frontend polls status endpoint every 2 seconds
- Two-phase import: confirmed orders first, then unconfirmed (pending) orders

**Backend:**
- `LetzshopHistoricalImportJob` model tracks: status, current_phase, current_page, shipments_fetched, orders_processed, confirmed_stats, declined_stats
- `POST /stores/{id}/import-history` starts background job, returns job_id immediately
- `GET /stores/{id}/import-history/{job_id}/status` returns current progress

**Frontend:**
- Progress panel shows: phase (confirmed/pending), page number, shipments fetched, orders processed
- Disabled "Import History" button during import with spinner
- Final result summary shows combined stats from both phases

**Key Discovery:**
- Letzshop API has NO "declined" shipment state
- Valid states: `awaiting_order_completion`, `unconfirmed`, `completed`, `accepted`, `confirmed`
- Declined items are tracked at inventory unit level with state `confirmed_unavailable`

### Filter for Declined Items ✅
Added ability to filter orders that have at least one declined/unavailable item.

**Backend:**
- `list_orders()` accepts `has_declined_items: bool` parameter
- Uses JSON string contains check: `inventory_units.cast(String).contains("confirmed_unavailable")`
- `get_order_stats()` returns `has_declined_items` count

**Frontend:**
- "Has Declined Items" toggle button in filters section
- Shows count badge when there are orders with declined items
- Toggles between all orders and filtered view

**API:**
- `GET /stores/{id}/orders?has_declined_items=true` - filter orders

### Order Date Display ✅
Orders now display the actual order date from Letzshop instead of the import date.

**Database:**
- Added `order_date` column to `LetzshopOrder` model
- Migration: `2362c2723a93_add_order_date_to_letzshop_orders.py`

**Backend:**
- `create_order()` extracts `completedAt` from Letzshop order data and stores as `order_date`
- `update_order_from_shipment()` populates `order_date` if not already set
- Date parsing handles ISO format with timezone (including `Z` suffix)

**Frontend:**
- Order table displays `order_date` with fallback to `created_at` for legacy orders
- Format: localized date/time string

**Note:** Existing orders imported before this change will continue showing `created_at` until re-imported via historical import.

### Search Filter ✅
Added search functionality to find orders by order number, customer name, or email.

**Backend:**
- `list_orders()` accepts `search: str` parameter
- Uses ILIKE for case-insensitive partial matching across:
  - `letzshop_order_number`
  - `customer_name`
  - `customer_email`

**Frontend:**
- Search input field with magnifying glass icon
- Debounced input (300ms) to avoid excessive API calls
- Clear button to reset search
- Resets to page 1 when search changes

**API:**
- `GET /stores/{id}/orders?search=query` - search orders

---

## Next Steps (TODO)

### Priority 1: Stock Management
When an order is confirmed/imported:
1. Match EAN from order to local product catalog
2. Decrease stock quantity for matched products
3. Handle cases where product not found (alert/log)

**Considerations:**
- Should stock decrease happen on import or only on confirmation?
- Need rollback mechanism if order is rejected
- Handle partial matches (some items found, some not)

### Priority 2: Invoice Generation
Use `customer_locale` to generate invoices in customer's language:
- Invoice template with multi-language support
- PDF generation

### Priority 3: Analytics Dashboard
Build sales analytics based on imported orders:
- Sales by product
- Sales by time period
- Customer statistics
- Revenue breakdown

---

## Files Modified (2025-12-16 to 2025-12-19)

| File | Changes |
|------|---------|
| `app/services/letzshop/client_service.py` | Added paginated query, updated all queries with EAN/locale/country |
| `app/services/letzshop/order_service.py` | Historical import, EAN matching, order stats, has_declined_items filter, search filter, order_date extraction |
| `models/database/letzshop.py` | Added locale/country/order_date columns, `LetzshopHistoricalImportJob` model |
| `models/database/product.py` | Added `gtin` and `gtin_type` columns for EAN matching |
| `models/schema/letzshop.py` | Added `LetzshopOrderStats`, `LetzshopHistoricalImportJobResponse`, `order_date` field |
| `app/api/v1/admin/letzshop.py` | Import-history endpoints, has_declined_items filter, search filter, order_date in response |
| `app/tasks/letzshop_tasks.py` | **NEW** - Background task for historical import with progress tracking |
| `app/templates/admin/partials/letzshop-orders-tab.html` | Import History button, progress panel, declined items filter, search input, order_date display |
| `static/admin/js/marketplace-letzshop.js` | Historical import polling, progress display, declined items filter, search functionality |
| `tests/unit/services/test_letzshop_service.py` | Added tests for new functionality |
| `scripts/test_historical_import.py` | Manual test script for historical import |
| `scripts/debug_historical_import.py` | **NEW** - Debug script for shipment states and declined items |
| `scripts/letzshop_introspect.py` | GraphQL schema introspection tool, tracking workaround tests |
| `alembic/versions/a9a86cef6cca_*.py` | Migration for locale/country columns |
| `alembic/versions/cb88bc9b5f86_*.py` | Migration for gtin columns on Product table |
| `alembic/versions/*_add_historical_import_jobs.py` | **NEW** - Migration for LetzshopHistoricalImportJob table |
| `alembic/versions/2362c2723a93_*.py` | **NEW** - Migration for order_date column |