orion/docs/implementation/letzshop-order-import-improvements.md

Letzshop Order Import - Improvement Plan

Current Status (2025-12-17)

Schema Discovery Complete ✅

After running GraphQL introspection queries, we have identified all available fields.

Available Fields Summary

Data	GraphQL Path	Notes
EAN/GTIN	variant.tradeId.number	The product barcode
Trade ID Type	variant.tradeId.parser	Format: gtin13, gtin14, gtin12, gtin8, isbn13, isbn10
Brand Name	product._brand { ... on Brand { name } }	Union type requires fragment
MPN	variant.mpn	Manufacturer Part Number
SKU	variant.sku	Merchant's internal SKU
Product Name	variant.product.name { en, fr, de }	Translated names
Price	variant.price	Unit price
Quantity	Count of inventoryUnits	Each unit = 1 item
Customer Language	order.locale	Language for invoice (en, fr, de)
Customer Country	order.shipAddress.country	Country object

Key Findings

1. EAN lives in tradeId - Not a direct field on Variant, but nested in tradeId.number
2. TradeIdParser enum values: gtin14, gtin13 (EAN-13), gtin12 (UPC), gtin8, isbn13, isbn10
3. Brand is a Union - Must use ... on Brand { name } fragment, also handles BrandUnknown
4. No quantity field - Each InventoryUnit represents 1 item; count units to get quantity

Updated GraphQL Query

query {
  shipments(state: unconfirmed) {
    nodes {
      id
      number
      state
      order {
        id
        number
        email
        total
        completedAt
        locale
        shipAddress {
          firstName
          lastName
          company
          streetName
          streetNumber
          city
          zipCode
          phone
          country {
            name
            iso
          }
        }
        billAddress {
          firstName
          lastName
          company
          streetName
          streetNumber
          city
          zipCode
          phone
          country {
            name
            iso
          }
        }
      }
      inventoryUnits {
        id
        state
        variant {
          id
          sku
          mpn
          price
          tradeId {
            number
            parser
          }
          product {
            name { en fr de }
            _brand {
              ... on Brand { name }
            }
          }
        }
      }
      tracking {
        code
        provider
      }
    }
  }
}

Implementation Steps

Step 1: Update GraphQL Queries ✅ DONE

Update in app/services/letzshop/client_service.py:

• QUERY_SHIPMENTS_UNCONFIRMED ✅
• QUERY_SHIPMENTS_CONFIRMED ✅
• QUERY_SHIPMENT_BY_ID ✅
• QUERY_SHIPMENTS_PAGINATED_TEMPLATE ✅ (new - for historical import)

Step 2: Update Order Service ✅ DONE

Updated create_order() and update_order_from_shipment() in app/services/letzshop/order_service.py:

• Extract tradeId.number as EAN ✅
• Store MPN if available ✅
• Store locale for invoice language ✅
• Store shipping/billing country ISO codes ✅
• Enrich inventory_units with EAN, MPN, SKU, product_name ✅

Database changes:

• Added customer_locale column to LetzshopOrder
• Added shipping_country_iso column to LetzshopOrder
• Added billing_country_iso column to LetzshopOrder
• Migration: a9a86cef6cca_add_letzshop_order_locale_and_country_.py

Step 3: Match Products by EAN ✅ DONE (Basic)

When importing orders:

• Use tradeId.number (EAN) to find matching local product ✅
• _match_eans_to_products() function added ✅
• Returns match statistics (products_matched, products_not_found) ✅

TODO for later:

• ⬜ Decrease stock for matched product (needs careful implementation)
• ⬜ Show match status in order detail view

Step 4: Update Frontend ✅ DONE (Historical Import)

• Added "Import History" button to Orders tab ✅
• Added historical import result display ✅
• Added importHistoricalOrders() JavaScript function ✅

TODO for later:

• ⬜ Show product details in individual order view (EAN, MPN, SKU, match status)

Step 5: Historical Import Feature ✅ DONE

Import all confirmed orders for:

• Sales analytics (how many products sold)
• Customer records
• Historical data

Implementation:

• Pagination support with get_all_shipments_paginated() ✅
• Deduplication by letzshop_order_id ✅
• EAN matching during import ✅
• Progress callback for large imports ✅

Endpoints Added:

• POST /api/v1/admin/letzshop/vendors/{id}/import-history - Import historical orders
• GET /api/v1/admin/letzshop/vendors/{id}/import-summary - Get import statistics

Frontend:

• "Import History" button in Orders tab
• Result display showing imported/updated/skipped counts

Tests:

• Unit tests in tests/unit/services/test_letzshop_service.py ✅
• Manual test script scripts/test_historical_import.py ✅

Test Results (2025-12-17)

Query Test: PASSED ✅

Example shipment:
  Shipment #: H43748338602
  Order #: R702236251
  Customer: miriana.leal@letzshop.lu
  Locale: fr <<<< LANGUAGE
  Total: 32.88 EUR

  Ship to: Miriana Leal Ferreira
  City: 1468 Luxembourg
  Country: LU

  Items (1):
    - Pocket POP! Keychains: Marvel Avengers Infinity War - Iron Spider
      SKU: 00889698273022
      MPN: None
      EAN: 00889698273022 (gtin14) <<<< BARCODE
      Price: 5.88 EUR

Known Issues / Letzshop API Bugs

Bug 1: _brand field causes server error

• Error: NoMethodError: undefined method 'demodulize' for nil
• Trigger: Querying _brand { ... on Brand { name } } on some products
• Workaround: Removed _brand from queries
• Status: To report to Letzshop

Bug 2: tracking field causes server error (ALL queries)

• Error: NoMethodError: undefined method 'demodulize' for nil
• Trigger: Including tracking { code provider } in ANY shipment query
• Tested and FAILS on:
  • Paginated queries: shipments(state: confirmed, first: 10) { nodes { tracking { code provider } } }
  • Non-paginated queries: shipments(state: confirmed) { nodes { tracking { code provider } } }
  • Single shipment queries: Also fails (Letzshop doesn't support node(id:) interface)
• Impact: Cannot retrieve tracking numbers and carrier info at all
• Workaround: None - tracking info is currently unavailable via API
• Status: CRITICAL - Must report to Letzshop
• Date discovered: 2025-12-17

Note: Letzshop automatically creates tracking when orders are confirmed. The carrier picks up parcels. But we cannot retrieve this info due to the API bug.

Bug 3: Product table missing gtin field ✅ FIXED

• Error: type object 'Product' has no attribute 'gtin'
• Cause: gtin field only existed on MarketplaceProduct (staging table), not on Product (operational table)
• Date discovered: 2025-12-17
• Date fixed: 2025-12-18
• Fix applied:
  1. Migration cb88bc9b5f86_add_gtin_columns_to_product_table.py adds:
     • gtin (String(50)) - the barcode number
     • gtin_type (String(20)) - the format type (gtin13, gtin14, etc.)
     • Indexes: idx_product_gtin, idx_product_vendor_gtin
  2. models/database/product.py updated with new columns
  3. _match_eans_to_products() now queries Product.gtin
  4. get_products_by_eans() now returns products by EAN lookup
• Status: COMPLETE

GTIN Types Reference:

Type	Digits	Common Name	Region/Use
gtin13	13	EAN-13	Europe (most common)
gtin12	12	UPC-A	North America
gtin14	14	ITF-14	Logistics/cases
gtin8	8	EAN-8	Small items
isbn13	13	ISBN-13	Books
isbn10	10	ISBN-10	Books (legacy)

Letzshop API returns:

• tradeId.number → store in gtin
• tradeId.parser → store in gtin_type

Letzshop Shipment States (from official docs):

Letzshop State	Our sync_status	Description
unconfirmed	pending	New order, needs vendor confirmation
confirmed	confirmed	At least one product confirmed
declined	rejected	All products rejected

Note: There is no "shipped" state in Letzshop. Shipping is tracked via the tracking field (code + provider), not as a state change.

---

Historical Confirmed Orders Import

Purpose

Import all historical confirmed orders from Letzshop to:

1. Sales Analytics - Track total products sold, revenue by product/category
2. Customer Records - Build customer database with order history
3. Inventory Reconciliation - Understand what was sold to reconcile stock

Implementation Plan

1. Add "Import Historical Orders" Feature

• New endpoint: POST /api/v1/admin/letzshop/vendors/{id}/import-history
• Parameters:
  • state: confirmed/shipped/delivered (default: confirmed)
  • since: Optional date filter (import orders after this date)
  • dry_run: Preview without saving

2. Pagination Support

Letzshop likely returns paginated results. Need to handle:

query {
  shipments(state: confirmed, first: 50, after: $cursor) {
    pageInfo {
      hasNextPage
      endCursor
    }
    nodes { ... }
  }
}

3. Deduplication

• Check if order already exists by letzshop_order_id before inserting
• Update existing orders if data changed

4. EAN Matching & Stock Adjustment

When importing historical orders:

• Match tradeId.number (EAN) to local products
• Calculate total quantity sold per product
• Option to adjust inventory based on historical sales

5. Customer Database

Extract and store customer data:

• Email (unique identifier)
• Name (from shipping address)
• Preferred language (from order.locale)
• Order count, total spent

6. UI: Historical Import Page

Admin interface to:

• Trigger historical import
• View import progress
• See summary: X orders imported, Y customers added, Z products matched

Data Flow

Letzshop API (confirmed shipments)
        │
        ▼
┌───────────────────────┐
│  Import Service       │
│  - Fetch all pages    │
│  - Deduplicate        │
│  - Match EAN to SKU   │
└───────────────────────┘
        │
        ▼
┌───────────────────────┐
│  Database             │
│  - letzshop_orders    │
│  - customers          │
│  - inventory updates  │
└───────────────────────┘
        │
        ▼
┌───────────────────────┐
│  Analytics Dashboard  │
│  - Sales by product   │
│  - Revenue over time  │
│  - Customer insights  │
└───────────────────────┘

---

Schema Reference

Variant Fields

baseAmount: String
baseAmountProduct: String
baseUnit: String
countOnHand: Int
id: ID!
images: [Image]!
inPresale: Boolean!
isMaster: Boolean!
mpn: String
price: Float!
priceCrossed: Float
pricePerUnit: Float
product: Product!
properties: [Property]!
releaseAt: Iso8601Time
sku: String
tradeId: TradeId
uniqueId: String
url: String!

TradeId Fields

isRestricted: Boolean
number: String!        # <-- THE EAN/GTIN
parser: TradeIdParser! # <-- Format identifier

TradeIdParser Enum

gtin14  - GTIN-14 (14 digits)
gtin13  - GTIN-13 / EAN-13 (13 digits, most common in Europe)
gtin12  - GTIN-12 / UPC-A (12 digits, common in North America)
gtin8   - GTIN-8 / EAN-8 (8 digits)
isbn13  - ISBN-13 (books)
isbn10  - ISBN-10 (books)

Brand (via BrandUnion)

BrandUnion = Brand | BrandUnknown

Brand fields:
  id: ID!
  name: String!
  identifier: String!
  descriptor: String
  logo: Attachment
  url: String!

InventoryUnit Fields

id: ID!
price: Float!
state: String!
taxRate: Float!
uniqueId: String
variant: Variant

Reference: Letzshop Frontend Shows

From the Letzshop merchant interface:

• Order number: R532332163
• Shipment number: H74683403433
• Product: "Pop! Rocks: DJ Khaled - DJ Khaled #237"
• Brand: Funko
• Internal merchant number: MH-FU-56757
• Price: 16,95 €
• Quantity: 1
• Shipping: 2,99 €
• Total: 19,94 €

---

Completed (2025-12-18)

Order Stats Fix ✅

• Issue: Order status cards (Pending, Confirmed, etc.) were showing incorrect counts
• Cause: Stats were calculated client-side from only the visible page of orders
• Fix:
  1. Added get_order_stats() method to LetzshopOrderService
  2. Added LetzshopOrderStats schema with pending/confirmed/rejected/shipped counts
  3. API now returns stats field with counts for ALL orders
  4. JavaScript uses server-side stats instead of client-side calculation
• Status: COMPLETE

Tracking Investigation ✅

• Issue: Letzshop API bug prevents querying tracking field
• Added: --tracking option to letzshop_introspect.py to investigate workarounds
• Findings: Bug is on Letzshop's side, no client-side workaround possible
• Recommendation: Store tracking info locally after setting via API

Item-Level Confirmation ✅

• Issue: Orders were being confirmed/declined at order level, but Letzshop requires item-level actions
• Letzshop Model:
  • Each inventoryUnit must be confirmed/declined individually via confirmInventoryUnits mutation
  • isAvailable: true = confirmed, isAvailable: false = declined
  • Inventory unit states: unconfirmed → confirmed_available / confirmed_unavailable / returned
  • Shipment states derived from items: unconfirmed / confirmed / declined
  • Partial confirmation allowed (some items confirmed, some declined)
• Fix:
  1. Order detail modal now shows each item with product details (name, EAN, SKU, MPN, price)
  2. Per-item confirm/decline buttons for pending items
  3. Admin API endpoints for single-item and bulk operations:
     • POST /vendors/{id}/orders/{id}/items/{id}/confirm
     • POST /vendors/{id}/orders/{id}/items/{id}/decline
  4. Order status automatically updates based on item states:
     • All items declined → order status = "declined"
     • Any item confirmed → order status = "confirmed"

Terminology Update ✅

• Changed "Rejected" to "Declined" throughout UI to match Letzshop terminology
• Internal sync_status value remains "rejected" for backwards compatibility
• Filter dropdown, status badges, and action buttons now use "Declined"
• Added "Declined" stats card to orders dashboard

Historical Import: Multiple Phases ✅

• Historical import now fetches both confirmed AND unconfirmed (pending) shipments
• Note: "declined" is NOT a valid Letzshop shipment state - declined items are tracked at inventory unit level
• Combined stats shown in import result

---

Completed (2025-12-19)

Historical Import Progress Bar ✅

Real-time progress feedback for historical import using background tasks with database polling.

Implementation:

• Background task (app/tasks/letzshop_tasks.py) runs historical import asynchronously
• Progress stored in LetzshopHistoricalImportJob database model
• Frontend polls status endpoint every 2 seconds
• Two-phase import: confirmed orders first, then unconfirmed (pending) orders

Backend:

• LetzshopHistoricalImportJob model tracks: status, current_phase, current_page, shipments_fetched, orders_processed, confirmed_stats, declined_stats
• POST /vendors/{id}/import-history starts background job, returns job_id immediately
• GET /vendors/{id}/import-history/{job_id}/status returns current progress

Frontend:

• Progress panel shows: phase (confirmed/pending), page number, shipments fetched, orders processed
• Disabled "Import History" button during import with spinner
• Final result summary shows combined stats from both phases

Key Discovery:

• Letzshop API has NO "declined" shipment state
• Valid states: awaiting_order_completion, unconfirmed, completed, accepted, confirmed
• Declined items are tracked at inventory unit level with state confirmed_unavailable

Filter for Declined Items ✅

Added ability to filter orders that have at least one declined/unavailable item.

Backend:

• list_orders() accepts has_declined_items: bool parameter
• Uses JSON string contains check: inventory_units.cast(String).contains("confirmed_unavailable")
• get_order_stats() returns has_declined_items count

Frontend:

• "Has Declined Items" toggle button in filters section
• Shows count badge when there are orders with declined items
• Toggles between all orders and filtered view

API:

• GET /vendors/{id}/orders?has_declined_items=true - filter orders

Order Date Display ✅

Orders now display the actual order date from Letzshop instead of the import date.

Database:

• Added order_date column to LetzshopOrder model
• Migration: 2362c2723a93_add_order_date_to_letzshop_orders.py

Backend:

• create_order() extracts completedAt from Letzshop order data and stores as order_date
• update_order_from_shipment() populates order_date if not already set
• Date parsing handles ISO format with timezone (including Z suffix)

Frontend:

• Order table displays order_date with fallback to created_at for legacy orders
• Format: localized date/time string

Note: Existing orders imported before this change will continue showing created_at until re-imported via historical import.

Search Filter ✅

Added search functionality to find orders by order number, customer name, or email.

Backend:

• list_orders() accepts search: str parameter
• Uses ILIKE for case-insensitive partial matching across:
  • letzshop_order_number
  • customer_name
  • customer_email

Frontend:

• Search input field with magnifying glass icon
• Debounced input (300ms) to avoid excessive API calls
• Clear button to reset search
• Resets to page 1 when search changes

API:

• GET /vendors/{id}/orders?search=query - search orders

---

Next Steps (TODO)

Priority 1: Stock Management

When an order is confirmed/imported:

1. Match EAN from order to local product catalog
2. Decrease stock quantity for matched products
3. Handle cases where product not found (alert/log)

Considerations:

• Should stock decrease happen on import or only on confirmation?
• Need rollback mechanism if order is rejected
• Handle partial matches (some items found, some not)

Priority 2: Invoice Generation

Use customer_locale to generate invoices in customer's language:

• Invoice template with multi-language support
• PDF generation

Priority 3: Analytics Dashboard

Build sales analytics based on imported orders:

• Sales by product
• Sales by time period
• Customer statistics
• Revenue breakdown

---

Files Modified (2025-12-16 to 2025-12-19)

File	Changes
app/services/letzshop/client_service.py	Added paginated query, updated all queries with EAN/locale/country
app/services/letzshop/order_service.py	Historical import, EAN matching, order stats, has_declined_items filter, search filter, order_date extraction
models/database/letzshop.py	Added locale/country/order_date columns, LetzshopHistoricalImportJob model
models/database/product.py	Added gtin and gtin_type columns for EAN matching
models/schema/letzshop.py	Added LetzshopOrderStats, LetzshopHistoricalImportJobResponse, order_date field
app/api/v1/admin/letzshop.py	Import-history endpoints, has_declined_items filter, search filter, order_date in response
app/tasks/letzshop_tasks.py	NEW - Background task for historical import with progress tracking
app/templates/admin/partials/letzshop-orders-tab.html	Import History button, progress panel, declined items filter, search input, order_date display
static/admin/js/marketplace-letzshop.js	Historical import polling, progress display, declined items filter, search functionality
tests/unit/services/test_letzshop_service.py	Added tests for new functionality
scripts/test_historical_import.py	Manual test script for historical import
scripts/debug_historical_import.py	NEW - Debug script for shipment states and declined items
scripts/letzshop_introspect.py	GraphQL schema introspection tool, tracking workaround tests
alembic/versions/a9a86cef6cca_*.py	Migration for locale/country columns
alembic/versions/cb88bc9b5f86_*.py	Migration for gtin columns on Product table
alembic/versions/*_add_historical_import_jobs.py	NEW - Migration for LetzshopHistoricalImportJob table
alembic/versions/2362c2723a93_*.py	NEW - Migration for order_date column