sboulahtit/orion
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
eeafe6389ff844b4b63ac9a46fa2bb724986e045
orion/README.md

737 lines
21 KiB
Markdown
Raw Blame History

# Ecommerce Backend API with Marketplace Support
A comprehensive FastAPI-based product management system with JWT authentication, marketplace-aware CSV import/export, multi-shop support, and advanced inventory management capabilities.
## Overview
This FastAPI application provides a complete ecommerce backend solution designed for marketplace operations. The system supports multiple vendors (shops), centralized product catalogs, advanced inventory management, and automated CSV processing with background job handling.
### Key Features
- **JWT Authentication** - Secure user registration, login, and role-based access control
- **Marketplace Integration** - Support for multiple marketplaces (Letzshop, Amazon, eBay, Etsy, Shopify, etc.)
- **Multi-Shop Management** - Shop creation, ownership validation, and product catalog management
- **Advanced MarketplaceProduct Management** - GTIN validation, price processing, and comprehensive filtering
- **Inventory Management** - Multi-location inventory tracking with add/remove/set operations
- **CSV Import/Export** - Background processing of marketplace CSV files with progress tracking
- **Rate Limiting** - Built-in request rate limiting for API protection
- **Admin Panel** - Administrative functions for user and shop management
- **Statistics & Analytics** - Comprehensive reporting on products, marketplaces, and inventory
### Technology Stack
- **FastAPI** - Modern, fast web framework for building APIs
- **SQLAlchemy** - SQL toolkit and Object-Relational Mapping (ORM)
- **PostgreSQL** - Primary database (SQLite supported for development)
- **JWT** - JSON Web Tokens for secure authentication
- **Pydantic** - Data validation using Python type annotations
- **Pandas** - Data processing for CSV operations
- **bcrypt** - Password hashing
- **Pytest** - Comprehensive testing framework
## Architecture
### Project Structure
```
wizamart/
├── main.py # FastAPI application entry point
├── app/
│ ├── core/
│ │ ├── config.py # Configuration settings
│ │ ├── database.py # Database setup
│ │ └── lifespan.py # App lifecycle management
│ ├── api/
│ │ ├── deps.py # Common dependencies
│ │ ├── main.py # API router setup
│ │ └── v1/ # API version 1 routes
│ │ ├── auth.py # Authentication endpoints
│ │ ├── products.py # MarketplaceProduct management
│ │ ├── inventory.py # Inventory operations
│ │ ├── shops.py # Shop management
│ │ ├── marketplace.py # Marketplace imports
│ │ ├── admin.py # Admin functions
│ │ └── stats.py # Statistics
│ ├── services/ # Business logic layer
│ └── tasks/ # Background task processing
├── models/
│ ├── database/
│ │ ├── __init__.py # Import all models for easy access
│ │ ├── base.py # Base model class and common mixins
│ │ ├── user.py # User, UserProfile models
│ │ ├── auth.py # Authentication-related models
│ │ ├── product.py # MarketplaceProduct, ProductVariant models
│ │ ├── inventory.py # Inventory, InventoryMovement models
│ │ ├── shop.py # Shop, ShopLocation models
│ │ ├── marketplace.py # Marketplace integration models
│ │ └── admin.py # Admin-specific models
│ └── api/
│ ├── __init__.py # Common imports
│ ├── base.py # Base Pydantic models
│ ├── auth.py # Login, Token, User response models
│ ├── product.py # MarketplaceProduct request/response models
│ ├── inventory.py # Inventory operation models
│ ├── shop.py # Shop management models
│ ├── marketplace.py # Marketplace import models
│ ├── admin.py # Admin operation models
│ └── stats.py # Statistics response models
├── utils/
│ ├── data_processing.py # GTIN and price processing
│ ├── csv_processor.py # CSV import/export
│ └── database.py # Database utilities
├── middleware/
│ ├── auth.py # JWT authentication
│ ├── rate_limiter.py # Rate limiting
│ ├── error_handler.py # Error handling
│ └── logging_middleware.py # Request logging
├── tests/ # Comprehensive test suite
└── requirements.txt # Dependencies
```
### Service Layer Architecture
The application follows a Service Layer Architecture pattern that separates HTTP concerns from business logic:
```
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ HTTP Client │ │ FastAPI App │ │ Database │
└─────────────────┘ └─────────────────┘ └─────────────────┘
│ │ │
│ HTTP Request │ │
├─────────────────────► │ │
│ │ │
│ ┌─────────────────┐ │
│ │ Router │ │
│ │ (HTTP Layer) │ │
│ └─────────────────┘ │
│ │ │
│ │ Delegates to │
│ ▼ │
│ ┌─────────────────┐ │
│ │ Service │ │
│ │ (Business Layer)│ │
│ └─────────────────┘ │
│ │ │
│ │ Database Operations │
│ ├─────────────────────► │
│ │ │
│ HTTP Response │ │
◄─────────────────────── │ │
```
## Development Tools
This project includes a comprehensive Makefile that provides all necessary development commands. Here are the key workflows:
### Quick Start Commands
```bash
# First-time setup (installs dependencies and sets up database)
make setup
# Start development environment (API + docs)
make dev-full
# Start API server only
make dev
# Start documentation server only
make docs
```
### Installation Options
```bash
# Install all dependencies (production + development + testing + docs)
make install-all
# Install only production dependencies
make install
# Install only development dependencies
make install-dev
# Install only testing dependencies
make install-test
# Install only documentation dependencies
make install-docs
```
### Development Workflow
```bash
# Format code (black + isort)
make format
# Run linting (flake8 + mypy)
make lint
# Format and lint together
make check
# Complete CI pipeline (format + lint + test with coverage)
make ci
# Quality assurance check (format + lint + test + docs check)
make qa
```
```bash
# Clone the repository
git clone <wizamart-repo>
cd wizamart-repo
# Create virtual environment
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Complete setup (recommended)
make setup
# Or install dependencies manually
make install-all
```
The `make setup` command will install all dependencies and set up the database automatically.
### Environment Configuration
Create a `.env` file in the project root:
```env
# Database Configuration
DATABASE_URL=postgresql://user:password@localhost/ecommerce
# For development, you can use SQLite:
# DATABASE_URL=sqlite:///./ecommerce.db
# JWT Configuration
SECRET_KEY=your-super-secret-key-change-in-production
ACCESS_TOKEN_EXPIRE_MINUTES=30
# API Configuration
ALLOWED_HOSTS=["*"]
RATE_LIMIT_REQUESTS=100
RATE_LIMIT_WINDOW=3600
# Application Settings
PROJECT_NAME="Ecommerce Backend API"
VERSION="2.2.0"
DEBUG=True
```
### Database Setup
The application will automatically create tables on startup. For production, consider using Alembic for migrations.
```bash
# For PostgreSQL - create database first
createdb ecommerce
# Run database migrations
make migrate-up
# Reset database if needed
make migrate-reset
```
### Running the Application
```bash
# Development server
make dev
# Development server with documentation
make dev-full
# Production server (manual)
uvicorn main:app --host 0.0.0.0 --port 8000 --workers 4
```
The `make dev-full` command starts both the API server and documentation server simultaneously.
The API will be available at:
- **API Documentation**: http://localhost:8000/docs (Swagger UI)
- **Alternative Docs**: http://localhost:8000/redoc
- **Health Check**: http://localhost:8000/health
- **MkDocs Documentation**: http://localhost:8001 (when using `make dev-full`)
## Authentication
### User Registration
```bash
curl -X POST "http://localhost:8000/api/v1/auth/register" \
-H "Content-Type: application/json" \
-d '{
"email": "user@example.com",
"username": "testuser",
"password": "securepassword123"
}'
```
### User Login
```bash
curl -X POST "http://localhost:8000/api/v1/auth/login" \
-H "Content-Type: application/json" \
-d '{
"username": "testuser",
"password": "securepassword123"
}'
```
### Using JWT Token
```bash
# Get token from login response and use in subsequent requests
curl -X GET "http://localhost:8000/api/v1/marketplace/product" \
-H "Authorization: Bearer YOUR_JWT_TOKEN"
```
## Core Features
### MarketplaceProduct Management
#### Create a MarketplaceProduct
```bash
curl -X POST "http://localhost:8000/api/v1/marketplace/product" \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"marketplace_product_id": "PROD001",
"title": "Amazing MarketplaceProduct",
"description": "An amazing product description",
"price": "29.99",
"currency": "EUR",
"brand": "BrandName",
"gtin": "1234567890123",
"availability": "in inventory",
"marketplace": "Letzshop",
"shop_name": "MyShop"
}'
```
#### Search and Filter Products
```bash
# Get all products
curl -X GET "http://localhost:8000/api/v1/marketplace/product" \
-H "Authorization: Bearer YOUR_TOKEN"
# Filter by marketplace
curl -X GET "http://localhost:8000/api/v1/marketplace/product?marketplace=Amazon&limit=50" \
-H "Authorization: Bearer YOUR_TOKEN"
# Search products
curl -X GET "http://localhost:8000/api/v1/marketplace/product?search=Amazing&brand=BrandName" \
-H "Authorization: Bearer YOUR_TOKEN"
```
### Inventory Management
#### Set Inventory Quantity
```bash
curl -X POST "http://localhost:8000/api/v1/inventory" \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"gtin": "1234567890123",
"location": "WAREHOUSE_A",
"quantity": 100
}'
```
#### Add Inventory
```bash
curl -X POST "http://localhost:8000/api/v1/inventory/add" \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"gtin": "1234567890123",
"location": "WAREHOUSE_A",
"quantity": 25
}'
```
#### Check Inventory Levels
```bash
curl -X GET "http://localhost:8000/api/v1/inventory/1234567890123" \
-H "Authorization: Bearer YOUR_TOKEN"
```
### Marketplace Import
#### Import Products from CSV
```bash
curl -X POST "http://localhost:8000/api/v1/marketplace/import-product" \
-H "Authorization: Bearer YOUR_TOKEN" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/products.csv",
"marketplace": "Amazon",
"shop_code": "MYSHOP",
"batch_size": 1000
}'
```
#### Check Import Status
```bash
curl -X GET "http://localhost:8000/api/v1/marketplace/import-status/1" \
-H "Authorization: Bearer YOUR_TOKEN"
```
### Export Data
#### Export Products to CSV
```bash
# Export all products
curl -X GET "http://localhost:8000/api/v1/marketplace/product" \
-H "Authorization: Bearer YOUR_TOKEN" \
-o products_export.csv
# Export with filters
curl -X GET "http://localhost:8000/api/v1/marketplace/product?marketplace=Amazon&shop_name=MyShop" \
-H "Authorization: Bearer YOUR_TOKEN" \
-o amazon_products.csv
```
## CSV Import Format
The system supports CSV imports with the following headers:
### Required Fields
- `product_id` - Unique product identifier
- `title` - MarketplaceProduct title
### Optional Fields
- `description` - MarketplaceProduct description
- `link` - MarketplaceProduct URL
- `image_link` - MarketplaceProduct image URL
- `availability` - Inventory availability (in inventory, out of inventory, preorder)
- `price` - MarketplaceProduct price
- `currency` - Price currency (EUR, USD, etc.)
- `brand` - MarketplaceProduct brand
- `gtin` - Global Trade Item Number (EAN/UPC)
- `google_product_category` - MarketplaceProduct category
- `marketplace` - Source marketplace
- `shop_name` - Shop/seller name
### Example CSV
```csv
product_id,title,description,price,currency,brand,gtin,marketplace,shop_name
PROD001,"Amazing Widget","The best widget ever",29.99,EUR,WidgetCorp,1234567890123,Letzshop,MyShop
PROD002,"Super Gadget","A fantastic gadget",19.99,EUR,GadgetInc,9876543210987,Amazon,TechStore
```
## API Reference
### Authentication Endpoints
- `POST /api/v1/auth/register` - Register new user
- `POST /api/v1/auth/login` - Login user
- `GET /api/v1/auth/me` - Get current user info
### Marketplace Endpoints
- `GET /api/v1/marketplace/product` - List marketplace products with filtering
- `POST /api/v1/marketplace/product` - Create new marketplace product
- `GET /api/v1/marketplace/product/{product_id}` - Get specific marketplace product
- `PUT /api/v1/marketplace/product/{product_id}` - Update marketplace product
- `DELETE /api/v1/marketplace/product/{product_id}` - Delete marketplace product
- `POST /api/v1/marketplace/import-product` - Start CSV import
- `GET /api/v1/marketplace/import-status/{job_id}` - Check import status
- `GET /api/v1/marketplace/import-jobs` - List import jobs
-
### Inventory Endpoints
- `POST /api/v1/inventory` - Set inventory quantity
- `POST /api/v1/inventory/add` - Add to inventory
- `POST /api/v1/inventory/remove` - Remove from inventory
- `GET /api/v1/inventory/{gtin}` - Get inventory by GTIN
- `GET /api/v1/inventory/{gtin}/total` - Get total inventory
- `GET /api/v1/inventory` - List all inventory entries
### Shop Endpoints
- `POST /api/v1/shop` - Create new shop
- `GET /api/v1/shop` - List shops
- `GET /api/v1/shop/{shop_code}` - Get specific shop
### Statistics Endpoints
- `GET /api/v1/stats` - Get general statistics
- `GET /api/v1/stats/marketplace-stats` - Get marketplace statistics
### Admin Endpoints (Admin only)
- `GET /api/v1/admin/users` - List all users
- `PUT /api/v1/admin/users/{user_id}/status` - Toggle user status
- `GET /api/v1/admin/shops` - List all shops
- `PUT /api/v1/admin/shops/{shop_id}/verify` - Verify/unverify shop
## Documentation
This project uses MkDocs for documentation. The Makefile provides comprehensive documentation management commands.
### Documentation Development
```bash
# Start documentation development server
make docs
# Start both API and documentation servers
make dev-full
# Build documentation site
make docs-build
# Check documentation for issues
make docs-check
# Update docs with latest API information
make docs-update
```
### Documentation Deployment
```bash
# Deploy to GitHub Pages
make docs-deploy
# Force deploy to GitHub Pages
make docs-deploy-force
# Build and deploy in one command
make docs-publish
# Preview built documentation locally
make docs-preview
```
### Documentation Maintenance
```bash
# Setup documentation environment
make setup-docs
# Clean documentation build files
make docs-clean
# Get help with MkDocs commands
make docs-help
```
## Database Schema
### Core Tables
- **users** - User accounts and authentication
- **products** - MarketplaceProduct catalog with marketplace info
- **inventory** - Inventory tracking by location and GTIN
- **shops** - Shop/seller information
- **shop_products** - Shop-specific product settings
- **marketplace_import_jobs** - Background import job tracking
### Key Relationships
- Users own shops (one-to-many)
- Products belong to marketplaces and shops
- Inventory entries are linked to products via GTIN
- Import jobs track user-initiated imports
## Security Features
- **JWT Authentication** - Secure token-based authentication
- **Password Hashing** - bcrypt for secure password storage
- **Role-Based Access** - User and admin role separation
- **Rate Limiting** - Protection against API abuse
- **Input Validation** - Comprehensive data validation
- **SQL Injection Protection** - Parameterized queries
- **CORS Configuration** - Cross-origin request handling
## Testing
This project includes a comprehensive test suite with multiple testing strategies.
### Running Tests
```bash
# Run all tests
make test
# Run tests with coverage report
make test-coverage
# Run only unit tests
make test-unit
# Run only integration tests
make test-integration
# Run fast tests (exclude slow ones)
make test-fast
# Run slow tests only
make test-slow
```
### Test Specific Components
```bash
# Authentication tests
make test-auth
# MarketplaceProduct management tests
make test-products
# Inventory management tests
make test-inventory
# Marketplace functionality tests
make test-marketplace
# Admin functionality tests
make test-admin
```
### Test Setup
```bash
# Install test dependencies only
make setup-test
# Complete test workflow (format + lint + coverage)
make full-test
```
### Test Coverage
The test suite includes:
- **Unit Tests** - Individual component testing
- **Integration Tests** - Full workflow testing
- **Security Tests** - Authentication, authorization, input validation
- **Performance Tests** - Load testing with large datasets
- **Error Handling Tests** - Edge cases and error conditions
## Deployment
### Development Deployment
```bash
# Docker deployment
make docker-up
# Staging deployment
make deploy-staging
# Production deployment
make deploy-prod
```
### Cleanup and Maintenance
```bash
# Clean test artifacts and cache files
make clean
# Clean all build artifacts (including documentation)
make clean-all
# Release readiness check
make release-check
```
## Performance Optimizations
- **Database Indexing** - Strategic indexes on key columns
- **Pagination** - Efficient data retrieval with skip/limit
- **Streaming Responses** - Memory-efficient CSV exports
- **Background Processing** - Async import job handling
- **Connection Pooling** - Efficient database connections
- **Query Optimization** - Optimized database queries
## Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run quality assurance checks
5. Ensure all tests pass
6. Submit a pull request
### Development Setup for Contributors
```bash
# Complete development setup
make setup
# Install only development dependencies
make install-dev
# Run quality assurance pipeline
make qa
# Run complete CI pipeline
make ci
```
### Code Quality Standards
```bash
# Format code automatically
make format
# Check code quality
make check
# Run full test suite
make full-test
# Prepare for release
make release-check
```
### Available Make Commands
For a complete list of available commands, run:
```bash
make help
```
This will display all available commands organized by category:
- **Setup**: Installation and environment setup
- **Development**: Development servers and workflows
- **Documentation**: Documentation building and deployment
- **Testing**: Various test execution options
- **Code Quality**: Formatting, linting, and quality checks
- **Database**: Migration and database management
- **Docker**: Container management
- **Cleanup**: Artifact removal and maintenance
- **Workflows**: Combined command sequences
## Changelog
### v2.2.0
- Added marketplace-aware product import
- Implemented multi-shop support
- Enhanced inventory management with location tracking
- Added comprehensive test suite
- Improved authentication and authorization
### v2.1.0
- Added JWT authentication
- Implemented role-based access control
- Added CSV import/export functionality
### v2.0.0
- Complete rewrite with FastAPI
- Added PostgreSQL support
- Implemented comprehensive API documentation
## Support & Documentation
- **API Documentation**: http://localhost:8000/docs
- **Health Check**: http://localhost:8000/health
- **Version Info**: http://localhost:8000/
For issues and feature requests, please create an issue in the repository.
Reference in New Issue View Git Blame Copy Permalink