revamping frontend logging system and reorganising documentation

This commit is contained in:
2025-10-28 21:07:26 +01:00
parent 5c80ba17c5
commit b0cc0385f8
68 changed files with 3481 additions and 624 deletions

View File

@@ -0,0 +1,235 @@
# Database Quick Reference Card
Quick reference for common database operations. See `DATABASE_SETUP_GUIDE.md` for detailed instructions.
---
## 🚀 Quick Start (New Developer)
```bash
# 1. Clone and setup
git clone <repo>
cd letzshop
python -m venv venv
venv\Scripts\activate # Windows
pip install -r requirements.txt
# 2. Initialize database
alembic upgrade head
python scripts/seed_database.py
# 3. Start server
python -m uvicorn app.main:app --reload
# 4. Login
# http://localhost:8000/admin/login
# Username: admin | Password: admin123
```
---
## 🔄 Reset Database (Clean Slate)
**Windows:**
```powershell
.\scripts\reset_database.ps1
```
**Linux/Mac:**
```bash
./scripts/reset_database.sh
```
**Manual:**
```bash
# 1. Delete migrations
Remove-Item alembic\versions\*.py -Exclude __init__.py
# 2. Delete database
Remove-Item wizamart.db
# 3. Regenerate
alembic revision --autogenerate -m "Initial migration"
alembic upgrade head
python scripts/seed_database.py
```
---
## 📝 Creating Migrations
```bash
# After modifying models in models/database/
# 1. Generate migration
alembic revision --autogenerate -m "Add field to vendor_themes"
# 2. Review the file
# Check: alembic/versions/<id>_add_field_to_vendor_themes.py
# 3. Apply migration
alembic upgrade head
```
---
## 🔍 Checking Status
```bash
# Current migration version
alembic current
# Migration history
alembic history --verbose
# List all tables
python -c "import sqlite3; conn = sqlite3.connect('wizamart.db'); cursor = conn.cursor(); cursor.execute('SELECT name FROM sqlite_master WHERE type=\"table\" ORDER BY name;'); [print(t[0]) for t in cursor.fetchall()]"
# Check specific table
python -c "import sqlite3; conn = sqlite3.connect('wizamart.db'); cursor = conn.cursor(); cursor.execute('PRAGMA table_info(vendor_themes)'); [print(col) for col in cursor.fetchall()]"
```
---
## ⬆️ Upgrading Database
```bash
# Upgrade to latest
alembic upgrade head
# Upgrade one version
alembic upgrade +1
# Upgrade to specific revision
alembic upgrade <revision_id>
```
---
## ⬇️ Rolling Back
```bash
# Downgrade one version
alembic downgrade -1
# Downgrade to specific revision
alembic downgrade <revision_id>
# Downgrade to empty database
alembic downgrade base
```
---
## 🌱 Seeding Data
```bash
# Seed development data
python scripts/seed_database.py
# Manual seed (Python shell)
python
>>> from scripts.seed_database import seed_database
>>> seed_database()
```
---
## 🐛 Common Fixes
### "No such table" Error
```bash
# Check model imported in alembic/env.py
# Then regenerate:
alembic revision --autogenerate -m "Add missing table"
alembic upgrade head
```
### "Database locked"
```bash
# Stop server
taskkill /F /IM python.exe # Windows
pkill python # Linux/Mac
# Try again
alembic upgrade head
```
### "Target database not up to date"
```bash
alembic stamp head
alembic upgrade head
```
### "Multiple head revisions"
```bash
alembic merge heads -m "Merge migrations"
alembic upgrade head
```
---
## 📦 Adding New Models
**Checklist:**
```bash
# 1. Create model file
# models/database/my_model.py
# 2. Import in alembic/env.py
# from models.database.my_model import MyModel
# 3. Generate migration
alembic revision --autogenerate -m "Add MyModel table"
# 4. Review migration file
# alembic/versions/<id>_add_mymodel_table.py
# 5. Apply migration
alembic upgrade head
# 6. Test
python -c "from models.database.my_model import MyModel; print('✓ Import works')"
```
---
## 🔗 Important Files
```
├── alembic/
│ ├── env.py ← Import ALL models here!
│ └── versions/ ← Migration files
├── app/core/
│ └── database.py ← Database connection
├── models/database/ ← SQLAlchemy models
├── scripts/
│ ├── seed_database.py ← Seed script
│ └── reset_database.ps1 ← Reset script
├── alembic.ini ← Alembic config
└── wizamart.db ← SQLite database
```
---
## 🆘 Getting Help
1. Check full guide: `docs/DATABASE_SETUP_GUIDE.md`
2. Check Alembic docs: https://alembic.sqlalchemy.org/
3. Ask team lead
---
## ⚠️ Important Notes
- ✅ Always review migrations before applying
- ✅ Test on local database first
- ✅ Keep migrations in version control
- ✅ Backup before destructive operations
- ❌ Never edit applied migrations
- ❌ Never delete migration files
---
**Last Updated:** 2025-10-27

View File

@@ -0,0 +1,883 @@
# Database Setup & Initialization Guide
## Overview
This guide walks you through setting up the LetzShop database from scratch. Whether you're a new developer joining the team or need to reset your local database, this document covers everything you need to know.
---
## Table of Contents
1. [Prerequisites](#prerequisites)
2. [First-Time Setup](#first-time-setup)
3. [Database Reset (Clean Slate)](#database-reset-clean-slate)
4. [Verifying Your Setup](#verifying-your-setup)
5. [Common Issues & Solutions](#common-issues--solutions)
6. [Database Migrations](#database-migrations)
7. [Seeding Data](#seeding-data)
---
## Prerequisites
### Required Software
- **Python 3.11+**
- **Alembic** (installed via `requirements.txt`)
- **SQLite** (built into Python) or **PostgreSQL** (for production-like setup)
### Environment Setup
```bash
# 1. Clone the repository
git clone <repository-url>
cd letzshop
# 2. Create virtual environment
python -m venv venv
# 3. Activate virtual environment
# Windows:
venv\Scripts\activate
# Linux/Mac:
source venv/bin/activate
# 4. Install dependencies
pip install -r requirements.txt
```
---
## First-Time Setup
### Step 1: Configure Database Connection
Create or update `.env` file in project root:
```env
# Database Configuration
DATABASE_URL=sqlite:///./wizamart.db
# For PostgreSQL (production):
# DATABASE_URL=postgresql://user:password@localhost:5432/letzshop
# Other required settings
SECRET_KEY=your-secret-key-here-change-in-production
ALGORITHM=HS256
ACCESS_TOKEN_EXPIRE_MINUTES=30
```
---
### Step 2: Initialize Alembic (First Time Only)
**If `alembic/` folder doesn't exist:**
```bash
# Initialize Alembic
alembic init alembic
```
**Configure `alembic.ini`:**
```ini
# alembic.ini
# Find this line and update:
sqlalchemy.url = sqlite:///./wizamart.db
# Or for PostgreSQL:
# sqlalchemy.url = postgresql://user:password@localhost:5432/letzshop
```
**Configure `alembic/env.py`:**
```python
# alembic/env.py
# Add this near the top (after imports)
import sys
from pathlib import Path
# Add project root to Python path
sys.path.append(str(Path(__file__).parents[1]))
# ============================================================================
# CRITICAL: Import ALL database models here
# ============================================================================
from models.database.user import User
from models.database.vendor import Vendor
from models.database.vendor_domain import VendorDomain
from models.database.vendor_theme import VendorTheme
from models.database.customer import Customer
from models.database.team import Team, TeamMember, TeamInvitation
from models.database.product import Product
from models.database.marketplace_product import MarketplaceProduct
from models.database.inventory import Inventory
from models.database.order import Order
from models.database.marketplace_import_job import MarketplaceImportJob
# Import Base
from app.core.database import Base
# Set target metadata
target_metadata = Base.metadata
```
---
### Step 3: Generate Initial Migration
```bash
# Generate migration from your models
alembic revision --autogenerate -m "Initial migration - all tables"
# You should see output listing all detected tables:
# INFO [alembic.autogenerate.compare] Detected added table 'users'
# INFO [alembic.autogenerate.compare] Detected added table 'vendors'
# INFO [alembic.autogenerate.compare] Detected added table 'vendor_themes'
# etc.
```
---
### Step 4: Apply Migration
```bash
# Apply the migration to create all tables
alembic upgrade head
# You should see:
# INFO [alembic.runtime.migration] Running upgrade -> abc123def456, Initial migration - all tables
```
---
### Step 5: Seed Initial Data
Create an admin user and test vendor:
```bash
python scripts/seed_database.py
```
**Or manually:**
```python
# Run Python interactive shell
python
# Then execute:
from app.core.database import SessionLocal
from models.database.user import User
from models.database.vendor import Vendor
from middleware.auth import AuthManager
from datetime import datetime, timezone
db = SessionLocal()
auth_manager = AuthManager()
# Create admin user
admin = User(
username="admin",
email="admin@letzshop.com",
hashed_password=auth_manager.hash_password("admin123"),
role="admin",
is_active=True,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc)
)
db.add(admin)
db.flush()
# Create test vendor
vendor = Vendor(
vendor_code="TESTVENDOR",
subdomain="testvendor",
name="Test Vendor",
description="Development test vendor",
owner_user_id=admin.id,
contact_email="contact@testvendor.com",
is_active=True,
is_verified=True,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc)
)
db.add(vendor)
db.commit()
print("✅ Admin user and test vendor created!")
db.close()
exit()
```
---
### Step 6: Start the Application
```bash
# Start FastAPI server
python -m uvicorn app.main:app --reload
# Server should start on http://localhost:8000
```
**Test the setup:**
- Admin login: http://localhost:8000/admin/login
- Username: `admin`
- Password: `admin123`
---
## Database Reset (Clean Slate)
Use this when you want to completely reset your database and start fresh.
⚠️ **WARNING:** This will delete ALL data! Make a backup if needed.
---
### Quick Reset Script (Windows PowerShell)
**Create `scripts/reset_database.ps1`:**
```powershell
# scripts/reset_database.ps1
Write-Host "🗑️ Database Reset Script" -ForegroundColor Cyan
Write-Host "This will delete ALL data!" -ForegroundColor Red
$confirm = Read-Host "Type 'YES' to continue"
if ($confirm -ne "YES") {
Write-Host "Aborted." -ForegroundColor Yellow
exit
}
Write-Host ""
Write-Host "Step 1: Backing up current database..." -ForegroundColor Yellow
if (Test-Path wizamart.db) {
$backupName = "wizamart_backup_$(Get-Date -Format 'yyyyMMdd_HHmmss').db"
Copy-Item wizamart.db $backupName
Write-Host "✓ Backup created: $backupName" -ForegroundColor Green
}
Write-Host ""
Write-Host "Step 2: Removing old migrations..." -ForegroundColor Yellow
Remove-Item alembic\versions\*.py -Exclude __init__.py -ErrorAction SilentlyContinue
Remove-Item -Recurse -Force alembic\versions\__pycache__ -ErrorAction SilentlyContinue
# Ensure __init__.py exists
if (-not (Test-Path alembic\versions\__init__.py)) {
New-Item -Path alembic\versions\__init__.py -ItemType File | Out-Null
}
Write-Host "✓ Old migrations removed" -ForegroundColor Green
Write-Host ""
Write-Host "Step 3: Deleting database..." -ForegroundColor Yellow
Remove-Item wizamart.db -ErrorAction SilentlyContinue
Write-Host "✓ Database deleted" -ForegroundColor Green
Write-Host ""
Write-Host "Step 4: Generating new migration..." -ForegroundColor Yellow
alembic revision --autogenerate -m "Initial migration - all tables"
Write-Host "✓ Migration generated" -ForegroundColor Green
Write-Host ""
Write-Host "Step 5: Applying migration..." -ForegroundColor Yellow
alembic upgrade head
Write-Host "✓ Migration applied" -ForegroundColor Green
Write-Host ""
Write-Host "Step 6: Verifying tables..." -ForegroundColor Yellow
python -c @"
import sqlite3
conn = sqlite3.connect('wizamart.db')
cursor = conn.cursor()
cursor.execute('SELECT name FROM sqlite_master WHERE type=\"table\" ORDER BY name;')
tables = [t[0] for t in cursor.fetchall()]
print('\n📊 Tables created:')
for t in tables:
print(f' {t}')
print(f'\n Total: {len(tables)} tables')
if 'vendor_themes' in tables:
print(' vendor_themes table confirmed!')
conn.close()
"@
Write-Host ""
Write-Host "Step 7: Seeding initial data..." -ForegroundColor Yellow
python scripts/seed_database.py
Write-Host ""
Write-Host "✅ Database reset complete!" -ForegroundColor Green
Write-Host ""
Write-Host "Next steps:" -ForegroundColor Cyan
Write-Host " 1. Start server: python -m uvicorn app.main:app --reload" -ForegroundColor White
Write-Host " 2. Login: http://localhost:8000/admin/login" -ForegroundColor White
Write-Host " 3. Username: admin | Password: admin123" -ForegroundColor White
```
**Run it:**
```powershell
# First time only - allow script execution
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser
# Run the reset
.\scripts\reset_database.ps1
```
---
### Quick Reset Script (Linux/Mac Bash)
**Create `scripts/reset_database.sh`:**
```bash
#!/bin/bash
# scripts/reset_database.sh
echo "🗑️ Database Reset Script"
echo "⚠️ This will delete ALL data!"
read -p "Type 'YES' to continue: " confirm
if [ "$confirm" != "YES" ]; then
echo "Aborted."
exit 1
fi
echo ""
echo "Step 1: Backing up current database..."
if [ -f wizamart.db ]; then
backup_name="wizamart_backup_$(date +%Y%m%d_%H%M%S).db"
cp wizamart.db "$backup_name"
echo "✓ Backup created: $backup_name"
fi
echo ""
echo "Step 2: Removing old migrations..."
find alembic/versions -type f -name "*.py" ! -name "__init__.py" -delete
rm -rf alembic/versions/__pycache__
touch alembic/versions/__init__.py
echo "✓ Old migrations removed"
echo ""
echo "Step 3: Deleting database..."
rm -f wizamart.db
echo "✓ Database deleted"
echo ""
echo "Step 4: Generating new migration..."
alembic revision --autogenerate -m "Initial migration - all tables"
echo "✓ Migration generated"
echo ""
echo "Step 5: Applying migration..."
alembic upgrade head
echo "✓ Migration applied"
echo ""
echo "Step 6: Verifying tables..."
python -c "
import sqlite3
conn = sqlite3.connect('wizamart.db')
cursor = conn.cursor()
cursor.execute('SELECT name FROM sqlite_master WHERE type=\"table\" ORDER BY name;')
tables = [t[0] for t in cursor.fetchall()]
print('\n📊 Tables created:')
for t in tables:
print(f' ✓ {t}')
print(f'\n✅ Total: {len(tables)} tables')
if 'vendor_themes' in tables:
print('✅ vendor_themes table confirmed!')
conn.close()
"
echo ""
echo "Step 7: Seeding initial data..."
python scripts/seed_database.py
echo ""
echo "✅ Database reset complete!"
echo ""
echo "Next steps:"
echo " 1. Start server: python -m uvicorn app.main:app --reload"
echo " 2. Login: http://localhost:8000/admin/login"
echo " 3. Username: admin | Password: admin123"
```
**Run it:**
```bash
chmod +x scripts/reset_database.sh
./scripts/reset_database.sh
```
---
### Manual Reset Steps
If you prefer to run commands manually:
```bash
# 1. Backup (optional)
cp wizamart.db wizamart_backup.db # Windows: copy wizamart.db wizamart_backup.db
# 2. Remove migrations
rm alembic/versions/*.py # Windows: Remove-Item alembic\versions\*.py -Exclude __init__.py
touch alembic/versions/__init__.py # Windows: New-Item -Path alembic\versions\__init__.py
# 3. Delete database
rm wizamart.db # Windows: Remove-Item wizamart.db
# 4. Generate migration
alembic revision --autogenerate -m "Initial migration - all tables"
# 5. Apply migration
alembic upgrade head
# 6. Seed data
python scripts/seed_database.py
```
---
## Verifying Your Setup
### Check Alembic Status
```bash
# Check current migration version
alembic current
# Should show:
# <revision_id> (head)
```
### Verify Tables Exist
**Using Python:**
```python
import sqlite3
conn = sqlite3.connect('wizamart.db')
cursor = conn.cursor()
# List all tables
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name;")
tables = cursor.fetchall()
print("📊 Database Tables:")
for table in tables:
print(f"{table[0]}")
# Check specific table structure
cursor.execute("PRAGMA table_info(vendor_themes);")
columns = cursor.fetchall()
print("\n📋 vendor_themes columns:")
for col in columns:
print(f" - {col[1]} ({col[2]})")
conn.close()
```
### Test Database Connection
```bash
# Start Python shell
python
# Test connection
from app.core.database import SessionLocal, engine
from sqlalchemy import text
db = SessionLocal()
result = db.execute(text("SELECT name FROM sqlite_master WHERE type='table';"))
tables = [row[0] for row in result]
print("✅ Database connection successful!")
print(f"📊 Tables found: {len(tables)}")
for table in tables:
print(f" - {table}")
db.close()
exit()
```
### Test Application Startup
```bash
# Start server
python -m uvicorn app.main:app --reload
# Should see:
# INFO: Uvicorn running on http://127.0.0.1:8000
# INFO: Application startup complete.
```
**No errors = successful setup!**
---
## Common Issues & Solutions
### Issue 1: "No such table" Error
**Symptom:**
```
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: vendor_themes
```
**Solution:**
```bash
# 1. Check if model is imported in alembic/env.py
# Open alembic/env.py and verify:
from models.database.vendor_theme import VendorTheme
# 2. Regenerate migration
alembic revision --autogenerate -m "Add missing tables"
# 3. Apply migration
alembic upgrade head
```
---
### Issue 2: "Target database is not up to date"
**Symptom:**
```
ERROR [alembic.util.messaging] Target database is not up to date.
```
**Solution:**
```bash
# Option 1: Stamp to current head
alembic stamp head
# Option 2: Check and upgrade
alembic current # See current version
alembic upgrade head # Upgrade to latest
# Option 3: Full reset (see Database Reset section)
```
---
### Issue 3: Database Locked
**Symptom:**
```
sqlite3.OperationalError: database is locked
```
**Solution:**
```bash
# 1. Stop the FastAPI server (Ctrl+C)
# 2. Kill any Python processes
# Windows:
taskkill /F /IM python.exe
# Linux/Mac:
pkill python
# 3. Close any database browser applications
# 4. Try again
alembic upgrade head
```
---
### Issue 4: Migration File Conflicts
**Symptom:**
```
Multiple head revisions are present
```
**Solution:**
```bash
# Option 1: Merge migrations
alembic merge heads -m "Merge migrations"
alembic upgrade head
# Option 2: Clean reset (recommended for development)
# See Database Reset section above
```
---
### Issue 5: Import Errors in alembic/env.py
**Symptom:**
```
ModuleNotFoundError: No module named 'models'
```
**Solution:**
Check `alembic/env.py` has correct path setup:
```python
# alembic/env.py (top of file)
import sys
from pathlib import Path
# Add project root to Python path
sys.path.append(str(Path(__file__).parents[1]))
```
---
## Database Migrations
### Creating New Migrations
When you add/modify models:
```bash
# 1. Update your model file (e.g., models/database/vendor_theme.py)
# 2. Generate migration
alembic revision --autogenerate -m "Add new field to vendor_themes"
# 3. Review the generated migration
# Check: alembic/versions/<revision>_add_new_field_to_vendor_themes.py
# 4. Apply migration
alembic upgrade head
```
---
### Migration Best Practices
**DO:**
- Review generated migrations before applying
- Use descriptive migration messages
- Test migrations on development database first
- Keep migrations small and focused
- Commit migrations to version control
**DON'T:**
- Edit applied migrations (create new ones instead)
- Delete migration files from version control
- Skip migration review
- Combine unrelated changes in one migration
---
### Rolling Back Migrations
```bash
# Downgrade one version
alembic downgrade -1
# Downgrade to specific revision
alembic downgrade <revision_id>
# Downgrade to base (empty database)
alembic downgrade base
```
---
## Seeding Data
### Development Data Seed
**Create `scripts/seed_database.py`:**
```python
# scripts/seed_database.py
"""
Seed the database with initial development data.
"""
from app.core.database import SessionLocal
from models.database.user import User
from models.database.vendor import Vendor
from app.core.security import get_password_hash
from datetime import datetime, timezone
def seed_database():
"""Seed database with initial data."""
db = SessionLocal()
try:
print("🌱 Seeding database...")
# Check if admin already exists
existing_admin = db.query(User).filter(User.username == "admin").first()
if existing_admin:
print("⚠️ Admin user already exists, skipping...")
return
# Create admin user
admin = User(
username="admin",
email="admin@letzshop.com",
hashed_password=get_password_hash("admin123"),
is_admin=True,
is_active=True,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc)
)
db.add(admin)
db.flush()
print(f"✓ Admin user created (ID: {admin.id})")
# Create test vendor
vendor = Vendor(
vendor_code="TESTVENDOR",
subdomain="testvendor",
name="Test Vendor",
description="Development test vendor",
owner_user_id=admin.id,
contact_email="contact@testvendor.com",
is_active=True,
is_verified=True,
created_at=datetime.now(timezone.utc),
updated_at=datetime.now(timezone.utc)
)
db.add(vendor)
db.flush()
print(f"✓ Test vendor created: {vendor.vendor_code}")
db.commit()
print("\n✅ Database seeded successfully!")
print("\n📝 Login Credentials:")
print(" URL: http://localhost:8000/admin/login")
print(" Username: admin")
print(" Password: admin123")
except Exception as e:
db.rollback()
print(f"❌ Error seeding database: {e}")
raise
finally:
db.close()
if __name__ == "__main__":
seed_database()
```
**Run it:**
```bash
python scripts/seed_database.py
```
---
### Production Data Migration
For production, create separate seed scripts:
```python
# scripts/seed_production.py
# - Create only essential admin user
# - No test data
# - Strong passwords from environment variables
```
---
## Quick Reference
### Essential Commands
```bash
# Check migration status
alembic current
# Generate new migration
alembic revision --autogenerate -m "Description"
# Apply migrations
alembic upgrade head
# Rollback one migration
alembic downgrade -1
# View migration history
alembic history
# Seed database
python scripts/seed_database.py
# Start server
python -m uvicorn app.main:app --reload
```
---
### File Locations
```
project/
├── alembic/
│ ├── versions/ # Migration files here
│ │ └── __init__.py
│ ├── env.py # Alembic configuration (import models here!)
│ └── script.py.mako
├── app/
│ └── core/
│ └── database.py # Database connection
├── models/
│ └── database/ # SQLAlchemy models
│ ├── user.py
│ ├── vendor.py
│ ├── vendor_theme.py
│ └── ...
├── scripts/
│ ├── seed_database.py # Development seed
│ └── reset_database.ps1 # Reset script
├── alembic.ini # Alembic config
├── wizamart.db # SQLite database (gitignored)
└── .env # Environment variables (gitignored)
```
---
## Getting Help
### Internal Resources
- **Architecture Guide:** `docs/PROPER_ARCHITECTURE_GUIDE.md`
- **Exception Handling:** `docs/EXCEPTION_PATTERN_EXPLAINED.md`
- **Admin Feature Guide:** `docs/ADMIN_FEATURE_INTEGRATION_GUIDE.md`
### External Resources
- [Alembic Documentation](https://alembic.sqlalchemy.org/)
- [SQLAlchemy Documentation](https://docs.sqlalchemy.org/)
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
---
## Contributing
When adding new models:
1. ✅ Create the model in `models/database/`
2. ✅ Import it in `alembic/env.py`
3. ✅ Generate migration: `alembic revision --autogenerate -m "Add XYZ model"`
4. ✅ Review the migration file
5. ✅ Test locally: `alembic upgrade head`
6. ✅ Commit both model and migration files
---
**Last Updated:** 2025-10-27
**Maintainer:** Development Team
**Questions?** Contact the team lead or check internal documentation.