orion/docs/getting-started/database-setup-guide.md

# Database Setup & Initialization Guide

## Overview

This guide walks you through setting up the Orion database from scratch. Whether you're a new developer joining the team or need to reset your local database, this document covers everything you need to know.

---

## Table of Contents

1. [Prerequisites](#prerequisites)
2. [First-Time Setup](#first-time-setup)
3. [Database Reset (Clean Slate)](#database-reset-clean-slate)
4. [Verifying Your Setup](#verifying-your-setup)
5. [Common Issues & Solutions](#common-issues-solutions)
6. [Database Migrations](#database-migrations)
7. [Seeding Data](#seeding-data)

---

## Prerequisites

### Required Software

- **Python 3.11+**
- **Alembic** (installed via `requirements.txt`)
- **SQLite** (built into Python) or **PostgreSQL** (for production-like setup)

### Environment Setup

```bash
# 1. Clone the repository
    git clone <orion-repo>
    cd orion-repo

# 2. Create virtual environment
    python -m venv venv

# 3. Activate virtual environment
# Windows:
    venv\Scripts\activate
# Linux/Mac:
    source venv/bin/activate

# 4. Install dependencies
    pip install -r requirements.txt
```

---

## First-Time Setup

### Step 1: Configure Database Connection

Create or update `.env` file in project root:

```env
# Database Configuration
DATABASE_URL=sqlite:///./orion.db

# For PostgreSQL (production):
# DATABASE_URL=postgresql://user:password@localhost:5432/orion

# Other required settings
SECRET_KEY=your-secret-key-here-change-in-production
ALGORITHM=HS256
ACCESS_TOKEN_EXPIRE_MINUTES=30
```

---

### Step 2: Initialize Alembic (First Time Only)

**If `alembic/` folder doesn't exist:**

```bash
# Initialize Alembic
alembic init alembic
```

**Configure `alembic.ini`:**

```ini
# alembic.ini
# Find this line and update:
sqlalchemy.url = sqlite:///./orion.db

# Or for PostgreSQL:
# sqlalchemy.url = postgresql://user:password@localhost:5432/orion
```

**Configure `alembic/env.py`:**

```python
# alembic/env.py

# Add this near the top (after imports)
import sys
from pathlib import Path

# Add project root to Python path
sys.path.append(str(Path(__file__).parents[1]))

# ============================================================================
# CRITICAL: Import ALL database models here
# ============================================================================
from models.database.user import User
from models.database.store import Store
from models.database.store_domain import StoreDomain
from models.database.store_theme import StoreTheme
from models.database.customer import Customer
from models.database.team import Team, TeamMember, TeamInvitation
from models.database.product import Product
from models.database.marketplace_product import MarketplaceProduct
from models.database.inventory import Inventory
from models.database.order import Order
from models.database.marketplace_import_job import MarketplaceImportJob

# Import Base
from app.core.database import Base

# Set target metadata
target_metadata = Base.metadata
```

---

### Step 3: Generate Initial Migration

```bash
# Generate migration from your models
alembic revision --autogenerate -m "Initial migration - all tables"

# You should see output listing all detected tables:
# INFO  [alembic.autogenerate.compare] Detected added table 'users'
# INFO  [alembic.autogenerate.compare] Detected added table 'stores'
# INFO  [alembic.autogenerate.compare] Detected added table 'store_themes'
# etc.
```

---

### Step 4: Apply Migration

```bash
# Apply the migration to create all tables
alembic upgrade head

# You should see:
# INFO  [alembic.runtime.migration] Running upgrade  -> abc123def456, Initial migration - all tables
```

---

### Step 5: Seed Initial Data

Create an admin user and test store:

```bash
python scripts/seed_database.py
```

**Or manually:**

```python
# Run Python interactive shell
python

# Then execute:
from app.core.database import SessionLocal
from models.database.user import User
from models.database.store import Store
from middleware.auth import AuthManager
from datetime import datetime, timezone

db = SessionLocal()
auth_manager = AuthManager()

# Create admin user
admin = User(
    username="admin",
    email="admin@orion.lu",
    hashed_password=auth_manager.hash_password("admin123"),
    role="admin",
    is_active=True,
    created_at=datetime.now(timezone.utc),
    updated_at=datetime.now(timezone.utc)
)
db.add(admin)
db.flush()

# Create test store
store = Store(
    store_code="TESTSTORE",
    subdomain="teststore",
    name="Test Store",
    description="Development test store",
    owner_user_id=admin.id,
    contact_email="contact@teststore.com",
    is_active=True,
    is_verified=True,
    created_at=datetime.now(timezone.utc),
    updated_at=datetime.now(timezone.utc)
)
db.add(store)

db.commit()
print("✅ Admin user and test store created!")
db.close()
exit()
```

---

### Step 6: Start the Application

```bash
# Start FastAPI server
python -m uvicorn app.main:app --reload

# Server should start on http://localhost:8000
```

**Test the setup:**
- Admin login: http://localhost:8000/admin/login
- Username: `admin`
- Password: `admin123`

---

## Database Reset (Clean Slate)

Use this when you want to completely reset your database and start fresh.

⚠️ **WARNING:** This will delete ALL data! Make a backup if needed.

---

### Quick Reset Script (Windows PowerShell)

**Create `scripts/reset_database.ps1`:**

```powershell
# scripts/reset_database.ps1
Write-Host "🗑️  Database Reset Script" -ForegroundColor Cyan
Write-Host "This will delete ALL data!" -ForegroundColor Red
$confirm = Read-Host "Type 'YES' to continue"

if ($confirm -ne "YES") {
    Write-Host "Aborted." -ForegroundColor Yellow
    exit
}

Write-Host ""
Write-Host "Step 1: Backing up current database..." -ForegroundColor Yellow
if (Test-Path orion.db) {
    $backupName = "orion_backup_$(Get-Date -Format 'yyyyMMdd_HHmmss').db"
    Copy-Item orion.db $backupName
    Write-Host "✓ Backup created: $backupName" -ForegroundColor Green
}

Write-Host ""
Write-Host "Step 2: Removing old migrations..." -ForegroundColor Yellow
Remove-Item alembic\versions\*.py -Exclude __init__.py -ErrorAction SilentlyContinue
Remove-Item -Recurse -Force alembic\versions\__pycache__ -ErrorAction SilentlyContinue

# Ensure __init__.py exists
if (-not (Test-Path alembic\versions\__init__.py)) {
    New-Item -Path alembic\versions\__init__.py -ItemType File | Out-Null
}
Write-Host "✓ Old migrations removed" -ForegroundColor Green

Write-Host ""
Write-Host "Step 3: Deleting database..." -ForegroundColor Yellow
Remove-Item orion.db -ErrorAction SilentlyContinue
Write-Host "✓ Database deleted" -ForegroundColor Green

Write-Host ""
Write-Host "Step 4: Generating new migration..." -ForegroundColor Yellow
alembic revision --autogenerate -m "Initial migration - all tables"
Write-Host "✓ Migration generated" -ForegroundColor Green

Write-Host ""
Write-Host "Step 5: Applying migration..." -ForegroundColor Yellow
alembic upgrade head
Write-Host "✓ Migration applied" -ForegroundColor Green

Write-Host ""
Write-Host "Step 6: Verifying tables..." -ForegroundColor Yellow
python -c @"
import sqlite3
conn = sqlite3.connect('orion.db')
cursor = conn.cursor()
cursor.execute('SELECT name FROM sqlite_master WHERE type=\"table\" ORDER BY name;')
tables = [t[0] for t in cursor.fetchall()]
print('\n📊 Tables created:')
for t in tables:
    print(f'  ✓ {t}')
print(f'\n✅ Total: {len(tables)} tables')
if 'store_themes' in tables:
    print('✅ store_themes table confirmed!')
conn.close()
"@

Write-Host ""
Write-Host "Step 7: Seeding initial data..." -ForegroundColor Yellow
python scripts/seed_database.py

Write-Host ""
Write-Host "✅ Database reset complete!" -ForegroundColor Green
Write-Host ""
Write-Host "Next steps:" -ForegroundColor Cyan
Write-Host "  1. Start server: python -m uvicorn app.main:app --reload" -ForegroundColor White
Write-Host "  2. Login: http://localhost:8000/admin/login" -ForegroundColor White
Write-Host "  3. Username: admin | Password: admin123" -ForegroundColor White
```

**Run it:**

```powershell
# First time only - allow script execution
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

# Run the reset
.\scripts\reset_database.ps1
```

---

### Quick Reset Script (Linux/Mac Bash)

**Create `scripts/reset_database.sh`:**

```bash
#!/bin/bash
# scripts/reset_database.sh

echo "🗑️  Database Reset Script"
echo "⚠️  This will delete ALL data!"
read -p "Type 'YES' to continue: " confirm

if [ "$confirm" != "YES" ]; then
    echo "Aborted."
    exit 1
fi

echo ""
echo "Step 1: Backing up current database..."
if [ -f orion.db ]; then
    backup_name="orion_backup_$(date +%Y%m%d_%H%M%S).db"
    cp orion.db "$backup_name"
    echo "✓ Backup created: $backup_name"
fi

echo ""
echo "Step 2: Removing old migrations..."
find alembic/versions -type f -name "*.py" ! -name "__init__.py" -delete
rm -rf alembic/versions/__pycache__
touch alembic/versions/__init__.py
echo "✓ Old migrations removed"

echo ""
echo "Step 3: Deleting database..."
rm -f orion.db
echo "✓ Database deleted"

echo ""
echo "Step 4: Generating new migration..."
alembic revision --autogenerate -m "Initial migration - all tables"
echo "✓ Migration generated"

echo ""
echo "Step 5: Applying migration..."
alembic upgrade head
echo "✓ Migration applied"

echo ""
echo "Step 6: Verifying tables..."
python -c "
import sqlite3
conn = sqlite3.connect('orion.db')
cursor = conn.cursor()
cursor.execute('SELECT name FROM sqlite_master WHERE type=\"table\" ORDER BY name;')
tables = [t[0] for t in cursor.fetchall()]
print('\n📊 Tables created:')
for t in tables:
    print(f'  ✓ {t}')
print(f'\n✅ Total: {len(tables)} tables')
if 'store_themes' in tables:
    print('✅ store_themes table confirmed!')
conn.close()
"

echo ""
echo "Step 7: Seeding initial data..."
python scripts/seed_database.py

echo ""
echo "✅ Database reset complete!"
echo ""
echo "Next steps:"
echo "  1. Start server: python -m uvicorn app.main:app --reload"
echo "  2. Login: http://localhost:8000/admin/login"
echo "  3. Username: admin | Password: admin123"
```

**Run it:**

```bash
chmod +x scripts/reset_database.sh
./scripts/reset_database.sh
```

---

### Manual Reset Steps

If you prefer to run commands manually:

```bash
# 1. Backup (optional)
cp orion.db orion_backup.db  # Windows: copy orion.db orion_backup.db

# 2. Remove migrations
rm alembic/versions/*.py  # Windows: Remove-Item alembic\versions\*.py -Exclude __init__.py
touch alembic/versions/__init__.py  # Windows: New-Item -Path alembic\versions\__init__.py

# 3. Delete database
rm orion.db  # Windows: Remove-Item orion.db

# 4. Generate migration
alembic revision --autogenerate -m "Initial migration - all tables"

# 5. Apply migration
alembic upgrade head

# 6. Seed data
python scripts/seed_database.py
```

---

## Verifying Your Setup

### Check Alembic Status

```bash
# Check current migration version
alembic current

# Should show:
# <revision_id> (head)
```

### Verify Tables Exist

**Using Python:**

```python
import sqlite3

conn = sqlite3.connect('orion.db')
cursor = conn.cursor()

# List all tables
cursor.execute("SELECT name FROM sqlite_master WHERE type='table' ORDER BY name;")
tables = cursor.fetchall()

print("📊 Database Tables:")
for table in tables:
    print(f"  ✓ {table[0]}")

# Check specific table structure
cursor.execute("PRAGMA table_info(store_themes);")
columns = cursor.fetchall()

print("\n📋 store_themes columns:")
for col in columns:
    print(f"  - {col[1]} ({col[2]})")

conn.close()
```

### Test Database Connection

```bash
# Start Python shell
python

# Test connection
from app.core.database import SessionLocal, engine
from sqlalchemy import text

db = SessionLocal()
result = db.execute(text("SELECT name FROM sqlite_master WHERE type='table';"))
tables = [row[0] for row in result]

print("✅ Database connection successful!")
print(f"📊 Tables found: {len(tables)}")
for table in tables:
    print(f"  - {table}")

db.close()
exit()
```

### Test Application Startup

```bash
# Start server
python -m uvicorn app.main:app --reload

# Should see:
# INFO:     Uvicorn running on http://127.0.0.1:8000
# INFO:     Application startup complete.
```

**No errors = successful setup!**

---

## Common Issues & Solutions

### Issue 1: "No such table" Error

**Symptom:**
```
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: store_themes
```

**Solution:**

```bash
# 1. Check if model is imported in alembic/env.py
# Open alembic/env.py and verify:
from models.database.store_theme import StoreTheme

# 2. Regenerate migration
alembic revision --autogenerate -m "Add missing tables"

# 3. Apply migration
alembic upgrade head
```

---

### Issue 2: "Target database is not up to date"

**Symptom:**
```
ERROR [alembic.util.messaging] Target database is not up to date.
```

**Solution:**

```bash
# Option 1: Stamp to current head
alembic stamp head

# Option 2: Check and upgrade
alembic current  # See current version
alembic upgrade head  # Upgrade to latest

# Option 3: Full reset (see Database Reset section)
```

---

### Issue 3: Database Locked

**Symptom:**
```
sqlite3.OperationalError: database is locked
```

**Solution:**

```bash
# 1. Stop the FastAPI server (Ctrl+C)

# 2. Kill any Python processes
# Windows:
taskkill /F /IM python.exe

# Linux/Mac:
pkill python

# 3. Close any database browser applications

# 4. Try again
alembic upgrade head
```

---

### Issue 4: Migration File Conflicts

**Symptom:**
```
Multiple head revisions are present
```

**Solution:**

```bash
# Option 1: Merge migrations
alembic merge heads -m "Merge migrations"
alembic upgrade head

# Option 2: Clean reset (recommended for development)
# See Database Reset section above
```

---

### Issue 5: Import Errors in alembic/env.py

**Symptom:**
```
ModuleNotFoundError: No module named 'models'
```

**Solution:**

Check `alembic/env.py` has correct path setup:

```python
# alembic/env.py (top of file)
import sys
from pathlib import Path

# Add project root to Python path
sys.path.append(str(Path(__file__).parents[1]))
```

---

## Database Migrations

### Creating New Migrations

When you add/modify models:

```bash
# 1. Update your model file (e.g., models/database/store_theme.py)

# 2. Generate migration
alembic revision --autogenerate -m "Add new field to store_themes"

# 3. Review the generated migration
# Check: alembic/versions/<revision>_add_new_field_to_store_themes.py

# 4. Apply migration
alembic upgrade head
```

---

### Migration Best Practices

✅ **DO:**
- Review generated migrations before applying
- Use descriptive migration messages
- Test migrations on development database first
- Keep migrations small and focused
- Commit migrations to version control

❌ **DON'T:**
- Edit applied migrations (create new ones instead)
- Delete migration files from version control
- Skip migration review
- Combine unrelated changes in one migration

---

### Rolling Back Migrations

```bash
# Downgrade one version
alembic downgrade -1

# Downgrade to specific revision
alembic downgrade <revision_id>

# Downgrade to base (empty database)
alembic downgrade base
```

---

## Seeding Data

### Development Data Seed

**Create `scripts/seed_database.py`:**

```python
# scripts/seed_database.py
"""
Seed the database with initial development data.
"""

from app.core.database import SessionLocal
from models.database.user import User
from models.database.store import Store
from app.core.security import get_password_hash
from datetime import datetime, timezone


def seed_database():
    """Seed database with initial data."""
    db = SessionLocal()

    try:
        print("🌱 Seeding database...")

        # Check if admin already exists
        existing_admin = db.query(User).filter(User.username == "admin").first()
        if existing_admin:
            print("⚠️  Admin user already exists, skipping...")
            return

        # Create admin user
        admin = User(
            username="admin",
            email="admin@orion.lu",
            hashed_password=get_password_hash("admin123"),
            is_admin=True,
            is_active=True,
            created_at=datetime.now(timezone.utc),
            updated_at=datetime.now(timezone.utc)
        )
        db.add(admin)
        db.flush()
        print(f"✓ Admin user created (ID: {admin.id})")

        # Create test store
        store = Store(
            store_code="TESTSTORE",
            subdomain="teststore",
            name="Test Store",
            description="Development test store",
            owner_user_id=admin.id,
            contact_email="contact@teststore.com",
            is_active=True,
            is_verified=True,
            created_at=datetime.now(timezone.utc),
            updated_at=datetime.now(timezone.utc)
        )
        db.add(store)
        db.flush()
        print(f"✓ Test store created: {store.store_code}")

        db.commit()

        print("\n✅ Database seeded successfully!")
        print("\n📝 Login Credentials:")
        print("   URL: http://localhost:8000/admin/login")
        print("   Username: admin")
        print("   Password: admin123")

    except Exception as e:
        db.rollback()
        print(f"❌ Error seeding database: {e}")
        raise
    finally:
        db.close()


if __name__ == "__main__":
    seed_database()
```

**Run it:**

```bash
python scripts/seed_database.py
```

---

### Production Data Migration

For production, create separate seed scripts:

```python
# scripts/seed_production.py
# - Create only essential admin user
# - No test data
# - Strong passwords from environment variables
```

---

## Quick Reference

### Essential Commands

```bash
# Check migration status
alembic current

# Generate new migration
alembic revision --autogenerate -m "Description"

# Apply migrations
alembic upgrade head

# Rollback one migration
alembic downgrade -1

# View migration history
alembic history

# Seed database
python scripts/seed_database.py

# Start server
python -m uvicorn app.main:app --reload
```

---

### File Locations

```
project/
├── alembic/
│   ├── versions/           # Migration files here
│   │   └── __init__.py
│   ├── env.py             # Alembic configuration (import models here!)
│   └── script.py.mako
├── app/
│   └── core/
│       └── database.py    # Database connection
├── models/
│   └── database/          # SQLAlchemy models
│       ├── user.py
│       ├── store.py
│       ├── store_theme.py
│       └── ...
├── scripts/
│   ├── seed_database.py   # Development seed
│   └── reset_database.ps1 # Reset script
├── alembic.ini            # Alembic config
├── orion.db           # SQLite database (gitignored)
└── .env                  # Environment variables (gitignored)
```

---

## Getting Help

### Internal Resources

- **Architecture Guide:** `docs/PROPER_ARCHITECTURE_GUIDE.md`
- **Exception Handling:** `docs/EXCEPTION_PATTERN_EXPLAINED.md`
- **Admin Feature Guide:** `docs/ADMIN_FEATURE_INTEGRATION_GUIDE.md`

### External Resources

- [Alembic Documentation](https://alembic.sqlalchemy.org/)
- [SQLAlchemy Documentation](https://docs.sqlalchemy.org/)
- [FastAPI Documentation](https://fastapi.tiangolo.com/)

---

## Contributing

When adding new models:

1. ✅ Create the model in `models/database/`
2. ✅ Import it in `alembic/env.py`
3. ✅ Generate migration: `alembic revision --autogenerate -m "Add XYZ model"`
4. ✅ Review the migration file
5. ✅ Test locally: `alembic upgrade head`
6. ✅ Commit both model and migration files

---

**Last Updated:** 2025-10-27
**Maintainer:** Development Team
**Questions?** Contact the team lead or check internal documentation.