orion/docs/development/database-seeder/database-init-guide.md

# Database Initialization Guide

**Orion Platform - Database Management Documentation**

This guide covers the database initialization, seeding, and management workflows for the Orion multi-tenant e-commerce platform.

---

## Table of Contents

1. [Overview](#overview)
2. [Architecture](#architecture)
3. [Configuration](#configuration)
4. [Workflows](#workflows)
5. [Command Reference](#command-reference)
6. [Extending the System](#extending-the-system)
7. [Best Practices](#best-practices)
8. [Troubleshooting](#troubleshooting)

---

## Overview

### Database Initialization Philosophy

The Orion platform uses a **two-tier initialization system**:

1. **Production Initialization** (`init_production.py`)
   - Creates essential platform infrastructure
   - Safe to run in all environments (dev/staging/prod)
   - Idempotent (safe to run multiple times)
   - No test/demo data

2. **Demo Data Seeding** (`seed_demo.py`)
   - Creates realistic test data for development
   - **Development only** - blocked in production
   - Configurable via settings
   - Multiple seeding modes

### Key Principles

- **Single Source of Truth**: All configuration flows through `app.core.config.settings`
- **Environment Aware**: Automatic detection of dev/staging/production environments
- **Type Safe**: Pydantic validates all configuration
- **Makefile First**: All operations via clean `make` commands
- **Production Safe**: Built-in validation and blocking of unsafe operations

---

## Architecture

### System Components

```
┌─────────────────────────────────────────────────────────────┐
│                          .env File                           │
│  (Environment-specific configuration)                        │
└─────────────────┬───────────────────────────────────────────┘
                  │
                  ▼
┌─────────────────────────────────────────────────────────────┐
│            app/core/config.py (Settings)                     │
│  - Environment detection (is_production, is_development)     │
│  - Admin initialization settings                             │
│  - Demo data configuration                                   │
│  - Platform limits and constraints                           │
└────────────┬─────────────────────┬──────────────────────────┘
             │                     │
             ▼                     ▼
┌─────────────────────┐  ┌─────────────────────┐
│ init_production.py  │  │   seed_demo.py      │
│                     │  │                     │
│ Creates:            │  │ Creates:            │
│ • Admin user        │  │ • Demo stores      │
│ • Admin settings    │  │ • Test customers    │
│ • Role templates    │  │ • Sample products   │
│ • RBAC schema       │  │ • Demo orders       │
│                     │  │ • Themes/domains    │
│                     │  │                     │
│ Safe for prod: ✅   │  │ Safe for prod: ❌   │
└─────────────────────┘  └─────────────────────┘
             │                     │
             └──────────┬──────────┘
                        ▼
              ┌──────────────────┐
              │   Database       │
              └──────────────────┘
```

### Data Flow

1. **Configuration** → `.env` file defines environment and settings
2. **Settings Loading** → `app/core/config.py` loads and validates configuration
3. **Script Execution** → Scripts access configuration via `settings` singleton
4. **Database Operations** → Scripts create/update database records
5. **Validation** → Environment checks and warnings ensure safe operations

---

## Configuration

### Environment Variables (.env)

All database initialization is controlled through environment variables in `.env`:

```bash
# =============================================================================
# ENVIRONMENT CONFIGURATION
# =============================================================================
ENVIRONMENT=development              # development | staging | production

# =============================================================================
# ADMIN INITIALIZATION
# =============================================================================
ADMIN_EMAIL=admin@orion.lu
ADMIN_USERNAME=admin
ADMIN_PASSWORD=admin123              # ⚠️ CHANGE IN PRODUCTION!
ADMIN_FIRST_NAME=Platform
ADMIN_LAST_NAME=Administrator

# =============================================================================
# DEMO DATA CONFIGURATION (Development only)
# =============================================================================
SEED_DEMO_STORES=3                  # Number of demo stores
SEED_CUSTOMERS_PER_STORE=15         # Customers per store
SEED_PRODUCTS_PER_STORE=20          # Products per store
SEED_ORDERS_PER_STORE=10            # Orders per store

# =============================================================================
# PLATFORM LIMITS
# =============================================================================
MAX_STORES_PER_USER=5
MAX_TEAM_MEMBERS_PER_STORE=50
INVITATION_EXPIRY_DAYS=7
```

### Settings Framework

Access configuration in code through the settings singleton:

```python
from app.core.config import settings

# Environment detection
if settings.is_production:
    # Production-specific code
    pass
elif settings.is_development:
    # Development-specific code
    pass

# Access configuration
admin_email = settings.admin_email
demo_store_count = settings.seed_demo_stores
max_stores = settings.max_stores_per_user

# Environment info
from app.core.config import print_environment_info
print_environment_info()
```

### Production Validation

The system includes automatic validation:

```python
from app.core.config import validate_production_settings

# Check for insecure production settings
warnings = validate_production_settings()
if warnings:
    for warning in warnings:
        print(warning)
```

---

## Workflows

### First-Time Production Setup

**Prerequisites**: Database migrations applied

```bash
# 1. Configure environment
cat > .env << EOF
ENVIRONMENT=production
DATABASE_URL=postgresql://user:pass@localhost/orion
ADMIN_EMAIL=admin@yourmerchant.com
ADMIN_USERNAME=admin
ADMIN_PASSWORD=SecurePassword123!
JWT_SECRET_KEY=your-secret-key-here
EOF

# 2. Run migrations
make migrate-up

# 3. Initialize production essentials
make init-prod

# 4. Verify
python -c "from app.core.config import validate_production_settings; \
           print('✅ OK' if not validate_production_settings() else 'Fix warnings')"
```

**What happens**:
- Platform admin user created
- Essential admin settings configured
- Role templates initialized
- RBAC schema verified

**Security**: Always change default credentials in production!

---

### First-Time Development Setup

**Prerequisites**: Virtual environment activated

```bash
# Complete setup (migrations + init + demo data)
make db-setup
```

This single command runs:
1. `make migrate-up` - Apply database migrations
2. `make init-prod` - Create admin user and settings
3. `make seed-demo` - Create demo stores and test data

**Alternative: Step-by-step**
```bash
# 1. Apply migrations
make migrate-up

# 2. Create admin user
make init-prod

# 3. Create demo data
make seed-demo        # 3 stores
# OR
make seed-demo-minimal  # 1 store only
```

---

### Daily Development Workflow

#### Starting Development
```bash
# Start the development server
make dev

# API available at: http://localhost:8000
# Admin panel: http://localhost:8000/admin/login
# API docs: http://localhost:8000/docs
```

#### Refreshing Demo Data
```bash
# Add more demo data (idempotent)
make seed-demo

# Start fresh with new demo data
make seed-demo-reset
```

#### Database Changes
```bash
# 1. Modify your SQLAlchemy models
# 2. Create migration
make migrate-create message="add_new_feature"

# 3. Review the generated migration in alembic/versions/

# 4. Apply migration
make migrate-up

# 5. Update seed scripts if needed
```

---

### Production Deployment Workflow

```bash
# 1. Ensure production .env is configured
ENVIRONMENT=production
ADMIN_PASSWORD=SecurePassword123!  # Not default!
JWT_SECRET_KEY=your-secret-key     # Not default!

# 2. Apply migrations
make migrate-up

# 3. Initialize (if first deployment)
make init-prod

# 4. Create stores manually via admin panel
# DO NOT run seed-demo in production!
```

**Note**: `seed_demo.py` will refuse to run if `ENVIRONMENT=production`

---

## Command Reference

### Database Migrations

```bash
# Create a new migration
make migrate-create message="description"

# Create empty migration template
make migrate-create-manual message="description"

# Apply pending migrations
make migrate-up

# Rollback last migration
make migrate-down

# Check migration status
make migrate-status
```

### Initialization & Seeding

```bash
# Production initialization (SAFE for all environments)
make init-prod

# Demo data seeding (DEVELOPMENT ONLY)
make seed-demo           # Create 3 stores with full demo data
make seed-demo-minimal   # Create 1 store with minimal data
make seed-demo-reset     # DELETE ALL DATA and reseed (DANGEROUS!)

# Complete workflows
make db-setup            # Full setup: migrate + init + seed
make db-reset            # Nuclear option: rollback + migrate + reset
```

### Database Utilities

```bash
# Backup database
make backup-db

# Verify setup
make verify-setup

# Check migration history
make migrate-status
```

### Help Commands

```bash
# General help
make help

# Database-specific help
make help-db
```

---

## Extending the System

### Adding New Admin Settings

Edit `scripts/seed/init_production.py` to add new platform settings:

```python
def create_admin_settings(db: Session) -> int:
    """Create essential admin settings."""

    default_settings = [
        # ... existing settings ...

        # Add your new setting
        {
            "key": "your_new_setting",
            "value": str(settings.your_new_setting),  # From config
            "value_type": "string",  # string | integer | boolean
            "description": "Description of your setting",
            "is_public": False,  # True if visible to stores
        },
    ]
    # ... rest of function
```

### Adding New Demo Data

Edit `scripts/seed/seed_demo.py` to extend demo data creation:

```python
def seed_demo_data(db: Session, auth_manager: AuthManager):
    """Seed demo data for development."""

    # ... existing steps ...

    # Add your new demo data
    print_step(7, "Creating your demo data...")
    create_your_demo_data(db, stores)

    # ... commit
```

Create a new function for your data:

```python
def create_your_demo_data(db: Session, stores: List[Store]) -> List[YourModel]:
    """Create demo data for your feature."""

    items = []
    for store in stores:
        # Create demo items for this store
        item = YourModel(
            store_id=store.id,
            # ... your fields
        )
        db.add(item)
        items.append(item)

    db.flush()
    print_success(f"Created {len(items)} demo items")
    return items
```

### Adding Configuration Options

#### 1. Add to Settings Class

Edit `app/core/config.py`:

```python
class Settings(BaseSettings):
    # ... existing settings ...

    # Your new settings
    your_new_setting: str = "default_value"
    your_numeric_setting: int = 42
    your_boolean_setting: bool = True

    # Optional with validation
    your_list_setting: List[str] = ["item1", "item2"]
```

#### 2. Add to .env File

```bash
# Your new configuration
YOUR_NEW_SETTING=custom_value
YOUR_NUMERIC_SETTING=100
YOUR_BOOLEAN_SETTING=False
```

#### 3. Use in Code

```python
from app.core.config import settings

# Access your settings
value = settings.your_new_setting
if settings.your_boolean_setting:
    # Do something
```

### Adding New Demo Store Configurations

Edit the `DEMO_STORES` list in `scripts/seed/seed_demo.py`:

```python
DEMO_STORES = [
    # ... existing stores ...

    # Your new demo store
    {
        "store_code": "YOURSHOP",
        "name": "Your Shop Name",
        "subdomain": "yourshop",
        "description": "Your shop description",
        "theme_preset": "modern",  # or "classic", "vibrant"
        "custom_domain": "yourshop.example.com",  # or None
    },
]
```

Also add a corresponding user in `DEMO_STORE_USERS`.

### Creating Custom Seeding Modes

Add new mode detection in `seed_demo.py`:

```python
# Mode detection (from Makefile)
SEED_MODE = os.getenv('SEED_MODE', 'normal')  # normal, minimal, reset, custom

# In seed_demo_data function
if SEED_MODE == 'custom':
    # Your custom seeding logic
    create_custom_demo_data(db, auth_manager)
```

Add corresponding Makefile command:

```makefile
seed-demo-custom:
	@echo 🎪 Seeding custom demo data...
	@set SEED_MODE=custom&& $(PYTHON) scripts/seed/seed_demo.py
	@echo ✅ Custom demo seeding completed
```

---

## Best Practices

### Configuration Management

#### ✅ DO:
- Use `settings` for all configuration access
- Define defaults in `Settings` class
- Override via `.env` for environment-specific values
- Use type hints in Settings class
- Document each setting's purpose

#### ❌ DON'T:
- Use `os.getenv()` directly in scripts
- Hard-code configuration values
- Store secrets in code
- Ignore type validation

### Environment Detection

#### ✅ DO:
```python
from app.core.config import settings

if settings.is_production:
    # Production logic
elif settings.is_development:
    # Development logic
```

#### ❌ DON'T:
```python
# Don't check environment manually
import os
if os.getenv("ENVIRONMENT") == "production":
    # This bypasses settings validation
```

### Demo Data Creation

#### ✅ DO:
- Make demo data realistic and useful for testing
- Use idempotent checks (check existence before creating)
- Make demo data configurable via settings
- Include variety in demo data (different states, types)
- Document demo credentials clearly

#### ❌ DON'T:
- Hard-code demo data counts
- Create unrealistic test data
- Use production-like credentials in demo data
- Skip existence checks (causes duplicates)

### Script Development

#### ✅ DO:
- Use helper functions (`print_success`, `print_error`)
- Include comprehensive error handling
- Make operations idempotent
- Validate prerequisites (migrations, admin user)
- Provide clear, actionable error messages

#### ❌ DON'T:
- Assume database state
- Skip transaction management (db.commit())
- Ignore exceptions
- Mix production and demo logic in same function

### Makefile Commands

#### ✅ DO:
- Use descriptive command names
- Group related commands
- Provide `help` targets
- Use environment variables for modes
- Echo what's happening

#### ❌ DON'T:
- Pass complex arguments to scripts
- Create ambiguous command names
- Skip error checking
- Hide what commands do

---

## Security Considerations

### Production Initialization

**Critical**: Always change default credentials before production deployment!

```bash
# INSECURE - Default values
ADMIN_PASSWORD=admin123
JWT_SECRET_KEY=change-this-in-production

# SECURE - Custom values
ADMIN_PASSWORD=YourSecurePassword123!
JWT_SECRET_KEY=randomly-generated-secret-key-here
```

### Production Validation

The system automatically validates production settings:

```bash
# Warnings you might see:
⚠️  Using default admin password in production!
⚠️  Using default JWT secret key in production!
⚠️  Debug mode enabled in production!
⚠️  ALLOWED_HOSTS is set to wildcard (*) in production!
```

**Action Required**: Update `.env` with secure values

### Demo Data Protection

Demo seeding is automatically blocked in production:

```python
# In seed_demo.py
def check_environment():
    if settings.is_production:
        print_error("Cannot run demo seeding in production!")
        sys.exit(1)
```

This prevents accidental exposure of fake data in production.

### Credential Management

#### Production
- **Admin Password**: Strong, unique password
- **JWT Secret**: Randomly generated, never committed
- **Database Password**: Secure, rotated regularly
- **API Keys**: Use environment variables

#### Development
- **Admin Password**: `admin123` (documented, known to team)
- **Demo Users**: `password123` (clearly insecure)
- **Documentation**: All credentials documented in this guide

---

## Troubleshooting

### Common Issues

#### Issue: "Table doesn't exist" Error

**Cause**: Migrations not applied

**Solution**:
```bash
# Apply migrations
make migrate-up

# Verify
make migrate-status
```

#### Issue: "Admin user already exists"

**Cause**: This is normal! Scripts are idempotent.

**Solution**: No action needed. This message confirms the admin user exists.

```
⚠  Admin user already exists: admin@orion.lu
✓ All changes committed
```

#### Issue: "Cannot run demo seeding in production"

**Cause**: Correct behavior - safety feature working

**Solution**:
- If you're in development: Set `ENVIRONMENT=development` in `.env`
- If you're in production: Don't seed demo data! Create stores via admin panel

#### Issue: "Settings not found"

**Cause**: Configuration not properly imported

**Solution**:
```bash
# Verify settings work
python -c "from app.core.config import settings; print(settings.environment)"

# Check .env file exists
ls -la .env

# Verify Pydantic is installed
pip list | grep pydantic
```

#### Issue: Duplicate Demo Data

**Cause**: Running seed scripts multiple times without checking existence

**Solution**:
- Use `make seed-demo-reset` for fresh data
- Check scripts for proper idempotent checks
- Verify database state before seeding

#### Issue: Migration Conflicts

**Cause**: Multiple developers creating migrations simultaneously

**Solution**:
```bash
# Check migration status
make migrate-status

# Resolve conflicts in alembic/versions/
# Merge migrations or create a merge migration

# Apply resolved migrations
make migrate-up
```

### Debugging Commands

```bash
# Check environment configuration
python -c "from app.core.config import print_environment_info; print_environment_info()"

# Validate production settings
python -c "from app.core.config import validate_production_settings; \
           warnings = validate_production_settings(); \
           print('✅ OK' if not warnings else warnings)"

# Check database state
python -c "
from app.core.database import SessionLocal
from models.database.store import Store
from models.database.user import User
db = SessionLocal()
print(f'Users: {db.query(User).count()}')
print(f'Stores: {db.query(Store).count()}')
db.close()
"

# Test settings import
python -c "from app.core.config import settings; print(f'Env: {settings.environment}'); print(f'Admin: {settings.admin_email}')"
```

### Getting Help

1. **Check this guide** - Most common scenarios are documented
2. **Use help commands** - `make help-db` for command reference
3. **Check logs** - Review output from failed commands
4. **Verify configuration** - Ensure `.env` is properly configured
5. **Check migrations** - Ensure all migrations are applied

---

## Reference

### Default Credentials

#### Admin (After `make init-prod`)
```
URL:      http://localhost:8000/admin/login
Username: admin
Password: admin123  (⚠️ CHANGE IN PRODUCTION!)
Email:    admin@orion.lu
```

#### Demo Stores (After `make seed-demo`)
```
Store 1: store1@example.com / password123
Store 2: store2@example.com / password123
Store 3: store3@example.com / password123
```

**⚠️ All demo passwords are intentionally insecure for development use!**

### File Locations

```
orion/
├── .env                              # Environment configuration
├── Makefile                          # Command definitions
├── app/
│   └── core/
│       └── config.py                 # Settings framework
├── scripts/
│   ├── init_production.py            # Production initialization
│   └── seed_demo.py                  # Demo data seeding
└── alembic/
    └── versions/                     # Database migrations
```

### Key Commands Summary

```bash
# Setup
make setup                    # Complete initial setup
make db-setup                 # Database setup only

# Migrations
make migrate-up               # Apply migrations
make migrate-create message="msg"  # Create migration

# Initialization
make init-prod                # Production init (safe everywhere)

# Demo Data (Development only)
make seed-demo                # Full demo data
make seed-demo-minimal        # Minimal demo
make seed-demo-reset          # Reset and reseed

# Development
make dev                      # Start server
make test                     # Run tests

# Help
make help                     # All commands
make help-db                  # Database commands
```

---

## Appendices

### A. Environment Variable Reference

Complete list of database-related environment variables:

| Variable | Type | Default | Description |
|----------|------|---------|-------------|
| `ENVIRONMENT` | string | `development` | Environment mode (development/staging/production) |
| `DATABASE_URL` | string | `sqlite:///./orion.db` | Database connection string |
| `ADMIN_EMAIL` | string | `admin@orion.lu` | Platform admin email |
| `ADMIN_USERNAME` | string | `admin` | Platform admin username |
| `ADMIN_PASSWORD` | string | `admin123` | Platform admin password |
| `ADMIN_FIRST_NAME` | string | `Platform` | Admin first name |
| `ADMIN_LAST_NAME` | string | `Administrator` | Admin last name |
| `SEED_DEMO_STORES` | integer | `3` | Number of demo stores to create |
| `SEED_CUSTOMERS_PER_STORE` | integer | `15` | Demo customers per store |
| `SEED_PRODUCTS_PER_STORE` | integer | `20` | Demo products per store |
| `SEED_ORDERS_PER_STORE` | integer | `10` | Demo orders per store |
| `MAX_STORES_PER_USER` | integer | `5` | Maximum stores per user |
| `MAX_TEAM_MEMBERS_PER_STORE` | integer | `50` | Maximum team members per store |
| `INVITATION_EXPIRY_DAYS` | integer | `7` | Team invitation expiry days |

### B. Database Tables Created

#### Production Initialization Tables
- `users` - Platform users (admin, stores, team members)
- `admin_settings` - Platform configuration settings
- `roles` - RBAC role definitions

#### Demo Data Tables
- `stores` - Demo store accounts
- `store_users` - Store-user relationships
- `store_themes` - Store theme customizations
- `store_domains` - Custom domain configurations
- `customers` - Demo customer accounts
- `customer_addresses` - Customer address information
- `products` - Demo product catalog
- `marketplace_products` - Marketplace integrations
- `orders` - Demo order records
- `order_items` - Order line items

### C. Settings Properties Reference

Access these via `from app.core.config import settings`:

```python
# Environment Detection
settings.environment          # "development" | "staging" | "production"
settings.is_production        # bool
settings.is_development       # bool
settings.is_staging          # bool

# Admin Configuration
settings.admin_email         # str
settings.admin_username      # str
settings.admin_password      # str
settings.admin_first_name    # str
settings.admin_last_name     # str

# Demo Data Configuration
settings.seed_demo_stores            # int
settings.seed_customers_per_store    # int
settings.seed_products_per_store     # int
settings.seed_orders_per_store       # int

# Platform Limits
settings.max_stores_per_user         # int
settings.max_team_members_per_store  # int
settings.invitation_expiry_days       # int

# Database
settings.database_url        # str

# Platform
settings.platform_domain     # str
settings.project_name        # str
```

---

## Contributing

### Adding Features to This System

When extending the database initialization system:

1. **Update Settings** (`app/core/config.py`)
   - Add configuration variables
   - Include type hints
   - Provide sensible defaults
   - Document the setting

2. **Update Environment** (`.env`)
   - Add environment variables
   - Document their purpose
   - Provide examples

3. **Update Scripts**
   - Use `settings` for configuration
   - Include idempotent checks
   - Add helpful output messages
   - Handle errors gracefully

4. **Update Makefile**
   - Add commands if needed
   - Update help text
   - Follow naming conventions

5. **Update Documentation**
   - Document new features
   - Update command reference
   - Add examples
   - Include troubleshooting

### Code Review Checklist

When reviewing database initialization changes:

- [ ] Uses `settings` instead of direct environment access
- [ ] Includes idempotent checks
- [ ] Has appropriate environment detection
- [ ] Includes error handling
- [ ] Updates documentation
- [ ] Adds/updates tests
- [ ] Follows existing patterns
- [ ] Includes helpful output messages

---

## Changelog

### Current Version (v2.0)

**Database Initialization System**
- Settings-based configuration
- Environment-aware operations
- Makefile-driven commands
- Production safety features
- Comprehensive documentation

**Key Features**:
- Idempotent initialization scripts
- Automatic production validation
- Configurable demo data creation
- Clean command interface
- Type-safe configuration

---

## Support

For questions or issues:

1. **Check this documentation** - Most scenarios are covered
2. **Review command help** - `make help-db`
3. **Check settings** - Verify `.env` configuration
4. **Review logs** - Check script output for errors
5. **Contact the team** - Reach out if stuck

**Remember**: This system is designed to be safe, clear, and maintainable. When in doubt, refer to this guide!

---

**Last Updated**: 2025
**Maintained By**: Orion Platform Team
**Version**: 2.0