orion/docs/deployment/production.md

# Traditional VPS Deployment

This guide covers deploying Orion to a traditional VPS (Ubuntu 22.04+) without containers.

**Best for:** Teams who want direct server access and familiar Linux administration.

---

## Prerequisites

- Ubuntu 22.04 LTS or newer
- 4GB+ RAM recommended
- Root or sudo access
- Domain name with DNS configured

---

## Quick Start

```bash
# 1. Install system packages
sudo apt update && sudo apt upgrade -y
sudo apt install -y nginx postgresql-15 redis-server python3.11 python3.11-venv git curl

# 2. Create application user
sudo useradd -m -s /bin/bash orion

# 3. Setup PostgreSQL
sudo -u postgres createuser orion_user
sudo -u postgres createdb orion_db -O orion_user
sudo -u postgres psql -c "ALTER USER orion_user WITH PASSWORD 'your-secure-password';"

# 4. Clone and setup application
sudo su - orion
git clone <repository-url> ~/app
cd ~/app
python3.11 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

# 5. Configure environment
cp .env.example .env
nano .env  # Edit with production values

# 6. Initialize database
alembic upgrade head
python scripts/seed/init_production.py

# 7. Exit orion user
exit
```

---

## Systemd Services

### Main Application

```bash
sudo nano /etc/systemd/system/orion.service
```

```ini
[Unit]
Description=Orion API Server
After=network.target postgresql.service redis.service

[Service]
User=orion
Group=orion
WorkingDirectory=/home/orion/app
Environment="PATH=/home/orion/app/.venv/bin"
EnvironmentFile=/home/orion/app/.env
ExecStart=/home/orion/app/.venv/bin/uvicorn main:app --host 127.0.0.1 --port 8000 --workers 4
Restart=always
RestartSec=3
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

### Celery Worker

```bash
sudo nano /etc/systemd/system/orion-celery.service
```

```ini
[Unit]
Description=Orion Celery Worker
After=network.target redis.service postgresql.service

[Service]
User=orion
Group=orion
WorkingDirectory=/home/orion/app
Environment="PATH=/home/orion/app/.venv/bin"
EnvironmentFile=/home/orion/app/.env
ExecStart=/home/orion/app/.venv/bin/celery -A app.core.celery_config worker --loglevel=info -Q default,long_running,scheduled --concurrency=2
Restart=always
RestartSec=3
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target
```

### Celery Beat (Scheduler)

```bash
sudo nano /etc/systemd/system/orion-celery-beat.service
```

```ini
[Unit]
Description=Orion Celery Beat Scheduler
After=network.target redis.service

[Service]
User=orion
Group=orion
WorkingDirectory=/home/orion/app
Environment="PATH=/home/orion/app/.venv/bin"
EnvironmentFile=/home/orion/app/.env
ExecStart=/home/orion/app/.venv/bin/celery -A app.core.celery_config beat --loglevel=info
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target
```

### Flower (Task Monitoring)

```bash
sudo nano /etc/systemd/system/orion-flower.service
```

```ini
[Unit]
Description=Orion Flower Task Monitor
After=network.target redis.service

[Service]
User=orion
Group=orion
WorkingDirectory=/home/orion/app
Environment="PATH=/home/orion/app/.venv/bin"
EnvironmentFile=/home/orion/app/.env
ExecStart=/home/orion/app/.venv/bin/celery -A app.core.celery_config flower --port=5555
Restart=always
RestartSec=3

[Install]
WantedBy=multi-user.target
```

### Enable Services

```bash
sudo systemctl daemon-reload
sudo systemctl enable orion orion-celery orion-celery-beat orion-flower
sudo systemctl start orion orion-celery orion-celery-beat orion-flower
```

---

## Nginx Configuration

```bash
sudo nano /etc/nginx/sites-available/orion
```

```nginx
# Redirect HTTP to HTTPS
server {
    listen 80;
    server_name yourdomain.com www.yourdomain.com;
    return 301 https://$server_name$request_uri;
}

# Main HTTPS server
server {
    listen 443 ssl http2;
    server_name yourdomain.com www.yourdomain.com;

    # SSL (managed by Certbot)
    ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    # Security headers
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-XSS-Protection "1; mode=block" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;

    # Logging
    access_log /var/log/nginx/orion.access.log;
    error_log /var/log/nginx/orion.error.log;

    # Static files (served directly)
    location /static {
        alias /home/orion/app/static;
        expires 30d;
        add_header Cache-Control "public, immutable";
        access_log off;
    }

    # Uploaded files
    location /uploads {
        alias /home/orion/app/uploads;
        expires 7d;
        add_header Cache-Control "public";
    }

    # Application
    location / {
        proxy_pass http://127.0.0.1:8000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # WebSocket support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Timeouts
        proxy_connect_timeout 60s;
        proxy_send_timeout 60s;
        proxy_read_timeout 60s;
    }

    # Block sensitive files
    location ~ /\. {
        deny all;
    }
    location ~ \.env$ {
        deny all;
    }
}
```

### Enable Site

```bash
sudo ln -s /etc/nginx/sites-available/orion /etc/nginx/sites-enabled/
sudo nginx -t
sudo systemctl reload nginx
```

### SSL with Certbot

```bash
sudo apt install certbot python3-certbot-nginx
sudo certbot --nginx -d yourdomain.com -d www.yourdomain.com
```

---

## Firewall

```bash
sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw enable
```

---

## Daily Operations

### View Logs

```bash
# Application logs
sudo journalctl -u orion -f

# Celery logs
sudo journalctl -u orion-celery -f

# Nginx logs
sudo tail -f /var/log/nginx/orion.access.log
sudo tail -f /var/log/nginx/orion.error.log

# PostgreSQL logs
sudo tail -f /var/log/postgresql/postgresql-15-main.log
```

### Restart Services

```bash
sudo systemctl restart orion
sudo systemctl restart orion-celery
sudo systemctl restart nginx
```

### Database Access

```bash
# Connect as orion user
sudo -u postgres psql orion_db

# Or with password
psql -h localhost -U orion_user -d orion_db
```

### Deploy Updates

```bash
sudo su - orion
cd ~/app
git pull origin main
source .venv/bin/activate
pip install -r requirements.txt
alembic upgrade head
exit
sudo systemctl restart orion orion-celery
```

---

## Backups

!!! tip "Docker deployment"
    For Docker-based deployments, use the automated backup scripts (`scripts/backup.sh` and `scripts/restore.sh`) with systemd timer. See [Hetzner Server Setup — Step 17](hetzner-server-setup.md#step-17-backups).

### Database Backup Script (VPS without Docker)

```bash
sudo nano /home/orion/backup.sh
```

```bash
#!/bin/bash
BACKUP_DIR=/home/orion/backups
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory
mkdir -p $BACKUP_DIR

# Backup database
pg_dump -U orion_user orion_db | gzip > $BACKUP_DIR/db_$DATE.sql.gz

# Backup uploads
tar -czf $BACKUP_DIR/uploads_$DATE.tar.gz -C /home/orion/app uploads/

# Keep last 7 days
find $BACKUP_DIR -name "*.gz" -mtime +7 -delete

echo "Backup completed: $DATE"
```

```bash
chmod +x /home/orion/backup.sh
```

### Cron Job

```bash
sudo -u orion crontab -e
```

```cron
# Daily backup at 2 AM
0 2 * * * /home/orion/backup.sh >> /home/orion/backup.log 2>&1
```

---

## Monitoring

!!! tip "Docker deployment"
    For Docker-based deployments, a full Prometheus + Grafana + node-exporter + cAdvisor stack is included in `docker-compose.yml`. See [Hetzner Server Setup — Step 18](hetzner-server-setup.md#step-18-monitoring-observability) and [Observability Framework](../architecture/observability.md).

### Basic Health Check

```bash
curl -s http://localhost:8000/health | jq
```

### Process Monitoring

```bash
# Check all services
systemctl status orion orion-celery postgresql redis nginx

# Resource usage
htop
df -h
free -h
```

### Set Up Sentry (Error Tracking)

Sentry provides real-time error tracking and performance monitoring.

1. **Create a Sentry account** at [sentry.io](https://sentry.io) (free tier available)
2. **Create a new project** (Python/FastAPI)
3. **Add to `.env`**:
   ```env
   SENTRY_DSN=https://your-key@sentry.io/project-id
   SENTRY_ENVIRONMENT=production
   SENTRY_TRACES_SAMPLE_RATE=0.1
   ```
4. **Restart services**:
   ```bash
   sudo systemctl restart orion orion-celery
   ```

Sentry will now capture:
- Unhandled exceptions
- API errors with request context
- Celery task failures
- Performance traces (10% sample rate)

---

## Cloudflare R2 Storage

For production, use Cloudflare R2 instead of local storage for scalability and CDN integration.

### Setup

1. **Create R2 bucket** in CloudFlare dashboard
2. **Create API token** with Object Read/Write permissions
3. **Add to `.env`**:
   ```env
   STORAGE_BACKEND=r2
   R2_ACCOUNT_ID=your_account_id
   R2_ACCESS_KEY_ID=your_access_key
   R2_SECRET_ACCESS_KEY=your_secret_key
   R2_BUCKET_NAME=orion-media
   R2_PUBLIC_URL=https://media.yourdomain.com
   ```

See [CloudFlare Setup Guide](cloudflare.md) for detailed instructions.

---

## CloudFlare CDN & Proxy

For production, proxy your domain through CloudFlare for:
- Global CDN caching
- DDoS protection
- Free SSL certificates
- WAF (Web Application Firewall)

### Enable CloudFlare Headers

Add to `.env`:
```env
CLOUDFLARE_ENABLED=true
```

This enables proper handling of `CF-Connecting-IP` for real client IPs.

See [CloudFlare Setup Guide](cloudflare.md) for complete configuration.

---

## Troubleshooting

See [Infrastructure Guide - Troubleshooting](infrastructure.md#troubleshooting-guide) for detailed diagnostics.

### Quick Checks

```bash
# Is the app running?
systemctl status orion

# Can we connect to the database?
pg_isready -h localhost -U orion_user

# Is Redis running?
redis-cli ping

# Check open ports
ss -tlnp | grep -E '(8000|5432|6379|80|443)'

# View recent errors
journalctl -u orion --since "1 hour ago" | grep -i error
```