feat: add capacity planning docs, image upload system, and platform health monitoring
Documentation: - Add comprehensive capacity planning guide (docs/architecture/capacity-planning.md) - Add operations docs: platform-health, capacity-monitoring, image-storage - Link pricing strategy to capacity planning documentation - Update mkdocs.yml with new Operations section Image Upload System: - Add ImageService with WebP conversion and sharded directory structure - Generate multiple size variants (original, 800px, 200px) - Add storage stats endpoint for monitoring - Add Pillow dependency for image processing Platform Health Monitoring: - Add /admin/platform-health page with real-time metrics - Show CPU, memory, disk usage with progress bars - Display capacity thresholds with status indicators - Generate scaling recommendations automatically - Determine infrastructure tier based on usage - Add psutil dependency for system metrics Admin UI: - Add Capacity Monitor to Platform Health section in sidebar - Create platform-health.html template with stats cards - Create platform-health.js for Alpine.js state management 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
246
docs/operations/image-storage.md
Normal file
246
docs/operations/image-storage.md
Normal file
@@ -0,0 +1,246 @@
|
||||
# Image Storage System
|
||||
|
||||
Documentation for the platform's image storage and management system.
|
||||
|
||||
## Overview
|
||||
|
||||
The Wizamart platform uses a self-hosted image storage system with:
|
||||
|
||||
- **Sharded directory structure** for filesystem performance
|
||||
- **Automatic WebP conversion** for optimization
|
||||
- **Multiple size variants** for different use cases
|
||||
- **CDN-ready architecture** for scaling
|
||||
|
||||
## Storage Architecture
|
||||
|
||||
### Directory Structure
|
||||
|
||||
Images are stored in a sharded directory structure to prevent filesystem performance degradation:
|
||||
|
||||
```
|
||||
/static/uploads/
|
||||
└── products/
|
||||
├── 00/ # First 2 chars of hash
|
||||
│ ├── 1a/ # Next 2 chars
|
||||
│ │ ├── 001a2b3c_original.webp
|
||||
│ │ ├── 001a2b3c_800.webp
|
||||
│ │ └── 001a2b3c_200.webp
|
||||
│ └── 2b/
|
||||
│ └── ...
|
||||
├── 01/
|
||||
│ └── ...
|
||||
└── ff/
|
||||
└── ...
|
||||
```
|
||||
|
||||
### Hash Generation
|
||||
|
||||
The file hash is generated from:
|
||||
```python
|
||||
hash = md5(f"{vendor_id}:{product_id}:{timestamp}:{original_filename}")[:8]
|
||||
```
|
||||
|
||||
This ensures:
|
||||
- Unique file paths
|
||||
- Even distribution across directories
|
||||
- Predictable file locations
|
||||
|
||||
## Image Variants
|
||||
|
||||
Each uploaded image generates multiple variants:
|
||||
|
||||
| Variant | Max Dimensions | Format | Use Case |
|
||||
|---------|---------------|--------|----------|
|
||||
| `original` | As uploaded (max 2000px) | WebP | Detail view, zoom |
|
||||
| `800` | 800×800 | WebP | Product cards |
|
||||
| `200` | 200×200 | WebP | Thumbnails, grids |
|
||||
|
||||
### Size Estimates
|
||||
|
||||
| Original Size | After Processing | Storage per Image |
|
||||
|---------------|------------------|-------------------|
|
||||
| 2MB JPEG | ~200KB (original) + 80KB (800) + 15KB (200) | ~295KB |
|
||||
| 500KB JPEG | ~150KB (original) + 60KB (800) + 12KB (200) | ~222KB |
|
||||
| 100KB JPEG | ~80KB (original) + 40KB (800) + 10KB (200) | ~130KB |
|
||||
|
||||
**Average: ~200KB per image (all variants)**
|
||||
|
||||
## Upload Process
|
||||
|
||||
### API Endpoint
|
||||
|
||||
```http
|
||||
POST /api/v1/admin/images/upload
|
||||
Content-Type: multipart/form-data
|
||||
|
||||
file: <binary>
|
||||
vendor_id: 123
|
||||
product_id: 456 (optional, for product images)
|
||||
type: product|category|banner
|
||||
```
|
||||
|
||||
### Response
|
||||
|
||||
```json
|
||||
{
|
||||
"success": true,
|
||||
"image": {
|
||||
"id": "001a2b3c",
|
||||
"urls": {
|
||||
"original": "/uploads/products/00/1a/001a2b3c_original.webp",
|
||||
"medium": "/uploads/products/00/1a/001a2b3c_800.webp",
|
||||
"thumb": "/uploads/products/00/1a/001a2b3c_200.webp"
|
||||
},
|
||||
"size_bytes": 295000,
|
||||
"dimensions": {
|
||||
"width": 1200,
|
||||
"height": 1200
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Environment Variables
|
||||
|
||||
```bash
|
||||
# Image storage configuration
|
||||
IMAGE_UPLOAD_DIR=/var/www/uploads
|
||||
IMAGE_MAX_SIZE_MB=10
|
||||
IMAGE_ALLOWED_TYPES=jpg,jpeg,png,gif,webp
|
||||
IMAGE_QUALITY=85
|
||||
IMAGE_MAX_DIMENSION=2000
|
||||
```
|
||||
|
||||
### Python Configuration
|
||||
|
||||
```python
|
||||
# app/core/config.py
|
||||
class ImageSettings:
|
||||
UPLOAD_DIR: str = "/static/uploads"
|
||||
MAX_SIZE_MB: int = 10
|
||||
ALLOWED_TYPES: list = ["jpg", "jpeg", "png", "gif", "webp"]
|
||||
QUALITY: int = 85
|
||||
MAX_DIMENSION: int = 2000
|
||||
|
||||
# Generated sizes
|
||||
SIZES: dict = {
|
||||
"original": None, # No resize, just optimize
|
||||
"medium": 800,
|
||||
"thumb": 200,
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Guidelines
|
||||
|
||||
### Filesystem Limits
|
||||
|
||||
| Files per Directory | Status | Action |
|
||||
|---------------------|--------|--------|
|
||||
| < 10,000 | OK | None needed |
|
||||
| 10,000 - 50,000 | Monitor | Plan migration |
|
||||
| 50,000 - 100,000 | Warning | Increase sharding depth |
|
||||
| > 100,000 | Critical | Migrate to object storage |
|
||||
|
||||
### Capacity Planning
|
||||
|
||||
| Products | Images (5/product) | Total Files (3 sizes) | Storage |
|
||||
|----------|--------------------|-----------------------|---------|
|
||||
| 10,000 | 50,000 | 150,000 | 30 GB |
|
||||
| 50,000 | 250,000 | 750,000 | 150 GB |
|
||||
| 100,000 | 500,000 | 1,500,000 | 300 GB |
|
||||
| 500,000 | 2,500,000 | 7,500,000 | 1.5 TB |
|
||||
|
||||
## CDN Integration
|
||||
|
||||
For production deployments, configure a CDN for image delivery:
|
||||
|
||||
### Cloudflare (Recommended)
|
||||
|
||||
1. Set up Cloudflare for your domain
|
||||
2. Configure page rules for `/uploads/*`:
|
||||
- Cache Level: Cache Everything
|
||||
- Edge Cache TTL: 1 month
|
||||
- Browser Cache TTL: 1 week
|
||||
|
||||
### nginx Configuration
|
||||
|
||||
```nginx
|
||||
location /uploads/ {
|
||||
alias /var/www/uploads/;
|
||||
expires 30d;
|
||||
add_header Cache-Control "public, immutable";
|
||||
add_header X-Content-Type-Options nosniff;
|
||||
|
||||
# WebP fallback for older browsers
|
||||
location ~ \.(jpg|jpeg|png)$ {
|
||||
try_files $uri$webp_suffix $uri =404;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Maintenance
|
||||
|
||||
### Cleanup Orphaned Images
|
||||
|
||||
Remove images not referenced by any product:
|
||||
|
||||
```bash
|
||||
# Run via admin CLI
|
||||
python -m scripts.cleanup_orphaned_images --dry-run
|
||||
python -m scripts.cleanup_orphaned_images --execute
|
||||
```
|
||||
|
||||
### Regenerate Variants
|
||||
|
||||
If image quality settings change:
|
||||
|
||||
```bash
|
||||
# Regenerate all variants for a vendor
|
||||
python -m scripts.regenerate_images --vendor-id 123
|
||||
|
||||
# Regenerate all variants (use with caution)
|
||||
python -m scripts.regenerate_images --all
|
||||
```
|
||||
|
||||
## Monitoring
|
||||
|
||||
### Metrics to Track
|
||||
|
||||
- Total file count
|
||||
- Storage used (GB)
|
||||
- Files per directory (max)
|
||||
- Upload success rate
|
||||
- Average processing time
|
||||
|
||||
### Health Checks
|
||||
|
||||
The platform health page includes image storage metrics:
|
||||
|
||||
- Current file count
|
||||
- Storage usage
|
||||
- Directory distribution
|
||||
- Processing queue status
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
**Upload fails with "File too large"**
|
||||
- Check `IMAGE_MAX_SIZE_MB` setting
|
||||
- Verify nginx `client_max_body_size`
|
||||
|
||||
**Images not displaying**
|
||||
- Check file permissions (should be readable by web server)
|
||||
- Verify URL paths match actual file locations
|
||||
|
||||
**Slow uploads**
|
||||
- Check disk I/O performance
|
||||
- Consider async processing queue
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Capacity Planning](../architecture/capacity-planning.md)
|
||||
- [Platform Health](platform-health.md)
|
||||
- [Capacity Monitoring](capacity-monitoring.md)
|
||||
Reference in New Issue
Block a user