Celery workers were failing every task that touched the DB with
InvalidRequestError: "expression 'ContentPage' failed to locate a name".
Root cause: the worker process only loads task modules during startup,
not routes or services. Models reached only via SQLAlchemy
string-based relationships (e.g. Platform.relationship("ContentPage"))
were never imported, so the mapper couldn't resolve the name when the
first task tried to open a DB session. The FastAPI process avoids this
because api_router transitively imports the world; the worker doesn't.
Add _preload_all_module_models() in celery_config.py that walks the
module registry and importlib.import_module's each "app.modules.<code>.models"
package. Called at module import time, before any task runs.
Surfaced while finally getting loyalty.send_notification_email registered
on the worker — the task ran, hit the DB to load email settings, and
exploded on the unresolved Platform -> ContentPage relationship.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
When the FastAPI process called send_notification_email.delay() it
crashed with kombu OperationalError "[Errno 111] Connection refused"
because the published task was going to amqp://localhost// instead
of our configured redis broker.
Root cause: tasks decorated with @shared_task bind to whatever
Celery considers the "current default app" at decoration time. Our
celery_app (with redis broker) was never imported during FastAPI
startup, so the loyalty / billing / etc. task modules registered
their @shared_task functions against Celery's built-in default —
which has broker_url=amqp://localhost// and no RabbitMQ deployed.
Two-part fix:
1. app/core/celery_config.py — call celery_app.set_default() so
any @shared_task evaluated after this module loads binds to our
app rather than the built-in default.
2. main.py — import app.core.celery_config near the top, BEFORE
api_router (which transitively imports the task modules). The
side-effect of the import (creating celery_app + set_default())
must run before any @shared_task decorator fires. An # isort:
split below the import stops the import-sorter from alphabetising
it back behind app.api.main and re-introducing the bug.
User-visible effect: loyalty welcome emails (and every other
@shared_task-based notification) now actually queue + send.
Surfaced this while debugging why enrollment on prod produced a
loyalty_card row but no email_logs row.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Celery worker was OOM-killed (41 restarts) at 512MB with 4 concurrent
workers. Reduce concurrency to 2, increase worker limit to 768MB, and
reclaim memory from over-provisioned services (db 512→256, beat 256→128,
flower 256→192). Total allocation stays within 4GB server budget.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace all ~1,086 occurrences of Wizamart/wizamart/WIZAMART/WizaMart
with Orion/orion/ORION across 184 files. This includes database
identifiers, email addresses, domain references, R2 bucket names,
DNS prefixes, encryption salt, Celery app name, config defaults,
Docker configs, CI configs, documentation, seed data, and templates.
Renames homepage-wizamart.html template to homepage-orion.html.
Fixes duplicate file_pattern key in api.yaml architecture rule.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Phase 4 of module migration plan:
- Add ScheduledTask dataclass for declaring Celery Beat tasks
- Add tasks_path and scheduled_tasks fields to ModuleDefinition
- Create ModuleTask base class with database session management
- Create task discovery utilities (discover_module_tasks, build_beat_schedule)
- Update celery_config.py to discover and register module tasks
- Maintain backward compatibility with legacy task modules
Modules can now define tasks in their tasks/ directory and scheduled
tasks in their definition. The infrastructure supports gradual migration
of existing tasks from app/tasks/ to their respective modules.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Production quick wins for improved observability and scalability:
Sentry Error Tracking:
- Add sentry-sdk[fastapi] dependency
- Initialize Sentry in main.py with FastAPI/SQLAlchemy integrations
- Add Celery integration for background task error tracking
- Feature-flagged via SENTRY_DSN (disabled when empty)
Cloudflare R2 Storage:
- Add boto3 dependency for S3-compatible API
- Create storage_service.py with StorageBackend abstraction
- LocalStorageBackend for development (default)
- R2StorageBackend for production cloud storage
- Feature-flagged via STORAGE_BACKEND setting
CloudFlare CDN/Proxy:
- Create middleware/cloudflare.py for CF header handling
- Extract real client IP from CF-Connecting-IP
- Support CF-IPCountry for geo features
- Feature-flagged via CLOUDFLARE_ENABLED setting
Documentation:
- Add docs/deployment/cloudflare.md setup guide
- Update infrastructure.md with dev vs prod requirements
- Add enterprise upgrade checklist for scaling beyond 1000 users
- Update installation.md with new environment variables
All features are optional and disabled by default for development.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Rename second 'install' target to 'platform-install' to avoid conflict
- Add broker_connection_retry_on_startup=True for Celery 6.0 compatibility
- Update install.py references to use 'make platform-install'
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Migrate background tasks from FastAPI BackgroundTasks to Celery with Redis
for persistent task queuing, retries, and scheduled jobs.
Key changes:
- Add Celery configuration with Redis broker/backend
- Create task dispatcher with USE_CELERY feature flag for gradual rollout
- Add Celery task wrappers for all background operations:
- Marketplace imports
- Letzshop historical imports
- Product exports
- Code quality scans
- Test runs
- Subscription scheduled tasks (via Celery Beat)
- Add celery_task_id column to job tables for Flower integration
- Add Flower dashboard link to admin background tasks page
- Update docker-compose.yml with worker, beat, and flower services
- Add Makefile targets: celery-worker, celery-beat, celery-dev, flower
When USE_CELERY=false (default), system falls back to FastAPI BackgroundTasks
for development without Redis dependency.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
