feat: add Celery/Redis task queue with feature flag support
Migrate background tasks from FastAPI BackgroundTasks to Celery with Redis for persistent task queuing, retries, and scheduled jobs. Key changes: - Add Celery configuration with Redis broker/backend - Create task dispatcher with USE_CELERY feature flag for gradual rollout - Add Celery task wrappers for all background operations: - Marketplace imports - Letzshop historical imports - Product exports - Code quality scans - Test runs - Subscription scheduled tasks (via Celery Beat) - Add celery_task_id column to job tables for Flower integration - Add Flower dashboard link to admin background tasks page - Update docker-compose.yml with worker, beat, and flower services - Add Makefile targets: celery-worker, celery-beat, celery-dev, flower When USE_CELERY=false (default), system falls back to FastAPI BackgroundTasks for development without Redis dependency. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
@@ -31,6 +31,7 @@ class BackgroundTaskResponse(BaseModel):
|
||||
triggered_by: str | None
|
||||
error_message: str | None
|
||||
details: dict | None
|
||||
celery_task_id: str | None = None # Celery task ID for Flower linking
|
||||
|
||||
|
||||
class BackgroundTasksStatsResponse(BaseModel):
|
||||
@@ -77,6 +78,7 @@ def _convert_import_to_response(job) -> BackgroundTaskResponse:
|
||||
"errors": job.error_count,
|
||||
"total_processed": job.total_processed,
|
||||
},
|
||||
celery_task_id=getattr(job, "celery_task_id", None),
|
||||
)
|
||||
|
||||
|
||||
@@ -105,6 +107,7 @@ def _convert_test_run_to_response(run) -> BackgroundTaskResponse:
|
||||
"pass_rate": run.pass_rate,
|
||||
"git_branch": run.git_branch,
|
||||
},
|
||||
celery_task_id=getattr(run, "celery_task_id", None),
|
||||
)
|
||||
|
||||
|
||||
@@ -141,6 +144,7 @@ def _convert_scan_to_response(scan) -> BackgroundTaskResponse:
|
||||
"git_commit_hash": scan.git_commit_hash,
|
||||
"progress_message": scan.progress_message,
|
||||
},
|
||||
celery_task_id=getattr(scan, "celery_task_id", None),
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user