feat: add Celery/Redis task queue with feature flag support

Migrate background tasks from FastAPI BackgroundTasks to Celery with Redis
for persistent task queuing, retries, and scheduled jobs.

Key changes:
- Add Celery configuration with Redis broker/backend
- Create task dispatcher with USE_CELERY feature flag for gradual rollout
- Add Celery task wrappers for all background operations:
  - Marketplace imports
  - Letzshop historical imports
  - Product exports
  - Code quality scans
  - Test runs
  - Subscription scheduled tasks (via Celery Beat)
- Add celery_task_id column to job tables for Flower integration
- Add Flower dashboard link to admin background tasks page
- Update docker-compose.yml with worker, beat, and flower services
- Add Makefile targets: celery-worker, celery-beat, celery-dev, flower

When USE_CELERY=false (default), system falls back to FastAPI BackgroundTasks
for development without Redis dependency.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
2026-01-11 17:35:16 +01:00
parent 879ac0caea
commit 2792414395
30 changed files with 2218 additions and 79 deletions

View File

@@ -13,6 +13,15 @@
{% block content %}
{% call page_header_flex(title='Background Tasks', subtitle='Monitor running and completed background tasks') %}
<!-- Flower Dashboard Link (Celery Monitoring) -->
<a href="{{ config.FLOWER_URL | default('http://localhost:5555', true) }}"
target="_blank"
rel="noopener noreferrer"
class="flex items-center px-4 py-2 text-sm font-medium leading-5 text-white transition-colors duration-150 bg-purple-600 border border-transparent rounded-lg active:bg-purple-600 hover:bg-purple-700 focus:outline-none focus:shadow-outline-purple mr-2"
title="Open Flower dashboard for detailed Celery task monitoring">
<span x-html="$icon('chart-bar', 'w-4 h-4 mr-2')"></span>
Flower Dashboard
</a>
{{ refresh_button(variant='secondary') }}
{% endcall %}
@@ -208,6 +217,7 @@
<th class="px-4 py-3">Duration</th>
<th class="px-4 py-3">Triggered By</th>
<th class="px-4 py-3">Status</th>
<th class="px-4 py-3">Celery</th>
</tr>
</thead>
<tbody class="bg-white divide-y dark:divide-gray-700 dark:bg-gray-800">
@@ -241,6 +251,21 @@
x-text="task.status">
</span>
</td>
<td class="px-4 py-3 text-sm">
<template x-if="task.celery_task_id">
<a :href="'{{ config.FLOWER_URL | default('http://localhost:5555', true) }}/task/' + task.celery_task_id"
target="_blank"
rel="noopener noreferrer"
class="text-purple-600 hover:text-purple-800 dark:text-purple-400 dark:hover:text-purple-300 underline"
title="View in Flower">
<span x-html="$icon('external-link', 'w-4 h-4 inline')"></span>
<span x-text="task.celery_task_id.substring(0, 8) + '...'"></span>
</a>
</template>
<template x-if="!task.celery_task_id">
<span class="text-gray-400 text-xs">-</span>
</template>
</td>
</tr>
</template>
</tbody>