feat(prospecting): add batch delay + fix Celery error_message field

- Add PROSPECTING_BATCH_DELAY_SECONDS config (default 1.0s) — polite
  delay between prospects in batch scans to avoid rate limiting
- Apply delay to all 5 batch API endpoints and all Celery tasks
- Fix Celery tasks: error_message → error_log (matches model field)
- Add batch-scanning.md docs with rate limiting guide, scaling estimates
  for 70k+ URL imports, and pipeline order recommendations

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
2026-04-01 21:55:24 +02:00
parent 30f3dae5a3
commit 50a4fc38a7
4 changed files with 120 additions and 15 deletions

View File

@@ -26,6 +26,9 @@ class ModuleConfig(BaseSettings):
# Max concurrent HTTP requests for batch scanning
max_concurrent_requests: int = 10
# Delay between prospects in batch scans (seconds) — be polite to target sites
batch_delay_seconds: float = 1.0
model_config = {"env_prefix": "PROSPECTING_", "env_file": ".env", "extra": "ignore"}