feat(prospecting): add batch delay + fix Celery error_message field
- Add PROSPECTING_BATCH_DELAY_SECONDS config (default 1.0s) — polite delay between prospects in batch scans to avoid rate limiting - Apply delay to all 5 batch API endpoints and all Celery tasks - Fix Celery tasks: error_message → error_log (matches model field) - Add batch-scanning.md docs with rate limiting guide, scaling estimates for 70k+ URL imports, and pipeline order recommendations Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -26,6 +26,9 @@ class ModuleConfig(BaseSettings):
|
||||
# Max concurrent HTTP requests for batch scanning
|
||||
max_concurrent_requests: int = 10
|
||||
|
||||
# Delay between prospects in batch scans (seconds) — be polite to target sites
|
||||
batch_delay_seconds: float = 1.0
|
||||
|
||||
model_config = {"env_prefix": "PROSPECTING_", "env_file": ".env", "extra": "ignore"}
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user