- New scrape_content() method in enrichment_service: extracts meta
description, H1/H2 headings, paragraphs, images (filtered for size),
social links, service items, and detected languages using BeautifulSoup
- Scans 6 pages per prospect: /, /about, /a-propos, /services,
/nos-services, /contact
- Results stored as JSON in prospect.scraped_content_json
- New endpoints: POST /content-scrape/{id} and /content-scrape/batch
- Added to full_enrichment pipeline (Step 5, before security audit)
- CONTENT_SCRAPE job type for scan-jobs tracking
- "Content Scrape" batch button on scan-jobs page
- Add beautifulsoup4 to requirements.txt
Tested on batirenovation-strasbourg.fr: extracted 30 headings,
21 paragraphs, 13 images.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add PROSPECTING_BATCH_DELAY_SECONDS config (default 1.0s) — polite
delay between prospects in batch scans to avoid rate limiting
- Apply delay to all 5 batch API endpoints and all Celery tasks
- Fix Celery tasks: error_message → error_log (matches model field)
- Add batch-scanning.md docs with rate limiting guide, scaling estimates
for 70k+ URL imports, and pipeline order recommendations
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- SecurityReportService generates standalone branded HTML reports from
stored audit data (grade badge, simulated hacked site, detailed
findings, business impact, call-to-action with contact info)
- GET /security-audit/report/{prospect_id} returns HTMLResponse
- "Generate Report" button on prospect detail security tab opens
report in new browser tab (printable to PDF)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete security audit integration into the enrichment pipeline:
Backend:
- SecurityAuditService with 7 passive checks: HTTPS, SSL cert, security
headers, exposed files, cookies, server info, technology detection
- Constants file with SECURITY_HEADERS, EXPOSED_PATHS, SEVERITY_SCORES
- SecurityAuditResponse schema with JSON field validators + aliases
- Endpoints: POST /security-audit/{id}, POST /security-audit/batch
- Added to full_enrichment pipeline (Step 5, before scoring)
- get_pending_security_audit() query in prospect_service
Frontend:
- Security tab on prospect detail page with grade badge (A+ to F),
score/100, severity counts, HTTPS/SSL status, missing headers,
exposed files, technologies, and full findings list
- "Run Security Audit" button with loading state
- "Security Audit" batch button on scan-jobs page
Tested on batirenovation-strasbourg.fr: Grade D (50/100), 11 issues
found (missing headers, exposed wp-login, server version disclosure).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Move db.commit() from services to API endpoints and Celery tasks.
Services now use db.flush() only; endpoints own the transaction.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migrates scanning pipeline from marketing-.lu-domains app into Orion module.
Supports digital (domain scan) and offline (manual capture) lead channels
with enrichment, scoring, campaign management, and interaction tracking.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
