diff --git a/docs/proposals/end-to-end-prospecting-to-live-site.md b/docs/proposals/end-to-end-prospecting-to-live-site.md new file mode 100644 index 00000000..4507be7c --- /dev/null +++ b/docs/proposals/end-to-end-prospecting-to-live-site.md @@ -0,0 +1,293 @@ +# End-to-End Plan: Prospecting → Live Site + +## Context + +Full pipeline from discovering a prospect to delivering a live hosted website. Three workstreams in dependency order, with AI content enhancement deferred to a later phase. + +## Work Order + +``` +Workstream 1: Fix Hosting Foundation ← FIRST (unblocks everything) +Workstream 2: Security Audit Pipeline ← sales tool, enriches prospect data +Workstream 3: POC Builder + Templates ← the wow factor (AI deferred) +Workstream 4: AI Content Enhancement ← DEFERRED (provider TBD) +``` + +## The Full Journey (target state) + +``` +1. DISCOVER → Prospect created (digital scan or manual capture) +2. ENRICH → HTTP check, tech scan, performance, contacts, security audit +3. SCORE → Opportunity score + security grade → prioritize leads +4. REPORT → Generate branded security report for sales meeting +5. DEMO → Launch live hacker console to demonstrate vulnerabilities +6. BUILD POC → Apply industry template, populate with scraped content +7. PREVIEW → Client sees POC at acme.hostwizard.lu +8. PROPOSE → Send proposal with security report + POC link +9. ACCEPT → Create merchant, assign store, create subscription +10. GO LIVE → Assign custom domain (acme.lu), DNS + SSL configured +``` + +Steps 1-3 work today. Steps 4-10 need the work below. + +--- + +## Workstream 1: Fix Hosting Foundation + +**Goal:** Make site creation work, fix the merchant/prospect requirement, clean up the lifecycle. + +**Proposal:** `docs/proposals/hosting-site-creation-fix.md` + +### 1.1 Fix Site Creation Schema + Service + +| File | Change | +|---|---| +| `hosting/schemas/hosted_site.py` | Add `merchant_id: int | None`, `prospect_id: int | None`, `model_validator` requiring at least one | +| `hosting/services/hosted_site_service.py` | Rewrite `create()`: if merchant_id → use it; if prospect_id → auto-create merchant from prospect. Remove system merchant hack. Remove `create_from_prospect()`. | +| `hosting/routes/api/admin_sites.py` | Remove `POST /from-prospect/{prospect_id}` endpoint. Main `POST /sites` handles both paths. | + +### 1.2 Fix Site Creation Template + +| File | Change | +|---|---| +| `hosting/templates/.../site-new.html` | Add merchant autocomplete dropdown (search existing merchants), prospect autocomplete dropdown (search existing prospects). Validate at least one selected. Auto-fill business_name/contacts from selection. | + +### 1.3 Simplify accept_proposal + +| File | Change | +|---|---| +| `hosting/services/hosted_site_service.py` | `accept_proposal()` — merchant already exists at creation time, so remove merchant creation logic. Only needs: create subscription, mark prospect CONVERTED. | + +### 1.4 Update Tests + +| File | Change | +|---|---| +| `hosting/tests/conftest.py` | Update `hosted_site` fixture to pass `merchant_id` | +| `hosting/tests/unit/test_hosted_site_service.py` | Update create calls, add validation tests, remove `TestHostedSiteFromProspect` | + +### Verification +```bash +python -m pytest app/modules/hosting/tests/ -x -q +# Manual: create site from /admin/hosting/sites/new with merchant + prospect +``` + +--- + +## Workstream 2: Security Audit Pipeline + +**Goal:** Add security scanning to enrichment pipeline + report generation + live demo. + +**Proposal:** `docs/proposals/security-audit-demo-poc-builder.md` + +### Phase 2A: Security Audit Service (scan + store results) + +**Already done:** `ProspectSecurityAudit` model, `last_security_audit_at` column, relationship. + +| File | Action | +|---|---| +| `prospecting/services/security_audit_constants.py` | **CREATE** — Migrate TRANSLATIONS, EXPOSED_PATHS, SECURITY_HEADERS, SEVERITY_SCORES from `scripts/security-audit/audit.py` | +| `prospecting/services/security_audit_service.py` | **CREATE** — `SecurityAuditService.run_audit(db, prospect)` with check_https, check_ssl, check_headers, check_exposed_files, check_cookies, check_server_info, check_technology | +| `prospecting/schemas/security_audit.py` | **CREATE** — Response schemas | +| `prospecting/migrations/versions/prospecting_002_security_audit.py` | **CREATE** — Table + column migration | +| `prospecting/services/prospect_service.py` | Add `get_pending_security_audit()` | +| `prospecting/routes/api/admin_enrichment.py` | Add `/security-audit/batch` + `/security-audit/{prospect_id}`. Add to `full_enrichment`. | +| `prospecting/static/admin/js/scan-jobs.js` | Add `security_audit: 'security-audit'` to batchRoutes | +| `prospecting/templates/.../scan-jobs.html` | Add "Security Audit" batch button | +| `prospecting/templates/.../prospect-detail.html` | Add "Security" tab: grade badge, score bar, findings by severity, "Run Audit" button | +| `prospecting/static/admin/js/prospect-detail.js` | Add `runSecurityAudit()`, tab handling | + +### Phase 2B: Security Report Generation + +| File | Action | +|---|---| +| `prospecting/services/security_report_service.py` | **CREATE** — `generate_html_report(audit, prospect, language)` → standalone HTML. Migrate report template from `scripts/security-audit/audit.py` | +| `prospecting/routes/api/admin_enrichment.py` | Add `GET /security-audit/{prospect_id}/report` → HTMLResponse | +| `prospecting/templates/.../prospect-detail.html` | "Generate Report" button + language selector on security tab | + +### Phase 2C: Live Demo Server + +| File | Action | +|---|---| +| `prospecting/models/demo_session.py` | **CREATE** — DemoSession model (prospect_id, status, port, pid) | +| `prospecting/services/demo_service.py` | **CREATE** — start_demo (clone + spawn), stop_demo, stop_all. Migrate SiteCloner + DemoHandler from `scripts/security-audit/demo.py` | +| `prospecting/services/demo_constants.py` | **CREATE** — HACKER_JS, HACKER_HTML_TEMPLATE, etc. | +| `prospecting/routes/api/admin_demo.py` | **CREATE** — start/stop/list endpoints | +| `prospecting/templates/.../prospect-detail.html` | "Launch Demo" / "Stop Demo" buttons on security tab | + +### Verification +```bash +python -m pytest app/modules/prospecting/tests/ -x -q +# Manual: run security audit on prospect 1 +curl -X POST .../enrichment/security-audit/1 +# Manual: generate report +curl .../enrichment/security-audit/1/report > report.html +# Manual: launch demo +curl -X POST .../demo/start/1 +``` + +--- + +## Workstream 3: POC Builder + Industry Templates + +**Goal:** One-click POC generation from prospect data using industry templates. Result is a near-final multi-page website. + +**Architecture decision:** `docs/proposals/hosting-architecture-decision.md` — reuse CMS + Store + StoreDomain. + +### Phase 3A: Content Scraping Enhancement + +Before building POCs, we need more content from prospect sites than just contacts. + +| File | Action | +|---|---| +| `prospecting/models/prospect.py` | Add `scraped_content_json` column (Text) — stores extracted page content | +| `prospecting/services/enrichment_service.py` | Add `scrape_content(db, prospect)` method: extract meta description, H1/H2 headings, first paragraphs, service/menu items, image URLs, business hours, social links using BeautifulSoup | +| `prospecting/services/prospect_service.py` | Add `get_pending_content_scrape()` | +| `prospecting/routes/api/admin_enrichment.py` | Add `/content-scrape/batch` + `/content-scrape/{prospect_id}` | +| Add to `full_enrichment` pipeline | After contact scrape, before scoring | +| Migration | Add `last_content_scrape_at`, `scraped_content_json` to prospects | + +**Scraped content structure:** +```json +{ + "meta_description": "Bati Rénovation — Construction et rénovation à Strasbourg", + "headings": ["Bati Rénovation", "Nos Services", "Nos Réalisations"], + "paragraphs": ["Entreprise spécialisée dans la rénovation...", ...], + "services": ["Rénovation intérieure", "Construction neuve", ...], + "images": ["https://site.fr/hero.jpg", "https://site.fr/project1.jpg"], + "social_links": {"facebook": "...", "instagram": "..."}, + "business_hours": "Lun-Ven 8h-18h", + "languages_detected": ["fr"] +} +``` + +### Phase 3B: Template Infrastructure + +| File | Action | +|---|---| +| `hosting/templates_library/manifest.json` | **CREATE** — Registry of all templates | +| `hosting/templates_library/generic/` | **CREATE** — Generic template (meta.json, pages/*.json, theme.json) | +| `hosting/templates_library/restaurant/` | **CREATE** — Restaurant template | +| `hosting/templates_library/construction/` | **CREATE** — Construction template | +| `hosting/templates_library/auto-parts/` | **CREATE** — Auto parts template | +| `hosting/templates_library/professional-services/` | **CREATE** — Professional services template | +| `hosting/services/template_service.py` | **CREATE** — `list_templates()`, `get_template(id)`, `validate_template()` | +| `hosting/schemas/template.py` | **CREATE** — `TemplateListResponse`, `TemplateDetailResponse` | +| `hosting/routes/api/admin_sites.py` | Add `GET /templates` endpoint | + +### Phase 3C: POC Builder Service + +| File | Action | +|---|---| +| `hosting/services/poc_builder_service.py` | **CREATE** — `build_poc(db, prospect_id, template_id, merchant_id)`: 1) Load template, 2) Load scraped content from prospect, 3) Create HostedSite + Store, 4) Populate ContentPages from template with prospect data replacing placeholders, 5) Apply StoreTheme from template, 6) Return HostedSite | +| `hosting/routes/api/admin_sites.py` | Add `POST /poc/build` endpoint | +| `hosting/templates/.../site-new.html` | Add template selector (cards with previews) to creation flow | + +**Placeholder replacement:** +``` +{{business_name}} → prospect.business_name or scraped H1 +{{city}} → prospect.city or extracted from address +{{phone}} → primary phone contact +{{email}} → primary email contact +{{address}} → address contact value +{{meta_description}} → scraped meta description +{{services}} → scraped service items (for service grid sections) +{{hero_image}} → first scraped image or template default +``` + +### Phase 3D: CMS Section Types for Hosting + +The existing CMS sections (hero, features, pricing, cta) cover basics. Hosting templates need a few more: + +| Section Type | Used By | Description | +|---|---|---| +| `services_grid` | construction, professional | Service cards with icons and descriptions | +| `gallery` | construction, restaurant | Image grid with lightbox | +| `team_grid` | professional | Team member cards (photo, name, role) | +| `menu_display` | restaurant | Menu categories with items and prices | +| `testimonials` | all | Customer review cards | +| `contact_form` | all | Configurable contact form (name, email, message) | +| `map_embed` | all | Google Maps embed from address | +| `hours_display` | restaurant, auto-parts | Business hours table | + +| File | Action | +|---|---| +| `cms/schemas/homepage_sections.py` | Add new section type schemas | +| `app/templates/storefront/sections/` | **CREATE** — Jinja2 partials for each new section type | +| `app/templates/storefront/content-page.html` | Update section renderer to handle new types | + +### Verification +```bash +python -m pytest app/modules/hosting/tests/ -x -q +python -m pytest app/modules/prospecting/tests/ -x -q +# Manual: +# 1. Create prospect for batirenovation-strasbourg.fr +# 2. Run full enrichment (includes content scrape) +# 3. Go to /admin/hosting/sites/new → select prospect + "construction" template +# 4. POC built → preview at subdomain +# 5. Accept → go live with custom domain +``` + +--- + +## Workstream 4: AI Content Enhancement (DEFERRED) + +**Provider:** TBD (Claude API or OpenAI) +**Scope:** TBD (full rewrite vs enhance + fill gaps) + +When ready, this adds an AI step to the POC builder: +1. After scraping content and before populating CMS pages +2. AI enhances scraped text: better headlines, professional descriptions, SEO meta +3. AI generates missing content: testimonials, about section, service descriptions +4. AI translates to additional languages (fr/de/en/lb) +5. Integrated as optional step — POC builder works without AI (just uses scraped text + template defaults) + +| File | Action (future) | +|---|---| +| `app/core/config.py` or `hosting/config.py` | Add AI provider config (API key, model, temperature) | +| `hosting/services/ai_content_service.py` | **CREATE** — `enhance_content(scraped, template, language)`, `generate_missing(template_sections, scraped)`, `translate(content, target_langs)` | +| `hosting/services/poc_builder_service.py` | Call AI service between scrape and populate steps | +| `.env.example` | Add `AI_PROVIDER`, `AI_API_KEY`, `AI_MODEL` | + +--- + +## Dependency Graph + +``` +Workstream 1 (hosting fix) + ├── 1.1 Schema + service ← no dependencies + ├── 1.2 Template form ← needs 1.1 + ├── 1.3 Simplify accept ← needs 1.1 + └── 1.4 Tests ← needs 1.1-1.3 + +Workstream 2 (security audit) + ├── 2A Service + endpoints ← no dependencies (model done) + ├── 2B Report generation ← needs 2A + └── 2C Live demo ← needs 2A conceptually + +Workstream 3 (POC builder) + ├── 3A Content scraping ← no dependencies + ├── 3B Template files ← no dependencies + ├── 3C POC builder service ← needs 1.1 + 3A + 3B + └── 3D CMS section types ← needs 3B (to know what sections templates use) + +Workstream 4 (AI) ← DEFERRED + └── needs 3C +``` + +**Parallelizable:** 1.1 + 2A + 3A + 3B can all start simultaneously. + +--- + +## Estimated Scope + +| Workstream | New Files | Modified Files | ~Lines | +|---|---|---|---| +| 1 (hosting fix) | 0 | 5 | ~200 | +| 2A (security service) | 4 | 6 | ~1200 | +| 2B (report) | 1 | 2 | ~400 | +| 2C (demo) | 4 | 3 | ~500 | +| 3A (content scrape) | 0 | 4 | ~200 | +| 3B (templates) | ~20 JSON | 2 | ~500 | +| 3C (POC builder) | 2 | 3 | ~300 | +| 3D (CMS sections) | ~8 | 2 | ~400 | +| **Total** | **~40** | **~27** | **~3700** |