# End-to-End Plan: Prospecting → Live Site

## Context

Full pipeline from discovering a prospect to delivering a live hosted website. Three workstreams in dependency order, with AI content enhancement deferred to a later phase.

## Work Order

```
Workstream 1: Fix Hosting Foundation          ← FIRST (unblocks everything)
Workstream 2: Security Audit Pipeline         ← sales tool, enriches prospect data
Workstream 3: POC Builder + Templates         ← the wow factor (AI deferred)
Workstream 4: AI Content Enhancement          ← DEFERRED (provider TBD)
```

## The Full Journey (target state)

```
1. DISCOVER    → Prospect created (digital scan or manual capture)
2. ENRICH      → HTTP check, tech scan, performance, contacts, security audit
3. SCORE       → Opportunity score + security grade → prioritize leads
4. REPORT      → Generate branded security report for sales meeting
5. DEMO        → Launch live hacker console to demonstrate vulnerabilities
6. BUILD POC   → Apply industry template, populate with scraped content
7. PREVIEW     → Client sees POC at acme.hostwizard.lu
8. PROPOSE     → Send proposal with security report + POC link
9. ACCEPT      → Create merchant, assign store, create subscription
10. GO LIVE    → Assign custom domain (acme.lu), DNS + SSL configured
```

Steps 1-3 work today. Steps 4-10 need the work below.

---

## Workstream 1: Fix Hosting Foundation

**Goal:** Make site creation work, fix the merchant/prospect requirement, clean up the lifecycle.

**Proposal:** `docs/proposals/hosting-site-creation-fix.md`

### 1.1 Fix Site Creation Schema + Service

| File | Change |
|---|---|
| `hosting/schemas/hosted_site.py` | Add `merchant_id: int | None`, `prospect_id: int | None`, `model_validator` requiring at least one |
| `hosting/services/hosted_site_service.py` | Rewrite `create()`: if merchant_id → use it; if prospect_id → auto-create merchant from prospect. Remove system merchant hack. Remove `create_from_prospect()`. |
| `hosting/routes/api/admin_sites.py` | Remove `POST /from-prospect/{prospect_id}` endpoint. Main `POST /sites` handles both paths. |

### 1.2 Fix Site Creation Template

| File | Change |
|---|---|
| `hosting/templates/.../site-new.html` | Add merchant autocomplete dropdown (search existing merchants), prospect autocomplete dropdown (search existing prospects). Validate at least one selected. Auto-fill business_name/contacts from selection. |

### 1.3 Simplify accept_proposal

| File | Change |
|---|---|
| `hosting/services/hosted_site_service.py` | `accept_proposal()` — merchant already exists at creation time, so remove merchant creation logic. Only needs: create subscription, mark prospect CONVERTED. |

### 1.4 Update Tests

| File | Change |
|---|---|
| `hosting/tests/conftest.py` | Update `hosted_site` fixture to pass `merchant_id` |
| `hosting/tests/unit/test_hosted_site_service.py` | Update create calls, add validation tests, remove `TestHostedSiteFromProspect` |

### Verification
```bash
python -m pytest app/modules/hosting/tests/ -x -q
# Manual: create site from /admin/hosting/sites/new with merchant + prospect
```

---

## Workstream 2: Security Audit Pipeline

**Goal:** Add security scanning to enrichment pipeline + report generation + live demo.

**Proposal:** `docs/proposals/security-audit-demo-poc-builder.md`

### Phase 2A: Security Audit Service (scan + store results)

**Already done:** `ProspectSecurityAudit` model, `last_security_audit_at` column, relationship.

| File | Action |
|---|---|
| `prospecting/services/security_audit_constants.py` | **CREATE** — Migrate TRANSLATIONS, EXPOSED_PATHS, SECURITY_HEADERS, SEVERITY_SCORES from `scripts/security-audit/audit.py` |
| `prospecting/services/security_audit_service.py` | **CREATE** — `SecurityAuditService.run_audit(db, prospect)` with check_https, check_ssl, check_headers, check_exposed_files, check_cookies, check_server_info, check_technology |
| `prospecting/schemas/security_audit.py` | **CREATE** — Response schemas |
| `prospecting/migrations/versions/prospecting_002_security_audit.py` | **CREATE** — Table + column migration |
| `prospecting/services/prospect_service.py` | Add `get_pending_security_audit()` |
| `prospecting/routes/api/admin_enrichment.py` | Add `/security-audit/batch` + `/security-audit/{prospect_id}`. Add to `full_enrichment`. |
| `prospecting/static/admin/js/scan-jobs.js` | Add `security_audit: 'security-audit'` to batchRoutes |
| `prospecting/templates/.../scan-jobs.html` | Add "Security Audit" batch button |
| `prospecting/templates/.../prospect-detail.html` | Add "Security" tab: grade badge, score bar, findings by severity, "Run Audit" button |
| `prospecting/static/admin/js/prospect-detail.js` | Add `runSecurityAudit()`, tab handling |

### Phase 2B: Security Report Generation

| File | Action |
|---|---|
| `prospecting/services/security_report_service.py` | **CREATE** — `generate_html_report(audit, prospect, language)` → standalone HTML. Migrate report template from `scripts/security-audit/audit.py` |
| `prospecting/routes/api/admin_enrichment.py` | Add `GET /security-audit/{prospect_id}/report` → HTMLResponse |
| `prospecting/templates/.../prospect-detail.html` | "Generate Report" button + language selector on security tab |

### Phase 2C: Live Demo Server

| File | Action |
|---|---|
| `prospecting/models/demo_session.py` | **CREATE** — DemoSession model (prospect_id, status, port, pid) |
| `prospecting/services/demo_service.py` | **CREATE** — start_demo (clone + spawn), stop_demo, stop_all. Migrate SiteCloner + DemoHandler from `scripts/security-audit/demo.py` |
| `prospecting/services/demo_constants.py` | **CREATE** — HACKER_JS, HACKER_HTML_TEMPLATE, etc. |
| `prospecting/routes/api/admin_demo.py` | **CREATE** — start/stop/list endpoints |
| `prospecting/templates/.../prospect-detail.html` | "Launch Demo" / "Stop Demo" buttons on security tab |

### Verification
```bash
python -m pytest app/modules/prospecting/tests/ -x -q
# Manual: run security audit on prospect 1
curl -X POST .../enrichment/security-audit/1
# Manual: generate report
curl .../enrichment/security-audit/1/report > report.html
# Manual: launch demo
curl -X POST .../demo/start/1
```

---

## Workstream 3: POC Builder + Industry Templates

**Goal:** One-click POC generation from prospect data using industry templates. Result is a near-final multi-page website.

**Architecture decision:** `docs/proposals/hosting-architecture-decision.md` — reuse CMS + Store + StoreDomain.

### Phase 3A: Content Scraping Enhancement

Before building POCs, we need more content from prospect sites than just contacts.

| File | Action |
|---|---|
| `prospecting/models/prospect.py` | Add `scraped_content_json` column (Text) — stores extracted page content |
| `prospecting/services/enrichment_service.py` | Add `scrape_content(db, prospect)` method: extract meta description, H1/H2 headings, first paragraphs, service/menu items, image URLs, business hours, social links using BeautifulSoup |
| `prospecting/services/prospect_service.py` | Add `get_pending_content_scrape()` |
| `prospecting/routes/api/admin_enrichment.py` | Add `/content-scrape/batch` + `/content-scrape/{prospect_id}` |
| Add to `full_enrichment` pipeline | After contact scrape, before scoring |
| Migration | Add `last_content_scrape_at`, `scraped_content_json` to prospects |

**Scraped content structure:**
```json
{
  "meta_description": "Bati Rénovation — Construction et rénovation à Strasbourg",
  "headings": ["Bati Rénovation", "Nos Services", "Nos Réalisations"],
  "paragraphs": ["Entreprise spécialisée dans la rénovation...", ...],
  "services": ["Rénovation intérieure", "Construction neuve", ...],
  "images": ["https://site.fr/hero.jpg", "https://site.fr/project1.jpg"],
  "social_links": {"facebook": "...", "instagram": "..."},
  "business_hours": "Lun-Ven 8h-18h",
  "languages_detected": ["fr"]
}
```

### Phase 3B: Template Infrastructure

| File | Action |
|---|---|
| `hosting/templates_library/manifest.json` | **CREATE** — Registry of all templates |
| `hosting/templates_library/generic/` | **CREATE** — Generic template (meta.json, pages/*.json, theme.json) |
| `hosting/templates_library/restaurant/` | **CREATE** — Restaurant template |
| `hosting/templates_library/construction/` | **CREATE** — Construction template |
| `hosting/templates_library/auto-parts/` | **CREATE** — Auto parts template |
| `hosting/templates_library/professional-services/` | **CREATE** — Professional services template |
| `hosting/services/template_service.py` | **CREATE** — `list_templates()`, `get_template(id)`, `validate_template()` |
| `hosting/schemas/template.py` | **CREATE** — `TemplateListResponse`, `TemplateDetailResponse` |
| `hosting/routes/api/admin_sites.py` | Add `GET /templates` endpoint |

### Phase 3C: POC Builder Service

| File | Action |
|---|---|
| `hosting/services/poc_builder_service.py` | **CREATE** — `build_poc(db, prospect_id, template_id, merchant_id)`: 1) Load template, 2) Load scraped content from prospect, 3) Create HostedSite + Store, 4) Populate ContentPages from template with prospect data replacing placeholders, 5) Apply StoreTheme from template, 6) Return HostedSite |
| `hosting/routes/api/admin_sites.py` | Add `POST /poc/build` endpoint |
| `hosting/templates/.../site-new.html` | Add template selector (cards with previews) to creation flow |

**Placeholder replacement:**
```
{{business_name}}  → prospect.business_name or scraped H1
{{city}}           → prospect.city or extracted from address
{{phone}}          → primary phone contact
{{email}}          → primary email contact
{{address}}        → address contact value
{{meta_description}} → scraped meta description
{{services}}       → scraped service items (for service grid sections)
{{hero_image}}     → first scraped image or template default
```

### Phase 3D: CMS Section Types for Hosting

The existing CMS sections (hero, features, pricing, cta) cover basics. Hosting templates need a few more:

| Section Type | Used By | Description |
|---|---|---|
| `services_grid` | construction, professional | Service cards with icons and descriptions |
| `gallery` | construction, restaurant | Image grid with lightbox |
| `team_grid` | professional | Team member cards (photo, name, role) |
| `menu_display` | restaurant | Menu categories with items and prices |
| `testimonials` | all | Customer review cards |
| `contact_form` | all | Configurable contact form (name, email, message) |
| `map_embed` | all | Google Maps embed from address |
| `hours_display` | restaurant, auto-parts | Business hours table |

| File | Action |
|---|---|
| `cms/schemas/homepage_sections.py` | Add new section type schemas |
| `app/templates/storefront/sections/` | **CREATE** — Jinja2 partials for each new section type |
| `app/templates/storefront/content-page.html` | Update section renderer to handle new types |

### Verification
```bash
python -m pytest app/modules/hosting/tests/ -x -q
python -m pytest app/modules/prospecting/tests/ -x -q
# Manual:
# 1. Create prospect for batirenovation-strasbourg.fr
# 2. Run full enrichment (includes content scrape)
# 3. Go to /admin/hosting/sites/new → select prospect + "construction" template
# 4. POC built → preview at subdomain
# 5. Accept → go live with custom domain
```

---

## Workstream 4: AI Content Enhancement (DEFERRED)

**Provider:** TBD (Claude API or OpenAI)
**Scope:** TBD (full rewrite vs enhance + fill gaps)

When ready, this adds an AI step to the POC builder:
1. After scraping content and before populating CMS pages
2. AI enhances scraped text: better headlines, professional descriptions, SEO meta
3. AI generates missing content: testimonials, about section, service descriptions
4. AI translates to additional languages (fr/de/en/lb)
5. Integrated as optional step — POC builder works without AI (just uses scraped text + template defaults)

| File | Action (future) |
|---|---|
| `app/core/config.py` or `hosting/config.py` | Add AI provider config (API key, model, temperature) |
| `hosting/services/ai_content_service.py` | **CREATE** — `enhance_content(scraped, template, language)`, `generate_missing(template_sections, scraped)`, `translate(content, target_langs)` |
| `hosting/services/poc_builder_service.py` | Call AI service between scrape and populate steps |
| `.env.example` | Add `AI_PROVIDER`, `AI_API_KEY`, `AI_MODEL` |

---

## Dependency Graph

```
Workstream 1 (hosting fix)
    ├── 1.1 Schema + service     ← no dependencies
    ├── 1.2 Template form        ← needs 1.1
    ├── 1.3 Simplify accept      ← needs 1.1
    └── 1.4 Tests                ← needs 1.1-1.3

Workstream 2 (security audit)
    ├── 2A Service + endpoints   ← no dependencies (model done)
    ├── 2B Report generation     ← needs 2A
    └── 2C Live demo             ← needs 2A conceptually

Workstream 3 (POC builder)
    ├── 3A Content scraping      ← no dependencies
    ├── 3B Template files        ← no dependencies
    ├── 3C POC builder service   ← needs 1.1 + 3A + 3B
    └── 3D CMS section types     ← needs 3B (to know what sections templates use)

Workstream 4 (AI) ← DEFERRED
    └── needs 3C
```

**Parallelizable:** 1.1 + 2A + 3A + 3B can all start simultaneously.

---

## Estimated Scope

| Workstream | New Files | Modified Files | ~Lines |
|---|---|---|---|
| 1 (hosting fix) | 0 | 5 | ~200 |
| 2A (security service) | 4 | 6 | ~1200 |
| 2B (report) | 1 | 2 | ~400 |
| 2C (demo) | 4 | 3 | ~500 |
| 3A (content scrape) | 0 | 4 | ~200 |
| 3B (templates) | ~20 JSON | 2 | ~500 |
| 3C (POC builder) | 2 | 3 | ~300 |
| 3D (CMS sections) | ~8 | 2 | ~400 |
| **Total** | **~40** | **~27** | **~3700** |