Omjee73/Vulnerability_Scanner

 ## 📋 Table of Contents - [Features](#-features) - [Architecture](#-architecture) - [Complete Folder Structure](#-complete-folder-structure) - [Layer-by-Layer Breakdown](#-layer-by-layer-breakdown) - [Layer 1 — Presentation](#layer-1--presentation-react--nginx) - [Layer 2 — API Gateway](#layer-2--api-gateway-flask) - [Layer 3 — Processing](#layer-3--processing-celery) - [Layer 4 — Scanning Engine](#layer-4--scanning-engine-OmniTricks) - [Layer 5 — Intelligence (Disabled)](#layer-5--intelligence-disabled) - [Layer 6 — Data Layer](#layer-6--data-layer-postgresql--redis) - [Security Tools (19+)](#-security-tools-19) - [API Endpoints](#-api-endpoints) - [Scan Workflow](#-scan-workflow) - [External Platform Plugin Mode](#-external-platform-plugin-mode) - [Hybrid Deployment (Subscription Model)](#-hybrid-deployment-subscription-model) - [Quick Start](#-quick-start) - [Docker Registry (ghcr.io)](#-docker-registry-ghcrio) - [Environment Variables](#-environment-variables) - [Frontend Pages & Components](#-frontend-pages--components) - [Database Schema](#-database-schema) - [Nginx Reverse Proxy](#-nginx-reverse-proxy) - [Troubleshooting](#-troubleshooting) ## ✨ Features - **19+ Security Tools** — subfinder, nuclei, httpx, nmap, naabu, masscan, nikto, ffuf, dirsearch, whatweb, dnsx, tlsx, testssl, wafw00f, gobuster, sqlmap, wpscan, curl, gau, waybackurls, dig - **AI-Powered Analysis** — Automatic target classification, risk scoring (0–100), attack vector identification, optimal tool selection per target - **Real-Time Streaming** — Server-Sent Events (SSE) push live scan logs to a terminal UI as tools execute - **3 Scan Modes** — OmniTricks AI (full), Levelwise Parallel, Ultra-Fast - **Parallel Execution** — 10 concurrent subdomain scans, each running 8–14 tools simultaneously - **Subdomain Discovery** — Subfinder (recursive + all sources) + DNS wordlist expansion - **PDF Export** — Download scan results as formatted PDF reports - **Asset Management** — Track companies, domains, IPs, and endpoints - **JWT Auth + OTP** — Email-based OTP verification, password reset, session cookies - **Rate Limiting** — Redis-backed sliding-window rate limiter on auth endpoints - **6 Color Themes** — Blue, Green, Purple, Red, Gray, Black - **Analytics Dashboard** — Scan trends, severity distribution, vulnerability counts via Recharts - **Docker-First** — Single `docker compose up --build` deploys everything ## External Platform Plugin Mode This repo can run in two ways: - **Standalone platform mode:** users log in to DeepTrustxAI, start scans from the React UI, and view results inside this platform. - **Plugin scanner mode:** another control plane/platform owns auth, billing, tenants, roles, and credentials. That platform calls this repo as a scanner service over HTTP. The plugin adapter lives in the API Gateway: services/api-gateway/app/routes/cp_scanner.py ### Plugin Endpoints | Method | Path | Called by | Purpose | |---|---|---|---| | `GET` | `/api/manifest` | External platform | Reads scanner metadata, dashboard URL, and credential schema | | `POST` | `/api/scan` | External platform | Starts a scan through the local Celery/scanning-engine flow | | `POST` | `/api/scans//view-token` | External platform | Issues a short-lived report secret | | `GET` | `/dashboard?scanId=...&secret=...` | Browser redirect | Shows a simple scan report after secret validation | When `/api/scan` is accepted, the adapter: 1. Verifies the external platform HMAC signature. 2. Creates a local `scans` record. 3. Queues the normal `hexstrike_scan` Celery task. 4. Stores CP mapping in `cp_scan_jobs`. 5. Starts a status monitor. 6. Sends signed callbacks to the external platform when the scan is running and when it completes or fails. ### Plugin Environment Variables Set these in `.env` before using plugin mode: CP_SCANNER_SHARED_SECRET=change-me-to-a-long-random-shared-secret CP_SCANNER_PUBLIC_BASE_URL=https://your-scanner.example.com CP_SCANNER_KEY=omnitricks CP_SCANNER_NAME=OmniTricks Vulnerability Scanner CP_SCANNER_VERSION=1.0.0 CP_SCANNER_CATEGORY=network CP_SCANNER_CALLBACK_POLL_SECONDS=5 CP_SCANNER_CALLBACK_MAX_SECONDS=7200 CP_SCANNER_VIEW_SECRET_TTL_MINUTES=60 `CP_SCANNER_SHARED_SECRET` must match the secret configured in the external platform. ### HMAC Contract Plugin mode uses this signature format: signature = HMAC_SHA256(sharedSecret, rawBody + "." + nonce) Required headers on incoming external-platform requests: Content-Type: application/json X-Scanner-Nonce: X-Scanner-Signature: The adapter uses the same format for outgoing callbacks. ### Scan Dispatch Body The external platform should call: POST /api/scan Example JSON body: { "jobId": "cp-job-123", "orgId": "org-123", "callbackUrl": "https://cp.example.com/scanner-callbacks/omnitricks/cp-job-123", "nonce": "same-nonce-for-this-job", "credentials": { "domain": "example.com", "execution_mode": "local" } } Successful response: { "scanId": "local-scan-id", "status": "accepted" } The external platform must send `domain` inside `credentials`. The scanner does not persist external platform credentials. ### Callback Bodies When work starts: { "status": "running" } When work completes: { "status": "completed", "viewSecret": "short-lived-secret", "summary": { "critical": 0, "high": 1, "medium": 2, "low": 3, "info": 4 } } When work fails: { "status": "failed", "error": "reason" } ### View Report Flow When a user clicks "View report" in the external platform: 1. External platform calls `POST /api/scans//view-token` with HMAC headers. 2. This scanner returns `{ "viewSecret": "..." }`. 3. External platform redirects the browser to: https://your-scanner.example.com/dashboard?scanId=&secret= The dashboard validates the secret against the hash stored in PostgreSQL and returns `401` for invalid or expired secrets. ## 🏗 Architecture ┌──────────────────────────────────────────────────────────────────────────┐ │ DeepTrustxAI VULNERABILITY SCANNER │ │ 6-Layer Microservices Architecture │ └──────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────┐ │ Layer 1: PRESENTATION │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ React 19 + TypeScript + Vite 6 + TailwindCSS │ │ │ │ Built → static assets → served by Nginx on :80 │ │ │ │ Reverse proxy: /api/ /auth/ /rescan/ → api-gateway:4000 │ │ │ └─────────────────────────────────────────────────────────────────┘ │ └────────────────────────────────┬────────────────────────────────────────┘ │ HTTP / SSE ┌────────────────────────────────▼────────────────────────────────────────┐ │ Layer 2: API GATEWAY │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ Flask 2.2 + Gunicorn (4 workers) on :4000 │ │ │ │ JWT Auth (cookies) │ Rate Limiting │ CORS │ Flask-Mail │ │ │ │ Routes: auth, scans, assets, results, stats │ │ │ │ Dispatches Celery tasks │ Direct SSE streaming │ │ │ └──────────┬───────────────────────────────┬──────────────────────┘ │ └─────────────┼───────────────────────────────┼──────────────────────────-─┘ │ Celery Tasks │ Direct HTTP ┌─────────────▼─────────────────┐ ┌──────────▼──────────────────────────┐ │ Layer 3: PROCESSING │ │ Layer 4: SCANNING ENGINE │ │ ┌───────────────────────┐ │ │ ┌────────────────────────────┐ │ │ │ Celery Worker (4 conc) │ │ │ │ Flask + Gunicorn on :8888 │ │ │ │ Celery Beat (scheduler)│ │ │ │ 19+ security tools │ │ │ │ OmniTricksManager │─────┼──│→ │ AI analysis engine │ │ │ │ Redis log streaming │ │ │ │ Tool orchestration │ │ │ └───────────────────────┘ │ │ │ Parallel execution (8 thr) │ │ └────────────────────────────────┘ │ └────────────────────────────┘ │ └────────────────────────────────────-─┘ │ │ ┌─────────────▼────────────────────────────────▼──────────────────────────┐ │ Layer 6: DATA LAYER │ │ ┌──────────────────────┐ ┌───────────────────────────────────────┐ │ │ │ PostgreSQL 16 on :5432│ │ Redis 7 (Alpine) on :6379 │ │ │ │ DB: deeptrustxai_db │ │ Celery broker (db 0) │ │ │ │ JSONB tables │ │ Celery results (db 1) │ │ │ │ Indexed for perf │ │ SSE log streaming │ │ │ └──────────────────────┘ │ Rate limiting counters │ │ │ └───────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ ### Service Connection Diagram Browser (:80) │ ├── Static Assets ──→ Nginx (presentation container) │ ├── /api/* ──────────→ Nginx proxy ──→ api-gateway (:4000) │ │ │ ├── PostgreSQL (reads/writes scans, users, assets) │ ├── Redis (rate limiting, SSE log reads) │ │ │ ├── [Celery Task] ──→ celery-worker │ │ │ │ │ ├── OmniTricksManager HTTP Client │ │ │ └── scanning-engine (:8888) │ │ │ ├── subfinder, nuclei, httpx... │ │ │ └── 19+ tools in parallel │ │ │ │ │ ├── PostgreSQL (store results) │ │ └── Redis (push SSE logs) │ │ │ └── [Direct SSE] ──→ scanning-engine (:8888) │ ├── /auth/* ─────────→ Nginx proxy ──→ api-gateway (:4000) │ └── /rescan/* ───────→ Nginx proxy ──→ api-gateway (:4000) ──→ SSE stream ## Hybrid Deployment (Subscription Model) For enterprise rollout where scanning runs inside client network Docker nodes and UI/API/DB stay on your central server, see: - `HYBRID_SUBSCRIPTION_DEPLOYMENT_GUIDE.md` ### Docker Compose Services | # | Service | Container Name | Port | Image | Role | |---|---------|---------------|------|-------|------| | 1 | `presentation` | DeepTrustxAI-frontend | **80** | node:22 → nginx:1.27 | React SPA + Nginx reverse proxy | | 2 | `api-gateway` | DeepTrustxAI-api-gateway | **4000** | python:3.12-slim | Flask REST API + JWT auth | | 3 | `celery-worker` | DeepTrustxAI-celery-worker | — | python:3.12-slim | Async scan task execution | | 4 | `celery-beat` | DeepTrustxAI-celery-beat | — | python:3.12-slim | Scheduled task runner | | 5 | `scanning-engine` | DeepTrustxAI-scanning-engine | **8888** | python:3.12-slim | OmniTricks tool orchestrator | | 6 | `postgres` | DeepTrustxAI-postgres | **5432** | postgres:16-alpine | Primary database | | 7 | `redis` | DeepTrustxAI-redis | **6379** | redis:7-alpine | Task broker + cache | Network: `DeepTrustxAI-network` (bridge). Volumes: `postgres-data`, `redis-data`, `nuclei-templates`, `scanning-data`. ## 📁 Complete Folder Structure Vulnerability_Scanner/ │ ├── docker-compose.yml # 7-service orchestration (6 layers) ├── .env # Environment variables (secrets, API keys) ├── .gitignore ├── .dockerignore ├── README.md # ← This file ├── VERSION # Current release version (e.g. 1.0.0) ├── registry-push.ps1 # PowerShell script: build, tag, push images to ghcr.io │ ├── services/ │ │ │ ├── presentation/ # ── LAYER 1: React Frontend + Nginx ── │ │ ├── Dockerfile # Multi-stage: node:22-alpine (build) → nginx:1.27-alpine (serve) │ │ ├── nginx.conf # SPA routing + reverse proxy to api-gateway │ │ ├── package.json # React 19.1, Vite 6.3.5, TailwindCSS, 30+ deps │ │ ├── package-lock.json │ │ ├── index.html # Vite entry point │ │ ├── vite.config.ts │ │ ├── tsconfig.json │ │ ├── tsconfig.app.json │ │ ├── tsconfig.node.json │ │ ├── tailwind.config.js │ │ ├── tailwind.config.cjs │ │ ├── postcss.config.cjs │ │ ├── eslint.config.js │ │ ├── public/ │ │ │ ├── vite.svg │ │ │ ├── grid.svg │ │ │ ├── iitm.jpg │ │ │ └── IIT_Madras_Logo.svg.png │ │ └── src/ │ │ ├── main.tsx # Entry: AuthProvider → ThemeProvider → App │ │ ├── App.tsx # React Router v7 — all route definitions │ │ ├── App.css # Global animations, gradients, scan-line effects │ │ ├── index.css # Tailwind directives + CSS custom properties │ │ ├── vite-env.d.ts # Vite type declarations │ │ │ │ │ ├── config/ │ │ │ └── api.ts # API_BASE_URL from VITE_API_URL env var │ │ │ │ │ ├── context/ │ │ │ ├── AuthContext.tsx # JWT session: login/logout/register, user state │ │ │ └── ThemeContext.tsx # 6 color palettes via CSS custom properties │ │ │ │ │ ├── types/ │ │ │ └── subdomain.ts # TS interfaces: Subdomain, Vulnerability, TrendData │ │ │ │ │ ├── pages/ │ │ │ ├── About.tsx # 8-step workflow visualization │ │ │ └── Profile.tsx # Edit name + change password │ │ │ │ │ ├── components/ │ │ │ ├── HeroSection.tsx # Landing page — animated domain input │ │ │ │ │ │ │ ├── auth/ │ │ │ │ ├── Login.tsx # Split-screen email/password login │ │ │ │ ├── Register.tsx # Registration form │ │ │ │ ├── ForgotPassword.tsx # Password reset request │ │ │ │ ├── VerificationPage.tsx # Email verification callback │ │ │ │ ├── verify-otp.tsx # 6-digit OTP input │ │ │ │ └── ProtectedRoute.tsx # Auth guard (redirect to /login) │ │ │ │ │ │ │ ├── layout/ │ │ │ │ ├── Layout.tsx # Shell: Navbar + visual effects + footer │ │ │ │ └── Navbar.tsx # Nav links, theme picker, user menu │ │ │ │ │ │ │ ├── scan/ │ │ │ │ ├── OmniTricksScan.tsx # Core scan page: POST scan → SSE → results │ │ │ │ ├── SubdomainCardSubfinder.tsx # Individual subdomain result card │ │ │ │ ├── subdomainDetailsSubfinder.tsx # Expanded subdomain details │ │ │ │ ├── DirectoryTree.tsx # Directory brute-force results tree │ │ │ │ └── PDFDownloadSubfinder.tsx # Export scan results to PDF (jspdf) │ │ │ │ │ │ │ ├── Statistics/ │ │ │ │ └── ReconDashboard.tsx # Charts: scan trends, severity, history │ │ │ │ │ │ │ ├── terminal/ │ │ │ │ └── Terminal.tsx # Live SSE log viewer (portal, macOS style) │ │ │ │ │ │ │ ├── asset/ │ │ │ │ └── AssetForm.tsx # CRUD: companies, domains, IPs, endpoints │ │ │ │ │ │ │ └── ui/ │ │ │ └── LoadingSpinner.tsx # Spinning loader icon │ │ │ │ │ └── assets/ │ │ └── react.svg │ │ │ ├── api-gateway/ # ── LAYER 2: Flask REST API ── │ │ ├── Dockerfile # python:3.12-slim, gunicorn 4 workers, port 4000 │ │ ├── requirements.txt # Flask 2.2.5, celery, psycopg2, redis, flask-mail, etc. │ │ └── app/ │ │ ├── __init__.py │ │ ├── main.py # Flask app factory, blueprint registration, CORS, JWT │ │ ├── tasks.py # Celery task stubs (dispatched to processing worker) │ │ ├── OmniTricks_manager.py # HTTP client → scanning-engine (connection pooling) │ │ │ │ │ ├── middleware/ │ │ │ ├── __init__.py │ │ │ └── rate_limit.py # Redis sliding-window rate limiter + domain sanitizer │ │ │ │ │ ├── models/ │ │ │ ├── __init__.py │ │ │ └── user.py # User model: email, password, OTP, reset tokens │ │ │ │ │ └── routes/ │ │ ├── __init__.py │ │ ├── auth.py # 10 endpoints: register, login, OTP, password flows │ │ ├── scans.py # 17 endpoints: create/list/stream scans, SSE streaming │ │ ├── assets.py # 3 endpoints: CRUD asset groups │ │ ├── results.py # 8 endpoints: scan results, ports, ffuf, ZAP alerts │ │ └── stats.py # 2 endpoints: dashboard statistics + trends │ │ │ ├── processing/ # ── LAYER 3: Celery Workers ── │ │ ├── Dockerfile # python:3.12-slim, celery worker --concurrency=4 │ │ ├── requirements.txt # celery, psycopg2, redis, requests, aiohttp │ │ ├── tasks.py # 3 Celery tasks: OmniTricks, levelwise, ultra-fast │ │ ├── OmniTricks_manager.py # HTTP client → scanning-engine (ThreadPoolExecutor) │ │ └── utils/ │ │ ├── __init__.py │ │ ├── redis_log_handler.py # Custom logging.Handler → Redis rpush for SSE │ │ └── ollama_risk_analyzer.py # AI risk analysis (DISABLED — entirely commented out) │ │ │ ├── scanning-engine/ # ── LAYER 4: OmniTricks Tool Orchestrator ── │ │ ├── Dockerfile # python:3.12-slim + nmap, masscan, nikto, naabu, testssl │ │ ├── requirements.txt # flask, gunicorn, aiohttp, requests │ │ ├── init_tools.py # Build-time script: downloads 7 Go binaries │ │ ├── tool_tracker.py # Standalone tool version & status tracker (runs on startup) │ │ ├── entrypoint.sh # Startup: nuclei template update → tool tracker → gunicorn │ │ ├── app/ │ │ │ ├── __init__.py │ │ │ └── OmniTricks_server.py # ~2200 lines: 8 Flask routes, AI engine, 19 tool runners │ │ ├── utils/ │ │ │ └── tool_manager.py # ToolManager: downloads ProjectDiscovery Go binaries │ │ └── wordlists/ │ │ └── common.txt # Directory fuzzing wordlist (ffuf, dirsearch, gobuster) │ │ │ ├── OmniTricks-intelligence/ # ── LAYER 5: AI Analysis (DISABLED) ── │ │ ├── Dockerfile # python:3.12-slim, gunicorn on :8889 │ │ ├── requirements.txt │ │ └── app/ │ │ ├── __init__.py │ │ └── intelligence_server.py # Ollama/Mistral integration (not currently active) │ │ │ └── data/ # ── LAYER 6: Database Configuration ── │ ├── postgres/ │ │ └── init-db.js # Creates 6 collections + 11 indexes on first boot │ └── redis/ │ └── redis.conf # 512MB maxmemory, AOF persistence, allkeys-lru │ ├── Backend/ # Legacy monolith (pre-microservices, not used by Docker) └── Frontend/ # Legacy monolith (pre-microservices, not used by Docker) ## 🔬 Layer-by-Layer Breakdown ### Layer 1 — Presentation (React + Nginx) | Item | Detail | |------|--------| | **Framework** | React 19.1 + TypeScript 5.8 | | **Bundler** | Vite 6.3.5 (HMR in dev, optimized build for prod) | | **Styling** | TailwindCSS 3.4 + CSS custom properties for themes | | **Icons** | lucide-react 0.514 | | **Charts** | Recharts 2.15 (LineChart, BarChart, PieChart) | | **Routing** | React Router v7.6 (client-side) | | **SSE Client** | event-source-polyfill (supports credentials/cookies) | | **PDF Export** | jspdf 3.0 | | **Maps** | Leaflet 1.9 + react-leaflet 5.0 + mapbox-gl 3.12 | | **Animations** | framer-motion 12.17 + typewriter-effect + tsparticles | | **UI Components** | Radix UI (accordion, dialog, tabs, tooltip, slot) | | **Notifications** | react-hot-toast + sweetalert2 | | **Build** | Multi-stage Docker: `node:22-alpine` → `nginx:1.27-alpine` | | **Port** | **80** (Nginx) | ### Layer 2 — API Gateway (Flask) | Item | Detail | |------|--------| | **Framework** | Flask 2.2.5 | | **Server** | Gunicorn (4 workers, 600s timeout, keep-alive 5s) | | **Auth** | flask-jwt-extended 4.3.1 (HTTP-only cookies, CSRF disabled) | | **Database Client** | psycopg2 / SQLAlchemy → PostgreSQL `deeptrustxai_db` | | **Cache/Broker** | Redis 5.0.1 (rate limiting, Celery broker connection) | | **Email** | Flask-Mail 0.10.0 via Gmail SMTP (OTP delivery) | | **Rate Limiting** | Custom Redis sliding-window decorator | | **Port** | **4000** | **Blueprints Registered:** | Blueprint | Prefix | Endpoints | Purpose | |-----------|--------|-----------|---------| | `auth_bp` | `/auth` | 10 | Registration, login, OTP, password flows | | `scans_bp` | mixed | 17 | Scan CRUD, SSE streaming, task dispatch | | `assets_bp` | `/api` | 3 | Asset group management | | `results_bp` | mixed | 8 | Scan result queries, port data, ZAP alerts | | `stats_bp` | mixed | 2 | Dashboard statistics and trends | ### Layer 3 — Processing (Celery) | Item | Detail | |------|--------| | **Engine** | Celery 5.3.1 | | **Worker Concurrency** | 4 processes | | **Beat** | Separate container for scheduled tasks | | **Broker** | Redis db 0 | | **Result Backend** | Redis db 1 | | **Scan Client** | OmniTricksManager → HTTP to scanning-engine | | **Log Streaming** | RedisLogHandler → `scan_logs:{scan_id}` (max 1000, 24h TTL) | **Active Celery Tasks:** | Task Name | Description | |-----------|-------------| | `OmniTricks_scan` | Full AI scan: analyze → discover subdomains → select tools → parallel execution of 10+ tools per subdomain → store results in PostgreSQL → stream logs to Redis | | `levelwise_parallel_scan` | Same pipeline but tracks `levels_completed: 3` for progress UI | | `OmniTricks_ultra_parallel_scan` | Fast-path alias that delegates to `OmniTricks_scan` | ### Layer 4 — Scanning Engine (OmniTricks) | Item | Detail | |------|--------| | **Framework** | Flask 3.0 | | **Server** | Gunicorn (4 workers, 8 threads, 300s timeout) | | **Binary Tools** | 7 Go binaries (ProjectDiscovery) downloaded at Docker build | | **System Tools** | nmap, masscan, nikto, testssl, dirsearch, wafw00f, whois | | **Nuclei Templates** | Auto-updated on every container start via `entrypoint.sh`, persisted in `nuclei-templates` volume | | **Parallelism** | ThreadPoolExecutor (8 workers per subdomain scan) | | **AI Engine** | Target classification, risk scoring, tech detection, tool selection | | **Port** | **8888** | ### Layer 5 — Intelligence (Disabled) Ollama + Mistral-based AI risk analysis. Commented out in `docker-compose.yml` and all processing tasks. When enabled, would run on port **8889** and provide: - Natural-language vulnerability assessments per subdomain - Batch risk analysis across scan results - Automatic model management (pull, health checks) ### Layer 6 — Data Layer (PostgreSQL + Redis) **PostgreSQL 16** — Database: `deeptrustxai_db` | Collection | Purpose | Key Indexes | |------------|---------|-------------| | `users` | User accounts (email, password, OTP) | `email` (unique), `is_verified` | | `scans` | Scan metadata & lifecycle status | `(user_id, created_at)` desc, `status`, `domain`, `scan_type` | | `subdomain_results` | Per-subdomain tool findings & risk scores | `scan_id`, `subdomain`, `risk_score` (desc) | | `scan_results` | Legacy knockpy scan results | `scan_id`, `domain`, `created_at` (desc) | | `scan_results_subfinder` | Subfinder-pipeline scan results | `scan_id`, `domain` | | `assets` | Company asset groups (domains, IPs) | `user_id`, `domain`, `created_at` (desc) | **Redis 7 (Alpine):** | Function | Key Pattern | Detail | |----------|-------------|--------| | Celery Broker | db 0 | Task queue (JSON serialization) | | Celery Results | db 1 | Task result storage | | Rate Limiting | `ratelimit:{endpoint}:{ip}` | Sliding-window counters (db 0) | | SSE Log Stream | `scan_logs:{scan_id}` | Max 1000 entries, 24h TTL, rpush/ltrim | Config: 512MB maxmemory, `allkeys-lru` eviction, AOF persistence (`appendfsync everysec`), RDB snapshots (900/1, 300/10, 60/10000). ## 🔧 Security Tools (19+) ### ProjectDiscovery Go Binaries (auto-downloaded at Docker build via `init_tools.py`) | # | Tool | Version | GitHub Repo | Purpose | |---|------|---------|-------------|---------| | 1 | **subfinder** | v2.6.3 | projectdiscovery/subfinder | Passive + active subdomain enumeration (recursive, all sources) | | 2 | **httpx** | v1.3.7 | projectdiscovery/httpx | HTTP probing: status codes, titles, TLS, tech detection, CDN, CNAME | | 3 | **nuclei** | v3.1.5 | projectdiscovery/nuclei | Template-based vuln scanning: CVE detection, misconfigs, exposures. Templates auto-updated on container start and persisted via Docker volume | | 4 | **dnsx** | v1.2.3 | projectdiscovery/dnsx | DNS resolution & validation: A records, TTL, resolver info | | 5 | **ffuf** | v2.1.0 | ffuf/ffuf | Web fuzzing & directory brute-force (auto-calibration, wordlist) | | 6 | **naabu** | v2.3.4 | projectdiscovery/naabu | Fast port scanning: SYN/CONNECT, top 1000 ports, rate 1000/s | | 7 | **tlsx** | v1.1.2 | projectdiscovery/tlsx | TLS certificate inspection: versions, ciphers, SAN, expiry | ### System-Installed Tools (Dockerfile apt/git/pip) | # | Tool | Install Method | Purpose | |---|------|---------------|---------| | 8 | **nmap** | apt | Port scanning + service/version detection + OS fingerprinting | | 9 | **masscan** | apt | Mass IP port scanner (ports 1–5000 + high ports, rate 1000/s) | | 10 | **nikto** | git clone → `/opt/nikto/` | Web server vulnerability scanner (misconfigs, outdated software, dangerous files) | | 11 | **testssl.sh** | git clone → `/opt/testssl/` | TLS/SSL testing: cipher suites, protocols, Heartbleed, POODLE, DROWN | | 12 | **dirsearch** | pip install | Directory/file scanner with JSON output and wordlist support | | 13 | **wafw00f** | pip install | Web Application Firewall detection and identification | | 14 | **whatweb** | system (scanning-engine reads via subprocess) | Web technology fingerprinting: server, CMS, frameworks, versions | | 15 | **whois** | apt | Domain registration and ownership lookup | | 16 | **curl** | apt | HTTP header inspection, technology detection from response headers | ### Referenced Tools (expected on system PATH, optional) | # | Tool | Purpose | |---|------|---------| | 17 | ~~**amass**~~ | ~~Passive subdomain enumeration (OWASP Amass)~~ — **REMOVED**: redundant with subfinder | | 18 | **gobuster** | Directory/file brute-forcing | | 19 | **wpscan** | WordPress vulnerability scanner (plugins, users, themes) | | 20 | **sqlmap** | SQL injection detection and exploitation | | 21 | **waybackurls** | Fetch historical URLs from Wayback Machine | | 22 | **gau** | Get All URLs from multiple web archives | ## 📡 API Endpoints ### Health Check | Method | Path | Auth | Description | |--------|------|------|-------------| | `GET` | `/health` | No | API Gateway health status | ### Authentication (10 endpoints) | Method | Path | Auth | Rate Limit | Description | |--------|------|------|------------|-------------| | `POST` | `/auth/register` | No | 10/60s | Create account + send OTP email | | `POST` | `/auth/login` | No | 20/60s | Email/password login → set JWT cookie | | `POST` | `/auth/verify-otp` | No | 10/60s | Verify 6-digit OTP → auto-login | | `POST` | `/auth/resend-otp` | No | 5/60s | Regenerate and resend OTP | | `POST` | `/auth/logout` | No | — | Clear JWT cookies | | `GET` | `/auth/me` | **Yes** | — | Get current user profile | | `PUT` | `/auth/profile` | **Yes** | — | Update user display name | | `POST` | `/auth/change-password` | **Yes** | — | Change password (requires current) | | `POST` | `/auth/forgot-password` | No | 5/60s | Send password reset OTP/token | | `POST` | `/auth/reset-password` | No | 5/60s | Reset password with OTP/token | ### Scan Management (17 endpoints) | Method | Path | Auth | Description | |--------|------|------|-------------| | `POST` | `/api/scans/OmniTricks` | **Yes** | Queue OmniTricks AI scan (Celery task) | | `GET` | `/api/scans/OmniTricks` | **Yes** | List user's OmniTricks scans (`?mode=`, `?limit=`) | | `GET` | `/api/scans/OmniTricks/:id` | **Yes** | Get single scan metadata | | `GET` | `/api/scans/OmniTricks/:id/stream` | **Yes** | SSE real-time log stream from Redis | | `GET` | `/api/scans/OmniTricks/:id/results` | **Yes** | Aggregated scan results from PostgreSQL JSONB tables | | `POST` | `/api/scans/levelwise` | **Yes** | Queue levelwise parallel scan | | `GET` | `/api/scans/levelwise/:id` | **Yes** | Levelwise scan progress and status | | `POST` | `/api/scans/ultra-fast` | **Yes** | Queue ultra-fast scan (`?mode=ultra/async`) | | `GET` | `/api/scans/ultra-fast` | **Yes** | List ultra-fast scans | | `GET` | `/api/scans/ultra-fast/:id` | **Yes** | Ultra-fast scan status | | `GET` | `/api/scans/ultra-fast/:id/results` | **Yes** | Paginated results (`?page=`, `?limit=`, `?severity=`) | | `GET` | `/rescan/stream` | **Yes** | Direct OmniTricks AI SSE stream (`?domain=`) | | `GET` | `/rescan/stream_subfinder_dnsx_httpx` | **Yes** | Subfinder pipeline SSE (`?domain=`) | | `GET` | `/scan/comprehensive` | **Yes** | Levelwise parallel SSE (`?domain=`) | | `GET` | `/api/OmniTricks/health` | **Yes** | Check scanning-engine health | | `POST` | `/api/trigger_background_scan` | **Yes** | Fire-and-forget Celery scan | | `GET` | `/api/background_scan_status/:task_id` | **Yes** | Check Celery task state | ### Asset Management (3 endpoints) | Method | Path | Auth | Description | |--------|------|------|-------------| | `POST` | `/api/assets` | **Yes** | Create asset group (company, domains, IPs, endpoints) | | `GET` | `/api/assets` | **Yes** | List all user's asset groups | | `DELETE` | `/api/assets/:id` | **Yes** | Delete asset group (owner only) | ### Results (8 endpoints) | Method | Path | Auth | Description | |--------|------|------|-------------| | `GET` | `/results` | **Yes** | Legacy scan results (`?scan_id=`) | | `GET` | `/resultssubfinder` | **Yes** | Latest subfinder scan results | | `GET` | `/resultssubfinderchart` | **Yes** | Subfinder chart data (`?scan_id=`) | | `GET` | `/recent-scan-json` | **Yes** | Recent scans JSON (`?scan_type=`, `?limit=`) | | `GET` | `/api/getPorts` | No | Open ports for a domain (`?subdomain=`) | | `GET` | `/api/getPorts_subfinder` | No | Ports from subfinder/OmniTricks scans (`?subdomain=`) | | `GET` | `/api/getFfuf_subfinder` | No | Ffuf directory results (`?subdomain=`) | | `GET` | `/api/getZapAlerts` | No | ZAP security alerts (`?subdomain=`) | ### Statistics (2 endpoints) | Method | Path | Auth | Description | |--------|------|------|-------------| | `GET` | `/api/statistics` | **Yes** | Dashboard stats: totals, trends, severity breakdown, top alerts | | `GET` | `/scan-trends` | **Yes** | Scan trend data for line charts | ### Scanning Engine Internal API (port 8888, not exposed to frontend) | Method | Path | Description | |--------|------|-------------| | `GET` | `/health` | Engine health + tool count + version | | `GET` | `/api/tools` | List all 19 available tools with categories/priorities | | `POST` | `/api/analyze` | AI target analysis: DNS, HTTP, port scan, WAF, risk, techs | | `POST` | `/api/tools/optimal` | AI-recommended tools for a target (max 12) | | `POST` | `/api/intelligence/generate-command` | Generate CLI command for any tool | | `POST` | `/api/scan/subdomain` | Execute tools against a single subdomain (parallel) | | `POST` | `/api/scan/batch` | Batch subdomain scan (stub) | | `POST` | `/api/discover/subdomains` | Discovery: subfinder + wordlist expansion | ## 🔄 Scan Workflow ┌─────────────────────────────────────────────────────────────────┐ │ USER enters domain in browser (HeroSection → navigates /scan) │ └──────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ FRONTEND: POST /api/scans/OmniTricks { domain: "example.com" } │ │ Opens SSE connection to /api/scans/OmniTricks/:id/stream │ └──────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ API GATEWAY: Creates scan in PostgreSQL (status: "queued") │ │ Dispatches Celery task → OmniTricks_scan(scan_id, domain) │ └──────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ CELERY WORKER picks up task │ │ Creates RedisLogHandler → streams logs to scan_logs:{scan_id} │ │ Creates OmniTricksManager(scanning-engine:8888) │ │ Updates PostgreSQL: status → "running" │ └──────────────────────────┬──────────────────────────────────────┘ │ ┌────────────────┼────────────────┐ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Step 1: │ │ Step 2: │ │ Step 3: │ │ ANALYZE │ │ DISCOVER │ │ SELECT TOOLS │ │ │ │ │ │ │ │ POST /api/ │ │ POST /api/ │ │ POST /api/ │ │ analyze │ │ discover/ │ │ tools/ │ │ │ │ subdomains │ │ optimal │ │ • DNS lookup │ │ │ │ │ │ • HTTP probe │ │ • subfinder │ │ • AI selects │ │ • 14 ports │ │ • subfinder │ │ best tools │ │ • WAF detect │ │ • wordlist │ │ • Per-tool │ │ • Risk score │ │ expansion │ │ params │ │ • Tech detect│ │ • Dedup │ │ • Max 12 │ │ • Classify │ │ • Max 150 │ │ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └─────────────────┼─────────────────┘ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Step 4: PARALLEL SCAN — ThreadPoolExecutor (10 workers) │ │ │ │ For EACH subdomain (up to 150): │ │ POST /api/scan/subdomain { subdomain, tools[] } │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ ThreadPoolExecutor (8 workers) runs tools in parallel: │ │ │ │ │ │ │ │ httpx ─────→ HTTP probe + tech fingerprint │ │ │ │ nuclei ────→ CVE detection + vulnerability templates │ │ │ │ nmap ──────→ Port scan + service/version detection │ │ │ │ naabu ─────→ Fast port scan │ │ │ │ dnsx ──────→ DNS records │ │ │ │ tlsx ──────→ TLS certificate inspection │ │ │ │ whatweb ───→ Technology fingerprinting │ │ │ │ ffuf ──────→ Directory brute-force │ │ │ │ nikto ─────→ Web vulnerability scan │ │ │ │ masscan ───→ Mass port scan │ │ │ │ testssl ───→ SSL/TLS testing │ │ │ │ dirsearch ─→ Directory scanning │ │ │ │ wafw00f ───→ WAF detection │ │ │ │ curl ──────→ Header analysis │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ Results merged → Risk score computed (0-100) │ │ CVEs extracted → Vulnerabilities categorized by severity │ └──────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ STORAGE & STREAMING │ │ │ │ • Each subdomain result → PostgreSQL subdomain_results table │ │ • Scan status → PostgreSQL scans (completed, total_subdomains) │ │ • Logs → Redis scan_logs:{scan_id} → SSE → Terminal component │ └──────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ FRONTEND renders results: │ │ • SubdomainCardSubfinder for each discovered subdomain │ │ • Risk score badges (CRITICAL / HIGH / MEDIUM / LOW / MINIMAL) │ │ • Expandable details: ports, techs, vulns, directories │ │ • PDF export via PDFDownloadSubfinder │ │ • Analytics in ReconDashboard (charts, history) │ └─────────────────────────────────────────────────────────────────┘ ## 🚀 Quick Start ### Prerequisites - [Docker Desktop](https://www.docker.com/products/docker-desktop/) (Windows/Mac) or Docker Engine (Linux) - Docker Compose v2+ - 4GB+ RAM available for Docker - Ports **80**, **4000**, **6379**, **8888**, **27017** available ### 1. Clone & Configure git clone cd Vulnerability_Scanner ### 2. Set Environment Variables Edit `.env` file: # ── Required ── JWT_SECRET_KEY=change-this-to-a-strong-random-string # ── Email (for OTP verification) ── MAIL_SERVER=smtp.gmail.com MAIL_PORT=587 MAIL_USE_TLS=True MAIL_USERNAME=your-email@gmail.com MAIL_PASSWORD=your-gmail-app-password # ── Auto-configured by docker-compose (override if needed) ── POSTGRES_PASSWORD=Omjee@7379 POSTGRES_URL_PASSWORD=Omjee%407379 DATABASE_URL=postgresql://vulnerscan_user:Omjee%407379@postgres:5432/deeptrustxai_db CELERY_BROKER_URL=redis://redis:6379/0 OmniTricks_URL=http://scanning-engine:8888 ### 3. Build & Launch docker compose up --build -d This will: 1. Build 5 Docker images (presentation, api-gateway, processing, scanning-engine, OmniTricks-intelligence) 2. Pull 2 official images (postgres:16-alpine, redis:7-alpine) 3. Download 7 Go security tool binaries during scanning-engine build 4. Install all Python and Node.js dependencies 5. Create PostgreSQL JSONB tables and indexes on first boot 6. Start all 7 containers on the `DeepTrustxAI-network` bridge ### 4. Access the Platform | Service | URL | |---------|-----| | **Dashboard** | http://localhost | | **API Gateway** | http://localhost:4000/health | | **Scanning Engine** | http://localhost:8888/health | ### 5. First Use 1. Open **http://localhost** → Redirects to `/login` 2. Click **Register** → Enter email + password 3. Check your email for a **6-digit OTP** code 4. Enter OTP → You're automatically logged in 5. Type a domain on the home page (e.g. `example.com`) → Click **Scan** 6. Watch **real-time scan logs** in the terminal panel as 19+ tools execute 7. View results as **subdomain cards** with risk scores, technologies, open ports, and vulnerabilities 8. Export to **PDF** or analyze on the **Dashboard** ### Common Commands # Start all services docker compose up -d # Rebuild after code changes docker compose up --build -d # Rebuild single service docker compose up -d --build scanning-engine # View live logs docker compose logs -f api-gateway docker compose logs -f celery-worker docker compose logs -f scanning-engine # Stop all services docker compose down # Stop + delete all data (⚠️ destructive) docker compose down -v # Check container status docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" ## 📦 Docker Registry (ghcr.io) All DeepTrustxAI Docker images are published to **GitHub Container Registry** (`ghcr.io`) for easy deployment on any machine without rebuilding. ### Registry Details | Item | Value | |------|-------| | **Registry** | `ghcr.io` (GitHub Container Registry) | | **Owner** | `omjee73` | | **Image Prefix** | `ghcr.io/omjee73/` | | **Version File** | `VERSION` (project root) | | **Push Script** | `registry-push.ps1` (PowerShell) | | **Packages URL** | https://github.com/omjee73?tab=packages | ### Published Images | # | Image Name | Service(s) | Size | Dockerfile | |---|------------|------------|------|------------| | 1 | `ghcr.io/omjee73/DeepTrustxAI-frontend` | presentation | ~77 MB | `services/presentation/Dockerfile` | | 2 | `ghcr.io/omjee73/DeepTrustxAI-api-gateway` | api-gateway | ~263 MB | `services/api-gateway/Dockerfile` | | 3 | `ghcr.io/omjee73/DeepTrustxAI-processing` | celery-worker, celery-beat | ~442 MB | `services/processing/Dockerfile` | | 4 | `ghcr.io/omjee73/DeepTrustxAI-scanning-engine` | scanning-engine | ~1.5 GB | `services/scanning-engine/Dockerfile` | ### Tagging Strategy Each image gets **two tags** on every push: | Tag | Example | Purpose | |-----|---------|---------| | `v` | `v1.0.0` | Immutable version tag — never overwritten | | `latest` | `latest` | Always points to the newest push | Version is read from the `VERSION` file (project root). Override with `-Version "1.1.0"`. ### Pull Images (Deploy on Any Machine) To deploy **without building** — just pull pre-built images: # Login to ghcr.io (needs a GitHub PAT with read:packages scope) echo YOUR_GITHUB_PAT | docker login ghcr.io -u omjee73 --password-stdin # Pull all images docker compose pull # Start everything docker compose up -d Or pull individual images: # Pull specific image (latest) docker pull ghcr.io/omjee73/DeepTrustxAI-frontend:latest docker pull ghcr.io/omjee73/DeepTrustxAI-api-gateway:latest docker pull ghcr.io/omjee73/DeepTrustxAI-processing:latest docker pull ghcr.io/omjee73/DeepTrustxAI-scanning-engine:latest # Pull specific version docker pull ghcr.io/omjee73/DeepTrustxAI-scanning-engine:v1.0.0 ### Push Images (registry-push.ps1) The `registry-push.ps1` PowerShell script handles building, tagging, and pushing all images. **Prerequisites:** - Docker Desktop running - GitHub Personal Access Token (PAT) with `write:packages` + `read:packages` scope - Create PAT at: https://github.com/settings/tokens/new **Available Commands:** # Step 1: Login to ghcr.io (interactive — prompts for PAT securely) .\registry-push.ps1 login # Step 2: Build + Tag + Push all 4 images .\registry-push.ps1 push # Push with a specific version (also updates VERSION file) .\registry-push.ps1 push -Version "1.1.0" # Push only one service .\registry-push.ps1 push -Service scanning-engine .\registry-push.ps1 push -Service frontend .\registry-push.ps1 push -Service api-gateway .\registry-push.ps1 push -Service processing # Build + Tag only (no push to registry) .\registry-push.ps1 build # List all local DeepTrustxAI images with tags and sizes .\registry-push.ps1 list **Script Parameters:** | Parameter | Values | Default | Description | |-----------|--------|---------|-------------| | `Action` | `push`, `login`, `build`, `list` | `push` | What operation to perform | | `-Version` | e.g. `"1.1.0"` | reads `VERSION` file | Override version tag | | `-Service` | `frontend`, `api-gateway`, `scanning-engine`, `processing` | all | Push only one service | ### Typical Workflow # 1. Make code changes # 2. Update VERSION file (e.g. 1.0.0 → 1.1.0) # 3. Login (only needed once per session) .\registry-push.ps1 login # 4. Build + Push all images .\registry-push.ps1 push # 5. Verify on GitHub # Visit: https://github.com/omjee73?tab=packages # 6. Logout (security) docker logout ghcr.io ## 🖥 VM image-pulling & required credentials When deploying to a VM (DigitalOcean droplet), images are expected to be pulled at runtime rather than preinstalled. The provided setup scripts now run `docker-compose pull` before `up`. Required credentials and tokens (short list): - DigitalOcean API Token (for Terraform/DO API & DNS automations) - SSH key pair for droplet access - GHCR / Docker registry credentials (GHCR_USER + GHCR_TOKEN) if private images - AGENT_REGISTRATION_TOKEN (server-side) for agent auto-register - AGENT_API_KEY (per-agent secret) for HMAC-signed agent requests - PostgreSQL credentials (POSTGRES_PASSWORD) - Redis access (REDIS_URL) - JWT_SECRET / JWT_SECRET_KEY for API auth - Let's Encrypt email + DNS API token (DIGITALOCEAN_TOKEN) for automated certs - WireGuard keys (server_private/server_public + client keys) — generate with `services/wireguard-setup/generate_keys.sh` Commands to ensure VM pulls images: # optional: docker login ghcr.io export GHCR_USER=youruser export GHCR_TOKEN=ghp_xxx sudo docker login ghcr.io -u "$GHCR_USER" --password-stdin <<< "$GHCR_TOKEN" # pull images and start sudo docker-compose pull --ignore-pull-failures sudo docker-compose up -d Keep secrets in the droplet environment (systemd unit or .env) and do NOT commit them to git. ### Deploy on a New Machine # 1. Clone the repo git clone https://github.com/Omjee73/Vulnerability_Scanner.git cd Vulnerability_Scanner # 2. Create .env file with your secrets cp .env.example .env # edit with your values # 3. Login to ghcr.io echo YOUR_GITHUB_PAT | docker login ghcr.io -u omjee73 --password-stdin # 4. Pull pre-built images + start docker compose pull docker compose up -d # 5. Access at http://localhost No need to build anything — all images are pre-built and pulled from the registry. ### Making Images Public By default GitHub packages are **private**. To make them public (no PAT needed to pull): 1. Go to https://github.com/omjee73?tab=packages 2. Click on each package (e.g. `DeepTrustxAI-frontend`) 3. Click **Package settings** (right sidebar) 4. Scroll to **Danger Zone** → **Change visibility** → **Public** 5. Repeat for all 4 packages Once public, anyone can pull without authentication: docker pull ghcr.io/omjee73/DeepTrustxAI-frontend:latest ## ⚙️ Environment Variables | Variable | Default | Service | Description | |----------|---------|---------|-------------| | `JWT_SECRET_KEY` | `your-secret-key-here` | api-gateway | **Change this.** JWT signing key | | `DATABASE_URL` | `postgresql://vulnerscan_user:...@postgres:5432/deeptrustxai_db` | api-gateway, processing | PostgreSQL connection | | `CELERY_BROKER_URL` | `redis://redis:6379/0` | api-gateway, processing | Celery task queue | | `CELERY_RESULT_BACKEND` | `redis://redis:6379/1` | processing | Celery result storage | | `REDIS_URL` | `redis://redis:6379/0` | api-gateway, processing | Rate limits + SSE logs | | `OmniTricks_URL` | `http://scanning-engine:8888` | api-gateway, processing | Scanning engine API | | `MAIL_SERVER` | `smtp.gmail.com` | api-gateway | SMTP server for OTP emails | | `MAIL_PORT` | `587` | api-gateway | SMTP port | | `MAIL_USE_TLS` | `True` | api-gateway | Enable TLS for email | | `MAIL_USERNAME` | — | api-gateway | SMTP username | | `MAIL_PASSWORD` | — | api-gateway | SMTP app password | | `AUTO_SCAN_DOMAIN` | `iitm.ac.in` | processing | Default auto-scan target | | `TOOLS_DIR` | `/app/tools` | scanning-engine | Security tool binary directory | | `COHERE_API_KEY` | — | api-gateway | Optional: Cohere LLM for risk analysis | | `VITE_API_URL` | `http://localhost:4000` | presentation (build-time) | API URL baked into frontend bundle | ## 🎨 Frontend Pages & Components ### Route Map | Path | Component | Access | Description | |------|-----------|--------|-------------| | `/login` | `Login` | Public | Split-screen email/password login | | `/register` | `Register` | Public | Account creation form | | `/forgot-password` | `ForgotPassword` | Public | Password reset request | | `/verify-otp` | `VerifyOtp` | Public | 6-digit OTP input | | `/auth/verify` | `VerificationPage` | Public | Email verification callback | | `/` | `HeroSection` | Protected | Home page — animated domain input | | `/scan` | `OmniTricksScan` | Protected | Core scan page: SSE terminal + results | | `/dashboard` | `ReconDashboard` | Protected | Analytics: charts, trends, scan history | | `/assets` | `AssetForm` | Protected | Asset CRUD: companies, domains, IPs | | `/about` | `About` | Protected | 8-step workflow visualization | | `/profile` | `Profile` | Protected | Edit profile + change password | ### Component Responsibilities | Component | File | Purpose | |-----------|------|---------| | **HeroSection** | `components/HeroSection.tsx` | Landing page: animated title with typewriter, domain input, navigates to `/scan?domain=` | | **OmniTricksScan** | `components/scan/OmniTricksScan.tsx` | Main scan orchestrator: POST scan → open SSE → render Terminal → fetch & display results | | **Terminal** | `components/terminal/Terminal.tsx` | Portal-mounted live log viewer with macOS-style title bar, auto-scroll, SSE consumption | | **SubdomainCardSubfinder** | `components/scan/SubdomainCardSubfinder.tsx` | Compact result card per subdomain: IP, status, risk badge, ports, tech tags | | **subdomainDetailsSubfinder** | `components/scan/subdomainDetailsSubfinder.tsx` | Expanded detail view: httpx, nuclei, nmap, ffuf data | | **DirectoryTree** | `components/scan/DirectoryTree.tsx` | Tree view for directory brute-force results (ffuf/dirsearch) | | **PDFDownloadSubfinder** | `components/scan/PDFDownloadSubfinder.tsx` | Generates formatted PDF report from scan results (jspdf) | | **ReconDashboard** | `components/Statistics/ReconDashboard.tsx` | Analytics: Recharts LineChart, severity pie, scan history table, stat cards | | **AssetForm** | `components/asset/AssetForm.tsx` | CRUD interface for company assets: domains, IPs, endpoints, API keys | | **Layout** | `components/layout/Layout.tsx` | Page shell: Navbar + scan-line animation + grid overlay + footer | | **Navbar** | `components/layout/Navbar.tsx` | Navigation links, theme color picker, user dropdown menu | | **Login** | `components/auth/Login.tsx` | Split-screen login with animated background | | **Register** | `components/auth/Register.tsx` | Registration with email + password fields | | **ProtectedRoute** | `components/auth/ProtectedRoute.tsx` | Auth guard: shows spinner while loading, redirects to `/login` if unauthenticated | | **Profile** | `pages/Profile.tsx` | Update display name, change password, view account info | | **About** | `pages/About.tsx` | 8-step workflow flowchart, feature cards, platform description | | **LoadingSpinner** | `components/ui/LoadingSpinner.tsx` | Lucide `Loader2` spinning animation | ### Authentication Flow Register → POST /auth/register → OTP sent to email ↓ /verify-otp → POST /auth/verify-otp → JWT set in HTTP-only cookie → redirect / ↓ Every page load → AuthProvider calls GET /auth/me → hydrates user state ↓ ProtectedRoute checks isAuthenticated → allows or redirects to /login ↓ Logout → POST /auth/logout → clears JWT cookie → redirect /login ### Theme System 6 color palettes persisted in `localStorage`, applied via CSS custom properties on `:root`: | Palette | Primary | Secondary | Accent | |---------|---------|-----------|--------| | Blue | `#3B82F6` | `#1E40AF` | `#60A5FA` | | Green | `#10B981` | `#047857` | `#34D399` | | **Purple** (default) | `#8B5CF6` | `#5B21B6` | `#A78BFA` | | Red | `#EF4444` | `#991B1B` | `#F87171` | | Gray | `#6B7280` | `#374151` | `#9CA3AF` | | Black | `#1F2937` | `#111827` | `#4B5563` | Components reference themes via `var(--color-primary)`, `var(--color-background)`, etc. ## 🗄 Database Schema ### PostgreSQL Document Tables **`users`** { "_id": "text-id", "email": "user@example.com", "password_hash": "$2b$12$...", "organization": "example.com", "name": "user", "created_at": "2026-03-08T00:00:00Z", "last_login": "2026-03-08T12:00:00Z", "is_verified": true, "otp": "123456", "otp_expires": "2026-03-08T00:10:00Z" } **`scans`** { "_id": "uuid-string", "domain": "example.com", "user_id": "text-id", "status": "completed", "scan_type": "OmniTricks", "scan_engine": "OmniTricks-ai", "created_at": "2026-03-08T00:00:00Z", "started_at": "2026-03-08T00:00:01Z", "completed_at": "2026-03-08T00:05:00Z", "total_subdomains": 42 } **`subdomain_results`** { "_id": "text-id", "scan_id": "uuid-string", "domain": "example.com", "subdomain": "api.example.com", "ip": "192.168.1.1", "status_code": 200, "technologies": ["nginx", "React"], "ports": [80, 443, 8080], "vulnerabilities": [ { "name": "CVE-2024-1234", "severity": "high", "description": "..." } ], "risk_analysis": { "risk_score": 72, "risk_level": "HIGH" }, "scanned_at": "2026-03-08T00:02:00Z" } **`assets`** { "_id": "text-id", "user_id": "text-id", "company_name": "Example Corp", "domains": ["example.com", "example.org"], "ip_addresses": ["192.168.1.0/24"], "endpoints": ["https://api.example.com/v1"], "created_at": "2026-03-08T00:00:00Z" } ## 🌐 Nginx Reverse Proxy The presentation container serves the React SPA and proxies API requests to the api-gateway. | Location | Target | Special Config | |----------|--------|----------------| | `/` | SPA static files | `try_files $uri $uri/ /index.html` (client routing) | | `/api/` | `http://api-gateway:4000` | SSE: `proxy_buffering off`, `proxy_read_timeout 600s` | | `/auth/` | `http://api-gateway:4000` | Standard proxy headers (Host, X-Real-IP, X-Forwarded-For) | | `/rescan/` | `http://api-gateway:4000` | SSE: `proxy_buffering off`, `proxy_read_timeout 1200s` | | `/results` | `http://api-gateway:4000` | Standard proxy | | `/resultssubfinder` | `http://api-gateway:4000` | Standard proxy | | `/resultssubfinderchart` | `http://api-gateway:4000` | Standard proxy | | `/scan-trends` | `http://api-gateway:4000` | Standard proxy | **Gzip:** Enabled for text/css/json/javascript/xml (min 256 bytes). ## 🐛 Troubleshooting | Issue | Solution | |-------|----------| | **Port 80 already in use** | Stop other web servers: `netstat -ano \| findstr :80` then kill the PID | | **Docker build fails on scanning-engine** | Network issue downloading Go binaries. `init_tools.py` has `\|\| true` to continue gracefully | | **Frontend shows "unhealthy"** | Healthcheck uses `127.0.0.1` — check `docker logs DeepTrustxAI-frontend` | | **Celery tasks stuck in "queued"** | Check worker: `docker logs DeepTrustxAI-celery-worker` — may need Redis connectivity | | **OTP email not received** | Verify `MAIL_USERNAME` and `MAIL_PASSWORD` in `.env`. Gmail requires [App Password](https://support.google.com/accounts/answer/185833) | | **PostgreSQL connection refused** | Wait 20s for healthcheck. Check: `docker logs DeepTrustxAI-postgres` | | **Rate limited (HTTP 429)** | Auth endpoints have sliding-window limits — wait 60s and retry | | **No scan results returned** | Check scanning-engine: `docker logs DeepTrustxAI-scanning-engine` | | **WSL2 DNS resolution fails** | Docker compose sets `dns: [1.1.1.1, 8.8.8.8]` — check host network config | | **TS errors in VS Code** | `node_modules` not installed locally. Run `cd services/presentation && npm install` for IDE support | ### Useful Debug Commands # Container health overview docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" # Tail service logs docker compose logs -f api-gateway docker compose logs -f scanning-engine docker compose logs -f celery-worker # Shell into a container docker exec -it DeepTrustxAI-api-gateway /bin/bash docker exec -it DeepTrustxAI-scanning-engine /bin/bash # PostgreSQL queries docker exec -it DeepTrustxAI-postgres psql -U vulnerscan_user -d deeptrustxai_db -c "select count(*) from users;" docker exec -it DeepTrustxAI-postgres psql -U vulnerscan_user -d deeptrustxai_db -c "select doc from scans order by doc->>'created_at' desc limit 5;" # Redis health docker exec -it DeepTrustxAI-redis redis-cli ping docker exec -it DeepTrustxAI-redis redis-cli info memory # Celery inspection docker exec -it DeepTrustxAI-celery-worker celery -A tasks inspect active docker exec -it DeepTrustxAI-celery-worker celery -A tasks inspect registered # Test scanning engine directly curl http://localhost:8888/health | python -m json.tool curl http://localhost:8888/api/tools | python -m json.tool # Test API gateway curl http://localhost:4000/health | python -m json.tool ## 📄 License MIT License — see [LICENSE](LICENSE) for details.

**Built for security researchers and penetration testers** *DeepTrustxAI Vulnerability Scanner — 6-Layer Microservices Architecture*

标签：Celery, Flask, React, Syscalls, 安全侦察, 实时处理, 密码管理, 微服务架构, 插件系统, 漏洞扫描, 自动化渗透测试