Omjee73/Vulnerability_Scanner

## 📋 目录 - [Features](#-features) - [Architecture](#-architecture) - [Complete Folder Structure](#-complete-folder-structure) - [Layer-by-Layer Breakdown](#-layer-by-layer-breakdown) - [Layer 1 — Presentation](#layer-1--presentation-react--nginx) - [Layer 2 — API Gateway](#layer-2--api-gateway-flask) - [Layer 3 — Processing](#layer-3--processing-celery) - [Layer 4 — Scanning Engine](#layer-4--scanning-engine-hexstrike) - [Layer 5 — Intelligence (Disabled)](#layer-5--intelligence-disabled) - [Layer 6 — Data Layer](#layer-6--data-layer-mongodb--redis) - [Security Tools (19+)](#-security-tools-19) - [API Endpoints](#-api-endpoints) - [Scan Workflow](#-scan-workflow) - [Quick Start](#-quick-start) - [Docker Registry (ghcr.io)](#-docker-registry-ghcrio) - [Environment Variables](#-environment-variables) - [Frontend Pages & Components](#-frontend-pages--components) - [Database Schema](#-database-schema) - [Nginx Reverse Proxy](#-nginx-reverse-proxy) - [Troubleshooting](#-troubleshooting) ## ✨ 特性 - **19+ Security Tools** — subfinder, nuclei, httpx, nmap, naabu, masscan, nikto, ffuf, dirsearch, whatweb, dnsx, tlsx, testssl, wafw00f, gobuster, sqlmap, wpscan, curl, gau, waybackurls, dig - **AI-Powered Analysis** — Automatic target classification, risk scoring (0–100), attack vector identification, optimal tool selection per target - **Real-Time Streaming** — Server-Sent Events (SSE) push live scan logs to a terminal UI as tools execute - **3 Scan Modes** — HexStrike AI (full), Levelwise Parallel, Ultra-Fast - **Parallel Execution** — 10 concurrent subdomain scans, each running 8–14 tools simultaneously - **Subdomain Discovery** — Subfinder (recursive + all sources) + DNS wordlist expansion - **PDF Export** — Download scan results as formatted PDF reports - **Asset Management** — Track companies, domains, IPs, and endpoints - **JWT Auth + OTP** — Email-based OTP verification, password reset, session cookies - **Rate Limiting** — Redis-backed sliding-window rate limiter on auth endpoints - **6 Color Themes** — Blue, Green, Purple, Red, Gray, Black - **Analytics Dashboard** — Scan trends, severity distribution, vulnerability counts via Recharts - **Docker-First** — Single `docker compose up --build` deploys everything ## 🏗 架构 ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ CYSTAR VULNERABILITY SCANNER │ │ 6-Layer Microservices Architecture │ └──────────────────────────────────────────────────────────────────────────┘ ┌─────────────────────────────────────────────────────────────────────────┐ │ Layer 1: PRESENTATION │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ React 19 + TypeScript + Vite 6 + TailwindCSS │ │ │ │ Built → static assets → served by Nginx on :80 │ │ │ │ Reverse proxy: /api/ /auth/ /rescan/ → api-gateway:4000 │ │ │ └─────────────────────────────────────────────────────────────────┘ │ └────────────────────────────────┬────────────────────────────────────────┘ │ HTTP / SSE ┌────────────────────────────────▼────────────────────────────────────────┐ │ Layer 2: API GATEWAY │ │ ┌─────────────────────────────────────────────────────────────────┐ │ │ │ Flask 2.2 + Gunicorn (4 workers) on :4000 │ │ │ │ JWT Auth (cookies) │ Rate Limiting │ CORS │ Flask-Mail │ │ │ │ Routes: auth, scans, assets, results, stats │ │ │ │ Dispatches Celery tasks │ Direct SSE streaming │ │ │ └──────────┬───────────────────────────────┬──────────────────────┘ │ └─────────────┼───────────────────────────────┼──────────────────────────-─┘ │ Celery Tasks │ Direct HTTP ┌─────────────▼─────────────────┐ ┌──────────▼──────────────────────────┐ │ Layer 3: PROCESSING │ │ Layer 4: SCANNING ENGINE │ │ ┌───────────────────────┐ │ │ ┌────────────────────────────┐ │ │ │ Celery Worker (4 conc) │ │ │ │ Flask + Gunicorn on :8888 │ │ │ │ Celery Beat (scheduler)│ │ │ │ 19+ security tools │ │ │ │ HexStrikeManager │─────┼──│→ │ AI analysis engine │ │ │ │ Redis log streaming │ │ │ │ Tool orchestration │ │ │ └───────────────────────┘ │ │ │ Parallel execution (8 thr) │ │ └────────────────────────────────┘ │ └────────────────────────────┘ │ └────────────────────────────────────-─┘ │ │ ┌─────────────▼────────────────────────────────▼──────────────────────────┐ │ Layer 6: DATA LAYER │ │ ┌──────────────────────┐ ┌───────────────────────────────────────┐ │ │ │ MongoDB 7.0 on :27017│ │ Redis 7 (Alpine) on :6379 │ │ │ │ DB: subdomain_scanner│ │ Celery broker (db 0) │ │ │ │ 6 collections │ │ Celery results (db 1) │ │ │ │ Indexed for perf │ │ SSE log streaming │ │ │ └──────────────────────┘ │ Rate limiting counters │ │ │ └───────────────────────────────────────┘ │ └──────────────────────────────────────────────────────────────────────────┘ ``` ### 服务连接图 ``` Browser (:80) │ ├── Static Assets ──→ Nginx (presentation container) │ ├── /api/* ──────────→ Nginx proxy ──→ api-gateway (:4000) │ │ │ ├── MongoDB (reads/writes scans, users, assets) │ ├── Redis (rate limiting, SSE log reads) │ │ │ ├── [Celery Task] ──→ celery-worker │ │ │ │ │ ├── HexStrikeManager HTTP Client │ │ │ └── scanning-engine (:8888) │ │ │ ├── subfinder, nuclei, httpx... │ │ │ └── 19+ tools in parallel │ │ │ │ │ ├── MongoDB (store results) │ │ └── Redis (push SSE logs) │ │ │ └── [Direct SSE] ──→ scanning-engine (:8888) │ ├── /auth/* ─────────→ Nginx proxy ──→ api-gateway (:4000) │ └── /rescan/* ───────→ Nginx proxy ──→ api-gateway (:4000) ──→ SSE stream ``` ### Docker Compose 服务 | # | Service | Container Name | Port | Image | Role | |---|---------|---------------|------|-------|------| | 1 | `presentation` | cystar-frontend | **80** | node:22 → nginx:1.27 | React SPA + Nginx reverse proxy | | 2 | `api-gateway` | cystar-api-gateway | **4000** | python:3.12-slim | Flask REST API + JWT auth | | 3 | `celery-worker` | cystar-celery-worker | — | python:3.12-slim | Async scan task execution | | 4 | `celery-beat` | cystar-celery-beat | — | python:3.12-slim | Scheduled task runner | | 5 | `scanning-engine` | cystar-scanning-engine | **8888** | python:3.12-slim | HexStrike tool orchestrator | | 6 | `mongo` | cystar-mongo | **27017** | mongo:7.0 | Primary database | | 7 | `redis` | cystar-redis | **6379** | redis:7-alpine | Task broker + cache | Network: `cystar-network` (bridge). Volumes: `mongo-data`, `redis-data`, `nuclei-templates`, `scanning-data`. ## 📁 完整文件夹结构 ``` Vulnerability_Scanner/ │ ├── docker-compose.yml # 7-service orchestration (6 layers) ├── .env # Environment variables (secrets, API keys) ├── .gitignore ├── .dockerignore ├── README.md # ← This file ├── VERSION # Current release version (e.g. 1.0.0) ├── registry-push.ps1 # PowerShell script: build, tag, push images to ghcr.io │ ├── services/ │ │ │ ├── presentation/ # ── LAYER 1: React Frontend + Nginx ── │ │ ├── Dockerfile # Multi-stage: node:22-alpine (build) → nginx:1.27-alpine (serve) │ │ ├── nginx.conf # SPA routing + reverse proxy to api-gateway │ │ ├── package.json # React 19.1, Vite 6.3.5, TailwindCSS, 30+ deps │ │ ├── package-lock.json │ │ ├── index.html # Vite entry point │ │ ├── vite.config.ts │ │ ├── tsconfig.json │ │ ├── tsconfig.app.json │ │ ├── tsconfig.node.json │ │ ├── tailwind.config.js │ │ ├── tailwind.config.cjs │ │ ├── postcss.config.cjs │ │ ├── eslint.config.js │ │ ├── public/ │ │ │ ├── vite.svg │ │ │ ├── grid.svg │ │ │ ├── iitm.jpg │ │ │ └── IIT_Madras_Logo.svg.png │ │ └── src/ │ │ ├── main.tsx # Entry: AuthProvider → ThemeProvider → App │ │ ├── App.tsx # React Router v7 — all route definitions │ │ ├── App.css # Global animations, gradients, scan-line effects │ │ ├── index.css # Tailwind directives + CSS custom properties │ │ ├── vite-env.d.ts # Vite type declarations │ │ │ │ │ ├── config/ │ │ │ └── api.ts # API_BASE_URL from VITE_API_URL env var │ │ │ │ │ ├── context/ │ │ │ ├── AuthContext.tsx # JWT session: login/logout/register, user state │ │ │ └── ThemeContext.tsx # 6 color palettes via CSS custom properties │ │ │ │ │ ├── types/ │ │ │ └── subdomain.ts # TS interfaces: Subdomain, Vulnerability, TrendData │ │ │ │ │ ├── pages/ │ │ │ ├── About.tsx # 8-step workflow visualization │ │ │ └── Profile.tsx # Edit name + change password │ │ │ │ │ ├── components/ │ │ │ ├── HeroSection.tsx # Landing page — animated domain input │ │ │ │ │ │ │ ├── auth/ │ │ │ │ ├── Login.tsx # Split-screen email/password login │ │ │ │ ├── Register.tsx # Registration form │ │ │ │ ├── ForgotPassword.tsx # Password reset request │ │ │ │ ├── VerificationPage.tsx # Email verification callback │ │ │ │ ├── verify-otp.tsx # 6-digit OTP input │ │ │ │ └── ProtectedRoute.tsx # Auth guard (redirect to /login) │ │ │ │ │ │ │ ├── layout/ │ │ │ │ ├── Layout.tsx # Shell: Navbar + visual effects + footer │ │ │ │ └── Navbar.tsx # Nav links, theme picker, user menu │ │ │ │ │ │ │ ├── scan/ │ │ │ │ ├── HexStrikeScan.tsx # Core scan page: POST scan → SSE → results │ │ │ │ ├── SubdomainCardSubfinder.tsx # Individual subdomain result card │ │ │ │ ├── subdomainDetailsSubfinder.tsx # Expanded subdomain details │ │ │ │ ├── DirectoryTree.tsx # Directory brute-force results tree │ │ │ │ └── PDFDownloadSubfinder.tsx # Export scan results to PDF (jspdf) │ │ │ │ │ │ │ ├── Statistics/ │ │ │ │ └── ReconDashboard.tsx # Charts: scan trends, severity, history │ │ │ │ │ │ │ ├── terminal/ │ │ │ │ └── Terminal.tsx # Live SSE log viewer (portal, macOS style) │ │ │ │ │ │ │ ├── asset/ │ │ │ │ └── AssetForm.tsx # CRUD: companies, domains, IPs, endpoints │ │ │ │ │ │ │ └── ui/ │ │ │ └── LoadingSpinner.tsx # Spinning loader icon │ │ │ │ │ └── assets/ │ │ └── react.svg │ │ │ ├── api-gateway/ # ── LAYER 2: Flask REST API ── │ │ ├── Dockerfile # python:3.12-slim, gunicorn 4 workers, port 4000 │ │ ├── requirements.txt # Flask 2.2.5, celery, pymongo, redis, flask-mail, etc. │ │ └── app/ │ │ ├── __init__.py │ │ ├── main.py # Flask app factory, blueprint registration, CORS, JWT │ │ ├── tasks.py # Celery task stubs (dispatched to processing worker) │ │ ├── hexstrike_manager.py # HTTP client → scanning-engine (connection pooling) │ │ │ │ │ ├── middleware/ │ │ │ ├── __init__.py │ │ │ └── rate_limit.py # Redis sliding-window rate limiter + domain sanitizer │ │ │ │ │ ├── models/ │ │ │ ├── __init__.py │ │ │ └── user.py # User model: email, password, OTP, reset tokens │ │ │ │ │ └── routes/ │ │ ├── __init__.py │ │ ├── auth.py # 10 endpoints: register, login, OTP, password flows │ │ ├── scans.py # 17 endpoints: create/list/stream scans, SSE streaming │ │ ├── assets.py # 3 endpoints: CRUD asset groups │ │ ├── results.py # 8 endpoints: scan results, ports, ffuf, ZAP alerts │ │ └── stats.py # 2 endpoints: dashboard statistics + trends │ │ │ ├── processing/ # ── LAYER 3: Celery Workers ── │ │ ├── Dockerfile # python:3.12-slim, celery worker --concurrency=4 │ │ ├── requirements.txt # celery, pymongo, redis, requests, aiohttp │ │ ├── tasks.py # 3 Celery tasks: hexstrike, levelwise, ultra-fast │ │ ├── hexstrike_manager.py # HTTP client → scanning-engine (ThreadPoolExecutor) │ │ └── utils/ │ │ ├── __init__.py │ │ ├── redis_log_handler.py # Custom logging.Handler → Redis rpush for SSE │ │ └── ollama_risk_analyzer.py # AI risk analysis (DISABLED — entirely commented out) │ │ │ ├── scanning-engine/ # ── LAYER 4: HexStrike Tool Orchestrator ── │ │ ├── Dockerfile # python:3.12-slim + nmap, masscan, nikto, naabu, testssl │ │ ├── requirements.txt # flask, gunicorn, aiohttp, requests │ │ ├── init_tools.py # Build-time script: downloads 7 Go binaries │ │ ├── tool_tracker.py # Standalone tool version & status tracker (runs on startup) │ │ ├── entrypoint.sh # Startup: nuclei template update → tool tracker → gunicorn │ │ ├── app/ │ │ │ ├── __init__.py │ │ │ └── hexstrike_server.py # ~2200 lines: 8 Flask routes, AI engine, 19 tool runners │ │ ├── utils/ │ │ │ └── tool_manager.py # ToolManager: downloads ProjectDiscovery Go binaries │ │ └── wordlists/ │ │ └── common.txt # Directory fuzzing wordlist (ffuf, dirsearch, gobuster) │ │ │ ├── hexstrike-intelligence/ # ── LAYER 5: AI Analysis (DISABLED) ── │ │ ├── Dockerfile # python:3.12-slim, gunicorn on :8889 │ │ ├── requirements.txt │ │ └── app/ │ │ ├── __init__.py │ │ └── intelligence_server.py # Ollama/Mistral integration (not currently active) │ │ │ └── data/ # ── LAYER 6: Database Configuration ── │ ├── mongo/ │ │ └── init-db.js # Creates 6 collections + 11 indexes on first boot │ └── redis/ │ └── redis.conf # 512MB maxmemory, AOF persistence, allkeys-lru │ ├── Backend/ # Legacy monolith (pre-microservices, not used by Docker) └── Frontend/ # Legacy monolith (pre-microservices, not used by Docker) ``` ## 🔬 分层拆解 ### 第 1 层 — 表示层 (React + Nginx) | Item | Detail | |------|--------| | **Framework** | React 19.1 + TypeScript 5.8 | | **Bundler** | Vite 6.3.5 (HMR in dev, optimized build for prod) | | **Styling** | TailwindCSS 3.4 + CSS custom properties for themes | | **Icons** | lucide-react 0.514 | | **Charts** | Recharts 2.15 (LineChart, BarChart, PieChart) | | **Routing** | React Router v7.6 (client-side) | | **SSE Client** | event-source-polyfill (supports credentials/cookies) | | **PDF Export** | jspdf 3.0 | | **Maps** | Leaflet 1.9 + react-leaflet 5.0 + mapbox-gl 3.12 | | **Animations** | framer-motion 12.17 + typewriter-effect + tsparticles | | **UI Components** | Radix UI (accordion, dialog, tabs, tooltip, slot) | | **Notifications** | react-hot-toast + sweetalert2 | | **Build** | Multi-stage Docker: `node:22-alpine` → `nginx:1.27-alpine` | | **Port** | **80** (Nginx) | ### 第 2 层 — API 网关 (Flask) | Item | Detail | |------|--------| | **Framework** | Flask 2.2.5 | | **Server** | Gunicorn (4 workers, 600s timeout, keep-alive 5s) | | **Auth** | flask-jwt-extended 4.3.1 (HTTP-only cookies, CSRF disabled) | | **Database Client** | PyMongo 4.6.1 → MongoDB `subdomain_scanner` | | **Cache/Broker** | Redis 5.0.1 (rate limiting, Celery broker connection) | | **Email** | Flask-Mail 0.10.0 via Gmail SMTP (OTP delivery) | | **Rate Limiting** | Custom Redis sliding-window decorator | | **Port** | **4000** | **Blueprints Registered:** | Blueprint | Prefix | Endpoints | Purpose | |-----------|--------|-----------|---------| |auth_bp` | `/auth` | 10 | Registration, login, OTP, password flows | | `scans_bp` | mixed | 17 | Scan CRUD, SSE streaming, task dispatch | | `assets_bp` | `/api` | 3 | Asset group management | | `results_bp` | mixed | 8 | Scan result queries, port data, ZAP alerts | | `stats_bp` | mixed | 2 | Dashboard statistics and trends | ### 第 3 层 — 处理层 (Celery) | Item | Detail | |------|--------| | **Engine** | Celery 5.3.1 | | **Worker Concurrency** | 4 processes | | **Beat** | Separate container for scheduled tasks | | **Broker** | Redis db 0 | | **Result Backend** | Redis db 1 | | **Scan Client** | HexStrikeManager → HTTP to scanning-engine | | **Log Streaming** | RedisLogHandler → `scan_logs:{scan_id}` (max 1000, 24h TTL) | **Active Celery Tasks:** | Task Name | Description | |-----------|-------------| | `hexstrike_scan` | Full AI scan: analyze → discover subdomains → select tools → parallel execution of 10+ tools per subdomain → store results in MongoDB → stream logs to Redis | | `levelwise_parallel_scan` | Same pipeline but tracks `levels_completed: 3` for progress UI | | `hexstrike_ultra_parallel_scan` | Fast-path alias that delegates to `hexstrike_scan` | ### 第 4 层 — 扫描引擎 (HexStrike) | Item | Detail | |------|--------| | **Framework** | Flask 3.0 | | **Server** | Gunicorn (4 workers, 8 threads, 300s timeout) | | **Binary Tools** | 7 Go binaries (ProjectDiscovery) downloaded at Docker build | | **System Tools** | nmap, masscan, nikto, testssl, dirsearch, wafw00f, whois | | **Nuclei Templates** | Auto-updated on every container start via `entrypoint.sh`, persisted in `nuclei-templates` volume | | **Parallelism** | ThreadPoolExecutor (8 workers per subdomain scan) | | **AI Engine** | Target classification, risk scoring, tech detection, tool selection | | **Port** | **8888** | ### 第 5 层 — 情报（已禁用） Ollama + Mistral-based AI risk analysis. Commented out in `docker-compose.yml` and all processing tasks. When enabled, would run on port **8889** and provide: - Natural-language vulnerability assessments per subdomain - Batch risk analysis across scan results - Automatic model management (pull, health checks) ### 第 6 层 — 数据层 (MongoDB + Redis) **MongoDB 7.0** — Database: `subdomain_scanner` | Collection | Purpose | Key Indexes | |------------|---------|-------------| | `users` | User accounts (email, password, OTP) | `email` (unique), `is_verified` | | `scans` | Scan metadata & lifecycle status | `(user_id, created_at)` desc, `status`, `domain`, `scan_type` | | `subdomain_results` | Per-subdomain tool findings & risk scores | `scan_id`, `subdomain`, `risk_score` (desc) | | `scan_results` | Legacy knockpy scan results | `scan_id`, `domain`, `created_at` (desc) | | `scan_results_subfinder` | Subfinder-pipeline scan results | `scan_id`, `domain` | | `assets` | Company asset groups (domains, IPs) | `user_id`, `domain`, `created_at` (desc) | **Redis 7 (Alpine):** | Function | Key Pattern | Detail | |----------|-------------|--------| | Celery Broker | db 0 | Task queue (JSON serialization) | | Celery Results | db 1 | Task result storage | | Rate Limiting | `ratelimit:{endpoint}:{ip}` | Sliding-window counters (db 0) | | SSE Log Stream | `scan_logs:{scan_id}` | Max 1000 entries, 24h TTL, rpush/ltrim | Config: 512MB maxmemory, `allkeys-lru` eviction, AOF persistence (`appendfsync everysec`), RDB snapshots (900/1, 300/10, 60/10000). ## 🔧 安全工具 (19+) ### ProjectDiscovery Go 二进制文件（通过 `init_tools.py` 在 Docker 构建时自动下载） | # | Tool | Version | GitHub Repo | Purpose | |---|------|---------|-------------|---------| | 1 | **subfinder** | v2.6.3 | projectdiscovery/subfinder | Passive + active subdomain enumeration (recursive, all sources) | | 2 | **httpx** | v1.3.7 | projectdiscovery/httpx | HTTP probing: status codes, titles, TLS, tech detection, CDN, CNAME | | 3 | **nuclei** | v3.1.5 | projectdiscovery/nuclei | Template-based vuln scanning: CVE detection, misconfigs, exposures. Templates auto-updated on container start and persisted via Docker volume | | 4 | **dnsx** | v1.2.3 | projectdiscovery/dnsx | DNS resolution & validation: A records, TTL, resolver info | | 5 | **ffuf** | v2.1.0 | ffuf/ffuf | Web fuzzing & directory brute-force (auto-calibration, wordlist) | | 6 | **naabu** | v2.3.4 | projectdiscovery/naabu | Fast port scanning: SYN/CONNECT, top 1000 ports, rate 1000/s | | 7 | **tlsx** | v1.1.2 | projectdiscovery/tlsx | TLS certificate inspection: versions, ciphers, SAN, expiry | ### 系统安装工具 (Dockerfile apt/git/pip) | # | Tool | Install Method | Purpose | |---|------|---------------|---------| | 8 | **nmap** | apt | Port scanning + service/version detection + OS fingerprinting | | 9 | **masscan** | apt | Mass IP port scanner (ports 1–5000 + high ports, rate 1000/s) | | 10 | **nikto** | git clone → `/opt/nikto/` | Web server vulnerability scanner (misconfigs, outdated software, dangerous files) | | 11 | **testssl.sh** | git clone → `/opt/testssl/` | TLS/SSL testing: cipher suites, protocols, Heartbleed, POODLE, DROWN | | 12 | **dirsearch** | pip install | Directory/file scanner with JSON output and wordlist support | | 13 | **wafw00f** | pip install | Web Application Firewall detection and identification | | 14 | **whatweb** | system (scanning-engine reads via subprocess) | Web technology fingerprinting: server, CMS, frameworks, versions | | 15 | **whois** | apt | Domain registration and ownership lookup | | 16 | **curl** | apt | HTTP header inspection, technology detection from response headers | ### 引用工具 (期望在系统 PATH 中，可选) | # | Tool | Purpose | |---|------|---------| | 17 | ~~**amass**~~ | ~~Passive subdomain enumeration (OWASP Amass)~~ — **REMOVED**: redundant with subfinder | | 18 | **gobuster** | Directory/file brute-forcing | | 19 | **wpscan** | WordPress vulnerability scanner (plugins, users, themes) | | 20 | **sqlmap** | SQL injection detection and exploitation | | 21 | **waybackurls** Fetch historical URLs from Wayback Machine | | 22 | **gau** | Get All URLs from multiple web archives | ## 📡 API 端点 ### 健康检查 | Method | Path | Auth | Description | |--------|------|------|-------------| | `GET` | `/health` | No | API Gateway health status | ### 身份验证 (10 个端点) | Method | Path | Auth | Rate Limit | Description | |--------|------|------|------------|-------------| | `POST` | `/auth/register` | No | 10/60s | Create account + send OTP email | | `POST` | `/auth/login` | No | 20/60s | Email/password login → set JWT cookie | | `POST` | `/auth/verify-otp` | No | 10/60s | Verify 6-digit OTP → auto-login | | `POST` | `/auth/resend-otp` | No | 5/60s | Regenerate and resend OTP | | `POST` | `/auth/logout` | No | — | Clear JWT cookies | | `GET` | `/auth/me` | **Yes** | — | Get current user profile | | `PUT` | `/auth/profile` | **Yes** | — | Update user display name | | `POST` | `/auth/change-password` | **Yes** | — | Change password (requires current) | | `POST` | `/auth/forgot-password` | No | 5/60s | Send password reset OTP/token | | `POST` | `/auth/reset-password` | No | 5/60s | Reset password with OTP/token | ### 扫描管理 (17 个端点) | Method | Path | Auth | Description | |--------|------|------|-------------| | `POST` | `/api/scans/hexstrike` | **Yes** | Queue HexStrike AI scan (Celery task) | | `GET` | `/api/scans/hexstrike` | **Yes** | List user's HexStrike scans (`?mode=`, `?limit=`) | | `GET` | `/api/scans/hexstrike/:id` | **Yes** | Get single scan metadata | | `GET` | `/api/scans/hexstrike/:id/stream` | **Yes** | SSE real-time log stream from Redis | | `GET` | `/api/scans/hexstrike/:id/results` | **Yes** | Aggregated scan results (MongoDB pipeline) | | `POST` | `/api/scans/levelwise` | **Yes** | Queue levelwise parallel scan | | `GET` | `/api/scans/levelwise/:id` | **Yes** | Levelwise scan progress and status | | `POST` | `/api/scans/ultra-fast` | **Yes** | Queue ultra-fast scan (`?mode=ultra/async`) | | `GET` | `/api/scans/ultra-fast` | **Yes** | List ultra-fast scans | | `GET` | `/api/scans/ultra-fast/:id` | **Yes** | Ultra-fast scan status | | `GET` | `/api/scans/ultra-fast/:id/results` | **Yes** | Paginated results (`?page=`, `?limit=`, `?severity=`) | | `GET` | `/rescan/stream` | **Yes** | Direct HexStrike AI SSE stream (`?domain=`) | | `GET` | `/rescan/stream_subfinder_dnsx_httpx` | **Yes** | Subfinder pipeline SSE (`?domain=`) | | `GET` | `/scan/comprehensive` | **Yes** | Levelwise parallel SSE (`?domain=`) | | `GET` | `/api/hexstrike/health` | **Yes** | Check scanning-engine health | | `POST` | `/api/trigger_background_scan` | **Yes** | Fire-and-forget Celery scan | | `GET` | `/api/background_scan_status/:task_id` | **Yes** | Check Celery task state | ### 资产管理 (3 个端点) | Method | Path | Auth | Description | |--------|------|------|-------------| | `POST` | `/api/assets` | **Yes** | Create asset group (company, domains, IPs, endpoints) | | `GET` | `/api/assets` | **Yes** | List all user's asset groups | | `DELETE` | `/api/assets/:id` | **Yes** | Delete asset group (owner only) | ### 结果 (8 个端点) | Method | Path | Auth | Description | |--------|------|------|-------------| | `GET` | `/results` | **Yes** | Legacy scan results (`?scan_id=`) | | `GET` | `/resultssubfinder` | **Yes** | Latest subfinder scan results | | `GET` | `/resultssubfinderchart` | **Yes** | Subfinder chart data (`?scan_id=`) | | `GET` | `/recent-scan-json` | **Yes** | Recent scans JSON (`?scan_type=`, `?limit=`) | | `GET` | `/api/getPorts` | No | Open ports for a domain (`?subdomain=`) | | `GET` | `/api/getPorts_subfinder` | No | Ports from subfinder/hexstrike scans (`?subdomain=`) | | `GET` | `/api/getFfuf_subfinder` | No | Ffuf directory results (`?subdomain=`) | | `GET` | `/api/getZapAlerts` | No | ZAP security alerts (`?subdomain=`) | ### 统计 (2 个端点) | Method | Path | Auth | Description | |--------|------|------|-------------| | `GET` | `/api/statistics` | **Yes** | Dashboard stats: totals, trends, severity breakdown, top alerts | | `GET` | `/scan-trends` | **Yes** | Scan trend data for line charts | ### 扫描引擎内部 API (端口 8888，不暴露给前端) | Method | Path | Description | |--------|------|-------------| | `GET` | `/health` | Engine health + tool count + version | | `GET` | `/api/tools` | List all 19 available tools with categories/priorities | | `POST` | `/api/analyze` | AI target analysis: DNS, HTTP, port scan, WAF, risk, techs | | `POST` | `/api/tools/optimal` | AI-recommended tools for a target (max 12) | | `POST` | `/api/intelligence/generate-command` | Generate CLI command for any tool | | `POST` | `/api/scan/subdomain` | Execute tools against a single subdomain (parallel) | | `POST` | `/api/scan/batch` | Batch subdomain scan (stub) | | `POST` | `/api/discover/subdomains` | Discovery: subfinder + wordlist expansion | ## 🔄 扫描流程 ``` ┌─────────────────────────────────────────────────────────────────┐ │ USER enters domain in browser (HeroSection → navigates /scan) │ └──────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ FRONTEND: POST /api/scans/hexstrike { domain: "example.com" } │ │ Opens SSE connection to /api/scans/hexstrike/:id/stream │ └──────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ API GATEWAY: Creates scan in MongoDB (status: "queued") │ │ Dispatches Celery task → hexstrike_scan(scan_id, domain) │ └──────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ CELERY WORKER picks up task │ │ Creates RedisLogHandler → streams logs to scan_logs:{scan_id} │ │ Creates HexStrikeManager(scanning-engine:8888) │ │ Updates MongoDB: status → "running" │ └──────────────────────────┬──────────────────────────────────────┘ │ ┌────────────────┼────────────────┐ ▼ ▼ ▼ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ Step 1: │ │ Step 2: │ │ Step 3: │ │ ANALYZE │ │ DISCOVER │ │ SELECT TOOLS │ │ │ │ │ │ │ │ POST /api/ │ │ POST /api/ │ │ POST /api/ │ │ analyze │ │ discover/ │ │ tools/ │ │ │ │ subdomains │ │ optimal │ │ • DNS lookup │ │ │ │ │ │ • HTTP probe │ │ • subfinder │ │ • AI selects │ │ • 14 ports │ │ • subfinder │ │ best tools │ │ • WAF detect │ │ • wordlist │ │ • Per-tool │ │ • Risk score │ │ expansion │ │ params │ │ • Tech detect│ │ • Dedup │ │ • Max 12 │ │ • Classify │ │ • Max 150 │ │ │ └──────┬───────┘ └──────┬───────┘ └──────┬───────┘ └─────────────────┼─────────────────┘ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ Step 4: PARALLEL SCAN — ThreadPoolExecutor (10 workers) │ │ │ │ For EACH subdomain (up to 150): │ │ POST /api/scan/subdomain { subdomain, tools[] } │ │ │ │ ┌─────────────────────────────────────────────────────────┐ │ │ │ ThreadPoolExecutor (8 workers) runs tools in parallel: │ │ │ │ │ │ │ │ httpx ─────→ HTTP probe + tech fingerprint │ │ │ │ nuclei ────→ CVE detection + vulnerability templates │ │ │ │ nmap ──────→ Port scan + service/version detection │ │ │ │ naabu ─────→ Fast port scan │ │ │ │ dnsx ──────→ DNS records │ │ │ │ tlsx ──────→ TLS certificate inspection │ │ │ │ whatweb ───→ Technology fingerprinting │ │ │ │ ffuf ──────→ Directory brute-force │ │ │ │ nikto ─────→ Web vulnerability scan │ │ │ │ masscan ───→ Mass port scan │ │ │ │ testssl ───→ SSL/TLS testing │ │ │ │ dirsearch ─→ Directory scanning │ │ │ │ wafw00f ───→ WAF detection │ │ │ │ curl ──────→ Header analysis │ │ │ └─────────────────────────────────────────────────────────┘ │ │ │ │ Results merged → Risk score computed (0-100) │ │ CVEs extracted → Vulnerabilities categorized by severity │ └──────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ STORAGE & STREAMING │ │ │ │ • Each subdomain result → MongoDB subdomain_results collection │ │ • Scan status → MongoDB scans (completed, total_subdomains) │ │ • Logs → Redis scan_logs:{scan_id} → SSE → Terminal component │ └──────────────────────────┬──────────────────────────────────────┘ │ ▼ ┌─────────────────────────────────────────────────────────────────┐ │ FRONTEND renders results: │ │ • SubdomainCardSubfinder for each discovered subdomain │ │ • Risk score badges (CRITICAL / HIGH / MEDIUM / LOW / MINIMAL) │ │ • Expandable details: ports, techs, vulns, directories │ │ • PDF export via PDFDownloadSubfinder │ │ • Analytics in ReconDashboard (charts, history) │ └─────────────────────────────────────────────────────────────────┘ ``` ## 🚀 快速开始 ### 先决条件 - [Docker Desktop](https://www.docker.com/products/docker-desktop/) (Windows/Mac) or Docker Engine (Linux) - Docker Compose v2+ - 4GB+ RAM available for Docker - Ports **80**, **4000**, **6379**, **8888**, **27017** available ### 1. 克隆并配置 ``` git clone cd Vulnerability_Scanner ``` ### 2. 设置环境变量 Edit `.env` file: ``` # ── 必需 ── JWT_SECRET_KEY=change-this-to-a-strong-random-string # ── 电子邮件（用于 OTP 验证） ── MAIL_SERVER=smtp.gmail.com MAIL_PORT=587 MAIL_USE_TLS=True MAIL_USERNAME=your-email@gmail.com MAIL_PASSWORD=your-gmail-app-password # ── 由 docker-compose 自动配置（必要时覆盖） ── MONGO_URI=mongodb://mongo:27017/ CELERY_BROKER_URL=redis://redis:6379/0 HEXSTRIKE_URL=http://scanning-engine:8888 ``` ### 3. 构建并启动 ``` docker compose up --build -d ``` This will: 1. Build 5 Docker images (presentation, api-gateway, processing, scanning-engine, hexstrike-intelligence) 2. Pull 2 official images (mongo:7.0, redis:7-alpine) 3. Download 7 Go security tool binaries during scanning-engine build 4. Install all Python and Node.js 5. Create MongoDB collections and indexes on first boot 6. Start all 7 containers on the `cystar-network` bridge ### 4. 访问平台 | Service | URL | |---------|-----| | **Dashboard** | http://localhost | | **API Gateway** | http://localhost:4000/health | | **Scanning Engine** | http://localhost:8888/health | ### 5. 首次使用 1. Open **http://localhost** → Redirects to `/login` 2. Click **Register** → Enter email + password 3. Check your email for a **6-digit OTP** code 4. Enter OTP → You're automatically logged in 5. Type a domain on the home page (e.g. `example.com`) → Click **Scan** 6. Watch **real-time scan logs** in the terminal panel as 19+ tools execute 7. View results as **subdomain cards** with risk scores, technologies, open ports, and vulnerabilities 8. Export to **PDF** or analyze on the **Dashboard** ### 常用命令 ``` # 启动所有服务 docker compose up -d # 在代码更改后重建 docker compose up --build -d # 重建单个服务 docker compose up -d --build scanning-engine # 查看实时日志 docker compose logs -f api-gateway docker compose logs -f celery-worker docker compose logs -f scanning-engine # 停止所有服务 docker compose down # 停止并删除所有数据（⚠️ 破坏性操作） docker compose down -v # 检查容器状态 docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" ``` ## 📦 Docker 注册表 (ghcr.io) All Cystar Docker images are published to **GitHub Container Registry** (`ghcr.io`) for easy deployment on any machine without rebuilding. ### 注册表详情 | Item | Value | |------|-------| | **Registry** | `ghcr.io` (GitHub Container Registry) | | **Owner** | `omjee73` | | **Image Prefix** | `ghcr.io/omjee73/` | | **Version File** | `VERSION` (project root) | | **Push Script** | `registry-push.ps1` (PowerShell) | | **Packages URL** | https://github.com/omjee73?tab=packages | ### 已发布镜像 | # | Image Name | Service(s) | Size | Dockerfile | |---|------------|------------|------|------------| | 1 | `ghcr.io/omjee73/cystar-frontend` | presentation | ~77 MB | `services/presentation/Dockerfile` | | 2 | `ghcr.io/omjee73/cystar-api-gateway` | api-gateway | ~263 MB | `services/api-gateway/Dockerfile` | | 3 | `ghcr.io/omjee73/cystar-processing` | celery-worker, celery-beat | ~442 MB | `services/processing/Dockerfile` | | 4 | `ghcr.io/omjee73/cystar-scanning-engine` | scanning-engine | ~1.5 GB | `services/scanning-engine/Dockerfile` | ### 标记策略 Each image gets **two tags** on every push: | Tag | Example | Purpose | |-----|---------|---------| | `v` | `v1.0.0` | Immutable version tag — never overwritten | | `latest` | `latest` | Always points to the newest push | Version is read from the `VERSION` file (project root). Override with `-Version "1.1.0"`. ### 拉取镜像（在任意机器上部署） To deploy **without building** — just pull pre-built images: ``` # 登录 ghcr.io（需要具有 read:packages 范围的 GitHub PAT） echo YOUR_GITHUB_PAT | docker login ghcr.io -u omjee73 --password-stdin # 拉取所有镜像 docker compose pull # 启动所有服务 docker compose up -d ``` Or pull individual images: ``` # 拉取特定镜像（最新版） docker pull ghcr.io/omjee73/cystar-frontend:latest docker pull ghcr.io/omjee73/cystar-api-gateway:latest docker pull ghcr.io/omjee73/cystar-processing:latest docker pull ghcr.io/omjee73/cystar-scanning-engine:latest # 拉取特定版本 docker pull ghcr.io/omjee73/cystar-scanning-engine:v1.0.0 ``` ### 推送镜像 (registry-push.ps1) The `registry-push.ps1` PowerShell script handles building, tagging, and pushing all images. **Prerequisites:** - Docker Desktop running - GitHub Personal Access Token (PAT) with `write:packages` + `read:packages` scope - Create PAT at: https://github.com/settings/tokens/new **Available Commands:** ``` # 步骤 1：登录 ghcr.io（交互式 — 安全提示输入 PAT） .\registry-push.ps1 login # 步骤 2：构建 + 标记 + 推送所有 4 个镜像 .\registry-push.ps1 push # 使用特定版本推送（同时更新 VERSION 文件） .\registry-push.ps1 push -Version "1.1.0" # 仅推送单个服务 .\registry-push.ps1 push -Service scanning-engine .\registry-push.ps1 push -Service frontend .\registry-push.ps1 push -Service api-gateway .\registry-push.ps1 push -Service processing # 仅构建 + 标记（不推送到注册表） .\registry-push.ps1 build # 列出所有带有标签和大小的本地 Cystar 镜像 .\registry-push.ps1 list ``` **Script Parameters:** | Parameter | Values | Default | Description | |-----------|--------|---------|-------------| | `Action` | `push`, `login`, `build`, `list` | `push` | What operation to perform | | `-Version` | e.g. `"1.1.0"` | reads `VERSION` file | Override version tag | | `-Service` | `frontend`, `api-gateway`, `scanning-engine`, `processing` | all | Push only one service | ### 典型工作流程 ``` # 1. 修改代码 # 2. 更新 VERSION 文件（例如 1.0.0 → 1.1.0） # 3. 登录（每个会话仅需一次） .\registry-push.ps1 login # 4. 构建 + 推送所有镜像 .\registry-push.ps1 push # 5. 在 GitHub 上验证 # 访问：https://github.com/omjee73?tab=packages # 6. 登出（安全） docker logout ghcr.io ``` ### 在新机器上部署 ``` # 1. 克隆仓库 git clone https://github.com/Omjee73/Vulnerability_Scanner.git cd Vulnerability_Scanner # 2. 创建 .env 文件并填写密钥 cp .env.example .env # edit with your values # 3. 登录 ghcr.io echo YOUR_GITHUB_PAT | docker login ghcr.io -u omjee73 --password-stdin # 4. 拉取预构建镜像并启动 docker compose pull docker compose up -d # 5. 访问 http://localhost ``` No need to build anything — all images are pre-built and pulled from the registry. ### 制作镜像为公开 By default GitHub packages are **private**. To make them public (no PAT needed to pull): 1. Go to https://github.com/omjee73?tab=packages 2. Click on each package (e.g. `cystar-frontend`) 3. Click **Package settings** (right sidebar) 4. Scroll to **Danger Zone** → **Change visibility** → **Public** 5. Repeat for all 4 packages Once public, anyone can pull without authentication: ``` docker pull ghcr.io/omjee73/cystar-frontend:latest ``` ## ⚙️ 环境变量 | Variable | Default | Service | Description | |----------|---------|---------|-------------| | `JWT_SECRET_KEY` | `your-secret-key-here` | api-gateway | **Change this.** JWT signing key | | `MONGO_URI` | `mongodb://mongo:27017/` | api-gateway, processing | MongoDB connection | | `CELERY_BROKER_URL` | `redis://redis:6379/0` | api-gateway, processing | Celery task queue | | `CELERY_RESULT_BACKEND` | `redis://redis:6379/1` | processing | Celery result storage | | `REDIS_URL` | `redis://redis:6379/0` | api-gateway, processing | Rate limits + SSE logs | | `HEXSTRIKE_URL` | `http://scanning-engine:8888` | api-gateway, processing | Scanning engine API | | `MAIL_SERVER` | `smtp.gmail.com` | api-gateway | SMTP server for OTP emails | | `MAIL_PORT` | `587` | api-gateway | SMTP port | | `MAIL_USE_TLS` | `True` | api-gateway | Enable TLS for email | | `MAIL_USERNAME` | — | api-gateway | SMTP username | | `MAIL_PASSWORD` | — | api-gateway | SMTP app password | | `AUTO_SCAN_DOMAIN` | `iitm.ac.in` | processing | Default auto-scan target | | `TOOLS_DIR` | `/app/tools` | scanning-engine | Security tool binary directory | | `COHERE_API_KEY` | — | api-gateway | Optional: Cohere LLM for risk analysis | | `VITE_API_URL` | `http://localhost:4000` | presentation (build-time) | API URL baked into frontend bundle | ## 🎨 前端页面与组件 ### 路由映射 | Path | Component | Access | Description | |------|-----------|--------|-------------| | `/login` | `Login` | Public | Split-screen email/password login | | `/register` | `Register` | Public | Account creation form | | `/forgot-password` | `ForgotPassword` | Public | Password reset request | | `/verify-otp` | `VerifyOtp` | Public | 6-digit OTP input | | `/auth/verify` | `VerificationPage` | Public | Email verification callback | | `/` | `HeroSection` | Protected | Home page — animated domain input | | `/scan` | `HexStrikeScan` | Protected | Core scan page: SSE terminal + results | | `/dashboard` | `ReconDashboard` | Protected | Analytics: charts, trends, scan history | | `/assets` | `AssetForm` | Protected | Asset CRUD: companies, domains, IPs | | `/about` | `About` | Protected | 8-step workflow visualization | | `/profile` | `Profile` | Protected | Edit profile + change password | ### 组件职责 | Component | | Purpose | |-----------|------|---------| | **HeroSection** | `components/HeroSection.tsx` | Landing page: animated title with typewriter, domain input, navigates to `/scan?domain=` | | **HexStrikeScan** | `components/scan/HexStrikeScan.tsx` | Main scan orchestrator: POST scan → open SSE → render Terminal → fetch & display results | | **Terminal** | `components/terminal/Terminal.tsx` | Portal-mounted live log viewer with macOS-style title bar, auto-scroll, SSE consumption | | **SubdomainCardSubfinder** | `components/scan/SubdomainCardSubfinder.tsx` | Compact result card per subdomain: IP, status, risk badge, ports, tech tags | | **subdomainDetailsSubfinder** | `components/scan/subdomainDetailsSubfinder.tsx` | Expanded detail view: httpx, nuclei, nmap, ffuf data | | **DirectoryTree** | `components/scan/DirectoryTree.tsx` | Tree view for directory brute-force results (ffuf/dirsearch) | | **PDFDownloadSubfinder** | `components/scan/PDFDownloadSubfinder.tsx` | Generates formatted PDF report from scan results (jspdf) | | **ReconDashboard** | `components/Statistics/ReconDashboard.tsx` | Analytics: Recharts LineChart, severity pie, scan history table, stat cards | | **AssetForm** | `components/asset/AssetForm.tsx` | CRUD interface for company assets: domains, IPs, endpoints, API keys | | **Layout** | `components/layout/Layout.tsx` | Page shell: Navbar + scan-line animation + grid overlay + footer | | **Navbar** | `components/layout/Navbar.tsx` | Navigation links, theme color picker, user dropdown menu | | **Login** | `components/auth/Login.tsx` | Split-screen login with animated background | | **Register** | `components/auth/Register.tsx` | Registration with email + password fields | | **ProtectedRoute** | `components/auth/ProtectedRoute.tsx` | Auth guard: shows spinner while loading, redirects to `/login` if unauthenticated | | **Profile** | `pages/Profile.tsx` | Update display name, change password, view account info | | **About** | `pages/About.tsx` | 8-step workflow flowchart, feature cards, platform description | | **LoadingSpinner** | `components/ui/LoadingSpinner.tsx` | Lucide `Loader2` spinning animation | ### 身份验证流程 ``` Register → POST /auth/register → OTP sent to email ↓ /verify-otp → POST /auth/verify-otp → JWT set in HTTP-only cookie → redirect / ↓ Every page load → AuthProvider calls GET /auth/me → hydrates user state ↓ ProtectedRoute checks isAuthenticated → allows or redirects to /login ↓ Logout → POST /auth/logout → clears JWT cookie → redirect /login ``` ### 主题系统 6 color palettes persisted in `localStorage`, applied via CSS custom properties on `:root`: | Palette | Primary | Secondary | Accent | |---------|---------|-----------|--------| | Blue | `#3B82F6` | `#1E40AF` | `#60A5FA` | | Green | `#10B981` | `#047857` | `#34D399` | | **Purple** (default) | `#8B5CF6` | `#5B21B6` | `#A78BFA` | | Red | `#EF4444` | `#991B1B` | `#F87171` | | Gray | `#6B7280` | `#374151` | `#9CA3AF` | | Black | `#1F2937` | `#111827` | `#4B5563` | Components reference themes via `var(--color-primary)`, `var(--color-background)`, etc. ## 🗄 数据库架构 ### MongoDB 集合 **`users`** ``` { "_id": "ObjectId", "email": "user@example.com", "password_hash": "$2b$12$...", "organization": "example.com", "name": "user", "created_at": "2026-03-08T00:00:00Z", "last_login": "2026-03-08T12:00:00Z", "is_verified": true, "otp": "123456", "otp_expires": "2026-03-08T00:10:00Z" } ``` **`scans`** ``` { "_id": "uuid-string", "domain": "example.com", "user_id": "ObjectId-string", "status": "completed", "scan_type": "hexstrike", "scan_engine": "hexstrike-ai", "created_at": "2026-03-08T00:00:00Z", "started_at": "2026-03-08T00:00:01Z", "completed_at": "2026-03-08T00:05:00Z", "total_subdomains": 42 } ``` **`subdomain_results`** ``` { "_id": "ObjectId", "scan_id": "uuid-string", "domain": "example.com", "subdomain": "api.example.com", "ip": "192.168.1.1", "status_code": 200, "technologies": ["nginx", "React"], "ports": [80, 443, 8080], "vulnerabilities": [ { "name": "CVE-2024-1234", "severity": "high", "description": "..." } ], "risk_analysis": { "risk_score": 72, "risk_level": "HIGH" }, "scanned_at": "2026-03-08T00:02:00Z" } ``` **`assets`** ``` { "_id": "ObjectId", "user_id": "ObjectId-string", "company_name": "Example Corp", "domains": ["example.com", "example.org"], "ip_addresses": ["192.168.1.0/24"], "endpoints": ["https://api.example.com/v1"], "created_at": "2026-03-08T00:00:00Z" } ``` ## 🌐 Nginx 反向代理 The presentation container serves the React SPA and proxies API requests to the api-gateway. | Location | Target | Special Config | |----------|--------|----------------| | `/` | SPA static files | `try_files $uri $uri/ /index.html` (client routing) | | `/api/` | `http://api-gateway:4000` | SSE: `proxy_buffering off`, `proxy_read_timeout 600s` | | `/auth/` | `http://api-gateway:4000` | Standard proxy headers (Host, X-Real-IP, X-Forwarded-For) | | `/rescan/` | `http://api-gateway:4000` | SSE: `proxy_buffering off`, `proxy_read_timeout 1200s` | | `/results` | `http://api-gateway:4000` | Standard proxy | | `/resultssubfinder` | `http://api-gateway:4000` | Standard proxy | | `/resultssubfinderchart` | `http://api-gateway:4000` | Standard proxy | | `/scan-trends` | `http://api-gateway:4000` | Standard proxy | **Gzip:** Enabled for text/css/json/javascript/xml (min 256 bytes). ## 🐛 故障排查 | Issue | Solution | |-------|----------| | **Port 80 already in use** | Stop other web servers: `netstat -ano \| findstr :80` then kill the PID | | **Docker build fails on scanning-engine** | Network issue downloading Go binaries. `init_tools.py` has `\|\| true` to continue gracefully | | **Frontend shows "unhealthy"** | Healthcheck uses `127.0.0.1` — check `docker logs cystar-frontend` | | **Celery tasks stuck in "queued"** | Check worker: `docker logs cystar-celery-worker` — may need Redis connectivity | | **OTP email not received** | Verify `MAIL_USERNAME` and `MAIL_PASSWORD` in `.env`. Gmail requires [App Password](https://support.google.com/accounts/answer/185833) | | **MongoDB connection refused** | Wait 20s for healthcheck. Check: `docker logs cystar-mongo` | | **Rate limited (HTTP 429)** | Auth endpoints have sliding-window limits — wait 60s and retry | | **No scan results returned** | Check scanning-engine: `docker logs cystar-scanning-engine` | | **WSL2 DNS resolution fails** | Docker compose sets `dns: [1.1.1.1, 8.8.8.8]` — check host network config | | **TS errors in VS Code** | `node_modules` not installed locally. Run `cd services/presentation && npm install` for IDE support | ### 有用的调试命令 ``` # 容器健康状态概览 docker ps --format "table {{.Names}}\t{{.Status}}\t{{.Ports}}" # 查看服务日志 docker compose logs -f api-gateway docker compose logs -f scanning-engine docker compose logs -f celery-worker # 进入容器终端 docker exec -it cystar-api-gateway /bin/bash docker exec -it cystar-scanning-engine /bin/bash # MongoDB 查询 docker exec -it cystar-mongo mongosh subdomain_scanner --eval "db.stats()" docker exec -it cystar-mongo mongosh subdomain_scanner --eval "db.users.countDocuments()" docker exec -it cystar-mongo mongosh subdomain_scanner --eval "db.scans.find().sort({created_at:-1}).limit(5).pretty()" # Redis 健康状态 docker exec -it cystar-redis redis-cli ping docker exec -it cystar-redis redis-cli info memory # Celery 检查 docker exec -it cystar-celery-worker celery -A tasks inspect active docker exec -it cystar-celery-worker celery -A tasks inspect registered # 直接测试扫描引擎 curl http://localhost:8888/health | python -m json.tool curl http://localhost:8888/api/tools | python -m json.tool # 测试 API 网关 curl http://localhost:4000/health | python -m json.tool ``` ## 📄 许可证 MIT License — see [LICENSE](LICENSE) for details.

**Built for security researchers and penetration testers** *Cystar Vulnerability Scanner — 6-Layer Microservices Architecture*

标签：AES-256, AI侦察, API网关, Celery, CI/CD安全, DevSecOps, Docker, Docker Compose, Flask, HexStrike, LIDS, Llama, MongoDB, Nginx, NIDS, PMD, Python, React, Redis, Syscalls, TypeScript, 上游代理, 人工智能安全, 企业安全, 企业级, 分布式系统, 合规性, 响应大小分析, 多工具集成, 威胁情报, 子域名突变, 安全专业人士, 安全工具集成, 安全扫描, 安全插件, 安全防御评估, 实时流处理, 容器化, 开发者工具, 开源安全工具, 异步任务, 微服务架构, 扫描引擎, 搜索引擎查询, 无后门, 时序注入, 漏洞评估, 版权保护, 网络安全, 网络空间测绘, 网络资产管理, 自动化渗透测试, 逆向工具, 逆向工程平台, 隐私保护, 高可用架构