ThuyLinhDo0808/Zero-day-Threat-Hunting-System

# Threat Hunter AI — 基于图的零日漏洞 / APT 检测 MVP 一种下一代网络防御系统，将流量建模为动态图 （IP、端口、进程作为节点；连接/syscalls 作为边），并使用 图神经网络（Graph Neural Network）实时标记异常子图——而不是 匹配已知的特征签名。 ## 1. 架构 ``` ┌────────────────────┐ │ Zeek / Suricata │ (simulated here by log_generator.py) │ conn.log, dns.log │ └─────────┬──────────┘ │ JSON events ▼ ┌────────────────────┐ │ Kafka topic: │ (simulated by an in-process asyncio │ "network-events" │ queue — see data_pipeline/kafka_producer.py) └─────────┬──────────┘ │ consume ▼ ┌────────────────────┐ │ Graph Builder │ data_pipeline/graph_builder.py │ (sliding window) │ → builds torch_geometric.data.Data └─────────┬──────────┘ │ Data(x, edge_index, edge_attr) ▼ ┌────────────────────┐ │ GNN Inference │ ml_model/inference.py │ (GraphSAGE, node │ → per-node anomaly score [0,1] │ anomaly scoring) │ → calibrated threshold + hysteresis └─────────┬──────────┘ │ scores + alerts ▼ ┌────────────────────┐ │ FastAPI backend │ backend/app/main.py │ REST + WebSocket │ → /ws/graph streams graph+alerts └─────────┬──────────┘ │ WebSocket (JSON) ▼ ┌────────────────────┐ │ Next.js dashboard │ frontend/ │ react-force-graph │ → live graph + alert panel └────────────────────┘ ``` Elasticsearch/Kibana 伴随该 pipeline 运行，用于长期日志存储 和分析师的深入查询（Logstash/Zeek → ES）。MVP 的实时检测 路径并不需要它们，因此它们在 `data_pipeline/README_ELK.md` 中进行说明，而未在代码中模拟——检测路径（Kafka → Graph → GNN → API → UI） 才是下方已实现且可运行的部分。 ## 2. 目录结构 ``` threat-hunter-ai/ ├── README.md ├── data_pipeline/ │ ├── log_generator.py # simulates Zeek conn.log JSON events (benign + attack bursts) │ ├── kafka_producer.py # thin wrapper: real kafka-python if available, else asyncio.Queue mock │ ├── graph_builder.py # sliding-window stream -> PyG Data objects, feature engineering │ └── README_ELK.md # notes on wiring real Zeek/Logstash/Elasticsearch later ├── ml_model/ │ ├── gnn_model.py # GraphSAGE/GAT anomaly model definition │ ├── train.py # training loop on labeled benign/malicious graph snapshots │ ├── inference.py # real-time scoring + thresholding/hysteresis to cut false positives │ └── dataset.py # synthetic labeled dataset generator (benign vs ransomware/APT patterns) ├── backend/ │ ├── requirements.txt │ └── app/ │ ├── main.py # FastAPI app, WebSocket /ws/graph, REST /alerts, /health │ ├── model_service.py # loads GNN once, exposes score_graph() │ └── stream_manager.py # bridges data_pipeline generator -> connected websocket clients └── frontend/ ├── package.json ├── lib/ │ └── useThreatSocket.js # WebSocket hook with reconnect + backoff ├── components/ │ ├── GraphView.jsx # react-force-graph-2d live network visualization │ ├── AlertPanel.jsx # scrolling list of high-confidence alerts │ └── Sidebar.jsx # stats: nodes, edges, avg score, threshold slider └── pages/ ├── index.jsx # dashboard layout └── _app.jsx ``` ## 3. 在本地运行 MVP（无需外部服务） ``` # 1. ML + backend cd backend pip install -r requirements.txt python -m app.main # starts FastAPI on :8000, auto-generates mock traffic internally # 2. Frontend cd frontend npm install npm run dev # Next.js on :3000, connects to ws://localhost:8000/ws/graph ``` 不需要 Kafka、Zeek 或 Elasticsearch 集群即可查看其端到端的运行情况： `data_pipeline/log_generator.py` 会合成真实的正常流量，并 定期注入勒索软件/APT 流量突发（C2 beaconing、横向移动扇出、 将海量文件写入特征签名转化为流特征），并且 当 `kafka` 包 / broker 不存在时，`kafka_producer.py` 会回退到内存异步队列，因此 `backend/app/main.py` 可以 独立运行整个循环。以后切换到真实的 Kafka + Zeek 只需更改一行配置 （参见 `README_ELK.md`）。

标签：AMSI绕过, Kafka, Metaprompt, SonarQube插件, 图神经网络, 威胁检测, 流计算, 网络安全, 网络流量分析, 自定义脚本, 逆向工具, 隐私保护