PrerikaP1331/STIX-Dashboard
GitHub: PrerikaP1331/STIX-Dashboard
一款基于 AI 与 RAG 技术的网络威胁情报分析仪表板，用于自动化摄取、校验、分类和可视化 STIX 威胁数据。
Stars: 0 | Forks: 0
# STIX AI 仪表板
一个基于 AI 的网络威胁情报（CTI）平台，用于摄取、分析和可视化 [STIX](https://oasis-open.github.io/cti-documentation/) 威胁捆绑包 — 内置 ML 分类、基于 RAG 的聊天、相似性搜索和可信度评分。
## 功能
| 功能 | 描述 |
|---|---|
| **版本检测与验证** | 自动检测 STIX 1.x / 2.0 / 2.1（XML 和 JSON）并验证 schema |
| **自动转换** | 将任何 STIX 版本无缝转换为 STIX 2.1 |
| **威胁图谱** | 支持节点类型过滤的交互式 PyVis 关系图 |
| **攻击映射** | 将威胁对象映射到 MITRE ATT&CK 技术 |
| **恶意软件分类器** | RAG + Mistral-7B 将恶意软件分类为 15 种家族类型 |
| **可信度评估** | 集成 VirusTotal 的多因素威胁可信度评分（0–100） |
| **STIX 聊天** | 由 RAG 驱动的聊天机器人，用于查询 STIX 和 MITRE ATT&CK 知识 |
| **SDO 相似性搜索** | 基于 FAISS 的 ego-graph 浏览器，用于查找相似的 STIX 对象 |
| **规范化输出** | 导出标准化的 STIX 2.1 捆绑包 |
| **文件浏览器** | 浏览和检查存储的 STIX 文件 |
## 架构
```
graph TB
subgraph Frontend[" Streamlit Frontend"]
A[app.py
Navigation Router] A --> P1[Dashboard] A --> P2[Version Detector & Validator] A --> P3[SDO Similarity Search] A --> P4[Attack Mapping] A --> P5[Malware Classifier] A --> P6[Credibility Assessment] A --> P7[STIX Chat] A --> P8[File Browser] A --> P9[Canonical Output] end subgraph Modules[" Backend Modules"] M1[enhanced_detector.py
Version & Format Detection] M2[enhanced_validator.py
Schema Validation] M3[converter.py
STIX 1.x → 2.1] M4[threat_classifier.py
Rule-based Classifier] M5[similarity_search.py
FAISS Similarity] M6[credibility_assessor.py
Score Engine] M7[rag_engine.py
ChromaDB + Ollama] M8[attack_type_mapper.py
MITRE Lookup] M9[storage.py
Session Storage] end subgraph AI[" AI / ML Layer"] AI1[Mistral-7B-Instruct
Malware Classification] AI2[all-MiniLM-L6-v2
Sentence Embeddings] AI3[FAISS Index
Vector Search] AI4[ChromaDB
RAG Knowledge Base] AI5[Ollama
Local LLM Inference] end subgraph Data[" Data Layer"] D1[STIX Bundles
JSON / XML] D2[MITRE ATT&CK
enterprise-attack.json] D3[Vector DB
faiss_db/] D4[Knowledge Base
knowledge_base/] end P1 & P5 & P6 --> Modules P7 --> M7 P3 --> M5 Modules --> AI AI --> Data ``` ## 数据流 ``` flowchart LR U([ User]) -->|Upload STIX\nJSON or XML| DET subgraph Pipeline["Processing Pipeline"] DET[ Version Detection\nSTIX 1.x / 2.0 / 2.1] --> VAL[ Schema Validation] VAL --> CONV[ Convert to STIX 2.1] CONV --> ANAL[ Analysis Engine] end subgraph Analysis["Analysis Modules"] ANAL --> TH[ Threat Classification] ANAL --> CR[ Credibility Score] ANAL --> GR[ Relationship Graph] ANAL --> ML[ Malware Classification] ANAL --> SIM[ Similarity Search] end TH & CR & GR & ML & SIM --> OUT([ Dashboard\nResults & Downloads]) ``` ## 恶意软件分类 Pipeline ``` sequenceDiagram participant U as User participant UI as Streamlit UI participant VDB as FAISS Vector DB participant LLM as Mistral-7B (HuggingFace) U->>UI: Upload STIX Bundle UI->>UI: Convert to STIX 2.1 UI->>UI: Extract malware objects loop For each malware object UI->>VDB: Semantic search (top-5 similar docs) VDB-->>UI: Relevant context chunks UI->>LLM: Prompt with context + malware description LLM-->>UI: JSON classification result UI->>UI: Validate against 15 allowed types end UI-->>U: Classification results + confidence scores ``` ## 可信度评估评分 ``` graph LR B[STIX 2.1 Bundle] --> F1[ Source credibility] B --> F2[ Timestamp validity] B --> F3[ Relationship richness] B --> F4[ Metadata completeness] B --> F5[ IOC validation via VirusTotal] B --> F6[ Object type diversity] F1 & F2 & F3 & F4 & F5 & F6 --> SCORE[ Credibility Score\n0 – 100] SCORE --> HIGH[" HIGH ≥ 75"] SCORE --> MED[" MEDIUM 45–74"] SCORE --> LOW[" LOW < 45"] ``` ## SDO 相似性搜索 Pipeline ``` flowchart TD A[STIX Bundle JSON] --> B[bundles_2_csv\nExtract SDOs to CSV] B --> C[summary_generator\nOllama-powered summaries] C --> D[clean_csv\nNormalize & dedupe] D --> E[similarity_search\nFAISS cosine similarity] E --> F[similarity_json\nTop-K similar IDs per object] F --> G[Ego Graph Explorer\nPyVis interactive graph] ``` ## STIX RAG 聊天 ``` flowchart LR KB[ Knowledge Base\nSTIX docs + MITRE data] -->|Chunk + Embed| VDB[(ChromaDB\nVector Store)] Q[ User Question] -->|Embed query| VDB VDB -->|Top-K relevant chunks| CTX[Context Window] CTX --> OL[ Ollama LLM] OL --> ANS[ Answer + Sources] ``` ## 项目结构 ``` stix-ai-dashboard/ ├── app.py # Navigation router ├── pages/ │ ├── 1_Dashboard.py # Main threat overview + graph │ ├── Version_Detector_and_Validator.py │ ├── SDO_Similarity_Search.py # Ego-graph explorer │ ├── Attack_Mapping.py # MITRE ATT&CK mapping │ ├── malware_classifier.py # ML-based malware typing │ ├── Credibility_Assessment_Module.py │ ├── STIX_Chat.py # RAG chatbot │ ├── Canonical_Output.py │ └── file_browser.py ├── modules/ │ ├── enhanced_detector.py # STIX version detection │ ├── enhanced_validator.py # Schema validation │ ├── converter.py # STIX 1.x → 2.1 converter │ ├── similarity_search.py # FAISS similarity │ ├── rag_engine.py # ChromaDB + Ollama RAG │ ├── credibility_assessor.py # Scoring engine │ ├── attack_type_mapper.py # MITRE lookup │ ├── storage.py # Session-based file storage │ └── ... ├── vector_db/ # FAISS malware vector store ├── knowledge_base/ # RAG source documents ├── enterprise-attack.json # MITRE ATT&CK dataset └── requirements.txt ``` ## 快速开始 ### 1. 安装依赖 ``` pip install -r requirements.txt ``` ### 2. （可选）启动 Ollama 以使用 STIX 聊天 ``` ollama serve ollama pull mistral # or your preferred model ``` ### 3. 运行应用 ``` streamlit run app.py ``` 在浏览器中打开 `http://localhost:8501`。 ## AI/ML 技术栈 | 组件 | 技术 | |---|---| | Sentence Embeddings | `sentence-transformers/all-MiniLM-L6-v2` | | 恶意软件 LLM | `mistralai/Mistral-7B-Instruct-v0.1` | | Vector Search | FAISS (`faiss-cpu`) | | RAG 知识库 | ChromaDB | | 本地 LLM 推理 | Ollama | | 图谱可视化 | PyVis + NetworkX | | 仪表板框架 | Streamlit | ## 核心依赖 ``` streamlit plotly pyvis stix2 stix2-validator stix2-elevator faiss-cpu chromadb sentence-transformers transformers torch langchain-community networkx pandas ollama ``` ## 安全与隐私 - 所有文件处理均在**本地**进行 — 不会向外部服务器发送任何数据（除非提供了 VirusTotal 密钥） - 上传的文件存储在**会话级临时目录**中，并会自动清理 - 通过 Streamlit `session_state` 隔离，确保无跨会话数据泄露
Navigation Router] A --> P1[Dashboard] A --> P2[Version Detector & Validator] A --> P3[SDO Similarity Search] A --> P4[Attack Mapping] A --> P5[Malware Classifier] A --> P6[Credibility Assessment] A --> P7[STIX Chat] A --> P8[File Browser] A --> P9[Canonical Output] end subgraph Modules[" Backend Modules"] M1[enhanced_detector.py
Version & Format Detection] M2[enhanced_validator.py
Schema Validation] M3[converter.py
STIX 1.x → 2.1] M4[threat_classifier.py
Rule-based Classifier] M5[similarity_search.py
FAISS Similarity] M6[credibility_assessor.py
Score Engine] M7[rag_engine.py
ChromaDB + Ollama] M8[attack_type_mapper.py
MITRE Lookup] M9[storage.py
Session Storage] end subgraph AI[" AI / ML Layer"] AI1[Mistral-7B-Instruct
Malware Classification] AI2[all-MiniLM-L6-v2
Sentence Embeddings] AI3[FAISS Index
Vector Search] AI4[ChromaDB
RAG Knowledge Base] AI5[Ollama
Local LLM Inference] end subgraph Data[" Data Layer"] D1[STIX Bundles
JSON / XML] D2[MITRE ATT&CK
enterprise-attack.json] D3[Vector DB
faiss_db/] D4[Knowledge Base
knowledge_base/] end P1 & P5 & P6 --> Modules P7 --> M7 P3 --> M5 Modules --> AI AI --> Data ``` ## 数据流 ``` flowchart LR U([ User]) -->|Upload STIX\nJSON or XML| DET subgraph Pipeline["Processing Pipeline"] DET[ Version Detection\nSTIX 1.x / 2.0 / 2.1] --> VAL[ Schema Validation] VAL --> CONV[ Convert to STIX 2.1] CONV --> ANAL[ Analysis Engine] end subgraph Analysis["Analysis Modules"] ANAL --> TH[ Threat Classification] ANAL --> CR[ Credibility Score] ANAL --> GR[ Relationship Graph] ANAL --> ML[ Malware Classification] ANAL --> SIM[ Similarity Search] end TH & CR & GR & ML & SIM --> OUT([ Dashboard\nResults & Downloads]) ``` ## 恶意软件分类 Pipeline ``` sequenceDiagram participant U as User participant UI as Streamlit UI participant VDB as FAISS Vector DB participant LLM as Mistral-7B (HuggingFace) U->>UI: Upload STIX Bundle UI->>UI: Convert to STIX 2.1 UI->>UI: Extract malware objects loop For each malware object UI->>VDB: Semantic search (top-5 similar docs) VDB-->>UI: Relevant context chunks UI->>LLM: Prompt with context + malware description LLM-->>UI: JSON classification result UI->>UI: Validate against 15 allowed types end UI-->>U: Classification results + confidence scores ``` ## 可信度评估评分 ``` graph LR B[STIX 2.1 Bundle] --> F1[ Source credibility] B --> F2[ Timestamp validity] B --> F3[ Relationship richness] B --> F4[ Metadata completeness] B --> F5[ IOC validation via VirusTotal] B --> F6[ Object type diversity] F1 & F2 & F3 & F4 & F5 & F6 --> SCORE[ Credibility Score\n0 – 100] SCORE --> HIGH[" HIGH ≥ 75"] SCORE --> MED[" MEDIUM 45–74"] SCORE --> LOW[" LOW < 45"] ``` ## SDO 相似性搜索 Pipeline ``` flowchart TD A[STIX Bundle JSON] --> B[bundles_2_csv\nExtract SDOs to CSV] B --> C[summary_generator\nOllama-powered summaries] C --> D[clean_csv\nNormalize & dedupe] D --> E[similarity_search\nFAISS cosine similarity] E --> F[similarity_json\nTop-K similar IDs per object] F --> G[Ego Graph Explorer\nPyVis interactive graph] ``` ## STIX RAG 聊天 ``` flowchart LR KB[ Knowledge Base\nSTIX docs + MITRE data] -->|Chunk + Embed| VDB[(ChromaDB\nVector Store)] Q[ User Question] -->|Embed query| VDB VDB -->|Top-K relevant chunks| CTX[Context Window] CTX --> OL[ Ollama LLM] OL --> ANS[ Answer + Sources] ``` ## 项目结构 ``` stix-ai-dashboard/ ├── app.py # Navigation router ├── pages/ │ ├── 1_Dashboard.py # Main threat overview + graph │ ├── Version_Detector_and_Validator.py │ ├── SDO_Similarity_Search.py # Ego-graph explorer │ ├── Attack_Mapping.py # MITRE ATT&CK mapping │ ├── malware_classifier.py # ML-based malware typing │ ├── Credibility_Assessment_Module.py │ ├── STIX_Chat.py # RAG chatbot │ ├── Canonical_Output.py │ └── file_browser.py ├── modules/ │ ├── enhanced_detector.py # STIX version detection │ ├── enhanced_validator.py # Schema validation │ ├── converter.py # STIX 1.x → 2.1 converter │ ├── similarity_search.py # FAISS similarity │ ├── rag_engine.py # ChromaDB + Ollama RAG │ ├── credibility_assessor.py # Scoring engine │ ├── attack_type_mapper.py # MITRE lookup │ ├── storage.py # Session-based file storage │ └── ... ├── vector_db/ # FAISS malware vector store ├── knowledge_base/ # RAG source documents ├── enterprise-attack.json # MITRE ATT&CK dataset └── requirements.txt ``` ## 快速开始 ### 1. 安装依赖 ``` pip install -r requirements.txt ``` ### 2. （可选）启动 Ollama 以使用 STIX 聊天 ``` ollama serve ollama pull mistral # or your preferred model ``` ### 3. 运行应用 ``` streamlit run app.py ``` 在浏览器中打开 `http://localhost:8501`。 ## AI/ML 技术栈 | 组件 | 技术 | |---|---| | Sentence Embeddings | `sentence-transformers/all-MiniLM-L6-v2` | | 恶意软件 LLM | `mistralai/Mistral-7B-Instruct-v0.1` | | Vector Search | FAISS (`faiss-cpu`) | | RAG 知识库 | ChromaDB | | 本地 LLM 推理 | Ollama | | 图谱可视化 | PyVis + NetworkX | | 仪表板框架 | Streamlit | ## 核心依赖 ``` streamlit plotly pyvis stix2 stix2-validator stix2-elevator faiss-cpu chromadb sentence-transformers transformers torch langchain-community networkx pandas ollama ``` ## 安全与隐私 - 所有文件处理均在**本地**进行 — 不会向外部服务器发送任何数据（除非提供了 VirusTotal 密钥） - 上传的文件存储在**会话级临时目录**中，并会自动清理 - 通过 Streamlit `session_state` 隔离，确保无跨会话数据泄露
标签：AI风险缓解, DLL 劫持, FAISS, Kubernetes, Mistral-7B, STIX, Streamlit, 凭据扫描, 只读文件系统, 大语言模型, 威胁情报, 开发者工具, 特权检测, 访问控制, 逆向工具