butbeautifulv/veil
GitHub: butbeautifulv/veil
基于 Neo4j 的威胁情报图谱系统，通过三层 NATS 管道自动聚合漏洞、检测规则和情报源并构建关联关系网络。
Stars: 0 | Forks: 0
# Veil (漏洞利用智能层)
[](LICENSE)
**Veil** 是一个基于 Neo4j 的威胁情报图谱：涵盖漏洞 (CVE, CWE, CPE)、LOLbins 风格工件、检测内容 (Sigma/YARA/Caldera)、TI 订阅源、SBOM 公告以及代码规则模板。运行时被拆分为三个隔离层 —— **scrape**、**pipeline** 和 **graph** —— 通过 NATS JetStream 连接 (`scrape.>` → `ingest.>`)。
**许可证：** [MIT](LICENSE) · **贡献指南：** [CONTRIBUTING.md](CONTRIBUTING.md) · **Agents / AI：** [AGENTS.md](AGENTS.md) · **安全政策：** [SECURITY.md](SECURITY.md) · **行为准则：** [CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)
## 架构
```
flowchart LR
subgraph scrape [scrape]
SW[scrape_worker]
SW -->|harvest| NATS1[NATS SCRAPE]
end
subgraph pipeline [pipeline]
PW[pipeline_worker]
NATS1 --> PW
PW -->|commit| NATS2[NATS INGEST]
end
subgraph graph [graph]
IW[ingest_worker]
Neo4j[(Neo4j)]
API[HTTP API]
NATS2 --> IW --> Neo4j
API --> Neo4j
end
```
| 层级 | 路径 | 角色 |
|-------|------|------|
| **Scrape** | [scrape/](scrape/) | 拉取订阅源，Vitess 账本，发布 `harvest` |
| **Pipeline** | [pipeline/](pipeline/) | 规范化/去重 → `commit` (包含通过 [pipeline/pkg/nvd/parse](pipeline/pkg/nvd/parse/) 处理的 NVD CWE/CPE) |
| **Graph** | [graph/](graph/) | MERGE 写入 Neo4j；[serve/](graph/serve/) 提供 HTTP API + MCP 读取 Bolt |
部署：[deploy/](deploy/) · 契约：[docs/ingest-contract.md](docs/ingest-contract.md) · 运行时：[docs/threatintel-runtime.md](docs/threatintel-runtime.md)
## 快速开始
### 仅 Graph (演示 API + 可选 pack 导入)
```
docker compose up --build -d
```
| 端点 | 默认值 |
|----------|---------|
| Neo4j Browser | http://localhost:7474 (`neo4j` / `neo4jpassword`) |
| HTTP API | http://localhost:8090 |
`graph-bootstrap` 在发布时会导入默认的 graph pack ([versions.env](versions.env) → `GRAPH_PACK_VERSION`，当前为 [veil-graph-v0.4.2](https://github.com/butbeautifulv/veil/releases/tag/veil-graph-v0.4.2))，除非设置了 `GRAPH_PACK_SKIP=1`。本地 ZIP：[docker-compose.testpack.yml](docker-compose.testpack.yml)。
```
curl -sS http://localhost:8090/health
curl -sS http://localhost:8090/v1/categories | jq .
```
### 完整 scrape pipeline
```
./scripts/ops/compose-up-full.sh
```
端到端冒烟测试 (默认使用最小数据源)：
```
./scripts/test/smoke-scrape-e2e.sh --up
./scripts/test/smoke-scrape-e2e.sh
```
快速丰富 graph pack (~25 分钟)：[scripts/graph-pack/profile-fast-rich.sh](scripts/graph-pack/profile-fast-rich.sh) — [docs/graph-pack.md](docs/graph-pack.md)。
## 文档索引
| 文档 | 内容 |
|----------|----------|
| [AGENTS.md](AGENTS.md) | Cursor/agents：请先阅读 [docs/coding-style.md](docs/coding-style.md) |
| [docs/threatintel-runtime.md](docs/threatintel-runtime.md) | Compose、端口、环境变量、引导、API、MCP、NATS |
| [deploy/README.md](deploy/README.md) | 各层 compose、扩缩容、冒烟测试、graph pack 发布 |
| [scrape/README.md](scrape/README.md) | Scrape 数据源和环境变量 |
| [pipeline/README.md](pipeline/README.md) | Pipeline worker 和规范化 |
| [graph/README.md](graph/README.md) | Ingest、API、MCP、Neo4j 客户端 |
| [graph/ingest/README.md](graph/ingest/README.md) | JetStream → Neo4j 消费者 |
| [docs/coding-style.md](docs/coding-style.md) | 架构、分层、PR 检查清单 |
| [docs/ontology-appsec.md](docs/ontology-appsec.md) | 标签、关系、路线图 |
| [docs/ingest-contract.md](docs/ingest-contract.md) | `harvest` / `commit`，JetStream |
| [graph/serve/](graph/serve/) | HTTP API + stdio MCP |
| [scripts/README.md](scripts/README.md) | 导出、packs、冒烟测试、去重 |
| [docs/graph-pack.md](docs/graph-pack.md) | Graph pack 导出、发布、导入 |
## Graph packs
参见 [docs/graph-pack.md](docs/graph-pack.md)。快捷路径：
```
./scripts/graph-pack/export-cypher.sh
# versions.env 中的 version 或: GRAPH_PACK_VERSION=v0.4.2
./scripts/graph-pack/build.sh
```
## MCP
```
cd graph/serve && go run ./cmd/mcp
```
详情：[graph/serve/](graph/serve/)，[docs/threatintel-runtime.md](docs/threatintel-runtime.md#mcp-stdio)。
## 冒烟测试 Cypher
```
MATCH (n) RETURN labels(n)[0] AS label, count(*) AS c ORDER BY c DESC LIMIT 20;
MATCH (v:Vulnerability)-[:HAS_CWE]->() RETURN count(*) AS has_cwe;
MATCH (v:Vulnerability)-[:AFFECTS]->(:CPE) RETURN count(*) AS affects;
```
## 测试
```
make test-scrape
make test-pipeline
make test-graph
```
标签：Caldera, CPE, CVE, DNS 解析, Docker Compose, EVTX分析, GPT, HTTP API, JetStream, LOLBins, NATS, Neo4j, PFX证书, SBOM, YARA, 云资产可视化, 威胁情报, 安全大数据, 安全情报, 安全资讯聚合, 实时处理, 密码管理, 开发者工具, 攻击检测, 数字签名, 数据流水线, 日志审计, 漏洞管理, 知识库安全, 硬件无关, 网络安全, 请求拦截, 隐私保护