unknownspy333/Threat-Intelligence-Aggregator

# 威胁情报聚合器 一个基于 Python 的安全自动化工具包，用于从多个威胁情报源收集、解析、规范化 并关联入侵指标 —— 无需 AI 或机器学习。 ## 项目结构 ``` ti_aggregator/ ├── main.py # Pipeline orchestrator — run this ├── feed_parser.py # IOC extraction from all feed formats ├── normalizer.py # Schema normalization & deduplication ├── correlator.py # Cross-feed correlation & severity rating ├── blocklist_generator.py # Blocklist export (TXT / CSV / JSON) ├── reporter.py # Threat report generation ├── requirements.txt # Python dependencies ├── sample_feeds/ │ ├── feed1.csv # Sample CSV feed │ ├── feed2.json # Sample JSON feed │ ├── feed3.txt # Sample plain-text feed │ └── feed4_stix.json # Sample STIX 2.x bundle └── output/ ├── blocklists/ # Generated blocklist files └── reports/ # Generated threat reports ``` ## 快速开始 ### 1. 安装依赖 ``` pip install -r requirements.txt ``` ### 2. 运行 pipeline ``` python main.py ``` ### 3. 检查输出 ``` output/ ├── blocklists/ │ ├── ip_blocklist.txt <- paste into firewall │ ├── ip_blocklist.csv │ ├── ip_blocklist.json │ ├── domain_blocklist.txt <- paste into DNS sinkhole │ ├── url_blocklist.txt <- import into web filter │ ├── hash_blocklist.txt <- import into EDR / AV │ ├── email_blocklist.txt │ └── firewall_ipset.txt <- Linux ipset / iptables └── reports/ ├── ti_report_YYYYMMDD_HHMMSS.json └── ti_report_YYYYMMDD_HHMMSS.csv ``` ## 添加情报源 编辑 `main.py` 中的 `FEEDS` 列表： ``` FEEDS = [ # Local CSV file {"name": "AbuseIPDB_Export", "type": "csv", "path": "feeds/abuseipdb.csv"}, # Local JSON file {"name": "Custom_Feed", "type": "json", "path": "feeds/custom.json"}, # Plain text file {"name": "Internal_IOCs", "type": "txt", "path": "feeds/internal.txt"}, # STIX 2.x bundle {"name": "MISP_Feed", "type": "stix", "path": "feeds/misp_bundle.json"}, # Remote URL (fetched at runtime) {"name": "URLhaus", "type": "url", "path": "https://urlhaus.abuse.ch/downloads/text/"}, ] ``` ## 支持的 IOC 类型 | 类型 | 类别 | 示例 | |---------|----------|------------------------------------------------------------------| | ip | network | 185.220.101.45 | | domain | network | malware.example.com | | url | network | https://evil.ru/payload.exe | | md5 | file | 44d88612fea8a8f36de82e1278abb02f | | sha1 | file | da39a3ee5e6b4b0d3255bfef95601890afd80709 | | sha256 | file | aabbccdd...11223344 (64 hex chars) | | email | identity | phish@attacker-domain.xyz | ## 严重性评级 | 严重性 | 条件 | 建议操作 | |----------|--------------------------------|--------------------------------------------| | High | 在 3 个及以上独立源中出现 | 立即阻断 + SOC 升级处理 | | Medium | 在 2 个源中出现 | 加入黑名单 + 分析师审查 | | Low | 在 1 个源中出现 | 仅监控 / 记录 | ## 支持的情报源格式 | 格式 | 描述 | |--------|----------------------------------------------------------| | TXT | 纯文本，每行一个指标 | | CSV | 列式结构；扫描所有列（与 schema 无关） | | JSON | 任意 JSON 结构；搜索整个文档 | | STIX | STIX 2.x bundles；解析 indicator pattern 字段 | | URL | 返回文本的任何远程 HTTP/HTTPS 情报源 | ## Pipeline 架构 ``` [Feed Files / URLs] | v feed_parser.py <- Load + extract IOCs (regex) | v normalizer.py <- Validate, deduplicate, add metadata | v correlator.py <- Cross-feed grouping + severity rating | +---------> blocklist_generator.py -> output/blocklists/ | +---------> reporter.py -> output/reports/ ``` ## 使用的库 除 `requests` 外均为标准库： - `re` — 通过编译后的 regex 模式提取 IOC - `json` / `csv` — 情报源解析和结构化输出 - `requests` — 获取远程情报源 - `ipaddress` — IPv4 验证（过滤私有/保留范围） - `collections` — 用于关联的 Counter 和 defaultdict - `datetime` — ISO 8601 UTC 时间戳 - `os` / `sys` — 文件 I/O 和路径管理 ## 控制台输出示例 ``` ============================================================== THREAT INTELLIGENCE AGGREGATOR — REPORT SUMMARY ============================================================== Generated : 2026-05-27 10:30:00 UTC Feeds : 4 - SampleFeed_CSV - SampleFeed_JSON - SampleFeed_TXT - SampleFeed_STIX -------------------------------------------------------------- Total unique IOCs : 47 High severity : 8 Medium severity : 14 Low severity : 25 -------------------------------------------------------------- IOC Type Breakdown: ip : 12 ############ domain : 9 ######### url : 8 ######## sha256 : 6 ###### md5 : 4 #### sha1 : 3 ### email : 5 ##### -------------------------------------------------------------- Top High-Risk Indicators (showing up to 15): [ip ] 185.220.101.45 feeds=4 [domain] malware.example.com feeds=4 [sha256] aabbccdd11223344... feeds=3 [ip ] 91.108.4.0 feeds=3 ============================================================== ```

标签：IOC聚合, Python, 威胁情报, 开发者工具, 无后门, 逆向工具, 黑名单生成