mig-Rwa/Custom-Log-Based-Intrusion-Detection-System
GitHub: mig-Rwa/Custom-Log-Based-Intrusion-Detection-System
一个轻量级 Python 入侵检测引擎，专门解析 Linux 身份验证日志，通过阈值与时间窗口关联算法检测暴力破解、权限提升等攻击行为，并自动映射到 MITRE ATT&CK 框架。
Stars: 0 | Forks: 0
# 🛡️ 基于 Log 的自定义入侵检测系统 (IDS)
一个轻量级的、基于 Python 的入侵检测系统，用于监控 Linux 身份验证日志和系统日志，检测暴力破解、权限提升尝试和侦察活动。生成结构化的、映射到 MITRE ATT&CK 的警报，并可选转发至 Splunk、ELK 或 Syslog 等 SIEM。
## 📋 目录
- [概述](#overview)
- [功能](#features)
- [架构](#architecture)
- [检测规则](#detection-rules)
- [项目结构](#project-structure)
- [安装](#installation)
- [用法](#usage)
- [警报输出](#alert-output)
- [SIEM 集成](#siem-integration)
- [配置](#configuration)
- [输出示例](#sample-output)
- [路线图](#roadmap)
- [贡献](#contributing)
- [许可证](#license)
## 概述
安全运营中心 (SOC) 依靠检测工程来实时识别威胁。本项目实现了一个**自定义 IDS**，它能够：
1. **解析** Linux 日志文件（`auth.log`、`syslog`）为结构化事件
2. **关联** 使用可配置的阈值和时间窗口逻辑对事件进行关联
3. **检测** 暴力破解、撞库、权限提升和侦察模式
4. **生成** 映射到 MITRE ATT&CK 框架的结构化 JSON 警报
5. **转发** 警报到 SIEM（Splunk HEC、Elasticsearch 或 Syslog）
它附带了包含嵌入式攻击场景的真实样本日志，使其开箱即用，适合学习、测试或演示。
## 功能
| Feature | Description |
|---------|-------------|
| **6 Detection Rules** | SSH brute-force, password spraying, root login, sudo abuse, port scan indicators, session anomalies |
| **MITRE ATT&CK Mapping** | Every alert includes tactic, technique ID, and technique name |
| **Threshold + Time-Window Correlation** | Sliding window engine groups events by source IP and applies configurable thresholds |
| **IP & User Whitelisting** | Reduce false positives by whitelisting trusted IPs and service accounts |
| **Multiple Output Formats** | JSON, CEF (ArcSight), and flat log files |
| **SIEM Forwarding** | Built-in connectors for Splunk (HEC), Elasticsearch, and generic Syslog (UDP/TCP) |
| **Continuous Monitoring Mode** | Tail-mode file monitoring with configurable polling intervals |
| **Log Generator** | Built-in simulator generates realistic attack and benign traffic logs |
| **Log Rotation Handling** | Detects and recovers from log rotation during continuous monitoring |
| **Zero External Dependencies for Core** | Only `PyYAML` required; `requests` optional for HTTP-based SIEM forwarding |
## 架构
```
┌──────────────────────────────────────────────────────────────┐
│ ids.py (Main) │
│ Orchestrator & CLI Entry Point │
└──────────┬───────────────┬──────────────────┬────────────────┘
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌─────────────┐ ┌───────────────┐
│ Log Parser │ │ Detection │ │ Alert │
│ │ │ Engine │ │ Manager │
│ • Syslog │ │ │ │ │
│ • auth.log │ │ • Pattern │ │ • Console │
│ • IP extract │ │ matching │ │ • JSON file │
│ • User │ │ • Threshold │ │ • Log file │
│ extract │ │ • Time- │ │ • CEF format │
│ • Whitelist │ │ window │ │ │
│ filtering │ │ • MITRE │ │ │
│ │ │ tagging │ │ │
└──────────────┘ └─────────────┘ └───────┬───────┘
│
▼
┌───────────────┐
│ SIEM │
│ Forwarder │
│ │
│ • Splunk HEC │
│ • Elastic API │
│ • Syslog UDP │
└───────────────┘
```
## 检测规则
| # | Rule | Severity | Threshold | Window | MITRE ATT&CK |
|---|------|----------|-----------|--------|---------------|
| 1 | SSH Brute-Force | HIGH | 5 failed logins | 60s | T1110.001 — Password Guessing |
| 2 | Invalid User / Password Spraying | MEDIUM | 3 attempts | 120s | T1110.003 — Password Spraying |
| 3 | Sudo Abuse / Privilege Escalation | HIGH | 3 failures | 300s | T1548.003 — Sudo Caching |
| 4 | Port Scan Indicators | MEDIUM | 10 connections | 30s | T1046 — Network Service Discovery |
| 5 | Direct Root Login Attempts | CRITICAL | 1 attempt | 60s | T1078.003 — Local Accounts |
| 6 | Rapid Session Anomalies | LOW | 10 sessions | 60s | T1078 — Valid Accounts |
所有规则均可在 `config/ids_config.yaml` 中配置。无需修改代码即可调整阈值、时间窗口、严重程度和模式。
## 项目结构
```
Custom-Log-Based-Intrusion-Detection-System/
│
├── ids.py # Main entry point & CLI
├── requirements.txt # Python dependencies
├── .gitignore
│
├── config/
│ └── ids_config.yaml # Detection rules, SIEM settings, whitelists
│
├── modules/
│ ├── __init__.py
│ ├── config_loader.py # YAML config parser with dot-notation access
│ ├── log_parser.py # Parses auth.log/syslog into structured events
│ ├── detection_engine.py # Threshold + time-window correlation engine
│ ├── alert_manager.py # Console, JSON, CEF, and log output
│ ├── siem_forwarder.py # Splunk HEC, ELK, Syslog connectors
│ └── log_generator.py # Realistic attack log simulator
│
├── logs/
│ ├── sample_auth.log # Pre-built auth.log with 6 attack scenarios
│ └── sample_syslog.log # Firewall blocks + system events
│
└── alerts/
└── .gitkeep # Alert output directory (JSON + logs)
```
## 安装
### 前置条件
- Python 3.8 或更高版本
- pip (Python 包管理器)
### 设置
```
# 克隆 repository
git clone https://github.com/mig-Rwa/Custom-Log-Based-Intrusion-Detection-System.git
cd Custom-Log-Based-Intrusion-Detection-System
# 创建 virtual environment (推荐)
python -m venv .venv
source .venv/bin/activate # Linux/Mac
# .venv\Scripts\activate # Windows
# 安装 dependencies
pip install -r requirements.txt
```
## 用法
### 单次分析 (默认)
分析样本日志文件并打印所有检测到的警报：
```
python ids.py
```
### 持续监控模式
实时监控日志文件，按可配置的间隔检查新条目：
```
python ids.py --continuous
```
### 生成新的样本日志
重新生成随机攻击 + 正常日志数据以供测试：
```
python ids.py --generate-logs
```
### 自定义配置
指向不同的配置文件：
```
python ids.py --config /path/to/custom_config.yaml
```
### 所有选项
```
usage: ids.py [-h] [--config CONFIG] [--continuous] [--generate-logs]
Custom Log-Based Intrusion Detection System
optional arguments:
-h, --help show this help message and exit
--config, -c Path to IDS configuration file
--continuous, -m Run in continuous monitoring mode
--generate-logs, -g Generate sample log files for testing
```
## 警报输出
### 控制台输出
警报以 ANSI 颜色编码的严重程度打印：
```
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
🚨 ALERT: ssh_brute_force
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Severity : HIGH
Alert ID : 4a22ee71-ca0
Description : Detect SSH brute-force attempts (multiple failed logins)
Source IP : 203.0.113.50
Target Users: admin
Event Count : 8 (threshold: 5)
Time Window : 60s
First Seen : 2026-03-01T02:14:01
Last Seen : 2026-03-01T02:14:15
Hostname : webserver
MITRE ATT&CK:
Tactic : Credential Access
Technique : T1110.001 — Brute Force: Password Guessing
Evidence (sample):
[1] Mar 1 02:14:01 webserver sshd[12001]: Failed password for admin...
[2] Mar 1 02:14:03 webserver sshd[12001]: Failed password for admin...
[3] Mar 1 02:14:05 webserver sshd[12002]: Failed password for admin...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
```
### JSON 文件 (`alerts/ids_alerts.json`)
```
{
"alert_id": "4a22ee71-ca0",
"timestamp": "2026-03-01T20:03:22.544",
"rule_name": "ssh_brute_force",
"severity": "HIGH",
"source_ip": "203.0.113.50",
"target_users": ["admin"],
"event_count": 8,
"mitre_attack": {
"tactic": "Credential Access",
"technique_id": "T1110.001",
"technique_name": "Brute Force: Password Guessing"
},
"evidence": ["..."]
}
```
### 事件日志 (`alerts/ids_events.log`)
单行警报格式，便于 `grep` 和日志聚合：
```
[2026-03-01T20:03:22] [HIGH] rule=ssh_brute_force src_ip=203.0.113.50 users=admin count=8 mitre=T1110.001
```
## SIEM 集成
### Splunk (HTTP Event Collector)
1. 在 Splunk 中，进入 **Settings → Data Inputs → HTTP Event Collector**
2. 创建一个索引为 `security` 的新令牌
3. 更新 `config/ids_config.yaml`：
```
siem:
enabled: true
type: "splunk"
splunk:
hec_url: "https://YOUR-SPLUNK:8088/services/collector/event"
hec_token: "YOUR-HEC-TOKEN"
index: "security"
sourcetype: "custom_ids"
```
### Elasticsearch (ELK Stack)
```
siem:
enabled: true
type: "elk"
elk:
elasticsearch_url: "http://YOUR-ELK:9200"
index: "ids-alerts"
api_key: "YOUR-API-KEY"
```
### 通用 Syslog (UDP/TCP)
```
siem:
enabled: true
type: "generic_syslog"
generic_syslog:
host: "YOUR-SYSLOG-SERVER"
port: 514
protocol: "udp"
```
## 配置
所有配置位于 `config/ids_config.yaml`。主要部分：
| Section | Purpose |
|---------|---------|
| `general` | 监控间隔、日志格式、时区 |
| `log_sources` | 日志文件路径（样本或实时） |
| `detection_rules` | 规则定义：模式、阈值、时间窗口、严重程度、MITRE 映射 |
| `alerting` | 输出文件路径、控制台开关、警报格式 |
| `siem` | SIEM 类型、连接详情、凭据 |
| `whitelist` | 要从检测中排除的可信 IP 和用户名 |
### 指向实时日志 (Linux)
```
log_sources:
auth_log: "/var/log/auth.log"
syslog: "/var/log/syslog"
```
### 添加自定义检测规则
```
detection_rules:
my_custom_rule:
enabled: true
description: "Detect something suspicious"
threshold: 3
time_window: 120
severity: "HIGH"
patterns:
- "your regex pattern here"
mitre_attack:
tactic: "Initial Access"
technique: "T1190"
name: "Exploit Public-Facing Application"
```
## 输出示例
针对包含的样本日志运行 `python ids.py` 会产生：
| Severity | Rule | Source IP | MITRE Technique |
|----------|------|-----------|-----------------|
| 🔴 CRITICAL | Root Login Attempts | 45.33.32.156 | T1078.003 |
| 🟠 HIGH | SSH Brute-Force (admin) | 203.0.113.50 | T1110.001 |
| 🟠 HIGH | SSH Brute-Force (root) | 45.33.32.156 | T1110.001 |
| 🟡 MEDIUM | Password Spraying | 198.51.100.23 | T1110.003 |
| 🟡 MEDIUM | Port Scan Indicators | 192.0.2.100 | T1046 |
| 🔵 LOW | Session Anomaly | — | T1078 |
## 路线图
- [ ] 源 IP 地址的 GeoIP 丰富
- [ ] Email / Slack / Webhook 警报通知
- [ ] 用于警报可视化的仪表板 (Flask/Streamlit)
- [ ] PCAP / Zeek 日志解析支持
- [ ] Sigma 规则格式导入
- [ ] 自动化威胁情报源集成
- [ ] Docker 容器部署
## 许可证
本项目根据 MIT 许可证授权。详情请参见 [LICENSE](LICENSE)。
## 致谢
- [MITRE ATT&CK Framework](https://attack.mitre.org/) 用于威胁分类
- Linux `auth.log` 和 `syslog` 标准
- 开源网络安全社区
标签：Cloudflare, CSV导出, ELK, MITRE ATT&CK, PE 加载器, Python, SIEM集成, SSH安全, Syslog, 云计算, 免杀技术, 入侵检测系统, 协议分析, 威胁 hunting, 安全数据湖, 安全运营, 密码喷射, 异常检测, 恶意代码分类, 扫描框架, 插件系统, 数据统计, 无后门, 暴力破解检测, 权限提升, 端口扫描, 网络安全, 自动化响应, 规则引擎, 逆向工具, 隐私保护