ayayoussfiii/soc-log-analytics
GitHub: ayayoussfiii/soc-log-analytics
基于 PySpark 和机器学习的 SOC 大数据日志实时分析系统，结合 Sigma 规则与异常检测实现混合威胁告警。
Stars: 0 | Forks: 0
# soc-log-analytics
# SOC 大数据日志分析
## 架构
```
Ingestion (Syslog/Kafka)
↓
Normalisation JSON commun
↓
ETL PySpark (Window aggregations + Feature engineering)
↓
Feature Store ──────────────────────────────┐
↓ ↓
Détection hybride Flask API
├── Rules engine (Sigma) POST /analyze
├── Isolation Forest ML GET /alerts
└── Score fusion POST /train
↓
Alert Store (Parquet / JSON)
↓
SOC Dashboard (Live feed + Export)
```
## 项目结构
```
soc-log-analytics/
├── ingestion/
│ ├── log_parser.py # Parser Syslog RFC 3164/5424, auth.log
│ ├── normalizer.py # Schéma JSON commun
│ └── kafka_watcher.py # File watcher / Kafka
├── etl/
│ ├── spark_pipeline.py # Pipeline PySpark complet
│ ├── feature_engineering.py
│ └── window_aggregations.py
├── detection/
│ ├── hybrid_engine.py # Moteur hybride principal
│ ├── sigma_rules.py # Règles Sigma
│ ├── isolation_forest.py # Modèle ML
│ └── score_fusion.py # Fusion des scores
├── store/
│ ├── feature_store.py # Brute-force counters, entropy
│ └── alert_store.py # Stockage alertes JSON/Parquet
├── api/
│ └── app.py # Flask API REST
├── dashboard/
│ └── soc_dashboard.py # Live feed + Export rapport
├── tests/
├── requirements.txt
├── docker-compose.yml
└── .env.example
```
## 安装说明
### 1. 克隆项目
```
git clone https://github.com/ayayoussfiii/soc-log-analytics.git
cd soc-log-analytics
```
### 2. 创建虚拟环境
```
python -m venv venv
venv\Scripts\activate # Windows
source venv/bin/activate # Linux/Mac
```
### 3. 安装依赖
```
pip install -r requirements.txt
```
### 4. 配置环境变量
```
cp .env.example .env
# 使用您的值编辑 .env
```
## 使用说明
### 启动 Flask API
```
python api/app.py
```
### 分析日志
```
curl -X POST http://localhost:5000/analyze \
-H "Content-Type: application/json" \
-d '{"log": "May 24 13:00:01 myhost sshd[1234]: Failed password for root from 192.168.1.1"}'
```
### 查看告警
```
curl http://localhost:5000/alerts
curl http://localhost:5000/alerts?risk=HIGH
```
### 启动 Dashboard
```
python dashboard/soc_dashboard.py
```
### 运行 PySpark pipeline
```
python etl/spark_pipeline.py data/logs data/output
```
## API Endpoints
| 方法 | Endpoint | 描述 |
|---------|----------|-------------|
| POST | `/analyze` | 分析原始日志 |
| GET | `/alerts` | 获取告警 |
| POST | `/train` | 训练 ML 模型 |
| GET | `/health` | API 状态 |
## MITRE ATT&CK 覆盖范围
| 技术 | ID | Sigma 规则 |
|-----------|----|-------------|
| Brute Force | T1110 | SIGMA-001 |
| Privilege Escalation | T1548 | SIGMA-002 |
| Valid Accounts | T1078 | SIGMA-003 |
| Password Spraying | T1110.003 | SIGMA-004 |
| Endpoint DoS | T1499 | SIGMA-005 |
## Docker
```
docker-compose up -d
```
## 📄 License
MIT License
标签：AMSI绕过, Apex, Flask, PE 加载器, PySpark, 大数据, 威胁检测, 安全运营, 扫描框架, 机器学习, 目录扫描, 网络测绘, 请求拦截, 逆向工具