aahson45/AMDO_ISP_IDS

# 延迟感知网络保护：面向 ISP 环境中通用 IDS 编排的硬件加速 ML 中间层 **数据科学硕士论文** **意大利坎帕尼亚大学** **行业合作伙伴：** TIM S.p.A. (Telecom Italia) **作者：** Ali Ahson ([aahson45@gmail.com](mailto:aahson45@gmail.com)) **导师：** Prof. Fiammetta Marulli **学年：** 2025/2026 ## 📌 1. 执行摘要 互联网服务提供商面临着一个运营权衡：要么执行浅层数据包转发以维持超低网络延迟，要么执行深度包检测以检测复杂的网络威胁。在转发路径中直接运行内联入侵检测设备会引入不可接受的延迟和抖动（5.0–50.0 毫秒），迫使运营商采用被动监控而非主动威胁阻断。 **ADMO (Adaptive Domain-Adapted Middleware Orchestrator)** 是一种带外机器学习中间件，旨在消除这种权衡[cite: 1, 4, 5]。ADMO 在脱离关键转发路径的 SPAN 镜像数据包遥测数据上运行，在增加 **0.00 毫秒同步转发延迟** 的同时，在实时网络流量上实现了 **95.16% 的准确率** 和 **97.96% 的 DoS 召回率**。 ## 📐 2. 系统架构 ``` [ INTERNET / WAN ] │ ▼ ┌─────────────────────────────────┐ │ pfSense Firewall (Netgate VM) │ <--- Data Plane (<0.10 ms) └─────────────────────────────────┘ │ │ (Mirrored SPAN Port) │ │ (Forwarded Traffic) ▼ ▼ ┌─────────────────────────────────────────┐ [ LAN / VICTIM HOST ] │ Suricata 7 / CICFlowMeter Receiver │ └─────────────────────────────────────────┘ │ ▼ (EVE JSON / Flow Streams)[cite: 1] ┌──────────────────────────────────────────────────────────────────────────┐ │ ANALYTICS PLANE (ADMO MIDDLEWARE) │ │ │ │ ┌────────────────────────┐ ┌───────────────────────────────┐ │ │ │ Tier 1 Fast-Path │ ------> │ Tier 3 Multi-Class │ │ │ │ Binary XGBoost │ (Pass) │ XGBoost / LightGBM Classifier│ │ │ └────────────────────────┘ └───────────────────────────────┘ │ │ │ (Threat Flagged) │ (Threat Flagged) │ │ └──────────────────┬───────────────────┘ │ │ ▼ │ │ ┌───────────────────────────────────┐ │ │ │ Universal Orchestrator Adapter │ │ │ └───────────────────────────────────┘ │ └──────────────────────────────────┬───────────────────────────────────────┘ │ ▼ (Asynchronous Kernel Call)[cite: 1] [ pfctl -t snort2c -T add ][cite: 1] ``` ## 🗂️ 3. 仓库目录结构 ``` ├── README.md # Main Research & Execution Guide ├── pipeline/ # Processing & Execution Scripts │ ├── 00_diagnose.py # Phase 1: Data Integrity & Health Diagnostics │ ├── 01_label_normalization.py # Phase 1: Taxonomy Normalization Across Datasets │ ├── 02_merge_and_clean.py # Phase 1: Cleaning & Infinite Value Filtering │ ├── 03_feature_selection.py # Phase 1: Random Forest & ANOVA Golden 12 Selection │ ├── 04_class_balance_report.py # Phase 1: Multi-Class Imbalance Analysis │ ├── 05_ctgan_smoteenn_augment.py # Phase 1: Generative CTGAN + SMOTEENN Minority Synthesis │ ├── 06_train_test_split.py # Phase 1: Stratified Temporal Train/Val/Test Split │ ├── 07_train_fastpath_binary.py # Phase 2: Tier 1 Fast-Path Binary XGBoost Training │ ├── 08_train_batch_aggregator.py # Phase 2: Tier 2 5-Minute Window Flow Aggregation │ ├── 09_train_full_classifier.py # Phase 2: Tier 3 Multi-Class XGBoost & LightGBM Training │ ├── 10_latency_bench.py # Phase 2: Type A ML Pipeline Benchmark (MP1–MP4) │ ├── 11_phase5_evaluate_lab_flows.py # Phase 5: Zero-Shot Static Model Failure Diagnostic │ ├── 12_phase5_admo_domain_adaptation.py# Phase 5: ADMO Hybrid Retraining & Adaptation │ ├── 13_phase6_realtime_inference.py # Phase 6: Micro-Batch Real-Time Streaming Engine │ ├── 14_phase6_shap_explainability.py # Phase 6: Native C++ TreeSHAP Attribution │ ├── 15_phase6_generate_thesis_artifacts.py # Phase 6: LaTeX Tables & PNG Figure Generator │ ├── 16_suricata_eve_orchestrator.py # Component 4: Suricata 7 EVE JSON Adapter │ ├── bench_baseline_latency.py # Phase 3: Type B Network Forwarding Latency │ └── generate_lab_traffic.py # Phase 4: Kali Linux Traffic Generator Node │ ├── models/ # Serialized Model Binaries │ ├── tier1_fastpath_binary.json # Static Baseline Tier 1 Binary Model │ ├── tier1_admo_adapted.json # Live Adapted Tier 1 Model │ ├── tier3_full_classifier_xgboost.json # Static Baseline Tier 3 Model │ ├── tier3_full_classifier_lightgbm.txt # Static LightGBM Multi-Class Model │ └── tier3_admo_adapted.json # Live Adapted Tier 3 Model │ ├── data/ # PCAP Captures & JSON Manifests │ ├── wan_capture.pcap # WAN Interface Capture (em0) │ ├── lan_capture.pcap # LAN Interface Capture (em1) │ ├── phase5_mixed_traffic.pcap # Live Lab Traffic Run │ ├── 06_split_manifest.json # Train/Val/Test Split Metadata │ └── label_mapping.json # Class Taxonomy Mappings │ ├── reports/ # Execution Summary Reports │ ├── 00_diagnose_report.txt │ ├── 01_label_report.txt │ ├── 02_merge_report.txt │ ├── 03_feature_selection_report.txt │ ├── 04_class_balance_report.txt │ ├── 05_augmentation_report.txt │ ├── 06_train_test_split_report.txt │ ├── 07_tier1_fastpath_report.txt │ ├── 08_batch_aggregator_report.txt │ ├── 09_tier3_full_classifier_report.txt │ ├── 10_latency_bench_report.txt │ └── phase3_baseline_latency_report.txt │ └── artifacts/ # Thesis Figures & LaTeX Source Tables ├── figure_confusion_matrix_comparison.png ├── figure_shap_feature_shift.png ├── table_admo_performance.tex └── table_realtime_metrics.tex ``` ## 💻 4. 前置条件与安装说明 环境要求 分析主机：运行 Python 3.10+ 的 Windows 10/11 或 FreeBSD 防火墙设备：pfSense 2.7.x (FreeBSD 14) 流量生成器：Kali Linux 2024.x (hping3, hydra, python3-scapy, python3-requests)[cite: 7] 安装依赖 安装整个 pipeline 所需的 Python 库 ``` & "C:\Users\usid\AppData\Local\Programs\Python\Python310\python.exe" -m pip install numpy pandas xgboost lightgbm scikit-learn matplotlib seaborn scapy python-pptx ``` ## 📊 5. 关键实验结果摘要 实时分类性能（3,039 个留出的实时测试流） |评估指标 | 零样本基线 | ADMO 自适应模型 | 性能提升| | --- | --- | --- | --- | |整体实时准确率 | 4.79% | 95.16% | +90.37% 📈| |DoS / DDoS 精确率 | 0.0000 | 0.9699| +0.9699 📈| |DoS / DDoS 召回率 | 0.0000| 0.9796 | +0.9796 📈| |DoS / DDoS F1-Score| 0.0000| 0.9747 | +0.9747 📈| 已识别的真实 DoS 流：2,834 / 2,893 个流（97.96% 召回率） ## 已处理的真实 BENIGN 流：58 / 146 个流 ## 实时流处理与 SLA 指标 | **指标名称** | **测量值** | **运营背景** | |------------------------------------|------------------------|-----------------------------------------------------| | 实时遥测流总数 | 4,341 个流 | 在实时微批次流处理下进行评估 | | 一级快速路径卸载比率 | 96.45% (4,187 / 4,341) | DoS 在亚毫秒级的二元分类层进行过滤 | | 三级慢速路径升级比率 | 3.55% (154 / 4,341) | 模糊的流升级至多分类层 | | 每流平均推理延迟 | 781.05 µs | 完全在 1.0 毫秒的实时 SLA 预算内 | | 系统流处理容量 | 1,280.33 个流/秒 | 在单个 CPU 核心上维持纯 Python 吞吐量 | | 增加的转发延迟 | 1,280.33 个流/秒 | 由带外 SPAN 镜像设计保证 | ## ##⚡ 7. 一体化自动执行脚本 运行此 PowerShell 命令以按顺序执行所有 pipeline 阶段： ``` $PYTHON="C:\Users\usid3\AppData\Local\Programs\Python\Python310\python.exe" $BASE="C:\Users\usid3\Downloads\Ali Ahson Thesis Software\pipeline" Write-Host "=== STARTING FULL ADMO PIPELINE EXECUTION ===" -ForegroundColor Green & $PYTHON "$BASE\09_train_full_classifier.py"[cite: 10] & $PYTHON "$BASE\10_latency_bench.py"[cite: 9] & $PYTHON "$BASE\bench_baseline_latency.py"[cite: 8] & $PYTHON "$BASE\11_phase5_evaluate_lab_flows.py"[cite: 6] & $PYTHON "$BASE\12_phase5_admo_domain_adaptation.py"[cite: 5] & $PYTHON "$BASE\13_phase6_realtime_inference.py"[cite: 4] & $PYTHON "$BASE\14_phase6_shap_explainability.py"[cite: 3] & $PYTHON "$BASE\15_phase6_generate_thesis_artifacts.py"[cite: 2] & $PYTHON "$BASE\16_suricata_eve_orchestrator.py"[cite: 1] & $PYTHON "$BASE\generate_deck.py" Write-Host "=== ALL PIPELINE STAGES COMPLETED SUCCESSFULLY ===" -ForegroundColor Green ```

标签：Metaprompt, 逆向工具