H21lab/Anomaly-Detection

# 💡 更新 具备异常检测功能的在线应用程序 [**logs-analyzer**]( input.json Run script: cat input.pcap.json | python ad_tf.py -i normal.pcap.json \ -a anomaly.pcap.json -f field_1 field_2 .... field_n For fields the name of the fields from json ek should be used, e.g.: tshark -T ek -x -r ./res/input.pcap.gz | python ad_tf.py \ -i res/normal.json -a res/anomaly.json -f tcp_tcp_flags_raw \ tcp_tcp_dstport_raw Output pcap ad_test.pcap The script uses the tshark ek jsons including the raw hex data generated from pcaps by command as described above. The fields arguments are used for anomaly detection. The fields are used as columns, hashed and used as input to tensorflow neural classifier network. The neural classifier network is first trained with normal.pcap.json input with label 0 and with anomaly.pcap.json input with label 1. After training then from stdin is read the input.pcap.json and evaluated. The neural network predicts the label. The output pcap contains then the frames predicted by neural network as anomalies with label 1. ``` # 使用 tshark 的简单异常检测 ``` Simple script to help to detect anomalies in pcap file. Input is tshark ek json generate by: ./tshark -T ek -x -r trace.pcap > input.json Run script: cat input.json | python ad_simple.py field_1 field_2 .... field_n For fields the name of the fields from json ek should be used, e.g.: cat input.json | python ad_simple.py ip_ip_src ip_ip_dst Output pcap ad_test.pcap The script read the tshark ek json including the raw hex data. The input is generated from pcap using tshark. The fields arguments are used for simple anomaly detection. The behavior is similar like SQL GROUP BY command. The fields are hashed together and the output pcap contains the frames beginning with most unique combination of selected fields and descending to most frequent frames containing the selected fields. The following example cat input.json | python ad_simple.py ip_ip_src ip_ip_dst will generate pcap starting with less frequent combinations of source and dest IP pairs and descending to frames with common combinations. ``` ## 限制 本程序分发时希望其有用，但不提供任何保证。 ## 致谢 本代码由 Martin Kacer、H21 实验室创建，版权所有 2020。 https://www.h21lab.com

标签：Apex, JSON数据处理, pcap文件, Python, TensorFlow, tshark, 二进制发布, 分类器, 开源工具, 异常检测, 数据预处理, 无后门, 无监督学习, 机器学习, 流量解析, 深度学习, 监督学习, 神经网络, 网络安全, 脚本工具, 自动编码器, 逆向工具, 隐私保护