DedSecInside/TorBot
GitHub: DedSecInside/TorBot
TorBot 是一款专注于暗网的开源情报工具,旨在通过爬取 .onion 域名并可视化链接关系树来辅助安全研究人员进行暗网数据收集与分析。
Stars: 3961 | Forks: 660
████████╗ ██████╗ ██████╗ ██████╗ ██████╗ ████████╗
╚══██╔══╝██╔═══██╗██╔══██╗ ██╔══██╗██╔═████╗╚══██╔══╝
██║ ██║ ██║██████╔╝ ██████╔╝██║██╔██║ ██║
██║ ██║ ██║██╔══██╗ ██╔══██╗████╔╝██║ ██║
██║ ╚██████╔╝██║ ██║ ██████╔╝╚██████╔╝ ██║
╚═╝ ╚═════╝ ╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═╝
Open Source Intelligence Tool for the Dark Web
### 状态/社交链接
[](https://github.com/DedSecInside/TorBot/actions/workflows/pytest.yml)
[](https://github.com/DedSecInside/TorBot/actions/workflows/flake8.yml)
[](https://join.slack.com/t/dedsec-inside/shared_invite/zt-i4jcgj8w-1ecHiSdRubwuppAzz8~yfg)
[](https://www.codetriage.com/dedsecinside/torbot)
[]()
[]()
### 功能
1. Onion 爬虫 (.onion)
2. 返回页面标题或主机名(如果没有页面标题),以及包含网站简短描述的地址。
3. 将链接保存到数据库(未完成)
4. 输出网站的 HTML 或将其保存为 HTML 文件。(未完成)
5. 将链接树保存为 JSON 文件。
6. 爬取自定义域名
7. 检查链接是否存活
8. 内置更新程序
9. 构建链接关系的可视化树,可以快速查看或保存到文件
...(将会更新)
### 依赖项
- Tor(可选)
- Python ^3.9
- Poetry(可选)
### Python 依赖项
(更多详情请参见 pyproject.toml 或 requirements.txt)
## 安装说明
### TorBot
#### 使用 `venv`
* 如果使用 Python ^3.4,
```
python -m venv torbot_venv
source torbot_venv/bin/activate
pip install -r requirements.txt
pip install -e .
./main.py --help
```
#### 使用 `docker`
```
docker build -t {image_name} .
# 不使用 Tor 运行
docker run {image_name} poetry run python torbot -u https://example.com --depth 2 --visualize tree --save json --disable-socks5
# 使用 Tor 运行
docker run --network="host" {image_name} poetry run python torbot -u https://example.com --depth 2 --visualize tree --save json --disable-socks5
```
### 选项
usage: Gather and analyze data from Tor sites.
optional arguments:
-u URL, --url URL 指定要爬取的网站链接
--depth DEPTH 指定爬虫的最大深度(默认为 1)
-h, --help 显示此帮助信息并退出
--host 设置 SOCKS5 代理的 IP 地址(默认为 127.0.0.1)
--port 设置 SOCKS5 代理的端口(默认为 9050)
-v 显示 DEBUG 级别的日志,默认为 INFO
--version 显示 TorBot 的当前版本
--update 将 TorBot 更新到最新的稳定版本
-q, --quiet 阻止显示标题和 IP 地址
--save FORMAT 将结果保存到文件中。(tree, JSON)
--visualize FORMAT 将收集到的数据树可视化。(tree, JSON, table)
-i, --info Info 显示扫描站点的基本信息
* 注意:-u 是爬取所必需的参数
在此处阅读有关 torrc 的更多信息:[Torrc](https://github.com/DedSecInside/TorBoT/blob/master/Tor.md)
## 精选功能
- [x] 可视化模块重构
- [x] 为网络爬虫实现 BFS 搜索
- [x] 提高稳定性(优雅地处理错误,扩展测试覆盖率等)
- [x] 增加测试覆盖率
- [ ] 将最近的搜索结果保存到数据库
- [ ] 随机化 Tor 连接(随机 Header 和身份)
- [ ] 关键词/短语搜索
- [ ] 社交媒体集成
- [ ] 增加匿名性
- [ ] 截图捕获
## 贡献指南
### 发现了问题?
如果您在项目中遇到任何问题,请通过[在此处](https://github.com/DedSecInside/TorBot/issues/new/choose)创建新 issue 来告知我们。
### 开发者指南
我们欢迎对本项目做出贡献!请遵循以下几条准则:
1. Fork 该仓库并为您的贡献创建一个新分支。
2. 在向 `dev` 分支提交 pull request 之前,请确保运行 `pytest` 使您的代码通过所有测试。
3. 遵循 Python 代码的 [PEP8](https://www.python.org/dev/peps/pep-0008/) 风格指南。
4. 确保为任何新功能或更改添加适当的文档。
5. 提交 pull request 时,请提供对所做更改的详细描述。
## 参考资料
```
1. M. Glassman and M. J. Kang, “Intelligence in the internet age: The emergence and evolution of Open Source Intelligence (OSINT),” Comput. Human Behav., vol. 28, no. 2, pp. 673–682, 2012.
2. D. Bradbury, “In plain view: open source intelligence,” Comput. Fraud Secur., vol. 2011, no. 4, pp. 5–9, 2011.
3. B. Butler, B. Wardman, and N. Pratt, “REAPER: an automated, scalable solution for mass credential harvesting and OSINT,” 2016 APWG Symp. Electron. Crime Res., pp. 1–10, 2016.
4. B. Zantout and R. A. Haraty, “I2P Data Communication System I2P Data Communication System,” no. April 2002, 2014.
5. J. Qin, Y. Zhou, G. Lai, E. Reid, M. Sageman, and H. Chen, “The dark web portal project: collecting and analyzing the presence of terrorist groups on the web,” in Proceedings of the 2005 IEEE international conference on Intelligence and Security Informatics, 2005, pp. 623–624.
6. D. Moore, T. Rid, D. Moore, and T. Rid, “Cryptopolitik and the Darknet Cryptopolitik and the Darknet,” vol. 6338, 2016.
7. G. Weimann, “Going dark: Terrorism on the dark Web,” Stud. Confl. Terror., vol. 39, no. 3, pp. 195–206, 2016.
8. A. T. Zulkarnine, R. Frank, B. Monk, J. Mitchell, and G. Davies, “Surfacing collaborated networks in dark web to find illicit and criminal content,” in Intelligence and Security Informatics (ISI), 2016 IEEE Conference on, 2016, pp. 109–114.
9. T. Minárik and A.-M. Osula, “Tor does not stink: Use and abuse of the Tor anonymity network from the perspective of law,” Comput. Law Secur. Rev., vol. 32, no. 1, pp. 111–127, 2016.
10. K. Loesing, S. J. Murdoch, and R. Dingledine, “A Case Study on Measuring Statistical Data in the {T}or Anonymity Network,” in Proceedings of the Workshop on Ethics in Computer Security Research (WECSR 2010), 2010.
11. B. Nafziger, “Data Mining in the Dark : Darknet Intelligence Automation,” 2017.
12. I. Sanchez-Rola, D. Balzarotti, and I. Santos, “The onions have eyes: A comprehensive structure and privacy analysis of tor hidden services,” in Proceedings of the 26th International Conference on World Wide Web, 2017, pp. 1251–1260.
13. Mouli VR, Jevitha KP. “Web Services Attacks and Security-A Systematic Literature Review.”, Procedia Computer Science. 2016 Jan 1;93:870-7.
14. Cova M, Felmetsger V, Vigna G. "Vulnerability analysis of web-based applications. InTest and Analysis of Web Services" 2007 (pp. 363-394). Springer, Berlin, Heidelberg.
15. B. R. Holland, “Enabling Open Source Intelligence (OSINT) in private social networks,” 2012.
16. S. Nakamoto, “Bitcoin: A Peer-to-Peer Electronic Cash System,” Cryptogr. Mail. List https//metzdowd.com, 2009.
17. M. Wesam, A. Nabki, E. Fidalgo, E. Alegre, and I. De Paz, “Classifying Illegal Activities on Tor Network Based on Web Textual Contents”, vol. 1, pp. 35–43, 2017.
18. Sathyadevan S, Gangadharan S.“Crime analysis and prediction using data mining”. In Networks & Soft Computing (ICNSC), 2014 First International Conference on 2014 Aug 19 (pp. 406-412). IEEE.
19. Chau M, Chen H. "A machine learning approach to web page filtering using content and structure analysis. Decision Support Systems." 2008 Jan 1;44(2):482-94.
20. Ani R, Jose J, Wilson M, Deepa OS. “Modified Rotation Forest Ensemble Classifier for Medical Diagnosis in Decision Support Systems”, In Progress in Advanced Computing and Intelligent Engineering 2018 (pp. 137-146). Springer, Singapore.
21. Ani R, Augustine A, Akhil N.C. and Deepa O.S., 2016. “Random Forest Ensemble Classifier to Predict the Coronary Heart Disease Using Risk Factors”, In Proceedings of the International Conference on Soft Computing Systems (pp. 701-710). Springer, New Delhi.
```
## 维护者
- [X] [PS Narayanan](https://github.com/PSNAppz) - 共同所有者
- [X] [KingAkeem](https://github.com/KingAkeem) - 共同所有者
## 感谢所有贡献者
## 许可证
[GNU Public License](https://github.com/DedSecInside/TorBot/blob/dev/LICENSE.md)标签:BeEF, ESC4, OSINT, Python, Tor, 匿名网络, 命令控制, 威胁情报, 开发者工具, 数据采集, 无后门, 暗网, 暗网监控, 洋葱服务, 爬虫, 系统独立性, 网络安全, 网络安全, 请求拦截, 逆向工具, 隐私保护, 隐私保护