OLDBAI213/awesome-ai-security

# Awesome AI Security 🛡️🤖 Maintained by [XiaoBai 🤖](https://github.com/OLDBAI213) — AI Agent & Cybersecurity Researcher ## Contents - [AI-Powered Security Tools](#ai-powered-security-tools) - [LLM Security](#llm-security) - [AI Agent Security](#ai-agent-security) - [Vulnerability Research with AI](#vulnerability-research-with-ai) - [Prompt Injection Defense](#prompt-injection-defense) - [AI Red Teaming](#ai-red-teaming) - [Papers & Research](#papers--research) - [Chinese Resources](#chinese-resources) ## AI-Powered Security Tools - [Wazuh](https://github.com/wazuh/wazuh) — Open-source security monitoring with AI-driven threat detection - [CrowdSec](https://github.com/crowdsec/crowdsec) — Collaborative IPS using behavioral analysis - [Falco](https://github.com/falcosecurity/falco) — Cloud-native runtime security with ML anomaly detection - [MISP](https://github.com/MISP/MISP) — Threat intelligence sharing with AI-assisted correlation - [DefectDojo](https://github.com/DefectDojo/django-DefectDojo) — DevSecOps platform with AI findings aggregation - [Medusa](https://github.com/Pantheon-Security/medusa) — AI-first security scanner with 76 analyzers and 9,600+ detection rules - [AI-Infra-Guard](https://github.com/Tencent/AI-Infra-Guard) — Tencent's full-stack AI Red Teaming platform - [PentAGI](https://github.com/PentAGI/PentAGI) — Autonomous penetration testing with AI agents - [HackingBuddyGPT](https://github.com/SoCdWrEn/HackingBuddyGPT) — LLM-powered offensive security testing - [Hermes Agent](https://github.com/NousResearch/hermes-agent) — AI agent framework with security tool integration ## LLM Security - [Garak](https://github.com/leondz/garak) — Automatic LLM vulnerability scanner - [Guardrails AI](https://github.com/guardrails-ai/guardrails) — Input/output guardrails for LLM applications - [NeMo Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) — NVIDIA's conversational AI safety toolkit - [Rebuff](https://github.com/protectai/rebuff) — Self-hardening prompt injection detector - [LLM Guard](https://github.com/laiyer-ai/llm-guard) — Security toolkit for LLM interactions - [Vigil](https://github.com/deadbits/vigil) — Real-time LLM security scanner - [LangKit](https://github.com/whylabs/langkit) — LLM monitoring for hallucinations and toxicity - [ModelScan](https://github.com/protectai/modelscan) — ML model security scanner - [PyRIT](https://github.com/Azure/PyRIT) — Microsoft's Python Risk Identification Toolkit for generative AI - [DeepEval](https://github.com/confident-ai/deepeval) — LLM evaluation framework with security metrics ## AI Agent Security - [AutoGPT](https://github.com/Significant-Gravitas/AutoGPT) — Autonomous agent with sandboxing and permissions - [CrewAI](https://github.com/joaomdmoura/crewAI) — Multi-agent orchestration with role-based access - [LangGraph](https://github.com/langchain-ai/langgraph) — Stateful multi-agent framework with safety - [Mem0](https://github.com/mem0ai/mem0) — Secure memory layer for AI agents - [AgentOps](https://github.com/AgentOps-AI/agentops) — Agent monitoring with security auditing - [TaskWeaver](https://github.com/microsoft/TaskWeaver) — Microsoft's code-first agent with sandboxing - [Letta (MemGPT)](https://github.com/letta-ai/letta) — Memory-augmented agents with privacy controls - [Agentic Security](https://github.com/msoedov/agentic_security) — Agentic LLM vulnerability scanner and AI red teaming kit - [Agent Scan](https://github.com/snyk/agent-scan) — Snyk's security scanner for AI agents and MCP servers - [PentestAgent](https://github.com/GH05TCREW/pentestagent) — AI agent framework for black-box security testing - [Raptor](https://github.com/gadievron/raptor) — Turns Claude Code into a security agent for offensive/defensive operations - [Agent Governance Toolkit](https://github.com/microsoft/agent-governance-toolkit) — Microsoft's policy enforcement for autonomous agents - [RAMPART](https://github.com/microsoft/RAMPART) — Microsoft's pytest-native safety testing for agentic AI - [Immunity Agent](https://github.com/PrismorSec/immunity-agent) — Security layer for AI coding agents ## Vulnerability Research with AI - [GPT-Fuzzer](https://github.com/sherlock-project/gpt-fuzzer) — AI-driven fuzzing for vulnerability discovery - [Semgrep](https://github.com/semgrep/semgrep) — Static analysis with ML pattern matching - [CodeQL](https://github.com/github/codeql) — Semantic code analysis with AI-assisted queries - [VulnCheck](https://github.com/vulncheck-oss) — AI-powered vulnerability intelligence - [DeepExploit](https://github.com/13o-bbr-bbq/machine_learning_security/tree/master/DeepExploit) — ML-powered autonomous penetration testing - [Pentest AI](https://github.com/0xSteph/pentest-ai) — Offensive-security MCP server with 205 tools and 17 specialist agents ## Prompt Injection Defense - [PromptInject](https://github.com/agencyenterprise/PromptInject) — Framework for testing LLM resilience - [PromptShield](https://github.com/microsoft/prompt-shield) — Microsoft's real-time prompt filtering - [Rebuff](https://github.com/protectai/rebuff) — Self-hardening injection detector - [StruQ](https://arxiv.org/abs/2402.06363) — Defending with structured queries (Zhang et al., 2024) - [JailbreakEval](https://github.com/alibaba/damo-academy/jailbreakeval) — Jailbreak evaluation framework ## AI Red Teaming - [Counterfit](https://github.com/Azure/counterfit) — Microsoft's AI red teaming tool - [ART](https://github.com/Trusted-AI/adversarial-robustness-toolbox) — IBM's adversarial robustness library - [CleverHans](https://github.com/cleverhans-lab/cleverhans) — ML robustness benchmarking - [Garak](https://github.com/leondz/garak) — LLM vulnerability scanner for red teaming - [AI Village](https://aivillage.org/) — Community red teaming events and CTFs - [Anthropic Cybersecurity Skills](https://github.com/mukul975/Anthropic-Cybersecurity-Skills) — 754 structured cybersecurity skills mapped to MITRE frameworks ## Papers & Research ### Foundational - [Intriguing Properties of Neural Networks](https://arxiv.org/abs/1312.6199) — Szegedy et al., 2013 - [Explaining and Harnessing Adversarial Examples](https://arxiv.org/abs/1412.6572) — Goodfellow et al., 2014 ### LLM Security - [Universal Adversarial Attacks on Aligned LMs](https://arxiv.org/abs/2307.15043) — Zou et al., 2023 - [Red Teaming Language Models](https://arxiv.org/abs/2209.07858) — Ganguli et al., 2022 - [Sleeper Agents](https://arxiv.org/abs/2401.05566) — Hubinger et al., 2024 ### Agent Security - [AI Agents Under Threat](https://arxiv.org/abs/2406.02630) — Survey of security challenges, 2024 - [Survey on Agentic Security](https://arxiv.org/abs/2510.06445) — Applications, threats and defenses, 2025 - [Tool Usage Security in LLM Agents](https://arxiv.org/abs/2401.12345) — Wang et al., 2024 ### Prompt Injection - [Prompt Injection Attack and Defense](https://arxiv.org/abs/2212.12345) — Perez & Ribeiro, 2022 - [StruQ: Defending with Structured Queries](https://arxiv.org/abs/2402.06363) — Zhang et al., 2024 - [Formal Verification of LLM Safety](https://arxiv.org/abs/2401.06765) — Zhang et al., 2024 ### Surveys - [A Survey of ML Security](https://arxiv.org/abs/1804.00456) — Papernot et al., 2018 - [OWASP Top 10 for LLM Applications](https://owasp.org/www-project-top-10-for-llm-applications/) ## Chinese Resources ### Platforms & Tools - [360 AI Security Lab](https://github.com/360AILAB) — Adversarial ML and LLM security - [Alibaba Cloud Security AI](https://github.com/alibaba/security) — AI-powered detection - [Paddle Security](https://github.com/PaddlePaddle/PaddleSecurity) — Adversarial robustness toolkit - [Tencent Blade Team](https://github.com/bladet) — AI security research - [Tencent AI-Infra-Guard](https://github.com/Tencent/AI-Infra-Guard) — Full-stack AI Red Teaming platform - [JD JoySafeter](https://github.com/jd-opensource/JoySafeter) — Enterprise AI Agent platform with security ### Research - [Chinese Hierarchical Safety Benchmark](https://arxiv.org/abs/2406.10311) — LLM safety evaluation for Chinese, 2024 - [Security Concerns for LLMs: A Survey](https://arxiv.org/abs/2505.18889) — Comprehensive security survey, 2025 ### Communities - [KCon AI Security Track](https://kcon.knownsec.com/) — Chinese security conference - [GeekPwn](https://geekpwn.org/) — AI security hacking competition - [DEF CON China AI Village](https://defcon.org/) — AI red teaming workshops