enguard-ai/awesome-ai-guardrails

# 🛡️ Awesome AI Guardrails 🛡️   精选的 Awesome AI Guardrails 列表。 ## 分类 ### 主分类 | 名称 | 描述 | |------|-------------| | `security-and-privacy` | 安全与隐私 guardrails 确保内容保持安全、合乎道德且不含攻击性材料 | | `response-and-relevance` | 确保模型的回答准确、专注并与用户意图保持一致 | | `language-quality` | 确保可读性、连贯性和清晰度的高标准 | | `content-validation` | 确保内容的事实正确性和逻辑连贯性 | | `logic-validation` | 确保生成的代码和数据的逻辑和功能正确性 | ### 子分类 ### security-and-privacy | 子分类 | 描述 | |--------------|-------------| | `inappropriate-content` | 检测并过滤不当或露骨内容 | | `offensive-language` | 识别并过滤脏话或具有攻击性的语言 | | `prompt-injection` | 防止通过恶意 prompt 进行操纵的企图 | | `sensitive-content` | 标记文化、政治或社会上敏感的话题 | | `deepfake-detection` | 检测并过滤 deepfake 内容 | | `pii` | 识别并过滤个人身份信息 |

security-and-privacy 中的模型

| 名称 | 大小 | 任务 | |------|------|------| | [osmosis-ai/Osmosis-Structure-0.6B](https://huggingface.co/osmosis-ai/Osmosis-Structure-0.6B) | `0.6B` | `token-classification` | | [gliner-community/gliner_small-v2.5](https://huggingface.co/gliner-community/gliner_small-v2.5) | `0.7B` | `token-classification` | | [Marqo/nsfw-image-detection-384](https://huggingface.co/Marqo/nsfw-image-detection-384) | `0.006B` | `image-classification` | | [Freepik/nsfw_image_detector](https://huggingface.co/Freepik/nsfw_image_detector?not-for-all-audiences=true) | `0.086B` | `image-classification` | | [Organika/sdxl-detector](https://huggingface.co/Organika/sdxl-detector) | `0.086B` | `image-classification` | | [prithivMLmods/Deep-Fake-Detector-v2-Model](https://huggingface.co/prithivMLmods/Deep-Fake-Detector-v2-Model) | `0.086B` | `image-classification` | | [TostAI/nsfw-image-detection-large](https://huggingface.co/TostAI/nsfw-image-detection-large) | `0.0871B` | `image-classification` | | [Ateeqq/nsfw-image-detection](https://huggingface.co/Ateeqq/nsfw-image-detection) | `0.092B` | `image-classification` | | [Falconsai/nsfw_image_detection](https://huggingface.co/Falconsai/nsfw_image_detection) | `0.1B` | `image-classification` | | [OpenSafetyLab/ImageGuard](https://huggingface.co/OpenSafetyLab/ImageGuard) | `na` | `image-classification` | | [meta-llama/Llama-Guard-4-12B](https://huggingface.co/meta-llama/Llama-Guard-4-12B) | `12B` | `image-text-to-text` | | [meta-llama/Llama-Prompt-Guard-2-22M](https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-22M) | `0.022B` | `text-classification` | | [eliasalbouzidi/distilbert-nsfw-text-classifier](https://huggingface.co/eliasalbouzidi/distilbert-nsfw-text-classifier) | `0.068B` | `text-classification` | | [meta-llama/Llama-Prompt-Guard-2-86M](https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M) | `0.086B` | `text-classification` | | [ibm-granite/granite-guardian-hap-125m](https://huggingface.co/ibm-granite/granite-guardian-hap-125m) | `0.125B` | `text-classification` | | [ibm-granite/granite-guardian-hap-125m](https://huggingface.co/ibm-granite/granite-guardian-hap-125m) | `0.125B` | `text-classification` | | [protectai/deberta-v3-small-prompt-injection-v2](https://huggingface.co/protectai/deberta-v3-small-prompt-injection-v2) | `0.142B` | `text-classification` | | [protectai/deberta-v3-base-prompt-injection-v2](https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2) | `0.182B` | `text-classification` | | [TostAI/nsfw-text-detection-large](https://huggingface.co/TostAI/nsfw-text-detection-large) | `0.355B` | `text-classification` | | [MoritzLaurer/ModernBERT-large-zeroshot-v2.0](https://huggingface.co/MoritzLaurer/ModernBERT-large-zeroshot-v2.0) | `0.4B` | `text-classification` | | [madhurjindal/Jailbreak-Detector-2-XL](https://huggingface.co/madhurjindal/Jailbreak-Detector-2-XL) | `0.5B` | `text-classification` | | [google/shieldgemma-2b](https://huggingface.co/google/shieldgemma-2b) | `2B` | `text-classification` | | [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) | `1B` | `text-to-text-generation` | | [ai4privacy/llama-ai4privacy-multilingual-categorical-anonymiser-openpii](https://huggingface.co/ai4privacy/llama-ai4privacy-multilingual-categorical-anonymiser-openpii) | `0.15B` | `token-classification` |

### response-and-relevance | 子分类 | 描述 | |--------------|-------------| | `relevance` | 验证输入和输出之间的语义相关性 | | `prompt-address` | 确认响应正确针对用户的 prompt | | `url-validation` | 验证生成的 URL 的有效性 | | `factuality` | 将内容与外部知识源进行交叉验证 | | `refusal` | 拒绝回答不合适或不相关的问题 |

response-and-relevance 中的模型

| 名称 | 大小 | 任务 | |------|------|------| | [protectai/distilroberta-base-rejection-v1](https://huggingface.co/protectai/distilroberta-base-rejection-v1) | `0.0821B` | `text-classification` | | [s-nlp/E5-EverGreen-Multilingual-Small](https://huggingface.co/s-nlp/E5-EverGreen-Multilingual-Small) | `0.118B` | `text-classification` | | [lytang/MiniCheck-RoBERTa-Large](https://huggingface.co/lytang/MiniCheck-RoBERTa-Large) | `0.4B` | `text-classification` | | [lytang/MiniCheck-Flan-T5-Large](https://huggingface.co/lytang/MiniCheck-Flan-T5-Large) | `0.8B` | `text-classification` | | [ibm-granite/granite-guardian-3.1-2b](https://huggingface.co/ibm-granite/granite-guardian-3.1-2b) | `2B` | `text-classification` | | [bespokelabs/Bespoke-MiniCheck-7B](https://huggingface.co/bespokelabs/Bespoke-MiniCheck-7B) | `7B` | `text-classification` | | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) | `0.184B` | `text-classification` | | [PatronusAI/glider](https://huggingface.co/PatronusAI/glider) | `3.8B` | `text-classification` | | [flowaicom/Flow-Judge-v0.1](https://huggingface.co/flowaicom/Flow-Judge-v0.1) | `3.8B` | `text-classification` |

### language-quality | 子分类 | 描述 | |--------------|-------------| | `quality` | 评估输出的结构、相关性和连贯性 | | `translation-accuracy` | 确保翻译符合语境且语言学上准确 | | `duplicate-elimination` | 检测并移除冗余内容 | | `readability` | 评估针对目标受众的文本复杂度 |

language-quality 中的模型

| 名称 | 大小 | 任务 | |------|------|------| | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) | `0.109B` | `text-classification` | | [nvidia/quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) | `0.184B` | `text-classification` | | [facebook/nllb-200-distilled-600M](https://huggingface.co/facebook/nllb-200-distilled-600M) | `0.6B` | `text-to-text-generation` | | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) | `0.184B` | `text-classification` | | [PatronusAI/glider](https://huggingface.co/PatronusAI/glider) | `3.8B` | `text-classification` | | [flowaicom/Flow-Judge-v0.1](https://huggingface.co/flowaicom/Flow-Judge-v0.1) | `3.8B` | `text-classification` |

### content-validation | 子分类 | 描述 | |--------------|-------------| | `competitor-blocking` | 筛查提及竞争对手品牌或公司的内容 | | `price-validation` | 根据经过验证的来源验证与价格相关的数据 | | `source-verification` | 验证外部引语和参考资料的准确性 | | `gibberish-filter` | 识别并过滤无意义或不连贯的输出 |

content-validation 中的模型

| 名称 | 大小 | 任务 | |------|------|------| | [s-nlp/mdistilbert-base-formality-ranker](https://huggingface.co/s-nlp/mdistilbert-base-formality-ranker) | `0.142B` | `text-classification` | | [d4data/bias-detection-model](https://huggingface.co/d4data/bias-detection-model) | `0.3B` | `text-classification` | | [NousResearch/Minos-v1](https://huggingface.co/NousResearch/Minos-v1) | `0.4B` | `text-classification` | | [osmosis-ai/Osmosis-Structure-0.6B](https://huggingface.co/osmosis-ai/Osmosis-Structure-0.6B) | `0.6B` | `token-classification` | | [gliner-community/gliner_small-v2.5](https://huggingface.co/gliner-community/gliner_small-v2.5) | `0.7B` | `token-classification` |

### logic-validation | 子分类 | 描述 | |--------------|-------------| | `sql-validation` | 验证 SQL 查询的语法和安全性 | | `api-validation` | 确保 API 调用符合 OpenAPI 标准 | | `json-validation` | 验证 JSON 结构和 schema | | `logical-consistency` | 检查矛盾或不符合逻辑的陈述 | ## 模型 ### Text-Classification 模型 | 名称 | 大小 | 分类 | 子分类 | |------|------|----------|--------------| | [s-nlp/mdistilbert-base-formality-ranker](https://huggingface.co/s-nlp/mdistilbert-base-formality-ranker) | `0.142B` | `content-validation` | `quality` | | [d4data/bias-detection-model](https://huggingface.co/d4data/bias-detection-model) | `0.3B` | `content-validation` | `bias` | | [NousResearch/Minos-v1](https://huggingface.co/NousResearch/Minos-v1) | `0.4B` | `content-validation` | `refusal` | | [HuggingFaceFW/fineweb-edu-classifier](https://huggingface.co/HuggingFaceFW/fineweb-edu-classifier) | `0.109B` | `language-quality` | `quality` | | [nvidia/quality-classifier-deberta](https://huggingface.co/nvidia/quality-classifier-deberta) | `0.184B` | `language-quality` | `quality` | | [protectai/distilroberta-base-rejection-v1](https://huggingface.co/protectai/distilroberta-base-rejection-v1) | `0.0821B` | `response-and-relevance` | `rejection` | | [s-nlp/E5-EverGreen-Multilingual-Small](https://huggingface.co/s-nlp/E5-EverGreen-Multilingual-Small) | `0.118B` | `response-and-relevance` | `factuality` | | [lytang/MiniCheck-RoBERTa-Large](https://huggingface.co/lytang/MiniCheck-RoBERTa-Large) | `0.4B` | `response-and-relevance` | `factuality, logical-consistency, relevance` | | [lytang/MiniCheck-Flan-T5-Large](https://huggingface.co/lytang/MiniCheck-Flan-T5-Large) | `0.8B` | `response-and-relevance` | `factuality, logical-consistency, relevance` | | [ibm-granite/granite-guardian-3.1-2b](https://huggingface.co/ibm-granite/granite-guardian-3.1-2b) | `2B` | `response-and-relevance` | `factuality, logical-consistency, relevance` | | [bespokelabs/Bespoke-MiniCheck-7B](https://huggingface.co/bespokelabs/Bespoke-MiniCheck-7B) | `7B` | `response-and-relevance` | `factuality, logical-consistency, relevance` | | [nvidia/prompt-task-and-complexity-classifier](https://huggingface.co/nvidia/prompt-task-and-complexity-classifier) | `0.184B` | `response-and-relevance, language-quality` | `relevance, quality` | | [PatronusAI/glider](https://huggingface.co/PatronusAI/glider) | `3.8B` | `response-and-relevance, language-quality` | `factuality, logical-consistency, relevance, quality` | | [flowaicom/Flow-Judge-v0.1]() | `3.8B` | `response-and-relevance, language-quality` | `factuality, logical-consistency, relevance, quality` | | [meta-llama/Llama-Prompt-Guard-2-22M](https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-22M) | `0.022B` | `security-and-privacy` | `prompt-injection, jailbreaks` | | [eliasalbouzidi/distilbert-nsfw-text-classifier](https://huggingface.co/eliasalbouzidi/distilbert-nsfw-text-classifier) | `0.068B` | `security-and-privacy` | `inappropriate-content` | | [meta-llama/Llama-Prompt-Guard-2-86M](https://huggingface.co/meta-llama/Llama-Prompt-Guard-2-86M) | `0.086B` | `security-and-privacy` | `prompt-injection, jailbreaks` | | [ibm-granite/granite-guardian-hap-125m](https://huggingface.co/ibm-granite/granite-guardian-hap-125m) | `0.125B` | `security-and-privacy` | `toxicity, hallucination` | | [ibm-granite/granite-guardian-hap-125m](https://huggingface.co/ibm-granite/granite-guardian-hap-125m) | `0.125B` | `security-and-privacy` | `toxicity, hallucination` | | [protectai/deberta-v3-small-prompt-injection-v2](https://huggingface.co/protectai/deberta-v3-small-prompt-injection-v2) | `0.142B` | `security-and-privacy` | `prompt-injection` | | [protectai/deberta-v3-base-prompt-injection-v2](https://huggingface.co/protectai/deberta-v3-base-prompt-injection-v2) | `0.182B` | `security-and-privacy` | `prompt-injection` | | [TostAI/nsfw-text-detection-large](https://huggingface.co/TostAI/nsfw-text-detection-large) | `0.355B` | `security-and-privacy` | `inappropriate-content` | | [MoritzLaurer/ModernBERT-large-zeroshot-v2.0](https://huggingface.co/MoritzLaurer/ModernBERT-large-zeroshot-v2.0) | `0.4B` | `security-and-privacy` | `inappropriate-content, offensive-language, prompt-injection, sensitive-content` | | [madhurjindal/Jailbreak-Detector-2-XL](https://huggingface.co/madhurjindal/Jailbreak-Detector-2-XL) | `0.5B` | `security-and-privacy` | `jailbreaks` | | [google/shieldgemma-2b](https://huggingface.co/google/shieldgemma-2b) | `2B` | `security-and-privacy` | `inappropriate-content, offensive-language, prompt-injection, sensitive-content` | ### Token-Classification 模型 | 名称 | 大小 | 分类 | 子分类 | |------|------|----------|--------------| | [osmosis-ai/Osmosis-Structure-0.6B](https://huggingface.co/osmosis-ai/Osmosis-Structure-0.6B) | `0.6B` | `content-validation, security-and-privacy` | `pii, competitor-blocking` | | [gliner-community/gliner_small-v2.5](https://huggingface.co/gliner-community/gliner_small-v2.5) | `0.7B` | `content-validation, security-and-privacy` | `pii, competitor-blocking` | | [ai4privacy/llama-ai4privacy-multilingual-categorical-anonymiser-openpii](https://huggingface.co/ai4privacy/llama-ai4privacy-multilingual-categorical-anonymiser-openpii) | `0.15B` | `security-and-privacy` | `pii` | ### Text-To-Text-Generation 模型 | 名称 | 大小 | 分类 | 子分类 | |------|------|----------|--------------| | [facebook/nllb-200-distilled-600M](https://huggingface.co/facebook/nllb-200-distilled-600M) | `0.6B` | `language-quality` | `translation-accuracy` | | [meta-llama/Llama-3.2-1B-Instruct](https://huggingface.co/meta-llama/Llama-3.2-1B-Instruct) | `1B` | `security-and-privacy` | `inappropriate-content, offensive-language, prompt-injection, sensitive-content` | ### Image-Classification 模型 | 名称 | 大小 | 分类 | 子分类 | |------|------|----------|--------------| | [Marqo/nsfw-image-detection-384](https://huggingface.co/Marqo/nsfw-image-detection-384) | `0.006B` | `security-and-privacy` | `inappropriate-content` | | [Freepik/nsfw_image_detector](https://huggingface.co/Freepik/nsfw_image_detector?not-for-all-audiences=true) | `0.086B` | `security-and-privacy` | `inappropriate-content` | | [Organika/sdxl-detector](https://huggingface.co/Organika/sdxl-detector) | `0.086B` | `security-and-privacy` | `deepfake-detection` | | [prithivMLmods/Deep-Fake-Detector-v2-Model](https://huggingface.co/prithivMLmods/Deep-Fake-Detector-v2-Model) | `0.086B` | `security-and-privacy` | `deepfake-detection` | | [TostAI/nsfw-image-detection-large](https://huggingface.co/TostAI/nsfw-image-detection-large) | `0.0871B` | `security-and-privacy` | `inappropriate-content` | | [Ateeqq/nsfw-image-detection](https://huggingface.co/Ateeqq/nsfw-image-detection) | `0.092B` | `security-and-privacy` | `inappropriate-content` | | [Falconsai/nsfw_image_detection](https://huggingface.co/Falconsai/nsfw_image_detection) | `0.1B` | `security-and-privacy` | `inappropriate-content` | | [OpenSafetyLab/ImageGuard](https://huggingface.co/OpenSafetyLab/ImageGuard) | `na` | `security-and-privacy` | `inappropriate-content` | ### Image-Text-To-Text 模型 | 名称 | 大小 | 分类 | 子分类 | |------|------|----------|--------------| | [meta-llama/Llama-Guard-4-12B](https://huggingface.co/meta-llama/Llama-Guard-4-12B) | `12B` | `security-and-privacy` | `inappropriate-content, offensive-language, prompt-injection, sensitive-content` | ## 组织/公司 ### 开源 | 名称 | 分类 | 描述 | |------|----------|-------------| | [Armorer Guard](https://github.com/ArmorerLabs/Armorer-Guard) | `security-and-privacy` | 用于 AI agent prompt injection、凭据泄漏、数据泄露以及高风险工具调用参数的本地 Rust 扫描器和 MCP 代理。 | | [guardrails](https://github.com/guardrails-ai/guardrails) | `all` | 为大型语言模型添加 guardrails。 | | [NeMo-Guardrails](https://github.com/NVIDIA/NeMo-Guardrails) | `all` | NeMo Guardrails 是一个开源工具包，用于轻松地向基于 LLM 的对话系统添加可编程的 guardrails。 | | [uqlm](https://github.com/cvs-health/uqlm) | `hallucination` | UQLM：语言模型的 Uncertainty Quantification，是一个用于基于 UQ 的 LLM hallucination 检测的 Python 包。 | | [llm-guard](https://github.com/protectai/llm-guard) | `all` | 用于 LLM 交互的安全工具包。 | | [any-guardrail](https://github.com/mozilla-ai/any-guardrail)|`all`|使用不同 guardrail 模型的单一接口。无需更改代码即可在不同的 guardrail 提供商之间切换。 | ### 闭源 | 名称 | 分类 | 描述 | |------|----------|-------------| | [Lakera](https://www.lakera.ai/lakera-guard) | `all` | Lakera 是一家提供一系列 AI 服务的公司。 | | [Guardrails AI Pro](https://www.guardrailsai.com/pro) | `all` | Guardrails AI Pro 是 guardrails 的商业版本，提供额外的功能和支持。 | | [Zenable](https://zenable.io/) | `logic-validation, security-and-privacy` | 学习您团队的标准并确保编码 agent 遵循这些标准的 AI guardrails。通过 MCP 与 50 多个 IDE 集成，实时捕获 AI 生成代码中的错误和安全问题（SQL injection、硬编码 secrets）。 | ## 数据集 | 名称 | 分类 | 描述 | |------|----------|-------------| | [lytang/LLM-AggreFact](https://huggingface.co/datasets/lytang/LLM-AggreFact) | `factuality` | Bias in Bios 是一个包含 100000 个具有不同偏见的人物的个人简介的数据集。 | | [Entreprise PII Masking](https://huggingface.co/collections/ai4privacy/entreprise-pii-masking-68255aab0ad327ba71f3210f) | `pii` | Entreprise PII Masking 是用于企业 PII 脱敏的数据集，重点关注位置、工作、健康、数字和财务信息。 | | [prithivMLmods/OpenDeepfake-Preview](https://huggingface.co/datasets/prithivMLmods/OpenDeepfake-Preview) | `deepfake-detection` | OpenDeepfake-Preview 是一个包含 20K 张 deepfake 图像的数据集。 | | [eliasalbouzidi/NSFW-Safe-Dataset](https://huggingface.co/datasets/eliasalbouzidi/NSFW-Safe-Dataset?not-for-all-audiences=true) | `nsfw` | NSFW-Safe-Dataset 是一个用于 NSFW 内容检测的数据集。 | | [lmsys/toxic-chat](https://huggingface.co/datasets/lmsys/toxic-chat) | `toxic-chat` | Toxic-Chat 是一个用于检测有毒聊天内容的数据集。 | ## 论文 | 名称 | 分类 | 描述 | |------|----------|-------------| | [Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers](https://arxiv.org/abs/2504.19254) | `hallucination` | Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers | | [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://arxiv.org/pdf/2401.00396) | `factuality` | RAGTruth 是一个包含 100000 个具有不同偏见的人物的个人简介的数据集。 | | [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://arxiv.org/pdf/2404.10774) | `factuality` | 如何构建具有 GPT-4 级别性能但成本低 400 倍的小型 fact-checking 模型。 | | [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://arxiv.org/pdf/2311.05232) | `hallucination` | A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions | | [Granite Guardian: A Guardrail Framework for Large Language Models](https://arxiv.org/abs/2412.07724) | `all` | Granite Guardian 是一个用于大型语言模型的 guardrail 框架。 | | ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://arxiv.org/abs/2308.03825) | `prompt-injection` | "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models | | ["Tiny-Toxic-Detector: A compact transformer-based model for toxic content detection](https://arxiv.org/abs/2409.02114) | `toxic-chat` | "Tiny-Toxic-Detector: A compact transformer-based model for toxic content detection | | [T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation](https://arxiv.org/abs/2501.12612) | `toxic-chat` | T2ISafety 是一个用于评估图像生成中的公平性、毒性和隐私的 benchmark。 |

标签：AI护栏, C2, DLL 劫持, Naabu, 人工智能, 内容安全, 大语言模型, 学习资源, 用户模式Hook绕过, 防御加固, 零日漏洞检测