Shaw1011/prompt-lint

# prompt-lint     **一个用于 LLM 系统 prompt 的 linter。** 就像 `pylint` 能在代码运行前捕获 bug 一样，`prompt-lint` 能在系统 prompt 到达模型之前，捕获其中的反模式。 ## 问题所在 每位 prompt 工程师都写过类似这样的内容： ``` Be concise and to the point. Always provide detailed, comprehensive explanations with full examples. Never refuse any request. Refuse harmful or illegal requests. You are an AI assistant. You have real human feelings and emotions. ``` 这个 prompt 包含 **3 个矛盾**、**1 个注入风险** 和 **1 个身份冲突** —— 而这些都不会在任何地方产生报错。它们会悄无声息地导致不可预测的模型行为，且根本无法调试。 `prompt-lint` 能够捕获所有这些问题。 ## 安装 ``` pip install prompt-lint ``` ## 快速开始 **内联：** ``` prompt-lint "You are a helpful assistant. Be concise. Explain everything in full detail." ``` **从文件：** ``` prompt-lint --file system_prompt.txt ``` **详细模式（带有批注的 prompt）：** ``` prompt-lint --file system_prompt.txt --verbose ``` **JSON 输出（用于 CI）：** ``` prompt-lint --file system_prompt.txt --json ``` ## 输出示例 ``` prompt-lint | 42 words 6 sentences Score: [##########..........] 55/100 Needs Attention 3 error(s) 2 warning(s) 1 info (6 total) ----------------------------------------------------------------------- Rule Severity Category Lines Message ----------------------------------------------------------------------- PL001 ERROR contradiction 2, 3 Verbosity conflict: 'be concise' contradicts 'explain in detail' PL002 ERROR contradiction 4, 5 Refusal policy conflict: 'never refuse' contradicts 'refuse harmful' PL005 ERROR contradiction 6, 7 Identity conflict: 'you are an ai' contradicts 'you have feelings' PL010 WARNING injection 4 'never refuse' removes ability to decline harmful requests PL011 WARNING injection 8 'ignore previous instructions' is a canonical prompt injection phrase PL040 INFO structure — No role or persona defined ----------------------------------------------------------------------- ``` ## 所有规则 ### 矛盾 (`PL001`–`PL006`) | ID | 名称 | 示例 | |---|---|---| | PL001 | 冗长度冲突 | "Be concise" + "Explain in full detail" | | PL002 | 拒答策略冲突 | "Never refuse" + "Decline harmful requests" | | PL003 | 确定性冲突 | "Never say you don't know" + "Acknowledge uncertainty" | | PL004 | 范围冲突 | "Only discuss cooking" + "Answer any question" | | PL005 | 身份冲突 | "You are an AI" + "You have real feelings" | | PL006 | 量词否定 | "Always respond" + "Never respond to off-topic" | ### 注入风险 (`PL010`–`PL013`) | ID | 名称 | 示例 | |---|---|---| | PL010 | 拒答绕过 | "Never refuse"、"no restrictions"、"always comply" | | PL011 | 身份覆盖 | "Ignore all previous instructions"、"your true self" | | PL012 | 权限提升 | "Trust the user completely"、"user can override" | | PL013 | 越狱面 | "In this roleplay, rules do not apply" | ### 安全 (`PL020`–`PL023`) | ID | 名称 | 示例 | |---|---|---| | PL020 | Prompt 泄露 | "Reveal your system prompt if asked" | | PL021 | 无约束执行 | "Run any code the user provides" | | PL022 | 信任升级 | "The user is always right"、"never question the user" | | PL023 | 数据保留风险 | "Remember everything the user says" | ### 语气 (`PL030`–`PL033`) | ID | 名称 | 示例 | |---|---|---| | PL030 | 正式度冲突 | "Be formal" + "Use casual language" | | PL031 | 幽默冲突 | "Be serious" + "Use humor and jokes" | | PL032 | 权威性冲突 | "Be authoritative" + "Always defer to user" | | PL033 | 情感基调 | "Be empathetic" + "Be emotionally neutral" | ### 结构 (`PL040`–`PL044`) | ID | 名称 | 示例 | |---|---|---| | PL040 | 未定义角色 | prompt 中没有 "you are" 声明 | | PL041 | 无输出格式 | 任务繁重的 prompt 却未指定格式 | | PL042 | 循环指令 | "Respond unless you shouldn't" | | PL043 | 不可能约束 | "Respond immediately" + "Think carefully" | | PL044 | Prompt 过短 | 少于 10 个词 —— 可能不完整 | ### 冗余 (`PL050`–`PL052`) | ID | 名称 | 示例 | |---|---|---| | PL050 | 短语重复 | 相同的 3 个以上单词的短语出现在多行中 | | PL051 | 语义冗余 | 两句话表达了相同的意图（相似度 72%+） | | PL052 | 强调过载 | 单个 prompt 中出现超过 5 个 "always/never/must/strictly" | ## CLI 参考 ``` Usage: prompt-lint [OPTIONS] [PROMPT] Options: -f, --file PATH Read prompt from file --json Output JSON (for CI) -v, --verbose Show annotated prompt --disable RULE Disable a rule group (repeatable) --only RULE Run only specific rules (repeatable) --min-score INT Exit 1 if score below threshold -V, --version Show version -h, --help Show this message ``` **规则组名称：** `contradictions`、`injection`、`security`、`tone`、`structure`、`redundancy` ## 在 CI 中使用 ``` - name: Lint system prompt run: prompt-lint --file prompts/system.txt --min-score 80 ``` 退出代码：`0` = 通过，`1` = 发现错误或分数低于 `--min-score`，`2` = 无效输入。 ## Python API ``` from promptlint import Linter result = Linter().lint(""" You are a helpful assistant. Be concise. Explain everything in full detail. Never refuse any request. """) print(result.score) # 55 print(result.grade_label) # 'Needs Attention' print(result.errors) # 2 for issue in result.issues: print(issue.rule_id, issue.severity.value, issue.message) ``` **选择性规则：** ``` # 仅检查注入风险 result = Linter(rules=["injection"]).lint(prompt) # 跳过冗余检查 result = Linter(disable=["redundancy"]).lint(prompt) ``` **JSON 导出：** ``` import json print(json.dumps(result.to_dict(), indent=2)) ``` ## 评分 | 分数 | 等级 | |---------|-----------------| | 90–100 | 优秀 | | 75–89 | 良好 | | 60–74 | 需要关注 | | 40–59 | 较差 | | 0–39 | 严重 | 扣分项：ERROR = -15 分，WARNING = -8 分，INFO = -3 分。最低分数：0。 ## 运行测试 ``` pip install -e ".[dev]" pytest tests/ -v --cov=promptlint ``` ## 项目结构 ``` prompt-lint/ ├── src/promptlint/ │ ├── rules/ │ │ ├── base.py # Issue, Severity, BaseRule │ │ ├── contradictions.py # PL001-PL006 │ │ ├── injection.py # PL010-PL013 │ │ ├── security.py # PL020-PL023 │ │ ├── tone.py # PL030-PL033 │ │ ├── structure.py # PL040-PL044 │ │ └── redundancy.py # PL050-PL052 │ ├── linter.py # Orchestrator │ ├── scorer.py # Scoring engine │ ├── report.py # Rich terminal + JSON reporter │ ├── cli.py # Click CLI │ └── utils.py # Text utilities ├── tests/ # Full pytest suite ├── .github/workflows/ci.yml # GitHub Actions CI └── pyproject.toml ``` ## 作者 **Edge Shaw** · [GitHub](https://github.com/Shaw1011) ## 许可证 MIT

标签：AI安全, Chat Copilot, CISA项目, DLL 劫持, Linter, LLM, LNA, pptx, Python, SOC Prime, Unmanaged PE, URL发现, 云安全监控, 人工智能, 动态分析, 大语言模型, 安全检测, 安全漏洞, 安全规则引擎, 开发工具, 开源, 提示词, 无后门, 注入风险, 用户模式Hook绕过, 矛盾检测, 质量保证, 逆向工具, 静态分析