rocklambros/incident-rank-validation
GitHub: rocklambros/incident-rank-validation
基于贝叶斯推断和真实事件语料库，系统性验证社区投票的安全分类法排名与实际事件数据之间一致性的统计分析引擎。
Stars: 0 | Forks: 0
# incident-rank-validation
用于根据真实世界事件语料库验证排名安全分类法的引擎。给定一个社区投票的“Top N”列表和一个已标注事件的语料库，该引擎通过贝叶斯推断估计每个条目的潜在普遍性，并衡量基于投票的排名与基于事件推导的排名之间的一致性程度。
第一个生产周期针对 **OWASP Top 10 for LLM Applications (2026)**，使用了来自 GenAI Agentic Incidents 语料库的 7,714 起事件。
## 2026 OWASP LLM 周期结果
| 指标 | 数值 |
|---|---|
| 加权 Cohen's kappa | 0.203 \[-0.16, 0.57\] |
| 可测量条目 | 20 个中的 17 个 (85%) |
| 盲区条目 | LLM04, LLM08, LLM10 |
| 选择偏差 (Kruskal-Wallis) | H=0.55, p=0.46 (低) |
| 标记条目 | 5 (LLM01, LLM09, NEW-MTIE, NEW-PMP, NEW-WLA) |
| 语料库 B 一致性 | 46 个重叠事件中占 26% |
| MCMC 收敛度 | 所有 R-hat < 1.001 |
| 发布状态 | `non_publishable=True` (单一作者，INTERIM 审查者) |
0.203 的 kappa 值表明社区投票排名与基于事件数据推导的排名之间存在一般性的一致。有五个条目在投票位置和观察到的事件强度之间表现出统计学上的显著差异。三个条目（LLM04 Data/Model Poisoning、LLM08 Vector/Embedding Weaknesses、LLM10 Unbounded Consumption）无法测量，因为事件语料库缺乏估计其召回率的参考框架。
## 架构
```
graph TD
subgraph "Corpus Layer"
SRC["Source Corpus
(GenAI Agentic Incidents)"] SNAP["Vendored Snapshot
content-hashed + provenance.json"] ADAPT["Corpus Adapter
engine/adapters/"] SCHEMA["Canonical IncidentRecord
engine/schema.py"] end subgraph "Pre-registration Layer" TAX["Taxonomy Definitions
20 entries (16 ranked + 4 rollup)"] RUB["Frozen Rubric
hash-locked, vote-blind"] MAN["PreregManifest
hash-locked hyperparameters + seeds"] end subgraph "Classification Layer" S1["Stage-1 Classifier
deterministic rule-based"] S2["Stage-2 Classifier
LLM on RunPod
delimiter-fenced prompts"] end subgraph "Calibration Layer" GOLD["Gold-Set Labels
dual-coded, adjudicated"] BETA["Beta Posteriors
per-entry precision/recall"] CV["k-fold CV
fold-variance disclosure"] end subgraph "Inference Layer" NUTS["NUTS Sampler
NumPyro/JAX, CPU-pinned"] NB["Negative-Binomial
measurement-error model"] LAMB["Lambda Posteriors
latent prevalence per entry"] end subgraph "Decision Layer" VOTE["Vote Bootstrap
XLSX raw results"] CONC["Concordance
weighted kappa"] MEAS["Measurability Map"] FLAGS["Per-Entry Flags"] BIAS["Selection Bias
Kruskal-Wallis H"] TWIN["Robustness Twin
non-Bayesian cross-check"] end subgraph "Output Layer" RPT["Report
measurability-first"] REPRO["Reproduction Bundle
single-command regeneration"] THREATS["Threats Register"] NARR["Narrative Report
16 figures"] end SRC --> SNAP --> ADAPT --> SCHEMA TAX --> RUB --> MAN SCHEMA --> S1 --> S2 RUB --> S1 S2 --> GOLD --> BETA --> CV BETA --> NB SCHEMA --> NB --> NUTS --> LAMB MAN --> NUTS LAMB --> CONC VOTE --> CONC LAMB --> MEAS LAMB --> FLAGS LAMB --> BIAS LAMB --> TWIN CONC --> RPT MEAS --> RPT FLAGS --> RPT BIAS --> RPT TWIN --> RPT RPT --> REPRO RPT --> THREATS RPT --> NARR style SRC fill:#bbdefb,stroke:#1565c0,color:#0d47a1 style SNAP fill:#bbdefb,stroke:#1565c0,color:#0d47a1 style ADAPT fill:#bbdefb,stroke:#1565c0,color:#0d47a1 style SCHEMA fill:#bbdefb,stroke:#1565c0,color:#0d47a1 style TAX fill:#e1bee7,stroke:#7b1fa2,color:#4a148c style RUB fill:#e1bee7,stroke:#7b1fa2,color:#4a148c style MAN fill:#e1bee7,stroke:#7b1fa2,color:#4a148c style S1 fill:#ffe0b2,stroke:#e65100,color:#bf360c style S2 fill:#ffe0b2,stroke:#e65100,color:#bf360c style GOLD fill:#b2dfdb,stroke:#00695c,color:#004d40 style BETA fill:#b2dfdb,stroke:#00695c,color:#004d40 style CV fill:#b2dfdb,stroke:#00695c,color:#004d40 style NUTS fill:#c5cae9,stroke:#283593,color:#1a237e style NB fill:#c5cae9,stroke:#283593,color:#1a237e style LAMB fill:#c5cae9,stroke:#283593,color:#1a237e style VOTE fill:#ffccbc,stroke:#bf360c,color:#bf360c style CONC fill:#ffccbc,stroke:#bf360c,color:#bf360c style MEAS fill:#ffccbc,stroke:#bf360c,color:#bf360c style FLAGS fill:#ffccbc,stroke:#bf360c,color:#bf360c style BIAS fill:#ffccbc,stroke:#bf360c,color:#bf360c style TWIN fill:#ffccbc,stroke:#bf360c,color:#bf360c style RPT fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style REPRO fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style THREATS fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style NARR fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 ``` ## 流水线流程 该引擎强制执行严格的阶段控制流水线。每个阶段在运行前都会检查前置条件，并且无法访问后续阶段的数据。 ``` sequenceDiagram box rgb(225, 190, 231) Pre-registration participant P as prereg end box rgb(255, 224, 178) Classification participant C as classify end box rgb(178, 223, 219) Calibration participant Cal as calibrate end box rgb(197, 202, 233) Inference participant I as infer end box rgb(255, 204, 188) Decision participant D as decide end box rgb(200, 230, 201) Output participant R as report end Note over P: Hash-lock taxonomy,
rubric, hyperparameters,
PRNG seed P->>C: manifest.lock Note over C: Stage-1 deterministic +
Stage-2 LLM on residue C->>Cal: labeled_incidents.json Note over Cal: Gold-set sampling,
dual coding, Beta posteriors,
k-fold CV Cal->>I: posteriors.json rect rgb(232, 245, 233) Note over P,I: Vote-blind zone end Note over I: NUTS on CPU
(vote data structurally
inaccessible) I->>D: lambda_samples.npy rect rgb(255, 243, 224) Note over D,R: Vote-aware zone end Note over D: Vote data enters HERE
and only here D->>R: concordance + flags +
measurability + bias Note over R: Measurability map leads,
standing caveat,
reproduction bundle ``` ## 信息防火墙 投票排名数据仅在 `decide` 阶段进入流水线。CLI 在结构上防止了投票数据在早期阶段存在于周期目录中。这确保了分类法、评分标准、分类、校准和推断过程全部对投票不可见。 ``` graph LR subgraph blind ["Vote-Blind Zone"] PR[prereg] CL[classify] CA[calibrate] IN[infer] end subgraph aware ["Vote-Aware Zone"] DE[decide] RE[report] end VOTE["Vote XLSX"] VOTE -.->|"blocked"| PR VOTE -.->|"blocked"| CL VOTE -.->|"blocked"| CA VOTE -.->|"blocked"| IN VOTE ==>|"enters here"| DE PR --> CL --> CA --> IN --> DE --> RE style blind fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style aware fill:#fff3e0,stroke:#e65100,stroke-width:2px style PR fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style CL fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style CA fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style IN fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style DE fill:#ffe0b2,stroke:#e65100,color:#bf360c style RE fill:#ffe0b2,stroke:#e65100,color:#bf360c style VOTE fill:#ef9a9a,stroke:#c62828,color:#b71c1c linkStyle 0 stroke:#c62828,stroke-dasharray:5 linkStyle 1 stroke:#c62828,stroke-dasharray:5 linkStyle 2 stroke:#c62828,stroke-dasharray:5 linkStyle 3 stroke:#c62828,stroke-dasharray:5 linkStyle 4 stroke:#2e7d32,stroke-width:3px ``` ## 安装 需要 Python 3.12。 ``` git clone https://github.com/rocklambros/incident-rank-validation.git cd incident-rank-validation pip install uv uv sync ``` 用于生成叙述性报告 (matplotlib, seaborn, plotly)： ``` uv sync --group narrative ``` 验证安装： ``` uv run incident-rank --help ``` ## 快速开始：合成周期 运行一个合成的端到端验证周期以验证引擎是否正常工作。不需要外部数据或 API 密钥。 ``` uv run incident-rank run-synthetic \ --cycle projects/synthetic/cycles/2026 \ --corpus-mode synthetic ``` 这会针对具有已知真实属性的合成事件数据运行完整的流水线 (prereg, classify, calibrate, infer, decide, report)。 ## 运行真实周期 真实周期（如 OWASP LLM 2026 周期）通过 CLI 单独运行每个阶段： ``` # 1. Vendor 一个 corpus 快照 uv run incident-rank vendor-snapshot --project projects/owasp-llm # 2. 冻结 rubric（起草 + 裁决之后） uv run incident-rank freeze-rubric --cycle projects/owasp-llm/cycles/2026 # 3. 对 incidents 进行分类（Stage-1 + Stage-2） uv run incident-rank classify-real --cycle projects/owasp-llm/cycles/2026 # 4. 校准（gold-set 采样、Beta 后验、CV） uv run incident-rank cal-sample --cycle projects/owasp-llm/cycles/2026 uv run incident-rank cal-calibrate --cycle projects/owasp-llm/cycles/2026 uv run incident-rank cal-cv-stability --cycle projects/owasp-llm/cycles/2026 # 5. 运行 NUTS 推断（仅限 CPU，需要数小时） uv run incident-rank infer-real --cycle projects/owasp-llm/cycles/2026 # 6. 决策（投票数据在此输入） uv run incident-rank decide-real --cycle projects/owasp-llm/cycles/2026 # 7. 生成报告 + reproduction bundle uv run incident-rank report --cycle projects/owasp-llm/cycles/2026 uv run incident-rank repro-bundle --cycle projects/owasp-llm/cycles/2026 ``` ## 项目布局 ``` engine/ adapters/ Corpus adapters (GenAI Agentic, OWASP ASI, synthetic) calibrate/ Gold-set sampling, Beta posteriors, k-fold CV classify/ Stage-1 deterministic + Stage-2 LLM classifiers cli/ Click CLI commands and pipeline executor decide/ Concordance, measurability, selection bias, twin agreement erratum/ Post-hoc correction lineage with Merkle audit model/ NUTS inference (NumPyro/JAX), robustness twin, diagnostics monitoring/ Optional W&B logging prereg/ Pre-registration: manifest, rubric, gates, attestation report/ Report renderer, narrative charts, diff engine repro/ Reproduction bundle generator safety/ Corpus-mode enforcement snapshot/ Content-hashed vendoring, drift detection threats/ Threats register vote/ Vote-data loader and bootstrap projects/ owasp-llm/ OWASP LLM Top 10 2026 cycle data and results synthetic/ Synthetic validation cycle synthetic-stress/ Stress-test cycle (untuned hyperparameters) tests/ unit/ ~80 unit tests integration/ End-to-end pipeline tests proofs/ Mathematical property tests (never-falsely-low, frame-blind gate) security/ Prompt injection and delimiter tests for Stage-2 notebooks/ 2026_top_10_llm_update_what_the_data_says.ipynb Interactive analysis notebook 2026_top_10_llm_update_what_the_data_says.pdf PDF render of the notebook narrative/ Standalone narrative report with 16 figures docs/ HANDOFF.md Methodology spec (source of truth) PRD.md Phase map and pickup commands METHODOLOGY-CHANGELOG.md Semver-tagged methodology changes RUNBOOK.md Operational runbook GOLDSET-CODING-GUIDE.md Instructions for gold-set coders REVIEWERS.md Reviewer identification and attestation state ``` ## 核心概念 **可测量性映射。** 并非每个分类条目都能被测量。只有当事件语料库包含足够的事件*且*分类器能够检测到它们时，该条目才是可测量的。语料库缺乏参考框架（盲区）的条目将被报告为不可测量，而不是低普遍性。可测量性映射位于每个报告的开头。 **测量误差模型。** 引擎不会将分类器标签视为基准真相。使用带有 Beta 参数化精度和从 gold-set 校准中得出的召回率的负二项式模型，在估计每个条目的潜在普遍性时考虑了分类器误差。 **预注册。** 在任何推断运行之前，分类法、评分标准、超参数和 PRNG seed 都会在清单中进行哈希锁定。事后偏差会被自动检测并在报告中披露。 **透明度优先发布。** 结果永远不会被压制。如果 kappa 较低或可测量性有限，这将被作为发现报告出来，而不是被隐藏。每份报告都附带标准的范围和偏差警告。 ## 方法论 完整的方法论记录在 [`docs/HANDOFF.md`](docs/HANDOFF.md) 中。阶段执行历史记录在 [`docs/PRD.md`](docs/PRD.md) 中。方法论的变更通过 semver 标签记录在 [`docs/METHODOLOGY-CHANGELOG.md`](docs/METHODOLOGY-CHANGELOG.md) 中。 ### 版本历史 | 版本 | 计划 | 描述 | |---|---|---| | 0.1.0 | 计划 1 | 引擎基线 + 合成周期 | | 0.2.0 | 计划 2 | GenAI Agentic 语料库适配器 + snapshot vendoring | | 0.3.0 | 计划 3 | 评分标准冻结工作流 + 证明门控 | | 0.4.0 | 计划 4 | Gold-set 校准 + Beta 后验 + k-fold CV | | 1.0.0 | 计划 5 | 首次真实 OWASP LLM 2026 周期（内部） | | 1.1.0 | 计划 5 | Kappa 改进 + 流水线修复 | | 1.2.0 | 计划 6 | 语料库 B 验证 + 叙述性报告 | ## 测试 ``` # 完整 test suite uv run pytest # Type checking uv run mypy engine tests # Linting uv run ruff check . # Security scanning uv run semgrep --config .semgrep.yml --error engine/ ``` 测试套件包括数学属性测试 (`tests/proofs/`)，用于验证引擎永远不会将真正高普遍性的条目标记为低普遍性（绝不误判为低的保证）；以及安全测试 (`tests/security/`)，用于验证 Stage-2 LLM prompt 能够抵御通过事件文本进行的注入攻击。 ## 完整性控制 引擎强制执行多项结构完整性控制： - **哈希锁定的预注册** —— 分类法、评分标准、超参数和 PRNG seed 在推断运行前被冻结。 - **投票盲测** —— 如果周期目录中存在投票数据，CLI 将拒绝运行推断。 - **评分标准证明门控** —— 如果没有提交的评分标准证明（声明起草者是否看过语料库样本），分类将拒绝运行。 - **NUTS 诊断门控** —— 如果 R-hat > 1.01 或存在预热后的分叉，推断将拒绝输出报告。 - **Snapshot 绑定** —— gold-set 产物绑定到特定的语料库 snapshot hash。不匹配将中止运行。 - **可发布性推导** —— `non_publishable` 是根据审查者的证明状态计算得出的，而不是手动设置的。 ## 许可证 Apache-2.0。有关署名详细信息，请参见 [`NOTICE`](NOTICE)。 `projects/*/cycles/*/corpora/` 下的 Vendored 语料库数据在 [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) 下分发。 ## 链接 - [方法论规范](docs/HANDOFF.md) - [阶段映射](docs/PRD.md) - [方法论更新日志](docs/METHODOLOGY-CHANGELOG.md) - [操作手册](docs/RUNBOOK.md) - [安全策略](SECURITY.md) - [贡献指南](CONTRIBUTING.md)
(GenAI Agentic Incidents)"] SNAP["Vendored Snapshot
content-hashed + provenance.json"] ADAPT["Corpus Adapter
engine/adapters/"] SCHEMA["Canonical IncidentRecord
engine/schema.py"] end subgraph "Pre-registration Layer" TAX["Taxonomy Definitions
20 entries (16 ranked + 4 rollup)"] RUB["Frozen Rubric
hash-locked, vote-blind"] MAN["PreregManifest
hash-locked hyperparameters + seeds"] end subgraph "Classification Layer" S1["Stage-1 Classifier
deterministic rule-based"] S2["Stage-2 Classifier
LLM on RunPod
delimiter-fenced prompts"] end subgraph "Calibration Layer" GOLD["Gold-Set Labels
dual-coded, adjudicated"] BETA["Beta Posteriors
per-entry precision/recall"] CV["k-fold CV
fold-variance disclosure"] end subgraph "Inference Layer" NUTS["NUTS Sampler
NumPyro/JAX, CPU-pinned"] NB["Negative-Binomial
measurement-error model"] LAMB["Lambda Posteriors
latent prevalence per entry"] end subgraph "Decision Layer" VOTE["Vote Bootstrap
XLSX raw results"] CONC["Concordance
weighted kappa"] MEAS["Measurability Map"] FLAGS["Per-Entry Flags"] BIAS["Selection Bias
Kruskal-Wallis H"] TWIN["Robustness Twin
non-Bayesian cross-check"] end subgraph "Output Layer" RPT["Report
measurability-first"] REPRO["Reproduction Bundle
single-command regeneration"] THREATS["Threats Register"] NARR["Narrative Report
16 figures"] end SRC --> SNAP --> ADAPT --> SCHEMA TAX --> RUB --> MAN SCHEMA --> S1 --> S2 RUB --> S1 S2 --> GOLD --> BETA --> CV BETA --> NB SCHEMA --> NB --> NUTS --> LAMB MAN --> NUTS LAMB --> CONC VOTE --> CONC LAMB --> MEAS LAMB --> FLAGS LAMB --> BIAS LAMB --> TWIN CONC --> RPT MEAS --> RPT FLAGS --> RPT BIAS --> RPT TWIN --> RPT RPT --> REPRO RPT --> THREATS RPT --> NARR style SRC fill:#bbdefb,stroke:#1565c0,color:#0d47a1 style SNAP fill:#bbdefb,stroke:#1565c0,color:#0d47a1 style ADAPT fill:#bbdefb,stroke:#1565c0,color:#0d47a1 style SCHEMA fill:#bbdefb,stroke:#1565c0,color:#0d47a1 style TAX fill:#e1bee7,stroke:#7b1fa2,color:#4a148c style RUB fill:#e1bee7,stroke:#7b1fa2,color:#4a148c style MAN fill:#e1bee7,stroke:#7b1fa2,color:#4a148c style S1 fill:#ffe0b2,stroke:#e65100,color:#bf360c style S2 fill:#ffe0b2,stroke:#e65100,color:#bf360c style GOLD fill:#b2dfdb,stroke:#00695c,color:#004d40 style BETA fill:#b2dfdb,stroke:#00695c,color:#004d40 style CV fill:#b2dfdb,stroke:#00695c,color:#004d40 style NUTS fill:#c5cae9,stroke:#283593,color:#1a237e style NB fill:#c5cae9,stroke:#283593,color:#1a237e style LAMB fill:#c5cae9,stroke:#283593,color:#1a237e style VOTE fill:#ffccbc,stroke:#bf360c,color:#bf360c style CONC fill:#ffccbc,stroke:#bf360c,color:#bf360c style MEAS fill:#ffccbc,stroke:#bf360c,color:#bf360c style FLAGS fill:#ffccbc,stroke:#bf360c,color:#bf360c style BIAS fill:#ffccbc,stroke:#bf360c,color:#bf360c style TWIN fill:#ffccbc,stroke:#bf360c,color:#bf360c style RPT fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style REPRO fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style THREATS fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style NARR fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 ``` ## 流水线流程 该引擎强制执行严格的阶段控制流水线。每个阶段在运行前都会检查前置条件，并且无法访问后续阶段的数据。 ``` sequenceDiagram box rgb(225, 190, 231) Pre-registration participant P as prereg end box rgb(255, 224, 178) Classification participant C as classify end box rgb(178, 223, 219) Calibration participant Cal as calibrate end box rgb(197, 202, 233) Inference participant I as infer end box rgb(255, 204, 188) Decision participant D as decide end box rgb(200, 230, 201) Output participant R as report end Note over P: Hash-lock taxonomy,
rubric, hyperparameters,
PRNG seed P->>C: manifest.lock Note over C: Stage-1 deterministic +
Stage-2 LLM on residue C->>Cal: labeled_incidents.json Note over Cal: Gold-set sampling,
dual coding, Beta posteriors,
k-fold CV Cal->>I: posteriors.json rect rgb(232, 245, 233) Note over P,I: Vote-blind zone end Note over I: NUTS on CPU
(vote data structurally
inaccessible) I->>D: lambda_samples.npy rect rgb(255, 243, 224) Note over D,R: Vote-aware zone end Note over D: Vote data enters HERE
and only here D->>R: concordance + flags +
measurability + bias Note over R: Measurability map leads,
standing caveat,
reproduction bundle ``` ## 信息防火墙 投票排名数据仅在 `decide` 阶段进入流水线。CLI 在结构上防止了投票数据在早期阶段存在于周期目录中。这确保了分类法、评分标准、分类、校准和推断过程全部对投票不可见。 ``` graph LR subgraph blind ["Vote-Blind Zone"] PR[prereg] CL[classify] CA[calibrate] IN[infer] end subgraph aware ["Vote-Aware Zone"] DE[decide] RE[report] end VOTE["Vote XLSX"] VOTE -.->|"blocked"| PR VOTE -.->|"blocked"| CL VOTE -.->|"blocked"| CA VOTE -.->|"blocked"| IN VOTE ==>|"enters here"| DE PR --> CL --> CA --> IN --> DE --> RE style blind fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px style aware fill:#fff3e0,stroke:#e65100,stroke-width:2px style PR fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style CL fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style CA fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style IN fill:#c8e6c9,stroke:#2e7d32,color:#1b5e20 style DE fill:#ffe0b2,stroke:#e65100,color:#bf360c style RE fill:#ffe0b2,stroke:#e65100,color:#bf360c style VOTE fill:#ef9a9a,stroke:#c62828,color:#b71c1c linkStyle 0 stroke:#c62828,stroke-dasharray:5 linkStyle 1 stroke:#c62828,stroke-dasharray:5 linkStyle 2 stroke:#c62828,stroke-dasharray:5 linkStyle 3 stroke:#c62828,stroke-dasharray:5 linkStyle 4 stroke:#2e7d32,stroke-width:3px ``` ## 安装 需要 Python 3.12。 ``` git clone https://github.com/rocklambros/incident-rank-validation.git cd incident-rank-validation pip install uv uv sync ``` 用于生成叙述性报告 (matplotlib, seaborn, plotly)： ``` uv sync --group narrative ``` 验证安装： ``` uv run incident-rank --help ``` ## 快速开始：合成周期 运行一个合成的端到端验证周期以验证引擎是否正常工作。不需要外部数据或 API 密钥。 ``` uv run incident-rank run-synthetic \ --cycle projects/synthetic/cycles/2026 \ --corpus-mode synthetic ``` 这会针对具有已知真实属性的合成事件数据运行完整的流水线 (prereg, classify, calibrate, infer, decide, report)。 ## 运行真实周期 真实周期（如 OWASP LLM 2026 周期）通过 CLI 单独运行每个阶段： ``` # 1. Vendor 一个 corpus 快照 uv run incident-rank vendor-snapshot --project projects/owasp-llm # 2. 冻结 rubric（起草 + 裁决之后） uv run incident-rank freeze-rubric --cycle projects/owasp-llm/cycles/2026 # 3. 对 incidents 进行分类（Stage-1 + Stage-2） uv run incident-rank classify-real --cycle projects/owasp-llm/cycles/2026 # 4. 校准（gold-set 采样、Beta 后验、CV） uv run incident-rank cal-sample --cycle projects/owasp-llm/cycles/2026 uv run incident-rank cal-calibrate --cycle projects/owasp-llm/cycles/2026 uv run incident-rank cal-cv-stability --cycle projects/owasp-llm/cycles/2026 # 5. 运行 NUTS 推断（仅限 CPU，需要数小时） uv run incident-rank infer-real --cycle projects/owasp-llm/cycles/2026 # 6. 决策（投票数据在此输入） uv run incident-rank decide-real --cycle projects/owasp-llm/cycles/2026 # 7. 生成报告 + reproduction bundle uv run incident-rank report --cycle projects/owasp-llm/cycles/2026 uv run incident-rank repro-bundle --cycle projects/owasp-llm/cycles/2026 ``` ## 项目布局 ``` engine/ adapters/ Corpus adapters (GenAI Agentic, OWASP ASI, synthetic) calibrate/ Gold-set sampling, Beta posteriors, k-fold CV classify/ Stage-1 deterministic + Stage-2 LLM classifiers cli/ Click CLI commands and pipeline executor decide/ Concordance, measurability, selection bias, twin agreement erratum/ Post-hoc correction lineage with Merkle audit model/ NUTS inference (NumPyro/JAX), robustness twin, diagnostics monitoring/ Optional W&B logging prereg/ Pre-registration: manifest, rubric, gates, attestation report/ Report renderer, narrative charts, diff engine repro/ Reproduction bundle generator safety/ Corpus-mode enforcement snapshot/ Content-hashed vendoring, drift detection threats/ Threats register vote/ Vote-data loader and bootstrap projects/ owasp-llm/ OWASP LLM Top 10 2026 cycle data and results synthetic/ Synthetic validation cycle synthetic-stress/ Stress-test cycle (untuned hyperparameters) tests/ unit/ ~80 unit tests integration/ End-to-end pipeline tests proofs/ Mathematical property tests (never-falsely-low, frame-blind gate) security/ Prompt injection and delimiter tests for Stage-2 notebooks/ 2026_top_10_llm_update_what_the_data_says.ipynb Interactive analysis notebook 2026_top_10_llm_update_what_the_data_says.pdf PDF render of the notebook narrative/ Standalone narrative report with 16 figures docs/ HANDOFF.md Methodology spec (source of truth) PRD.md Phase map and pickup commands METHODOLOGY-CHANGELOG.md Semver-tagged methodology changes RUNBOOK.md Operational runbook GOLDSET-CODING-GUIDE.md Instructions for gold-set coders REVIEWERS.md Reviewer identification and attestation state ``` ## 核心概念 **可测量性映射。** 并非每个分类条目都能被测量。只有当事件语料库包含足够的事件*且*分类器能够检测到它们时，该条目才是可测量的。语料库缺乏参考框架（盲区）的条目将被报告为不可测量，而不是低普遍性。可测量性映射位于每个报告的开头。 **测量误差模型。** 引擎不会将分类器标签视为基准真相。使用带有 Beta 参数化精度和从 gold-set 校准中得出的召回率的负二项式模型，在估计每个条目的潜在普遍性时考虑了分类器误差。 **预注册。** 在任何推断运行之前，分类法、评分标准、超参数和 PRNG seed 都会在清单中进行哈希锁定。事后偏差会被自动检测并在报告中披露。 **透明度优先发布。** 结果永远不会被压制。如果 kappa 较低或可测量性有限，这将被作为发现报告出来，而不是被隐藏。每份报告都附带标准的范围和偏差警告。 ## 方法论 完整的方法论记录在 [`docs/HANDOFF.md`](docs/HANDOFF.md) 中。阶段执行历史记录在 [`docs/PRD.md`](docs/PRD.md) 中。方法论的变更通过 semver 标签记录在 [`docs/METHODOLOGY-CHANGELOG.md`](docs/METHODOLOGY-CHANGELOG.md) 中。 ### 版本历史 | 版本 | 计划 | 描述 | |---|---|---| | 0.1.0 | 计划 1 | 引擎基线 + 合成周期 | | 0.2.0 | 计划 2 | GenAI Agentic 语料库适配器 + snapshot vendoring | | 0.3.0 | 计划 3 | 评分标准冻结工作流 + 证明门控 | | 0.4.0 | 计划 4 | Gold-set 校准 + Beta 后验 + k-fold CV | | 1.0.0 | 计划 5 | 首次真实 OWASP LLM 2026 周期（内部） | | 1.1.0 | 计划 5 | Kappa 改进 + 流水线修复 | | 1.2.0 | 计划 6 | 语料库 B 验证 + 叙述性报告 | ## 测试 ``` # 完整 test suite uv run pytest # Type checking uv run mypy engine tests # Linting uv run ruff check . # Security scanning uv run semgrep --config .semgrep.yml --error engine/ ``` 测试套件包括数学属性测试 (`tests/proofs/`)，用于验证引擎永远不会将真正高普遍性的条目标记为低普遍性（绝不误判为低的保证）；以及安全测试 (`tests/security/`)，用于验证 Stage-2 LLM prompt 能够抵御通过事件文本进行的注入攻击。 ## 完整性控制 引擎强制执行多项结构完整性控制： - **哈希锁定的预注册** —— 分类法、评分标准、超参数和 PRNG seed 在推断运行前被冻结。 - **投票盲测** —— 如果周期目录中存在投票数据，CLI 将拒绝运行推断。 - **评分标准证明门控** —— 如果没有提交的评分标准证明（声明起草者是否看过语料库样本），分类将拒绝运行。 - **NUTS 诊断门控** —— 如果 R-hat > 1.01 或存在预热后的分叉，推断将拒绝输出报告。 - **Snapshot 绑定** —— gold-set 产物绑定到特定的语料库 snapshot hash。不匹配将中止运行。 - **可发布性推导** —— `non_publishable` 是根据审查者的证明状态计算得出的，而不是手动设置的。 ## 许可证 Apache-2.0。有关署名详细信息，请参见 [`NOTICE`](NOTICE)。 `projects/*/cycles/*/corpora/` 下的 Vendored 语料库数据在 [CC-BY-4.0](https://creativecommons.org/licenses/by/4.0/) 下分发。 ## 链接 - [方法论规范](docs/HANDOFF.md) - [阶段映射](docs/PRD.md) - [方法论更新日志](docs/METHODOLOGY-CHANGELOG.md) - [操作手册](docs/RUNBOOK.md) - [安全策略](SECURITY.md) - [贡献指南](CONTRIBUTING.md)
标签：Python, 反取证, 安全度量, 安全评估, 数据验证, 无后门, 统计分析, 贝叶斯推断, 逆向工具