rwilliamspbg-ops/Sovereign_Map_Federated_Learning

GitHub: rwilliamspbg-ops/Sovereign_Map_Federated_Learning

这是一个集成了拜占庭容错与TPM硬件信任根的生产级联邦学习框架,通过流式聚合技术实现大规模节点环境下的安全编排与低内存训练。

Stars: 2 | Forks: 0

# Sovereign Map 联邦学习 生产级联邦学习平台,结合了拜占庭容错聚合、信任验证、治理策略控制、代币经济遥测以及全栈可观测性。 ## 项目实时动态 [![Release](https://img.shields.io/github/v/release/rwilliamspbg-ops/Sovereign_Map_Federated_Learning?display_name=tag&style=flat-square&logo=github)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/releases) [![Build and Test](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/f363da7cc6011406.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/build.yml) [![Windows Client EXE Build](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/cf51f6d265011408.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/windows-client-exe.yml) [![macOS Client Smoke](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/543d25e72b011409.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/macos-client-smoke.yml) [![Observability CI](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/92cacfcc2f011410.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/observability-ci.yml) [![FedAvg Benchmark Compare](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/26836d41ac011411.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/fedavg-benchmark-compare.yml) [![API Spec Validation](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/2add611e9e011413.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/api-spec-validation.yml) [![API Docs Pages](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/5465b2782e011414.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/api-docs-pages.yml) [![Full Validation PR Gate](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/164bb36a67011415.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/full-validation-pr-gate.yml) [![CodeQL Security Analysis](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/9563c397db011417.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/codeql-analysis.yml) [![Security Supply Chain](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/f2f19555ca011418.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/security-supply-chain.yml) [![License](https://img.shields.io/github/license/rwilliamspbg-ops/Sovereign_Map_Federated_Learning?style=flat-square)](LICENSE) [![Metrics Upgrade](https://img.shields.io/badge/Metrics-Blockchain%20Bridge%20Integrated-0ea5e9?style=flat-square)](tokenomics_metrics_exporter.py) [![Dashboards Upgrade](https://img.shields.io/badge/Grafana-STARRED%20Live%20Dashboards-f59e0b?style=flat-square&logo=grafana&logoColor=white)](grafana/provisioning/dashboards) [![PySyft Demo](https://img.shields.io/badge/PySyft-Mohawk%20PoC%20Ready-10b981?style=flat-square)](examples/pysyft-integration) [![Prometheus Ready](https://img.shields.io/badge/Prometheus-Scrape%20Ready-ef4444?style=flat-square&logo=prometheus&logoColor=white)](prometheus.yml) [![Open Ecosystem](https://img.shields.io/badge/Open%20Ecosystem-Sprint%203%20Local--First-0ea5e9?style=flat-square)](docs/OPEN_ECOSYSTEM_FIRST_10_MINUTES.md) [![Marketplace Alerts](https://img.shields.io/badge/Alerts-Marketplace%20Guardrails-f97316?style=flat-square)](marketplace_alerts.yml) 文档入口:[docs/README.md](docs/README.md) ## 当前状态快照(2026 年 4 月) - `main` 分支上已激活跨平台 CI 覆盖。 - Linux 通道:`Build and Test`、`Lint Code Base`、`Reproducibility Check`、`Observability CI` - Windows 通道:`Windows Client EXE Build` - macOS 通道:`macOS Client Smoke` - Observability CI 现在在代码推送、拉取请求、定时漂移检查以及发布事件(`published`、`prereleased`)时运行。 - 当前贡献者基线验证命令: - `npm run test:ci` - `make observability-smoke` - `make alerts-test` - `~/.local/bin/black --check .` - `~/.local/bin/flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics` 有关当前路线图状态和剩余开放项,请参阅 [Documentation/Performance/OPS_OPTIMIZATION_ROADMAP_2026-03-24.md](Documentation/Performance/OPS_OPTIMIZATION_ROADMAP_2026-03-24.md) 和 [Documentation/Project/ROADMAP.md](Documentation/Project/ROADMAP.md)。 ## 量子 KEX 轮换演练(创世测试网) 该仓库包含一个面向公众的创世测试网演练,以演示后量子迁移准备情况并提供可审计的产物。 运行演练: ``` make quantum-kex-rotation-drill ``` 使用严格的非 Mock 后端强制执行运行: ``` make quantum-kex-rotation-drill-strict ``` 在 CI/本地验证生产模式护栏策略: ``` make quantum-kex-production-guardrail-check ``` 或使用显式端点/token 输入直接运行: ``` NODE_AGENT_BASE_URL=http://localhost:8082 \ MOHAWK_API_TOKEN_FILE=/run/secrets/mohawk_api_token \ bash scripts/quantum-kex-rotation-drill.sh ``` 运维手册和披露指南: - [Documentation/Security/QUANTUM_KEX_ROTATION_DRILL_RUNBOOK.md](Documentation/Security/QUANTUM_KEX_ROTATION_DRILL_RUNBOOK.md) 产物输出: - `artifacts/quantum-kex-rotation//` 生产状态说明: - 测试网演练可能使用模拟的回退后端进行演练。 - 生产级 PQC 就绪声明需要严格的非 Mock 强制执行,并且在 `drill-summary.json` 中设置 `production_pqc_ready=true`。 - `production_pqc_ready=true` 要求 Go 和 Python liboqs 适配器均报告 `available=true`。 ## 新贡献者快速通道 如果您刚刚克隆了仓库并希望快速运行测试,请使用此序列。 ### 1) 克隆并进入工作区 ``` git clone https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning.git cd Sovereign_Map_Federated_Learning ``` ### 2) 发现并运行维护者测试路径 ``` make help make fmt make lint make test ``` ### 3) 运行可选的可复现性检查 ``` make smoke make observability-smoke make quickstart-verify ``` 获取贡献指南的位置: - 完整贡献流程和 PR 清单:[CONTRIBUTING.md](CONTRIBUTING.md) - 快速贡献机会:[README.md#help-wanted-quick-wins](README.md#help-wanted-quick-wins) - 运行时验证预期:[README.md#contributor-first-steps](README.md#contributor-first-steps) - 运维仪表盘指标合约:[docs/OPERATIONS_DASHBOARD_METRIC_CONTRACT.md](docs/OPERATIONS_DASHBOARD_METRIC_CONTRACT.md) ## 验证和 CI 升级(2026 年 4 月) 统一的验证路径现在支持基于配置文件的执行、趋势 SLO 强制执行、产物差异摘要、浏览器运行时周期检查以及定时深度验证运行。 新增内容: - 必需式 PR 门控工作流:[.github/workflows/full-validation-pr-gate.yml](.github/workflows/full-validation-pr-gate.yml) - 定时深度工作流:[.github/workflows/full-validation-scheduled-deep.yml](.github/workflows/full-validation-scheduled-deep.yml) - 快速和深度测试套件配置文件:[tests/scripts/python/run_full_validation_suite.py](tests/scripts/python/run_full_validation_suite.py) - 趋势 SLO 检查器:[tests/scripts/ci/check_validation_trends.py](tests/scripts/ci/check_validation_trends.py) - CI 差异摘要写入器:[tests/scripts/ci/write_validation_diff_summary.py](tests/scripts/ci/write_validation_diff_summary.py) - 浏览器运行时 E2E 周期检查:[tests/scripts/python/test_browser_runtime_e2e.py](tests/scripts/python/test_browser_runtime_e2e.py) - Playwright 运行时产物:[tests/e2e/runtime-cadence.spec.js](tests/e2e/runtime-cadence.spec.js)、[tests/e2e/playwright.config.js](tests/e2e/playwright.config.js) 规范命令: ``` npm run test:setup npm run test:full:fast npm run test:full:deep npm run test:trends npm run test:summary:diff ``` 验证产物: - `test-results/full-validation/full_validation_.json` - `test-results/full-validation/full_validation_.md` - `test-results/full-validation/history.jsonl` 文档治理: - 文档维护手册:[docs/DOCUMENTATION_MAINTENANCE.md](docs/DOCUMENTATION_MAINTENANCE.md) - 测试设置详细信息和配置文件使用:[tests/docs/TEST_ENV_SETUP.md](tests/docs/TEST_ENV_SETUP.md) ## 移动端盾牌更新(2026 年 3 月) 移动端加固和应用商店打包流程现已在仓库内实现。 现已上线: - 面向 iOS Secure Enclave 和 Android StrongBox/Keystore 应用集成的硬件支持移动端签名器包装器。 - 两个移动平台上的规范签名梯度信封适配器,与后端验证器合约对齐。 - 用于验证签名移动端梯度的后端端点:`/mobile/verify_gradient`。 - 针对有效和无效移动端签名载荷的合约测试覆盖。 - 用于 Android Play Store 和 Apple App Store 提交流程的生产级应用商店包装包。 主要参考: - 移动端实现概述:[mobile-apps/MOBILE_APP_README.md](mobile-apps/MOBILE_APP_README.md) - Android 应用商店包装器:[mobile-apps/android-node-app/store-wrapper/README.md](mobile-apps/android-node-app/store-wrapper/README.md) - iOS 应用商店包装器:[mobile-apps/ios-node-app/store-wrapper/README.md](mobile-apps/ios-node-app/store-wrapper/README.md) - iOS 提交清单:[mobile-apps/ios-node-app/store-wrapper/APP_STORE_CONNECT_SUBMISSION_CHECKLIST.md](mobile-apps/ios-node-app/store-wrapper/APP_STORE_CONNECT_SUBMISSION_CHECKLIST.md) - 后端验证器端点实现:[sovereignmap_production_backend_v2.py](sovereignmap_production_backend_v2.py) - 移动端验证器合约测试:[tests/scripts/python/test_mobile_verify_gradient_contract.py](tests/scripts/python/test_mobile_verify_gradient_contract.py) ## 可观测性升级(2026 年 3 月) 此次升级将区块链和桥接执行遥测直接绑定到生产级 Grafana 表面。 新增内容: - 区块链和桥接运行时状态的新导出器指标: - `tokenomics_chain_height` - `tokenomics_bridge_transfers_total` - `tokenomics_bridge_routes_active` - `tokenomics_fl_verification_ratio` - `tokenomics_fl_average_confidence_bps` - 代币经济遥测载荷现在从后端运行时计算中发出这些字段。 - Grafana 运维和代币经济仪表盘现在包含一个专门的区块链和桥接运行时部分,涵盖验证、置信度、传输吞吐量、路由计数和链高度。 针对此升级更新的主要文件: - 指标导出器:[tokenomics_metrics_exporter.py](tokenomics_metrics_exporter.py) - 后端遥测载荷:[sovereignmap_production_backend_v2.py](sovereignmap_production_backend_v2.py) - 运维仪表盘:[grafana/provisioning/dashboards/operations_overview.json](grafana/provisioning/dashboards/operations_overview.json) - 代币经济仪表盘:[grafana/provisioning/dashboards/tokenomics_overview.json](grafana/provisioning/dashboards/tokenomics_overview.json) - Prometheus 抓取配置:[prometheus.yml](prometheus.yml) 仪表盘配置说明: - 规范仪表盘由 [grafana/provisioning/dashboards]( B{Traditional FL Aggregator} B --> C[Load full model deltas in memory] C --> D[High RAM footprint per round] A --> E{SovereignMap Mohawk Stream Aggregator} E --> F[Chunk updates into streaming windows] F --> G[Validate trust and policy per chunk] G --> H[Aggregate incrementally] H --> I[Low steady-state memory use] ``` ## 为什么选择 Mohawk Mohawk 风格的流式聚合将模型更新视为连续的分块流,而不是单一的张量载荷。这使得协调器能够以有界的工作内存执行验证、过滤和合并步骤。实际上,这使得在通用基础设施上实现高扇出节点参与成为可能:内存使用量随分块窗口大小缩放,而不是随完整全局更新大小缩放,同时信任和策略检查与聚合内联运行。 ## Mohawk Proto 公告 Sovereign Map 目前使用 Mohawk Proto 流式聚合方法作为其默认的高规模聚合路径。 公告(仅供参考):项目维护者将 Mohawk Proto 及相关聚合设计元素标记为临时专利公告材料。 - 本仓库中的实现仅供评估、研究和集成测试使用。 - 随着临时申请的进展,商业许可预期可能会发生变化。 - 有关企业/商业问题,请在 [GitHub Discussions](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/discussions) 中发起讨论。 ## 运行时版本矩阵 [![Go](https://img.shields.io/badge/Go-1.25%2B-00ADD8?style=flat-square&logo=go&logoColor=white)](README.md#prerequisites) [![Node.js](https://img.shields.io/badge/Node.js-20%2B-339933?style=flat-square&logo=node.js&logoColor=white)](README.md#prerequisites) [![npm](https://img.shields.io/badge/npm-10%2B-CB3837?style=flat-square&logo=npm&logoColor=white)](README.md#prerequisites) [![Python](https://img.shields.io/badge/Python-3.11%2B-3776AB?style=flat-square&logo=python&logoColor=white)](README.md#prerequisites) [![Docker Compose](https://img.shields.io/badge/Docker%20Compose-v2%20plugin-2496ED?style=flat-square&logo=docker&logoColor=white)](README.md#prerequisites) ## 平台概览 [![Capability: Federated Learning](https://img.shields.io/badge/Capability-Federated%20Learning-2f9e44?style=flat-square)](sovereignmap_production_backend_v2.py) [![Capability: Governance Policy](https://img.shields.io/badge/Capability-Governance%20Policy-5f3dc4?style=flat-square)](bridge-policies.json) [![Capability: Trust and Attestation](https://img.shields.io/badge/Capability-Trust%20and%20Attestation-0b7285?style=flat-square)](tpm_cert_manager.py) [![Capability: Tokenomics Telemetry](https://img.shields.io/badge/Capability-Tokenomics%20Telemetry-1971c2?style=flat-square)](tokenomics_metrics_exporter.py) [![Capability: Observability](https://img.shields.io/badge/Capability-Observability-e67700?style=flat-square)](prometheus.yml) [![Runtime: Docker Compose](https://img.shields.io/badge/Runtime-Docker%20Compose%20Supported-0284c7?style=flat-square&logo=docker&logoColor=white)](docker-compose.full.yml) ### 完整徽章组合 图例:按信任顺序从左到右扫描:质量门控 -> 安全/治理 -> SDK/发布 -> 部署表面 -> 设备/运行时占用空间 -> 社区信号。 核心质量门控: [![Build and Test](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/f363da7cc6011406.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/build.yml) [![Lint Code Base](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/aa8a8cf2dc011419.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/lint.yml) [![HIL Tests](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/5870b83706011420.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/hil-tests.yml) [![Reproducibility Check](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/acda053635011422.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/reproducibility-check.yml) [![Observability CI](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/92cacfcc2f011410.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/observability-ci.yml) [![OpenCV Go Tests](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/6c3eed9b6d011423.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/opencv-go-tests.yml) 安全和治理门控: [![CodeQL Security Analysis](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/9563c397db011417.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/codeql-analysis.yml) [![Security Supply Chain](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/f2f19555ca011418.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/security-supply-chain.yml) [![Secret Scan](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/5f3b7a9ebf011424.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/secret-scan.yml) [![Governance Check](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/8e750c0924011425.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/governance-check.yml) [![Workflow Action Pin Check](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/835a2a596e011427.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/workflow-action-pin-check.yml) [![Audit Check](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/deacfd3e9d011428.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/audit-check.yml) SDK 和发布工程: [![SDK Security](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/2a00af1523011429.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/sdk-security.yml) [![SDK Version](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/b2184e8e11011430.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/sdk-version.yml) [![SDK Publish](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/01a53a3e7a011432.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/sdk-publish.yml) [![SDK Provenance](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/d3c9e47b48011433.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/sdk-provenance.yml) [![SDK Channels](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/7d32a54dcc011434.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/sdk-channels.yml) [![Contributor Rankings](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/4b0cea3d09011436.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/contributor-rankings.yml) [![Docs Link Check](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/d259411de1011437.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/docs-link-check.yml) [![Test Artifacts Review](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/166a25dd99011439.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/test-artifacts-review.yml) 部署和打包: [![Build and Deploy](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/859158cd54011440.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/deploy.yml) [![Docker Build](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/6447fb3329011441.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/docker-build.yml) [![Windows Client EXE](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/cf51f6d265011408.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/windows-client-exe.yml) [![Phase 3D Production Deploy](https://static.pigsec.cn/wp-content/uploads/repos/2026/04/2732dd2e41011442.svg)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows/phase3d-production-deploy.yml) 设备和运行时占用空间: [![Web](https://img.shields.io/badge/Device-Web%20Dashboard-0ea5e9?style=flat-square&logo=google-chrome&logoColor=white)](frontend/src/HUD.jsx) [![Linux](https://img.shields.io/badge/OS-Linux%20Supported-f59e0b?style=flat-square&logo=linux&logoColor=white)](docker-compose.full.yml) [![macOS](https://img.shields.io/badge/OS-macOS%20Supported-111827?style=flat-square&logo=apple&logoColor=white)](README.md#quick-start) [![Windows](https://img.shields.io/badge/OS-Windows%20Supported-2563eb?style=flat-square&logo=windows&logoColor=white)](run-demo-windows.ps1) [![Android](https://img.shields.io/badge/Mobile-Android%20Supported-16a34a?style=flat-square&logo=android&logoColor=white)](mobile-apps/android-node-app) [![iOS](https://img.shields.io/badge/Mobile-iOS%20Supported-0f172a?style=flat-square&logo=apple&logoColor=white)](mobile-apps/ios-node-app) [![Raspberry Pi](https://img.shields.io/badge/Edge-Raspberry%20Pi%20Supported-c026d3?style=flat-square&logo=raspberry-pi&logoColor=white)](README.md#hardware-requirements) [![NVIDIA Jetson](https://img.shields.io/badge/Edge-NVIDIA%20Jetson%20Supported-15803d?style=flat-square&logo=nvidia&logoColor=white)](README.md#hardware-requirements) [![x86_64](https://img.shields.io/badge/Arch-x86__64%20Supported-1d4ed8?style=flat-square)](README.md#hardware-requirements) [![ARM64](https://img.shields.io/badge/Arch-ARM64%20Supported-9333ea?style=flat-square)](README.md#hardware-requirements) [![Docker](https://img.shields.io/badge/Runtime-Docker%20Compose%20Supported-0284c7?style=flat-square&logo=docker&logoColor=white)](docker-compose.full.yml) 社区和仓库信号: [![Last Commit](https://img.shields.io/github/last-commit/rwilliamspbg-ops/Sovereign_Map_Federated_Learning?style=flat-square&logo=git)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/commits/main) [![Repo Size](https://img.shields.io/github/repo-size/rwilliamspbg-ops/Sovereign_Map_Federated_Learning?style=flat-square&logo=files)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning) [![Contributors](https://img.shields.io/github/contributors/rwilliamspbg-ops/Sovereign_Map_Federated_Learning?style=flat-square&logo=github)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/graphs/contributors) [![Issues](https://img.shields.io/github/issues/rwilliamspbg-ops/Sovereign_Map_Federated_Learning?style=flat-square&logo=github)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/issues) [![Pull Requests](https://img.shields.io/github/issues-pr/rwilliamspbg-ops/Sovereign_Map_Federated_Learning?style=flat-square&logo=github)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/pulls) [![Stars](https://img.shields.io/github/stars/rwilliamspbg-ops/Sovereign_Map_Federated_Learning?style=flat-square&logo=github)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/stargazers) [![Forks](https://img.shields.io/github/forks/rwilliamspbg-ops/Sovereign_Map_Federated_Learning?style=flat-square&logo=github)](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/network/members) - 浏览所有工作流:[GitHub Actions workflow matrix](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/actions/workflows) - 贡献者和治理流程:[CONTRIBUTING.md](CONTRIBUTING.md) ## 本软件的用途 - 在分布式节点上运行联邦 ML 训练,其中原始数据必须保留在本地。 - 在对抗性或部分受信任的环境中运行拜占庭容错的模型聚合。 - 构建具有策略控制、证明信号和可审计遥测的信任感知 AI 基础设施。 - 通过 Prometheus 和 Grafana 表面监控实时 FL、代币经济和系统健康状况。 - 从本地 Docker 部署到大规模 Compose/Kubernetes 配置文件的原型设计和扩展。 ## 硬件要求 | 节点类别 | 最低(功能性) | 推荐(持续性) | | --- | --- | --- | | 边缘 CPU 节点 | Raspberry Pi 4(4 GB RAM),4 核 ARM CPU,32 GB 存储,Linux,TPM 2.0 设备访问权限 | Raspberry Pi 5 / x86 迷你 PC(8-16 GB RAM),NVMe 存储,TPM 2.0,稳定的有线网络 | | 边缘 GPU/NPU 节点 | Jetson Nano / 支持 Intel NPU 的边缘设备,8 GB RAM,CUDA/NPU 驱动程序 | NVIDIA Jetson Orin / 同等设备,16+ GB RAM,已调优的 CUDA/NPU 堆栈 | | 运维人员/聚合器 | 8 vCPU,16 GB RAM,SSD,Docker Compose | 16+ vCPU,32-64 GB RAM,NVMe,可选 GPU,独立的监控主机 | | 监控堆栈 | 用于 Prometheus + Grafana 的 2 vCPU,4 GB RAM | 4-8 vCPU,8-16 GB RAM,具有更长的保留时间和仪表盘并发 | 使用 [hardware_auto_tuner.py](hardware_auto_tuner.py) 自动分析主机能力并在大规模运行之前选择加速配置文件。 ## 技术简介 Sovereign Map 联邦学习是一个双平面运行时: 1. 聚合平面:基于 Flower 的联邦协调,具有拜占庭稳健策略逻辑和收敛跟踪。 2. 控制和遥测平面:用于健康、HUD、信任/策略操作、加入生命周期和指标发布的 Flask 服务。 核心特征: - 具有运行时收敛历史的拜占庭容错训练策略。 - 用于证明风格治理工作流的信任和验证 API。 - 用于运维人员控制注册的策略和加入管理端点。 - 用于运维和代币经济表面的 Prometheus 兼容指标导出器。 - 通过 Docker Compose 和 Kubernetes 清单进行多配置文件部署。 - 跨越 NPU、XPU、CUDA/ROCm、MPS 和 CPU 回退的硬件感知测试。 ## 系统布局 - 后端聚合和 API:[sovereignmap_production_backend_v2.py](sovereignmap_production_backend_v2.py) - 代币经济指标导出器:[tokenomics_metrics_exporter.py](tokenomics_metrics_exporter.py) - TPM 指标导出器:[tpm_metrics_exporter.py](tpm_metrics_exporter.py) - 前端 HUD:[frontend/src/HUD.jsx](frontend/src/HUD.jsx) - 主要 Compose 配置文件:[docker-compose.full.yml](docker-compose.full.yml) - Kubernetes 规模配置文件:[kubernetes-5000-node-manifests.yaml](kubernetes-5000-node-manifests.yaml) ## 视觉演练 此项目的视觉证明应被视为发布证据,而不是可选的装饰。 每次发布的预期屏幕截图产物: - 运维 HUD:信任分数、节点参与度、延迟墙和弹性指标。 - Grafana 运维概览:实时负载下的仪表盘组 + 趋势墙。 - Grafana 代币经济概览:铸造/桥接/验证器/钱包健康部分。 跟踪的资产位置: - `docs/screenshots/hud-operations-overview.png` - `docs/screenshots/grafana-operations-overview.png` - `docs/screenshots/grafana-llm-overview.png` - `docs/screenshots/grafana-tokenomics-overview.png` 捕获工作流和验收清单: - [docs/screenshots/README.md](docs/screenshots/README.md) 功能产物清单: - [docs/FEATURE_ARTIFACTS_2026-03-21.md](docs/FEATURE_ARTIFACTS_2026-03-21.md) 当前状态:屏幕截图路径已定义,发布捕获工作流已记录;请在每个标记的发布中附上渲染的 PNG/GIF 证据。 ### 仪表盘屏幕截图预览

Grafana Operations Overview Network Operations HUD Grafana LLM Overview Grafana Tokenomics Overview

## 双平面运行时数据流 ``` sequenceDiagram participant Node as Edge Node Client participant Flower as Flower Aggregation Plane (:8080) participant Mohawk as Mohawk Stream Aggregator participant Policy as Trust/Policy Gate participant API as Control Plane API (:8000) participant Prom as Prometheus participant Grafana as Grafana/HUD Node->>Flower: Submit model update (FitRes) Flower->>Mohawk: Forward update chunks Mohawk->>Policy: Validate adapter policy + attestation metadata Policy-->>Mohawk: Accept/Reject + reason labels Mohawk->>Flower: Incremental aggregate result Flower->>API: Publish round metrics + convergence snapshot API->>Prom: Expose /metrics and event-derived gauges/counters Prom->>Grafana: Scrape telemetry API->>Grafana: Serve /health, /ops/health, /hud_data, /metrics_summary ``` ## 功能映射 | 领域 | 运行时表面 | 目的 | | --- | --- | --- | | 联邦学习 | [sovereignmap_production_backend_v2.py](sovereignmap_production_backend_v2.py)、[src/client.py](src/client.py) | 轮次编排、聚合、收敛 | | 信任和证明 | [tpm_cert_manager.py](tpm_cert_manager.py)、[tpm_metrics_exporter.py](tpm_metrics_exporter.py)、[secure_communication.py](secure_communication.py) | 身份、验证、信任信号 | | 治理和策略 | [bridge-policies.json](bridge-policies.json)、[capabilities.json](capabilities.json) | 运行时控制和策略表面 | | 代币经济和经济 | [tokenomics_metrics_exporter.py](tokenomics_metrics_exporter.py)、[tokenomics_metrics_exporter.py](tokenomics_metrics_exporter.py) | 经济遥测和仪表盘输入 | | 可观测性 | [prometheus.yml](prometheus.yml)、[alertmanager.yml](alertmanager.yml)、[fl_slo_alerts.yml](fl_slo_alerts.yml) | 指标收集、告警、SLO 验证 | | 运维 | [deploy.sh](deploy.sh)、[deploy_demo.sh](deploy_demo.sh)、[Makefile](Makefile) | 部署和可重复的运维工作流 | ## 详细功能参考 ### 后端 API 功能 使用此快速索引直接跳转到 API 命令组和文档。 API 命令索引: - [Control Plane API commands](#control-plane-api-commands) - [Training Service API commands](#training-service-api-commands) - [TPM Exporter API commands](#tpm-exporter-api-commands) - [Tokenomics Exporter API commands](#tokenomics-exporter-api-commands) API 文档索引: - HTTP 示例:[docs/api/http-examples.md](docs/api/http-examples.md) - Swagger UI(多规范):[docs/api/swagger-ui.html](docs/api/swagger-ui.html) - OpenAPI 规范:[docs/api/openapi.yaml](docs/api/openapi.yaml)、[docs/api/openapi.training.yaml](docs/api/openapi.training.yaml)、[docs/api/openapi.tpm.yaml](docs/api/openapi.tpm.yaml)、[docs/api/openapi.tokenomics.yaml](docs/api/openapi.tokenomics.yaml) - Postman 集合:[docs/api/postman_collection.json](docs/api/postman_collection.json) - API 验证器命令:`npm run api:validate` - API 文档 CI:[.github/workflows/api-spec-validation.yml](.github/workflows/api-spec-validation.yml)、[.github/workflows/api-docs-pages.yml](.github/workflows/api-docs-pages.yml) #### Control Plane API 命令 ``` curl -s http://localhost:8000/status | jq curl -s http://localhost:8000/health | jq curl -s http://localhost:8000/ops/health | jq curl -s -X POST http://localhost:8000/trigger_fl | jq curl -s http://localhost:8000/metrics_summary | jq curl -s http://localhost:8000/convergence | jq ``` #### Training Service API 命令 ``` curl -s http://localhost:5001/health | jq curl -s http://localhost:8000/training/status | jq ``` #### TPM Exporter API 命令 ``` curl -s http://localhost:9091/health | jq curl -s http://localhost:9091/metrics/summary | jq ``` #### Tokenomics Exporter API 命令 ``` curl -s http://localhost:9105/health | jq ``` | 端点 | 方法 | 功能 | 职责 | | --- | --- | --- | --- | | /health | GET | health | 服务健康、飞地状态、HUD 遥测快照 | | /status | GET | status | 聚合器运行时身份和核心端口映射 | | /chat | POST | chat_query | 用于运维人员提示的 HUD 助手查询处理 | | /hud_data | GET | hud_data | HUD 指标,包括审计准确性和模拟计数器 | | /founders | GET | get_founders | 用于治理视图的创始签名身份列表 | | /trigger_fl | POST | trigger_fl_round | 手动 FL 轮次模拟和收敛更新 | | /create_enclave | POST | create_enclave | 飞地状态转换工作流 | | /convergence | GET | get_convergence | 用于绘图的收敛历史数组 | | /metrics_summary | GET | metrics_summary | 跨运行时域的聚合指标摘要 | | /model_registry | GET | model_registry_recent | 最近的持久化模型元数据和轮次快照 | | /simulate/ | POST | trigger_hud_simulation | 按场景类型记录 HUD 模拟事件 | | /ops/health | GET | ops_health | 运维依赖/系统快照(端口、内存/磁盘压力、Prometheus 可达性) | | /ops/events/recent | GET | ops_events_recent | 返回最近的运维事件以进行时间线重放 | | /ops/events | GET (SSE) | ops_events_stream | 用于实时运维遥测的服务器发送事件流 | ### 信任、策略和加入生命周期功能 | 端点 | 方法 | 功能 | 职责 | | --- | --- | --- | --- | | /trust_snapshot | GET | trust_snapshot | 当前信任模式、策略状态和策略历史 | | /verification_policy | POST | update_verification_policy | 运行时策略更新表面 | | /llm_policy | GET | llm_policy_view | 公开活动的 LLM 适配器验证策略 | | /join/policy | GET | join_policy_view | 加入引导策略和入职约束 | | /join/invite | POST | create_join_invite | 颁发具有有界 TTL 和权限的加入邀请 | | /join/register | POST | register_join_participant | 注册参与者证书和加入元数据 | | /join/registrations | GET | list_join_registrations | 已注册参与者的管理员列表 | | /join/revoke/ | POST | revoke_join_participant | 参与者证书的管理撤销 | ### 训练控制功能 | 端点 | 方法 | 功能 | 职责 | | --- | --- | --- | --- | | /training/start | POST | start_training | 为 HUD/运维 流程触发训练启动信号 | | /training/stop | POST | stop_training | 触发训练停止信号 | | /training/status | GET | training_status | 当前模拟的训练进度和指标视图 | ### 代币经济导出器功能 | 端点 | 方法 | 功能 | 职责 | | --- | --- | --- | --- | | /metrics | GET | metrics | 代币经济仪表的 Prometheus 指标端点 | | /health | GET | health | 代币经济导出器活跃度和源文件元数据 | | /event/tokenomics | POST | event_tokenomics | 接收代币经济事件并持久化规范载荷 | ### TPM 导出器功能 | 端点 | 方法 | 功能 | 职责 | | --- | --- | --- | --- | | /metrics | GET | metrics | TPM/信任指标的 Prometheus 指标端点 | | /metrics/summary | GET | metrics_summary | 聚合的 TPM/信任摘要快照 | | /health | GET | health | TPM 导出器活跃度和元数据 | | /event/attestation | POST | event_attestation | 接收证明事件载荷 | | /event/message | POST | event_message | 接收与信任相关的运维消息 | ### 端点合约说明 - 认证边界:`/join/registrations` 和 `/join/revoke/` 受管理员限制,需要有效的管理员授权标头。 - 认证边界:`/verification_policy` 通过 `X-API-Role` 支持角色感知的更新和可选的承载令牌连接。 - 状态码行为:`/create_enclave` 在配置过程中可能返回 `202`,一旦达到稳定状态转换则返回 `200`。 - 状态码行为:`/trigger_fl` 对于接受的异步执行可能返回 `202`,当轮次执行无法继续时返回非 2xx。 - 流式语义:`/ops/events` 是 SSE 端点,包括心跳事件以保持长连接客户端连接。 - 流式语义:`/ops/events/recent` 应用于附加到 SSE 之前回填时间线状态。 ### 关键内部运行时功能 | 功能 | 文件 | 职责 | | --- | --- | --- | | validate_llm_adapter_update | [sovereignmap_production_backend_v2.py](sovereignmap_production_backend_v2.py) | 传入客户端更新的策略验证门控 | | build_tokenomics_payload | [sovereignmap_production_backend_v2.py](sovereignmap_production_backend_v2.py) | 从 FL 状态构建代币经济发布载荷 | | publish_tokenomics_event | [sovereignmap_production_backend_v2.py](sovereignmap_production_backend_v2.py) | 将代币经济遥测发送到导出器端点 | | publish_tpm_attestation_events | [sovereignmap_production_backend_v2.py](sovereignmap_production_backend_v2.py) | 为信任指标管道发出证明事件 | | run_flower_server | [sovereignmap_production_backend_v2.py](sovereignmap_production_backend_v2.py) | 启动和配置 Flower 聚合服务器 | | run_flask_metrics | [sovereignmap_production_backend_v2.py](sovereignmap_production_backend_v2.py) | 启动用于控制和测的 Flask API 平面 | | create_app | [tokenomics_metrics_exporter.py](tokenomics_metrics_exporter.py) | 构建导出器应用程序和端点绑定 | ## 快速开始 ### 一键式全栈 如果您想要最快的生产级演练,请运行以下两条命令并立即打开 HUD + Grafana: ``` docker compose -f docker-compose.full.yml up -d --scale node-agent=5 make stack-verify ``` 此启动命令的作用: - 从 [docker-compose.full.yml](docker-compose.full.yml) 启动完整的运行时配置文件。 - 将 `node-agent` 服务扩展到 5 个副本。 - 在一个命令中启动 API、HUD 和可观测性服务。 推荐的验证序列: ``` docker compose -f docker-compose.full.yml up -d --scale node-agent=5 docker compose -f docker-compose.full.yml ps make stack-verify ``` 演练后的清理: ``` docker compose -f docker-compose.full.yml down ``` 在 2 分钟内的预期结果: - API 健康端点在 `:8000` 上响应。 - HUD 可在 `http://localhost:3000` 访问。 - Grafana 可在 `http://localhost:3001` 访问。 - Prometheus 可在 `http://localhost:9090` 访问。 ### 前置条件 - Go 1.25+ - Node.js 20+ - npm 10+ - Python 3.11+ - 带有 Compose 插件的 Docker ### 选项 A:创世引导 ``` git clone https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning.git cd Sovereign_Map_Federated_Learning ./genesis-launch.sh ``` ### 选项 B:本地全栈 ``` docker compose -f docker-compose.full.yml up -d --scale node-agent=5 docker compose -f docker-compose.full.yml ps ``` ### 验证堆栈健康状况(必需) ``` curl -s http://localhost:8000/status | jq curl -s http://localhost:8000/health | jq curl -s http://localhost:8000/ops/health | jq curl -s http://localhost:8000/training/status | jq ``` 控制平面 mTLS 验证(启用 mTLS 控制平面时的预期行为): ``` # 如果没有客户端证书,这应该会失败。 curl -kfsS https://localhost:8080/p2p/info || echo "expected: mTLS client certificate required" ``` 如果此请求在没有客户端证书的情况下成功,请在生产使用之前验证您的控制平面 TLS 策略。 预期检查点: - `/status` 返回服务身份和端口。 - `/ops/health` 报告 API、Flower 和 Prometheus 可达性。 - 前端 HUD 可在 `http://localhost:3000` 访问。 - Grafana 可在 `http://localhost:3001` 访问。 ### 首次训练轮次(Hello World) CLI 流程: ``` # 触发一个全局轮次 curl -s -X POST http://localhost:8000/trigger_fl | jq # 验证轮次已推进且指标已更新 curl -s http://localhost:8000/metrics_summary | jq '.federated_learning.current_round, .federated_learning.current_accuracy, .federated_learning.current_loss' curl -s http://localhost:8000/convergence | jq '.current_round, .current_accuracy, .current_loss' ``` UI 流程: 1. 打开 `http://localhost:3000`。 2. 确认 **Network Operations HUD** 是默认着陆视图。 3. 单击 **Run Global FL Epoch**。 4. 确认实时时间线显示 `TRAINING_ROUND` 事件且轮次指标增加。 工作流验证(GitHub Actions): ``` gh run list --branch main --limit 20 ``` ### 选项 C:带有自定义参与者缩放的全配置文件 ``` docker compose -f docker-compose.full.yml up -d --scale node-agent=10 ``` ## 构建和验证命令 ``` # Go 和 backend 测试 go test ./... -count=1 # Monorepo package 构建和测试 npm ci npm run build:libs npm run test:ci # Frontend 构建 npm --prefix frontend ci npm --prefix frontend run build ``` ### FedAvg 基准比较(基础 vs 当前) ``` make benchmark-fedavg-compare ``` 要与不同的基线和输出路径进行比较: ``` BASE_REF=origin/main BENCH_RUNS=3 REPORT_PATH=results/metrics/fedavg_benchmark_compare.md make benchmark-fedavg-compare ``` 生成的报告: - [results/metrics/fedavg_benchmark_compare.md](results/metrics/fedavg_benchmark_compare.md) CI 工作流: - [FedAvg Benchmark Compare workflow](.github/workflows/fedavg-benchmark-compare.yml) ## 贡献者第一步 在打开 PR 之前,运行维护者使用的相同快速检查: ``` # 发现所有可用的开发者目标 make help # 必需基线 make fmt make lint make test # 推荐的可复现性冒烟检查 make smoke # 必需的可视化证据资产 make screenshots-check ``` 对于面向运行时的更改(HUD、可观测性、策略端点),请在 PR 描述中至少包含一个本地验证产物: - `/health` 和 `/ops/health` 输出片段。 - 来自 HUD 或 Grafana 的一张屏幕截图。 - 显示成功 `trigger_fl` 轮次的命令日志。 ## 部署配置文件 - 标准运行时序列:[docker-compose.full.yml](docker-compose.full.yml) ## 仓库标准 - 贡献指南:[CONTRIBUTING.md](CONTRIBUTING.md) - 安全策略:[SECURITY.md](SECURITY.md) - 更改日志:[CHANGELOG.md](CHANGELOG.md) - 许可证:[LICENSE](LICENSE) ## 帮助招募:快速胜利 如果您想快速做出贡献,这些领域具有高影响力和低设置摩擦: - 针对 TPM 2.0 硬件变体和 Docker 运行时的测试矩阵扩展。 - Apple Silicon (MPS) 加速优化和基准基线。 - 针对高基数节点机队的额外 Grafana 面板调优。 - 针对拜占庭和分区模拟路径的更好的合成故障工作负载。 按标签直接跳转到未解决问题: - 良好的初级问题:[good first issue label](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/labels/good%20first%20issue) - 帮助招募:[help wanted label](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/labels/help%20wanted) - 文档:[documentation label](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/labels/documentation) - 可观测性:[observability label](https://github.com/rwilliamspbg-ops/Sovereign_Map_Federated_Learning/labels/observability) 贡献流程和编码标准在 [CONTRIBUTING.md]( 后端端点 -> HUD 计数器表面)。 - 后端和导出器模块通过 Python 语法检查成功编译。 ### CI 健全性检查已完成 在最新更改后,已在 main 上验证通过: - Build and Test - Lint Code Base - HIL Tests - Reproduducibility Check - Governance Check - Workflow Action Pin Check - CodeQL Security Analysis - Security Supply Chain - Secret Scan - Observability CI - Build and Deploy ### 运维建议 对于发布候选版本,使用 [docker-compose.full.yml](
标签:Apex, API规范, FedAvg, FTP漏洞扫描, Grafana, MITM代理, Mohawk协议, TPM 2.0, 代币经济学, 全栈可观测性, 分布式系统, 响应大小分析, 子域名突变, 安全编排, 拜占庭容错, 日志审计, 机器学习, 流式聚合, 生产级框架, 硬件信任根, 联邦学习, 自定义请求头, 请求拦截, 逆向工具, 遥测, 隐私计算