robert-mcdermott/ai-knowledge-graph

 # AI 驱动的知识图谱生成器 该系统接收非结构化文本文档，并使用您选择的 LLM 以“主语-谓语-宾语”（SPO）三元组的形式提取知识，然后将这些关系可视化为交互式知识图谱。 您可以在此处查看使用该项目创建的知识图谱演示：[工业革命知识图谱](https://robert-mcdermott.github.io/ai-knowledge-graph/) ## 功能 - **文本分块**：自动将大型文档分割成易于处理的块 - **知识提取**：使用 AI 识别实体及其关系 - **实体标准化**：确保在文档块之间保持实体命名的一致性 - **关系推断**：发现图中互不相连部分之间的额外关系 - **交互式可视化**：创建交互式的图谱可视化 - **兼容任何 OpenAI 兼容 API endpoint**：Ollama、LM Studio、OpenAI、vLLM、LiteLLM（提供对 AWS Bedrock、Azure OpenAI、Anthropic 及许多其他 LLM 服务的访问） ## 环境要求 - Python 3.11+ - 必需的依赖包（使用 `pip install -r requirements.txt` 或 `uv sync` 安装） ## 快速开始 1. 克隆此代码仓库 2. 安装依赖项：`pip install -r requirements.txt` 3. 在 `config.toml` 中配置您的设置 4. 运行系统： ``` python generate-graph.py --input your_text_file.txt --output knowledge_graph.html ``` 或者使用 UV： ``` uv run generate-graph.py --input your_text_file.txt --output knowledge_graph.html ``` 或者安装并以模块形式使用： ``` pip install --upgrade -e . generate-graph --input your_text_file.txt --output knowledge_graph.html ``` ## 配置 系统可以使用 `config.toml` 文件进行配置： ``` [llm] model = "gemma3" # Google open weight model api_key = "sk-1234" base_url = "http://localhost:11434/v1/chat/completions" # Local Ollama instance running locally (but can be any OpenAI compatible endpoint) max_tokens = 8192 temperature = 0.2 [chunking] chunk_size = 200 # Number of words per chunk overlap = 20 # Number of words to overlap between chunks [standardization] enabled = true # Enable entity standardization use_llm_for_entities = true # Use LLM for additional entity resolution [inference] enabled = true # Enable relationship inference use_llm_for_inference = true # Use LLM for relationship inference apply_transitive = true # Apply transitive inference rules ``` ## 命令行选项 - `--input FILE`：要处理的输入文本文件 - `--output FILE`：用于可视化的输出 HTML 文件（默认：knowledge_graph.html） - `--config FILE`：配置文件的路径（默认：config.toml） - `--debug`：启用调试输出，显示原始 LLM 响应 - `--no-standardize`：禁用实体标准化 - `--no-inference`：禁用关系推断 - `--test`：使用测试数据生成示例可视化 ### 使用说明信息 (--help) ``` generate-graph --help usage: generate-graph [-h] [--test] [--config CONFIG] [--output OUTPUT] [--input INPUT] [--debug] [--no-standardize] [--no-inference] Knowledge Graph Generator and Visualizer options: -h, --help show this help message and exit --test Generate a test visualization with sample data --config CONFIG Path to configuration file --output OUTPUT Output HTML file path --input INPUT Path to input text file (required unless --test is used) --debug Enable debug output (raw LLM responses and extracted JSON) --no-standardize Disable entity standardization --no-inference Disable relationship inference ``` ### 运行示例 **命令：** ``` generate-graph --input data/industrial-revolution.txt --output industrial-revolution-kg.html ``` **控制台输出：** ``` Using input text from file: data/industrial-revolution.txt ================================================== PHASE 1: INITIAL TRIPLE EXTRACTION ================================================== Processing text in 13 chunks (size: 100 words, overlap: 20 words) Processing chunk 1/13 (100 words) Processing chunk 2/13 (100 words) Processing chunk 3/13 (100 words) Processing chunk 4/13 (100 words) Processing chunk 5/13 (100 words) Processing chunk 6/13 (100 words) Processing chunk 7/13 (100 words) Processing chunk 8/13 (100 words) Processing chunk 9/13 (100 words) Processing chunk 10/13 (100 words) Processing chunk 11/13 (100 words) Processing chunk 12/13 (86 words) Processing chunk 13/13 (20 words) Extracted a total of 216 triples from all chunks ================================================== PHASE 2: ENTITY STANDARDIZATION ================================================== Starting with 216 triples and 201 unique entities Standardizing entity names across all triples... Applied LLM-based entity standardization for 15 entity groups Standardized 201 entities into 181 standard forms After standardization: 216 triples and 160 unique entities ================================================== PHASE 3: RELATIONSHIP INFERENCE ================================================== Starting with 216 triples Top 5 relationship types before inference: - enables: 20 occurrences - impacts: 15 occurrences - enabled: 12 occurrences - pioneered: 10 occurrences - invented: 9 occurrences Inferring additional relationships between entities... Identified 9 disconnected communities in the graph Inferred 3 new relationships between communities Inferred 3 new relationships between communities Inferred 3 new relationships between communities Inferred 3 new relationships between communities Inferred 3 new relationships between communities Inferred 3 new relationships between communities Inferred 3 new relationships between communities Inferred 3 new relationships between communities Inferred 3 new relationships between communities Inferred 3 new relationships between communities Inferred 9 new relationships within communities Inferred 2 new relationships within communities Inferred 88 relationships based on lexical similarity Added -22 inferred relationships Top 5 relationship types after inference: - related to: 65 occurrences - advances via Artificial Intelligence: 36 occurrences - pioneered via computing: 26 occurrences - enables via computing: 24 occurrences - enables: 21 occurrences Added 370 inferred relationships Final knowledge graph: 564 triples Saved raw knowledge graph data to /mnt/c/Users/rmcdermo/Documents/industrial-revolution-kg.json Processing 564 triples for visualization Found 161 unique nodes Found 355 inferred relationships Detected 9 communities using Louvain method Nodes in NetworkX graph: 161 Edges in NetworkX graph: 537 Knowledge graph visualization saved to /mnt/c/Users/rmcdermo/Documents/industrial-revolution-kg.html Graph Statistics: { "nodes": 161, "edges": 564, "original_edges": 209, "inferred_edges": 355, "communities": 9 } Knowledge Graph Statistics: Nodes: 161 Edges: 564 Communities: 9 To view the visualization, open the following file in your browser: file:///mnt/c/Users/rmcdermo/Documents/industrial-revolution-kg.html ``` ## 工作原理 1. **分块**：将文档分割成具有重叠部分的块，以适应 LLM 的上下文窗口 2. **第一遍 - SPO 提取**： - LLM 处理每个块以提取“主语-谓语-宾语”三元组 - 在 `process_with_llm` 函数中实现 - LLM 识别每个文本片段中的实体及其关系 - 收集所有块的结果以形成初始知识图谱 3. **第二遍 - 实体标准化**： - 通过文本规范化进行基础标准化 - 可选的 LLM 辅助实体对齐（由 `standardization.use_llm_for_entities` 配置控制） - 启用后，LLM 会审查图谱中的所有唯一实体，并识别指向同一概念的实体组 - 这解决了同一个实体在不同块中表现不同的情况（例如，“AI”、“artificial intelligence”、“AI system”） - 标准化有助于创建更加连贯且易于导航的知识图谱 4. **第三遍 - 关系推断**： - 自动推断传递关系 - 可选的 LLM 辅助推断，用于连接互不相连的图组件（由 `inference.use_llm_for_inference` 配置控制） - 启用后，LLM 会分析来自互不相连社区的代表性实体，并推断出合理的关系 - 这通过添加文本中未明确说明的逻辑连接来减少图谱的碎片化 - 基于规则和基于 LLM 的推断方法协同工作，以创建更全面的图谱 5. **可视化**：使用 PyVis 库生成交互式 HTML 可视化 第二遍和第三遍都是可选的，可以在配置中禁用，以尽量减少 LLM 的使用或手动控制这些过程。 ## 可视化功能 - **颜色编码的社区**：节点颜色代表不同的社区 - **节点大小**：根据重要性（度、中介性、特征向量中心性）调整节点大小 - **关系类型**：原始关系显示为实线，推断关系显示为虚线 - **交互控制**：缩放、平移、悬停查看详情、过滤及物理控制 - **浅色（默认）和深色模式主题**。 ## 项目布局 ``` . ├── config.toml # Main configuration file for the system ├── generate-graph.py # Entry point when run directly as a script ├── pyproject.toml # Python project metadata and build configuration ├── requirements.txt # Python dependencies for 'pip' users ├── uv.lock # Python dependencies for 'uv' users └── src/ # Source code ├── generate_graph.py # Main entry point script when run as a module └── knowledge_graph/ # Core package ├── __init__.py # Package initialization ├── config.py # Configuration loading and validation ├── entity_standardization.py # Entity standardization algorithms ├── llm.py # LLM interaction and response processing ├── main.py # Main program flow and orchestration ├── prompts.py # Centralized collection of LLM prompts ├── text_utils.py # Text processing and chunking utilities ├── visualization.py # Knowledge graph visualization generator └── templates/ # HTML templates for visualization └── graph_template.html # Base template for interactive graph ``` ## 程序流程 此图展示了程序流程。 ``` flowchart TD %% Main entry points A[main.py - Entry Point] --> B{Parse Arguments} %% Test mode branch B -->|--test flag| C[sample_data_visualization] C --> D[visualize_knowledge_graph] %% Normal processing branch B -->|normal processing| E[load_config] E --> F[process_text_in_chunks] %% Text processing F --> G[chunk_text] G --> H[process_with_llm] %% LLM processing H --> I[call_llm] I --> J[extract_json_from_text] %% Entity standardization phase F --> K{standardization enabled?} K -->|yes| L[standardize_entities] K -->|no| M{inference enabled?} L --> M %% Relationship inference phase M -->|yes| N[infer_relationships] M -->|no| O[visualize_knowledge_graph] N --> O %% Visualization components O --> P[_calculate_centrality_metrics] O --> Q[_detect_communities] O --> R[_calculate_node_sizes] O --> S[_add_nodes_and_edges_to_network] O --> T[_get_visualization_options] O --> U[_save_and_modify_html] %% Subprocesses L --> L1[_resolve_entities_with_llm] N --> N1[_identify_communities] N --> N2[_infer_relationships_with_llm] N --> N3[_infer_within_community_relationships] N --> N4[_apply_transitive_inference] N --> N5[_infer_relationships_by_lexical_similarity] N --> N6[_deduplicate_triples] %% File outputs U --> V[HTML Visualization] F --> W[JSON Data Export] %% Prompts usage Y[prompts.py] --> H Y --> L1 Y --> N2 Y --> N3 %% Module dependencies subgraph Modules main.py config.py text_utils.py llm.py entity_standardization.py visualization.py prompts.py end %% Phases subgraph Phase 1: Triple Extraction G H I J end subgraph Phase 2: Entity Standardization L L1 end subgraph Phase 3: Relationship Inference N N1 N2 N3 N4 N5 N6 end subgraph Phase 4: Visualization O P Q R S T U end ``` ## 程序流程说明 1. **入口点**：程序从 `main.py` 开始，解析命令行参数。 2. **模式选择**： - 如果提供了 `--test` 标志，则生成示例可视化 - 否则，处理输入的文本文件 3. **配置**：使用 `config.py` 从 `config.toml` 加载设置 4. **文本处理**： - 使用 `text_utils.py` 将文本分割成有重叠的块 - 使用 LLM 处理每个块以提取三元组 - 使用 `prompts.py` 中的 prompt 指导 LLM 的提取过程 5. **实体标准化**（可选）： - 标准化所有三元组中的实体名称 - 在模棱两可的情况下可以使用 LLM 进行实体消歧 - 使用 `prompts.py` 中的专用 prompt 进行实体消歧 6. **关系推断**（可选）： - 识别图谱中的社区 - 推断互不相连社区之间的关系 - 应用传递性推断和词汇相似性规则 - 使用 `prompts.py` 中的专用 prompt 进行关系推断 - 对三元组进行去重 7. **可视化**： - 计算中心性指标并进行社区检测 - 根据重要性确定节点的大小和颜色 - 使用 PyVis 创建交互式 HTML 可视化 - 使用模板自定义 HTML 8. **输出**： - 以 HTML 和 JSON 格式保存知识图谱 - 显示有关节点、边和社区的统计信息

标签：AI风险缓解, DLL 劫持, Petitpotam, Python安全, 人工智能, 只读文件系统, 大语言模型, 特权检测, 用户模式Hook绕过, 逆向工具