0xsj/overwatch-analysis
GitHub: 0xsj/overwatch-analysis
Overwatch 平台的机器学习与智能分析层，提供模型管理、推理执行、内容分析、行为基线、异常检测、机器人识别及协同行为检测等核心能力。
Stars: 2 | Forks: 0
# Overwatch 分析服务
分析服务是 Overwatch 平台的**机器学习与智能处理层**。它负责管理 ML 模型、运行推理、检测异常、分析内容以及识别协同行为。所有分析能力均通过该服务流转。
## 职责
| 负责内容 | 不负责内容 |
|--------------|-------------------|
| ML 模型生命周期管理 | 内部训练模型（负责编排外部训练） |
| 运行推理（嵌入式 + 外部） | 存储原始采集数据 |
| 内容分析（NER、情感、毒性） | 从来源采集数据 |
| 行为基线计算 | 告警路由或升级 |
| 异常检测 | 用户身份认证 |
| 机器人与协同检测 | 实体/事件 CRUD |
| 网络/图谱分析 | 报告生成 |
| 向下游发送分析事件 | |
## 架构
```
┌──────────────────────────────────────────────────────────────────┐
│ Analysis Service │
│ │
│ ┌────────────────────────────────────────────────────────────┐ │
│ │ gRPC API │ │
│ │ Models: Create, Deploy, Deprecate, List │ │
│ │ Training: Start, Cancel, Get, List │ │
│ │ Inference: Predict, PredictBatch, Feedback │ │
│ │ Content: Analyze, AnalyzeBatch, Get │ │
│ │ Baseline: Compute, Compare, List │ │
│ │ Detection: Bot, Coordination, Anomaly, Network │ │
│ └─────────────────────────────┬──────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────────┴──────────────────────────────┐ │
│ │ Command / Query Handlers │ │
│ │ (CQRS) │ │
│ └─────────────────────────────┬──────────────────────────────┘ │
│ │ │
│ ┌─────────────────────────────┴──────────────────────────────┐ │
│ │ Domain Services │ │
│ │ FeatureExtractor, BaselineService, BotDetectionService │ │
│ │ CoordinationService, AnomalyService │ │
│ └─────────────────────────────┬──────────────────────────────┘ │
│ │ │
│ ┌───────────────────────────┼───────────────────────────┐ │
│ │ │ │ │
│ ▼ ▼ ▼ │
│ ┌──────────┐ ┌──────────────┐ ┌──────────┐ │
│ │Embedded │ │ External │ │ Cross- │ │
│ │Inference │ │ ML Service │ │ Service │ │
│ │ │ │ (HTTP) │ │ Clients │ │
│ │- Rules │ │ │ │ │ │
│ │- Stats │ │- Python/ │ │- Entity │ │
│ │- ONNX │ │ FastAPI │ │- Event │ │
│ └──────────┘ └──────────────┘ │- Narrative│
│ └──────────┘ │
│ │
│ ┌──────────────────────┼──────────────────────┐ │
│ ▼ ▼ ▼ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ PostgreSQL │ │ Redis │ │ NATS │ │
│ │ (store) │ │ (cache) │ │ (events) │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │ │
└───────────────────────────────────────────────────────┼──────────┘
│
▼
Alert, Correlation, Ledger
```
## 核心概念
### 模型 (Models)
带版本控制和部署配置的 ML 模型定义。
```
{
"id": "01KF0X8HEBWPCCF9ASNGXX7YMM",
"tenant_id": null,
"name": "Bot Detector v2",
"description": "Identifies automated social media accounts",
"type": "BINARY_CLASSIFICATION",
"capabilities": ["BOT_DETECTION"],
"status": "DEPLOYED",
"version": "2.1.0",
"is_default": true,
"input_schema": { "features": ["posting_freq", "account_age", "follower_ratio"] },
"output_schema": { "bot_score": "float", "is_bot": "bool" },
"metrics": {
"accuracy": 0.94,
"precision": 0.91,
"recall": 0.89,
"f1_score": 0.90
},
"deployment": {
"endpoint_type": "embedded_rule",
"timeout_ms": 100
},
"created_at": "2026-01-15T12:00:00Z"
}
```
### 预测 (Predictions)
带有可选可解释性的推理结果。
```
{
"id": "01KF0X8HEBWPCCF9ASNGXX7YMP",
"tenant_id": "01KF0X8HEBWPCCF9ASNGXX7YMN",
"model_id": "01KF0X8HEBWPCCF9ASNGXX7YMM",
"model_version": "2.1.0",
"capability": "BOT_DETECTION",
"target_domain": "account",
"target_id": "acc_123456",
"input": {
"posting_freq": 45.2,
"account_age_days": 3,
"follower_ratio": 0.01
},
"output": {
"bot_score": 0.87,
"is_bot": true
},
"confidence": 0.87,
"explanations": [
{
"type": "feature_importance",
"feature": "posting_freq",
"contribution": 0.45,
"description": "Posting frequency 10x above normal"
}
],
"latency_ms": 12,
"created_at": "2026-01-15T12:00:00Z"
}
```
### 基线 (Baselines)
用于异常检测的学习到的正常行为。
```
{
"id": "01KF0X8HEBWPCCF9ASNGXX7YMQ",
"tenant_id": "01KF0X8HEBWPCCF9ASNGXX7YMN",
"entity_id": "ent_vessel_123",
"baseline_type": "route",
"statistics": {
"avg_speed_knots": { "mean": 12.5, "std": 2.1, "p5": 8.0, "p95": 16.0 }
},
"patterns": {
"typical_routes": ["port_a_to_port_b", "port_b_to_port_c"]
},
"window_start": "2025-01-01T00:00:00Z",
"window_end": "2026-01-01T00:00:00Z",
"sample_count": 45000,
"stability_score": 0.92,
"is_stale": false,
"created_at": "2026-01-01T00:00:00Z"
}
```
### 内容分析 (Content Analysis)
文本内容的 NLP 结果。
```
{
"id": "01KF0X8HEBWPCCF9ASNGXX7YMR",
"content_id": "post_123456",
"entities": [
{
"text": "John Smith",
"type": "PERSON",
"start_offset": 0,
"end_offset": 10,
"confidence": 0.95,
"resolved_entity_id": "ent_person_456"
}
],
"topics": [{ "topic": "shipping", "score": 0.85 }],
"sentiment": {
"label": "negative",
"score": -0.65,
"positive": 0.12,
"negative": 0.77,
"neutral": 0.11
},
"toxicity": { "is_toxic": false, "score": 0.08 },
"language": { "language_code": "en", "confidence": 0.99 },
"analyzed_at": "2026-01-15T12:00:00Z"
}
```
### 协同结果 (Coordination Result)
对协同虚假行为的检测。
```
{
"id": "01KF0X8HEBWPCCF9ASNGXX7YMS",
"tenant_id": "01KF0X8HEBWPCCF9ASNGXX7YMN",
"account_ids": ["acc_001", "acc_002", "acc_003"],
"coordination_score": 0.89,
"is_coordinated": true,
"signals": [
{
"type": "timing",
"score": 0.92,
"weight": 0.3,
"evidence": { "avg_delay_seconds": 2.5, "burst_count": 15 }
},
{
"type": "content_similarity",
"score": 0.85,
"weight": 0.25,
"evidence": { "jaccard_similarity": 0.78 }
}
],
"analyzed_at": "2026-01-15T12:00:00Z"
}
```
## 模型生命周期
```
┌──────────┐
│ DRAFT │
└────┬─────┘
│ start training
▼
┌──────────┐
│ TRAINING │◄────────────────┐
└────┬─────┘ │ retrain
│ complete │
▼ │
┌──────────┐ │
│VALIDATING│ │
└────┬─────┘ │
│ passes │
▼ │
┌──────────┐ │
deploy │ READY │─────────────────┘
┌─────────────┤ │
▼ └────┬─────┘
┌──────────┐ │ deprecate
│ DEPLOYED │ │
└────┬─────┘ │
│ undeploy │
└──────────────────┼─────────────────┐
▼ ▼
┌──────────┐ ┌──────────┐
│DEPRECATED│ │ FAILED │
└──────────┘ └──────────┘
```
## 推理架构
### 嵌入式预测器（进程内）
| 类型 | 用例 | 实现 |
|------|----------|----------------|
| **基于规则** | 机器人检测、协同信号 | Go 原生 |
| **统计学** | 异常检测、基线比对 | gonum/stat |
| **ONNX** | 预训练情感、NER、embeddings | onnxruntime-go |
### 外部预测器（HTTP/gRPC）
| 类型 | 用例 | 实现 |
|------|----------|----------------|
| **HTTP** | Python ML 服务、HuggingFace API | net/http client |
| **gRPC** | 高性能模型服务器 | gRPC client |
### 预测器选择
```
Model.Deployment.EndpointType:
"embedded_rule" ──────▶ EmbeddedRulePredictor
"embedded_stats" ──────▶ EmbeddedStatsPredictor
"embedded_onnx" ──────▶ EmbeddedONNXPredictor
"http" ──────▶ HTTPPredictor
"grpc" ──────▶ GRPCPredictor
```
## 检测流程
### 机器人检测
```
1. Client calls DetectBot(account_id)
2. Fetch account from Narrative service
3. Extract features (posting_freq, account_age, follower_ratio)
4. Get default model for BOT_DETECTION capability
5. Route to appropriate predictor
6. Run inference
7. Store prediction
8. Publish BotDetected event if is_bot=true
9. Return result
```
### 异常检测
```
1. Client calls DetectAnomalies(entity_id, metrics)
2. Fetch baseline for entity + baseline_type
3. If no baseline, compute from historical data
4. Compare current values to baseline statistics
5. Calculate z-scores
6. Classify severity (low, medium, high, critical)
7. Publish AnomalyDetected for significant deviations
8. Return anomalies
```
### 协同检测
```
1. Client calls DetectCoordination(account_ids)
2. Fetch posts for all accounts from Narrative
3. Analyze timing patterns (intervals, bursts)
4. Analyze content (similarity, shared phrases)
5. Analyze network (retweet patterns, mentions)
6. Calculate weighted coordination score
7. Store result
8. Publish CoordinationDetected if coordinated
9. Return result
```
## 事件流
**发布至：**
```
overwatch.analysis.model.created
overwatch.analysis.model.deployed
overwatch.analysis.model.deprecated
overwatch.analysis.training.started
overwatch.analysis.training.completed
overwatch.analysis.training.failed
overwatch.analysis.prediction.created
overwatch.analysis.feedback.submitted
overwatch.analysis.content.analyzed
overwatch.analysis.baseline.computed
overwatch.analysis.anomaly.detected
overwatch.analysis.network.analyzed
overwatch.analysis.coordination.detected
overwatch.analysis.bot.detected
```
**消费自：**
```
overwatch.entity.entity.created
overwatch.entity.entity.updated
overwatch.event.event.recorded
overwatch.narrative.post.ingested
```
**被消费方：**
- Ledger 服务（审计追踪）
- Alert 服务（触发告警）
- Correlation 服务（模式匹配）
## 来源证明 (Provenance)
每个检测事件均由分析服务签名：
```
{
"event_id": "01KF0...",
"event_type": "bot.detected",
"occurred_at": "2026-01-15T12:00:00Z",
"service_signer": {
"did": "did:key:z6MkAnalysisService...",
"signer_type": "SERVICE",
"signer_id": "analysis-service",
"signature": "base64..."
},
"payload": {
"account_id": "acc_123",
"bot_score": 0.87,
"is_bot": true,
"model_id": "01KF0...",
"model_version": "2.1.0"
}
}
```
## 数据库 Schema
```
CREATE TABLE models (
id TEXT PRIMARY KEY,
tenant_id TEXT,
name TEXT NOT NULL,
description TEXT,
type TEXT NOT NULL,
capabilities TEXT[] NOT NULL DEFAULT '{}',
status TEXT NOT NULL DEFAULT 'draft',
version TEXT NOT NULL,
parent_version_id TEXT REFERENCES models(id),
is_default BOOLEAN NOT NULL DEFAULT FALSE,
input_schema JSONB,
output_schema JSONB,
metrics JSONB,
config JSONB,
deployment JSONB,
created_by TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE training_jobs (
id TEXT PRIMARY KEY,
model_id TEXT NOT NULL REFERENCES models(id),
tenant_id TEXT,
status TEXT NOT NULL DEFAULT 'pending',
training_config JSONB,
data_window_start TIMESTAMPTZ,
data_window_end TIMESTAMPTZ,
progress_percent INTEGER,
current_stage TEXT,
output_model_version TEXT,
output_metrics JSONB,
error TEXT,
created_by TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
started_at TIMESTAMPTZ,
completed_at TIMESTAMPTZ
);
CREATE TABLE predictions (
id TEXT PRIMARY KEY,
tenant_id TEXT,
model_id TEXT NOT NULL,
model_version TEXT NOT NULL,
capability TEXT NOT NULL,
target_domain TEXT NOT NULL,
target_id TEXT NOT NULL,
input JSONB NOT NULL,
output JSONB NOT NULL,
confidence DOUBLE PRECISION NOT NULL,
explanations JSONB,
feedback_is_correct BOOLEAN,
feedback_label TEXT,
feedback_notes TEXT,
feedback_by TEXT,
feedback_at TIMESTAMPTZ,
latency_ms INTEGER NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE baselines (
id TEXT PRIMARY KEY,
tenant_id TEXT,
entity_id TEXT NOT NULL,
baseline_type TEXT NOT NULL,
statistics JSONB NOT NULL,
patterns JSONB,
window_start TIMESTAMPTZ NOT NULL,
window_end TIMESTAMPTZ NOT NULL,
sample_count BIGINT NOT NULL,
stability_score DOUBLE PRECISION,
is_stale BOOLEAN NOT NULL DEFAULT FALSE,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE(tenant_id, entity_id, baseline_type)
);
CREATE TABLE content_analyses (
id TEXT PRIMARY KEY,
content_id TEXT NOT NULL UNIQUE,
entities JSONB,
topics JSONB,
sentiment JSONB,
toxicity JSONB,
language JSONB,
embedding JSONB,
model_versions JSONB,
analyzed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE network_analyses (
id TEXT PRIMARY KEY,
tenant_id TEXT,
network_type TEXT NOT NULL,
seed_ids TEXT[] NOT NULL,
depth INTEGER NOT NULL,
node_count INTEGER NOT NULL,
edge_count INTEGER NOT NULL,
density DOUBLE PRECISION,
communities JSONB,
influential_nodes JSONB,
bridge_nodes JSONB,
analyzed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE coordination_results (
id TEXT PRIMARY KEY,
tenant_id TEXT,
account_ids TEXT[] NOT NULL,
coordination_score DOUBLE PRECISION NOT NULL,
is_coordinated BOOLEAN NOT NULL,
signals JSONB NOT NULL,
network JSONB,
analyzed_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_models_tenant ON models(tenant_id);
CREATE INDEX idx_models_capability ON models USING GIN(capabilities);
CREATE INDEX idx_models_status ON models(status);
CREATE INDEX idx_predictions_model ON predictions(model_id);
CREATE INDEX idx_predictions_target ON predictions(target_domain, target_id);
CREATE INDEX idx_predictions_capability ON predictions(capability);
CREATE INDEX idx_baselines_entity ON baselines(entity_id);
CREATE INDEX idx_content_analyses_content ON content_analyses(content_id);
CREATE INDEX idx_coordination_accounts ON coordination_results USING GIN(account_ids);
```
## 配置
```
# 服务器
ANALYSIS_SERVER_HOST=0.0.0.0
ANALYSIS_SERVER_PORT=50060
# 数据库
ANALYSIS_DATABASE_HOST=localhost
ANALYSIS_DATABASE_PORT=5450
ANALYSIS_DATABASE_USER=overwatch
ANALYSIS_DATABASE_PASSWORD=overwatch_dev
ANALYSIS_DATABASE_NAME=overwatch_analysis
ANALYSIS_DATABASE_SSL_MODE=disable
# Redis
ANALYSIS_REDIS_HOST=localhost
ANALYSIS_REDIS_PORT=6390
ANALYSIS_REDIS_DB=6
# NATS
ANALYSIS_NATS_URL=nats://localhost:4230
ANALYSIS_NATS_SUBJECT_PREFIX=overwatch.analysis
# Service Identity
ANALYSIS_SERVICE_IDENTITY_ID=analysis-service
ANALYSIS_SERVICE_IDENTITY_NAME=analysis
ANALYSIS_SERVICE_IDENTITY_GENERATE_IF_MISSING=true
# Cross-Service Clients
ANALYSIS_ENTITY_SERVICE_ADDR=localhost:50055
ANALYSIS_EVENT_SERVICE_ADDR=localhost:50056
ANALYSIS_NARRATIVE_SERVICE_ADDR=localhost:50061
# Inference
ANALYSIS_INFERENCE_DEFAULT_TIMEOUT_MS=5000
ANALYSIS_INFERENCE_MAX_BATCH_SIZE=100
# External ML Service (可选)
ANALYSIS_ML_SERVICE_URL=http://localhost:8000
ANALYSIS_ML_SERVICE_TIMEOUT_MS=30000
# ONNX Models (可选)
ANALYSIS_ONNX_MODELS_PATH=/models
# 阈值
ANALYSIS_BOT_THRESHOLD=0.5
ANALYSIS_COORDINATION_THRESHOLD=0.6
ANALYSIS_COORDINATION_MIN_ACCOUNTS=3
ANALYSIS_BASELINE_MIN_SAMPLES=100
ANALYSIS_BASELINE_STALE_DAYS=30
```
## gRPC API
```
service AnalysisService {
rpc Ping(PingRequest) returns (PingResponse);
// Model Management
rpc CreateModel(CreateModelRequest) returns (CreateModelResponse);
rpc GetModel(GetModelRequest) returns (GetModelResponse);
rpc UpdateModel(UpdateModelRequest) returns (UpdateModelResponse);
rpc DeleteModel(DeleteModelRequest) returns (DeleteModelResponse);
rpc ListModels(ListModelsRequest) returns (ListModelsResponse);
rpc GetDefaultModel(GetDefaultModelRequest) returns (GetDefaultModelResponse);
rpc SetDefaultModel(SetDefaultModelRequest) returns (SetDefaultModelResponse);
rpc DeployModel(DeployModelRequest) returns (DeployModelResponse);
rpc UndeployModel(UndeployModelRequest) returns (UndeployModelResponse);
rpc DeprecateModel(DeprecateModelRequest) returns (DeprecateModelResponse);
// Training
rpc StartTraining(StartTrainingRequest) returns (StartTrainingResponse);
rpc GetTrainingJob(GetTrainingJobRequest) returns (GetTrainingJobResponse);
rpc ListTrainingJobs(ListTrainingJobsRequest) returns (ListTrainingJobsResponse);
rpc CancelTrainingJob(CancelTrainingJobRequest) returns (CancelTrainingJobResponse);
// Inference
rpc Predict(PredictRequest) returns (PredictResponse);
rpc PredictBatch(PredictBatchRequest) returns (PredictBatchResponse);
rpc GetPrediction(GetPredictionRequest) returns (GetPredictionResponse);
rpc ListPredictions(ListPredictionsRequest) returns (ListPredictionsResponse);
rpc SubmitFeedback(SubmitFeedbackRequest) returns (SubmitFeedbackResponse);
// Content Analysis
rpc AnalyzeContent(AnalyzeContentRequest) returns (AnalyzeContentResponse);
rpc AnalyzeContentBatch(AnalyzeContentBatchRequest) returns (AnalyzeContentBatchResponse);
rpc GetContentAnalysis(GetContentAnalysisRequest) returns (GetContentAnalysisResponse);
// Baseline Analysis
rpc GetBaseline(GetBaselineRequest) returns (GetBaselineResponse);
rpc ComputeBaseline(ComputeBaselineRequest) returns (ComputeBaselineResponse);
rpc ListBaselines(ListBaselinesRequest) returns (ListBaselinesResponse);
rpc CompareToBaseline(CompareToBaselineRequest) returns (CompareToBaselineResponse);
// Similarity Analysis
rpc FindSimilar(FindSimilarRequest) returns (FindSimilarResponse);
rpc ComputeSimilarity(ComputeSimilarityRequest) returns (ComputeSimilarityResponse);
// Network Analysis
rpc AnalyzeNetwork(AnalyzeNetworkRequest) returns (AnalyzeNetworkResponse);
rpc GetNetworkAnalysis(GetNetworkAnalysisRequest) returns (GetNetworkAnalysisResponse);
// Coordination Detection
rpc DetectCoordination(DetectCoordinationRequest) returns (DetectCoordinationResponse);
rpc GetCoordinationResult(GetCoordinationResultRequest) returns (GetCoordinationResultResponse);
// Bot Detection
rpc DetectBot(DetectBotRequest) returns (DetectBotResponse);
rpc DetectBotsBatch(DetectBotsBatchRequest) returns (DetectBotsBatchResponse);
// Anomaly Detection
rpc DetectAnomalies(DetectAnomaliesRequest) returns (DetectAnomaliesResponse);
// Stats
rpc GetModelStats(GetModelStatsRequest) returns (GetModelStatsResponse);
rpc GetAnalysisStats(GetAnalysisStatsRequest) returns (GetAnalysisStatsResponse);
}
```
## 与其他服务的集成
```
┌─────────────────────────────────────────────────────────────────┐
│ Analysis Service │
└────────────────────────────────┬────────────────────────────────┘
│
┌───────────────────────────┼───────────────────────────┐
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Entity │ │ Event │ │ Narrative │
│ Service │ │ Service │ │ Service │
│ │ │ │ │ │
│ - Get entity │ │ - Get events │ │ - Get posts │
│ attributes │ │ for entity │ │ - Get account│
│ - Get links │ │ - Historical │ │ activity │
└──────────────┘ └──────────────┘ └──────────────┘
│ │ │
└────────────────────────┴────────────────────────┘
│
▼
┌────────────────────────┐
│ Feature Extraction │
│ & Analysis │
└────────────┬───────────┘
│
┌───────────────────────────┼───────────────────────────┐
│ │ │
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ Alert │ │ Correlation │ │ Ledger │
│ Service │ │ Service │ │ Service │
│ │ │ │ │ │
│ Receives: │ │ Receives: │ │ Receives: │
│ - bot.detect │ │ - anomaly │ │ - All events │
│ - anomaly │ │ - coordinat. │ │ for audit │
│ - coordinat. │ │ - analysis │ │ │
└──────────────┘ └──────────────┘ └──────────────┘
```
## 参考资料
- **Protos**: `overwatch-contracts/proto/analysis/v1/`
- **架构模式**: 六边形架构与 CQRS
- **来源签名**: `overwatch-pkg/provenance/`
- **推理**: 嵌入式（规则、统计、ONNX） + 外部（HTTP、gRPC）
标签：Apex, EVTX分析, GitHub, gRPC, HTTP/HTTPS抓包, ML模型管理, Naabu, NER, Python工具, 人工智能, 内容安全, 协同行为检测, 反欺诈, 图分析, 基线计算, 威胁情报, 威胁评分, 开发者工具, 异常检测, 归因分析, 态势感知, 情感分析, 情报分析, 推理引擎, 搜索引擎查询, 日志审计, 机器人检测, 机器学习, 模式识别, 毒性分析, 测试用例, 深度学习, 用户模式Hook绕过, 社交网络分析, 系统分析, 网络分析, 网络安全, 网络诊断, 舆情分析, 虚假信息检测, 行为建模, 足迹分析, 逆向工具, 配置错误, 隐私保护, 风险控制