amirhosssein0/kafka-and-security-lab

# Kafka & Security 实验室 基于 Kubernetes，使用 Kafka (Strimzi) 和 KEDA 自动扩缩容的事件驱动通知系统 — 由 Trivy、Checkov、Kyverno、Falco 和 Velero 提供安全保障。 本项目将真实的事件驱动架构（Kafka + KEDA 缩容至零自动扩缩容）与完整的 DevSecOps 流水线（左移扫描、准入控制、运行时安全和灾难恢复）相结合，通过 GitOps (ArgoCD) 部署在 Azure Kubernetes Service 上。 ## 架构 ``` ┌─────────────┐ HTTP POST │ │ ──────────────────▶ │ Producer │ /notify │ (Go/Gin) │ └──────┬──────┘ │ writes ▼ ┌──────────────────┐ │ Kafka Topic: │ │ notifications │ (Strimzi, KRaft mode, │ (3 partitions) │ single-broker) └────────┬─────────┘ │ reads (consumer group) ▼ ┌──────────────────┐ │ Consumer │ │ (retry + backoff)│──────▶ SMTP (email delivery) └────────┬─────────┘ │ on failure after max retries ▼ ┌──────────────────┐ │ Kafka Topic: │ │ notifications-dlq│ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ DLQ Handler │ │ (logs failures) │ └──────────────────┘ KEDA watches consumer-group lag on `notifications` topic and scales the Consumer deployment between 0 and 3 replicas (matching the topic's partition count). ``` **上述架构中包含的安全层：** | 层级 | 工具 | 时机 | |---|---|---| | 左移 (IaC 和镜像扫描) | Trivy, Checkov | 部署前 (CI) | | 准入控制 | Kyverno | 部署时 | | 运行时安全 | Falco | 部署后 (持续) | | 备份 / 灾难恢复 (DR) | Velero | 定期 + 按需 | ## 技术栈 | 类别 | 技术 | |---|---| | 消息传递 | Apache Kafka (Strimzi operator, KRaft 模式) | | 自动扩缩容 | KEDA (基于 Kafka 滞后，缩容至零) | | 基础设施 | Terraform (Azure: AKS, ACR, Key Vault, VNet, Storage) | | 容器编排 | Azure Kubernetes Service (AKS) | | 应用打包 | Helm | | GitOps | ArgoCD (App of Apps 模式) | | CI | GitHub Actions | | 镜像/IaC 扫描 | Trivy, Checkov | | 准入策略 | Kyverno | | 运行时安全 | Falco | | 备份与灾难恢复 | Velero | | 密钥 | Azure Key Vault + Workload Identity (CSI driver) | | 应用 | Go (producer, consumer, dlq-handler) | ## 前置条件 - Azure 订阅 - `az` CLI (已登录) - `kubectl` - `helm` - `terraform` (>= 1.9.0) - `docker` - `velero` CLI (用于备份/恢复操作) ## 快速开始 ### 1. 配置基础设施 ``` cd terraform/environments/dev terraform init terraform apply az aks get-credentials --resource-group rg-kafka-lab-dev --name aks-kafka-lab-dev ``` ### 2. 安装集群级工具 (一次性引导) ``` # Strimzi operator kubectl apply -f kafka/namespace.yaml kubectl create -f 'https://strimzi.io/install/latest?namespace=kafka' -n kafka # KEDA helm repo add kedacore https://kedacore.github.io/charts helm install keda kedacore/keda --namespace keda --create-namespace # Kyverno helm repo add kyverno https://kyverno.github.io/kyverno/ helm install kyverno kyverno/kyverno --namespace kyverno --create-namespace kubectl apply -f security/kyverno/policies/ # Falco helm repo add falcosecurity https://falcosecurity.github.io/charts helm install falco falcosecurity/falco --namespace falco --create-namespace \ --set tty=true --set-file customRules."custom-rules\.yaml"=security/falco/custom-rules.yaml # ArgoCD kubectl create namespace argocd kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml --server-side ``` ### 3. 部署 Kafka 集群和 topic ``` kubectl apply -f kafka/strimzi/kafka-cluster.yaml kubectl apply -f kafka/strimzi/kafka-topics.yaml ``` ### 4. 引导 GitOps (ArgoCD 将从这里接管) ``` kubectl apply -f app-of-apps.yaml ``` 随后 ArgoCD 将自动从 Git 同步 `notification-system`、`velero`、`velero-schedule` 和 Kyverno 策略。 ### 5. 端到端验证 ``` kubectl port-forward -n notifications svc/producer 8080:8080 & curl -X POST localhost:8080/notify \ -H "Content-Type: application/json" \ -d '{"email":"you@example.com","message":"hello from kafka-and-security-lab"}' ``` 检查 consumer 日志以确认投递： ``` kubectl logs -n notifications deployment/consumer --tail=20 ``` ### 6. 销毁环境 (成本控制) ``` cd terraform/environments/dev terraform destroy ``` 有关完整的脚本化拆解版本，请参见 `scripts/destroy.sh`。 ## 仓库结构 ``` . ├── apps/ # Go source for producer, consumer, dlq-handler ├── argocd/ # ArgoCD Application manifests (App of Apps) ├── docs/ # Architecture, DLQ pattern, security decisions ├── helm/notification-system/ # Helm chart for the three apps ├── kafka/strimzi/ # Kafka cluster + topic CRDs ├── security/ │ ├── falco/ # Custom Falco rules │ └── kyverno/policies/ # Admission control policies ├── terraform/ # Infrastructure as Code (Azure) ├── velero/ # Backup schedule + restore runbook ├── scripts/ # deploy.sh / destroy.sh helpers └── .github/workflows/ # CI pipelines ``` ## 延伸阅读 - [`docs/architecture.md`](docs/architecture.md) — 详细的架构解析 - [`docs/dlq-pattern.md`](docs/dlq-pattern.md) — 死信队列设计与重试逻辑 - [`docs/security-decisions.md`](docs/security-decisions.md) — Checkov 扫描结果：已修复与已接受风险及其原因说明 - [`velero/restore-runbook.md`](velero/restore-runbook.md) — 灾难恢复操作手册 ## 许可证 参见 [LICENSE](LICENSE)。

标签：子域名字典, 子域名突变, 日志审计, 软件成分分析