metal-stack/firewall-controller
GitHub: metal-stack/firewall-controller
一个运行在裸机防火墙上的 Kubernetes 控制器，负责将集群网络策略协调为 nftables 规则并集成 Suricata 入侵检测。
Stars: 61 | Forks: 5
# firewall-controller
该控制器安装在位于多个 Kubernetes worker 节点前方的裸机防火墙上，负责将 `ClusterwideNetworkPolicy` 协调为 nftables 规则，以控制进出 Kubernetes 集群的访问。
它还允许控制通过的流量速率，以在受限使用场景下限制网络资源。Nftable 和节点指标通过 `nftables-exporter` 和 `node-exporter` 公开，这些 IP 在 Kubernetes 集群中作为 Service 和 Endpoint 可见。
此外，防火墙上还管理着一个 IDS，用于检测已知的网络异常。为此使用了 [suricata](https://suricata.io/)。目前，仅报告有关已扫描数据包数量的基本统计数据。在未来的版本中，将提供对所有警报的访问。
该控制器通常通过 [firewall-controller-manager](https://github.com/metal-stack/firewall-controller-manager) (FCM) 进行设置，FCM 负责在我们的 [Gardener 集成](https://docs.metal-stack.io/stable/overview/kubernetes/) 中管理 metal-stack 防火墙的生命周期。
## 架构
firewall-controller 作用于通常运行在您的集群和提供商管理的集群（在 Gardener 术语中称为 "shoot" 和 "seed"）中的 3 个 CRD：
| CRD | API | 所在位置 | 用途 |
| -------------------------------- | ---------------------------- | ---------- | ------------------------------------------------------------------- |
| `ClusterwideNetworkPolicy` | `metal-stack.io/v1` | Shoot | 控制防火墙规则，可由用户提供 |
| `Firewall` (由 FCM 定义) | `firewall.metal-stack.io/v2` | Seed | 定义防火墙，包括速率限制、控制器版本等 ... |
| `FirewallMonitor` (由 FCM 定义) | `firewall.metal-stack.io/v2` | Shoot | 用作用户查看防火墙状态的概览 |
请注意，`clusterwidenetworkpolicy` 资源具有命名空间，必须位于 `firewall` 命名空间中，否则此控制器不会对其进行协调。
示例 `ClusterwideNetworkPolicy`：
```
apiVersion: metal-stack.io/v1
kind: ClusterwideNetworkPolicy
metadata:
namespace: firewall
name: clusterwidenetworkpolicy-sample
spec:
egress:
- to:
- cidr: 1.1.0.0/24
except:
- 1.1.1.0/16
- cidr: 8.8.8.8/32
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
- protocol: TCP
port: 8080
# Optional, if specified this is the way to specify a port range from port to endPort
endPort: 8088
```
## 自动生成的 Ingress 规则
对于集群中每一个类型为 `LoadBalancer` 的 `Service`，相应的 Ingress 规则将被自动生成。
如果未指定 `loadBalancerSourceRanges`，则允许来自任何源 IP 地址的传入流量访问该服务。
### DNS 策略配置
`ClusterwideNetworkPolicy` 资源还允许您定义基于 DNS 的 Egress 策略。它们允许您根据 DNS 名称或通过将名称与提供的模式匹配来过滤 Egress 流量。
要按特定域名过滤，您需要提供 `matchName` 字段：
```
apiVersion: metal-stack.io/v1
kind: ClusterwideNetworkPolicy
metadata:
namespace: firewall
name: clusterwidenetworkpolicy-fqdn
spec:
egress:
- toFQDNs:
- matchName: example.com
ports:
- protocol: UDP
port: 80
- protocol: TCP
port: 80
```
如果您想过滤匹配特定模式的 FQDN，可以使用 `matchPattern` 字段，该字段支持 `*` 通配符。以下示例允许流量访问 `.example` 顶级域名中所有资源的 80 端口：
```
apiVersion: metal-stack.io/v1
kind: ClusterwideNetworkPolicy
metadata:
namespace: firewall
name: clusterwidenetworkpolicy-fqdn-pattern
spec:
egress:
- toFQDNs:
- matchPattern: *.example
ports:
- protocol: UDP
port: 80
- protocol: TCP
port: 80
```
默认情况下，DNS 信息从 Google DNS（地址为 8.8.8.8:53）收集。首选 DNS 服务器可以通过 FCM 的 `Firewall` 资源进行更改，该资源由提供商管理。
## 状态
一旦 firewall-controller 运行，它将向 `FirewallMonitor` CRD 状态报告多项统计数据。可以通过运行以下命令进行检查：
```
kubectl get -n firewall fwmon
NAME MACHINE ID IMAGE SIZE LAST EVENT AGE
shoot--prod--seed-firewall-089f9 f4f8b200-deef-11e9-8000-3cecef22f910 firewall-ubuntu-2.0.20221025 n1-medium-x86 Phoned Home 18m
```
当使用 `-o yaml` 显示资源时，它包含有关流量计数、数据包丢弃和 IDS 的详细信息：
```
Status:
Last Run: 2020-06-17T13:18:58Z
Stats:
# Network traffic in bytes separated into external and internal in/out/total
Devices:
External:
In: 91696
Out: 34600
Total: 0
Internal:
In: 0
Out: 0
Total: 2678671
# IDS Statistics by interface
Idsstats:
vrf104009:
Drop: 1992
Invalidchecksums: 0
Packets: 4997276
# nftable rule statistics by rule name
Rules:
Accept:
BGP unnumbered:
Counter:
Bytes: 0
Packets: 0
SSH incoming connections:
Counter:
Bytes: 936
Packets: 16
accept established connections:
Counter:
Bytes: 21211168
Packets: 39785
accept icmp:
Counter:
Bytes: 0
Packets: 0
accept traffic for k8s service kube-system/vpn-shoot:
Counter:
Bytes: 360
Packets: 6
Drop:
drop invalid packets:
Counter:
Bytes: 52
Packets: 1
drop invalid packets from forwarding to prevent malicious activity:
Counter:
Bytes: 0
Packets: 0
drop invalid packets to prevent malicious activity:
Counter:
Bytes: 0
Packets: 0
drop packets with invalid ct state:
Counter:
Bytes: 0
Packets: 0
drop ping floods:
Counter:
Bytes: 0
Packets: 0
Other:
block bgp forward to machines:
Counter:
Bytes: 0
Packets: 0
count and log dropped packets:
Counter:
Bytes: 2528
Packets: 51
snat (networkid: internet):
Counter:
Bytes: 36960
Packets: 486
```
## Prometheus 集成
防火墙上运行着两个 Exporter，用于报告此机器的关键指标：
- node-exporter，用于报告机器特定指标，如 CPU、内存和磁盘使用情况，详情请参见 [node-exporter](https://github.com/prometheus/node_exporter)。
- nftables-exporter，用于报告 nftables 指标，请参见 [nftables-exporter](https://github.com/Sheridan/nftables_exporter)
这两个 Exporter 都作为 Service 公开：
```
kubectl get svc -n firewall
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nftables-exporter ClusterIP None 9630/TCP 13h
node-exporter ClusterIP None 9100/TCP 13h
```
这些 Service 位于虚拟 Endpoint 之前：
```
kubectl get ep -n firewall
NAME ENDPOINTS AGE
nftables-exporter 10.3.164.1:9630 13h
node-exporter 10.3.164.1:9100 13h
```
您可以在 Prometheus 安装配置中抓取这些 Service 以获取指标。
要进行检查，您可以运行：
```
curl nftables-exporter.firewall.svc.cluster.local:9630/metrics
curl node-exporter.firewall.svc.cluster.local:9100/metrics
```
## 防火墙日志
也可以使用以下命令实时查看被丢弃的数据包（从 [stern](https://github.com/stern/stern) 安装 stern）：
```
stern -n firewall drop
```
输出将如下所示：
```
droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:27 +0000 UTC {"ACTION":"Drop","DPT":"4000","DST":"1.2.3.4","ID":"54321","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"vlan179","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"38464","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"236","URGP":"0","WINDOW":"65535","timestamp":"2020-06-17 13:23:27 +0000 UTC"}
droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:34 +0000 UTC {"ACTION":"Drop","DPT":"2362","DST":"1.2.3.4","ID":"44545","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"40194","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"242","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:34 +0000 UTC"}
droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:30 +0000 UTC {"ACTION":"Accept","DPT":"650","DST":"1.2.3.4","ID":"12399","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"vlan179","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"40194","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"241","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:30 +0000 UTC"}
droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:34 +0000 UTC {"ACTION":"Accept","DPT":"2362","DST":"1.2.3.4","ID":"44545","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"40194","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"242","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:34 +0000 UTC"}
droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:10 +0000 UTC {"ACTION":"Accept","DPT":"63351","DST":"1.2.3.4","ID":"11855","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"vlan179","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"54589","SRC":"2.3.4.5","SYN":"","TOS":"0x00","TTL":"245","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:10 +0000 UTC"}
droptailer-6d556bd988-4g8gp droptailer 2020-06-17 13:23:51 +0000 UTC {"ACTION":"Accept","DPT":"8002","DST":"1.2.3.4","ID":"17539","IN":"vrf104009","LEN":"40","MAC":"ca:41:f9:80:fa:89:aa:bb:0e:62:8c:a6:08:00","OUT":"","PREC":"0x00","PROTO":"TCP","RES":"0x00","SPT":"47615","SRC":"2.3.4.5","SYN":"","TOS":"0x08","TTL":"239","URGP":"0","WINDOW":"1024","timestamp":"2020-06-17 13:23:51 +0000 UTC"}
```
您可以将 droptailer 日志转发到您现有的任何日志聚合基础设施中。
如果在防火墙规约 (spec) 中启用，除了被丢弃的连接外，还可以记录已接受的连接。
标签：EVTX分析, EVTX分析, Gardener, Go语言, K8sOperator, Linux网络安全, MacOS取证, Metal-Stack, Metaprompt, NetworkPolicy, nftables, Prometheus监控, Suricata, 子域名突变, 安全控制, 控制器, 日志审计, 服务暴露, 流量控制, 现代安全运营, 程序破解, 网络指标, 网络策略, 网络速率限制, 自定义请求头, 裸金属防火墙, 边界防火墙