Handle Edge Cloud Network with KubeBus Yulin Sun, yulin.sun@huawei.com Seattle Cloud Lab, Huawei R&D USA, Bellevue WA Agenda • Edge sample user scenarios • Edge network characteristics • Related work Sample Scenarios HiLens Campus surveillance Huawei Hilens Edge network characteristics • Edge Nodes running at private network • Connect to Cloud behind NAT gateway • Mightn’t have direct connection Cluster Management • There is cloud cluster, edge cluster, i.e. multiple nodes running in private network • Edge nodes, Edge cluster and cloud cluster needs acting as a single cluster Edge Node Management

Handle Edge Cloud Network with KubeBus Yulin Sun, yulin.sun@huawei.com Seattle Cloud Lab, Huawei R&D USA, Bellevue WA Agenda • Edge sample user scenarios • Edge network characteristics • Related work Sample Scenarios HiLens Campus surveillance Huawei Hilens Edge network characteristics • Edge Nodes running at private network • Connect to Cloud behind NAT gateway • Mightn’t have direct connection Cluster Management • There is cloud cluster, edge cluster, i.e. multiple nodes running in private network • Edge nodes, Edge cluster and cloud cluster needs acting as a single cluster Edge Node Management

Directory and LDAP Support 4 4 4 2 Pod and Network Security Policies 4 3 2 2 Configurable Adherence to CIS 4 3 2 2 Global RBAC Policies 4 2 3 2 2.4 Shared Tools and Services Once Users must use a browser-based workflow to perform authentication. 3.2.2 Pod and Network Security Policies • SUSE Rancher: 4 • OpenShift: 3 • Tanzu: 2 • Anthos: 2 3.2.2.1 SUSE Rancher downstream clusters. This ensures conformance and reduces the risk of human error when changing policies. PSPs can be created and edited through the UI. SUSE Rancher also ships with OPA Gatekeeper as

controller should only be used where Pod Security Policies cannot be used on the cluster, as it can interact poorly with certain Pod Security Policies Several system services (such as nginx-ingress ) Mitigation Make sure nodes with role:controlplane are on the same local network as your nodes with role:worker . Use network ACLs to restrict connections to the kubelet port (10250/tcp) on worker nodes that the Container Network Interface file permissions are set to 644 or more restrictive (Not Scored) Notes This is a manual check. Audit ( /var/lib/cni/networks/k8s-pod-network ) Note This may return

Configure Kernel Runtime Parameters Configure etcd user and group Ensure that all Namespaces have Network Policies defined Reference Hardened RKE cluster.yml configuration Reference Hardened RKE Template configuration Namespaces have Network Policies defined Running different applications on the same Kubernetes cluster creates a risk of one compromised application attacking a neighboring application. Network segmentation supposed to. A network policy is a specification of how selections of pods are allowed to communicate with each other and other network endpoints. Network Policies are namespace scoped. When a network policy

Configure Kernel Runtime Parameters Configure etcd user and group Ensure that all Namespaces have Network Policies defined Reference Hardened RKE cluster.yml configuration Reference Hardened RKE Template configuration Namespaces have Network Policies defined Running different applications on the same Kubernetes cluster creates a risk of one compromised application attacking a neighboring application. Network segmentation v2.4 5 network policy is a specification of how selections of pods are allowed to communicate with each other and other network endpoints. Network Policies are namespace scoped. When a network policy

categorization, see Appendix A on page 38. Title ID Risk Inability To Secure Control Plane Network Communications 004 High Lack of Security Related Documentation 016 High Lack of VirtualService Assessment Google / NCC Group Confidential Finding Details Finding Inability To Secure Control Plane Network Communications Risk High Impact: High, Exploitability: Medium Identifier NCC-GOIST2005-004 Category Istio should not have any plaintext endpoints exposed via its control plane and should enforce all network communications use mTLS (or at minimum, TLS) for communi- cations within the istio-system namespace

集群上监控的一个关键指标是每个 etcd 集群成员上的 etcd 网络对延迟 的 p99 百分比。使用 Prometheus 跟踪指标数据。 histogram_quantile（ （0.99, rate(etcd_network_peer_round_trip_time_seconds_bucket[2m]) 指标报 告 etcd 在成员间复制客户端请求的时间。确保它小于 50 ms。 其他 其他资 资源 源 $ data:text/plain;charset=US- ASCII,%23%20turn%20on%20Receive%20Flow%20Steering%20%28RFS%29%20for%20all %20network%20interfaces%0ASUBSYSTEM%3D%3D%22net%22%2C%20ACTION%3D% 3D%22add%22%2C%20RUN%7Bprogram%7D%2B%3D%22/bin/bash%20- 程规 规格示例 格示例 $ oc delete -f 05-worker-kernelarg-hpav.yaml network="net01"/> ... <

Workload Edge Operations Ingress Policies Egress Policies WAF / IDS Firewall User AuthN/Z Data Loss Prevention Certificate Authority K8s Network Policy K8s RBAC Audit Logging Image security Edge Security Cluster security Service Proxy Ingress 1. Define ingress security policies to control accesses to services. Deploy web application firewall to defend against DDoS, injection security Egress 2. Define egress security policies to defend against data exfiltration, botnet attacks. 3. Define firewall and virtual private network to lock down external access. Edge security

(Automated) 1.1.9 Ensure that the Container Network Interface file permissions are set to 644 or more restrictive (Manual) 1.1.10 Ensure that the Container Network Interface file ownership is set to root:root 6 Ensure that Service Account Tokens are only mounted where necessary (Manual) 5.2 Pod Security Policies 5.2.1 Minimize the admission of privileged containers (Manual) 5.2.2 Minimize the admission of host IPC namespace (Automated) 5.2.4 Minimize the admission of containers wishing to share the host network namespace (Automated) 5.2.5 Minimize the admission of containers with allowPrivilegeEscalation (Automated)