Kailun Qin, Ant Group Putting an Invisible Shield on Kubernetes Secrets Agenda • K8s Secrets: Overview • TEE-based K8s Secrets Protection: Solution • Production Experience @ Ant Group • Demo • Summary Introducing mutual (remote / local) attestations between entities Production Experience @ Ant Group KMS Plugin • Workflow • Encryption • Decryption • Engineering decisions • apiserver is responsible

本⽂概述了Kubernetes集群中所需的各种组件。 Master组件 Master组件提供K8s集群的控制⾯板。Master对集群进⾏全局决策（例如调度），以及检测和响应集群事件（例如：当 replication controller所设置的 replicas 不够时，启动⼀个新的Pod）。 Master可在集群中的任意节点上运⾏。然⽽，简单起⻅，设置脚本通常在同⼀个VM上启动所有Master组件，并且不会 个单独的进程，但为了降低复杂性，它们都被编译成独⽴的⼆进制⽂件并运⾏在⼀个进程中。 这些控制器包括： Node Controller：当节点挂掉时，负责响应。 Replication Controller：负责维护系统中每个replication controller对象具有正确数量的Pod。 Endpoints Controller：填充Endpoint对象（即：连接Service＆Pod）。 Service “core”（由于没有明确的组名称，通常称为“legacy”）组，它的REST路径是 /api/v1 。例如 apiVersion: v1 。 2. 命名组是REST路径 /apis/$GROUP_NAME/$VERSION ，并使⽤ apiVersion: $GROUP_NAME/$VERSION （例如 apiVersion: batch/v1 ）。 ⽀持的API组的完整列表可详⻅：Kubernetes API reference

influence pod placements 10 Why use Zones? Kubernetes will automatically spread the pods in replication controllers or services across zones - to reduce the impact of zone failures How it works: • Active discussions regarding Kubernetes enhancements going on now in Resource Management Working Group – please join in • See Issue #49964 14 Using a NUMA aware hypervisor to solve issues now VM control plane starts first, and Prodsystems before others 27 The VMware SIG Charter Link to join group: https://groups.google.com/forum/#!forum/kubernetes-sig-vmware Link to join Slack: https://kubernetes

influence pod placements 10 Why use Zones? Kubernetes will automatically spread the pods in replication controllers or services across zones - to reduce the impact of zone failures How it works: • Active discussions regarding Kubernetes enhancements going on now in Resource Management Working Group – please join in • See Issue #49964 14 Using a NUMA aware hypervisor to solve issues now VM

得被管理对象能够被精细的分组管理，同时实现了整 个集群的高可用性。 17 www.h3c.com Confidential 秘密 17 17 K8s基本概念和术语介绍（RC） RC Replication Controller（副本管理器）和RS（Replica Set）： RC定义了一个期望的场景，即声明某种Pod的副本数量在任意时刻都符合某个预期值。RC包含如下几部分： Pod期待的副本数（replicas） 标pod实例的数量刚好等于此rc的期望值，如果有过多pod运行，系统就会停掉一些，否则会创建一些。通过rc， kubenetnes实现了用户应用集群的高可靠性，并大大减少了很多运维工作。 Replication Controller在k8s 1.2版本之后升级成了新的概念，Replica Set（下一代RC），Replicas Set支持基于集 合的标签选择器，而RC只支持基于等式的标签选择器。 "128Mi" 29 www.h3c.com Confidential 秘密 29 29 K8s基本概念和术语介绍 资源对象分类： 类别 名称 资源对象 Pod、 Replica Set、 Replication Controller、 Deployment、 StatefulSet、 Daemon Set、 Job、 CronJob、HorizontalPodAutoscaling 配置对象 Node、

Services Replication – нужен Zookeeper Можно использовать любой имеющийся ZK cluster Install Zookeeper: manifests/zookeeper/quick-start-persistent-volume/zookeeper-1-node-create.sh Replication apiVersion:

Container Label: app=backend IP: 172.17.10.2 Port: 80 Label: app=MyApp Container Container Replication Controller Label: app=MyApp Replicas: 2 Service <10.0.0.11>:<9376> Label: app=MyApp Endpoints:

Cluster ReplicaSet.yaml ContainerImage1 Replicas: 3 ContainerImage2 Replicas: 2 复制控制器Replication Controller • 自动恢复 • 手动扩展 • 滚动更新 • 多版本追踪 Worker (Container Host) P1R1 Worker (Container

CustomResourceDefinition metadata: name: ingressroutes.traefik.containo.us spec: scope: Namespaced group: traefik.containo.us version: v1alpha1 names: kind: IngressRoute plural: ingressroutes ion metadata: name: ingressroutetcps.traefik.containo.us spec: scope: Namespaced group: traefik.containo.us version: v1alpha1 names: kind: IngressRouteTCP plural: ingressroutetcps CustomResourceDefinition metadata: name: middlewares.traefik.containo.us spec: scope: Namespaced group: traefik.containo.us version: v1alpha1 names: kind: Middleware plural: middlewares

Pod Node Pod Unified logging、monitoring、alert with PaaS Consistent data Node group of build nodes Node group of user applications Scheduling customization Cluster Resource Auto Scaling kubelet task can push metric to gateway if needed • Cluster autoscaler will add/remove node from build group for scaling • HA is guaranteed by cluster HA, k8s Job controller and cluster autoscaler, can also