Kailun Qin, Ant Group Putting an Invisible Shield on Kubernetes Secrets Agenda • K8s Secrets: Overview • TEE-based K8s Secrets Protection: Solution • Production Experience @ Ant Group • Demo • Summary Introducing mutual (remote / local) attestations between entities Production Experience @ Ant Group KMS Plugin • Workflow • Encryption • Decryption • Engineering decisions • apiserver is responsible

Balancer, Security) master master api api © 2017 Rancher Labs, Inc. Kubernetes 1.7的扩展特性 • API aggregation(beta) • CustomResourceDefinitions(beta) • Support for extensible admission controllers • Pluggable © 2017 Rancher Labs, Inc. API Aggregation • What API aggregation provides • Extended with additional APIs • Build your own API server • Requirements of aggregation layer • Running Kubernetes 1.7 your own resource group, version and kind. • Your API server could be build and run now • Build as an image and run in a cluster © 2017 Rancher Labs, Inc. API Server Aggregation Architecture ETCD

“core”（由于没有明确的组名称，通常称为“legacy”）组，它的REST路径是 /api/v1 。例如 apiVersion: v1 。 2. 命名组是REST路径 /apis/$GROUP_NAME/$VERSION ，并使⽤ apiVersion: $GROUP_NAME/$VERSION （例如 apiVersion: batch/v1 ）。 ⽀持的API组的完整列表可详⻅：Kubernetes API reference Deployment描述，如果应⽤对该spec的更改，则Deployment Controller将以可控的速率，来将实际状态更改为期 望状态。（Deployment对象当前是 extensions API Group 的⼀部分。） 您可以操作Label进⾏调试。由于Kubernetes Replication Controller和Service使⽤Label来匹配Pod，因此可通过删 除相关Label来 describe nodes 命令检查Node的容量和数量。 例如： $ kubectl describe nodes e2e-test-minion-group-4lw4 Name: e2e-test-minion-group-4lw4 [ ... lines removed for clarity ...] Capacity: alpha.kubernetes.io/nvidia-gpu:

Launch in Context Unstructured Data Logs Messages VMware vRealize Log Insight Log analytics, aggregation, and search Virtual Applications vRealize Ops, Log Insight For Comprehensive Visibility 32

Alauda-Jenkins-Plugin/DSL • 流水线模版 • 图形化模块 • 用户打通 • 权限同步 • Jenkins/Pipeline CRDs/Custom Controllers/API Aggregation 数字化转型引领者 助力企业获得持续创新的核心能力

CustomResourceDefinition metadata: name: ingressroutes.traefik.containo.us spec: scope: Namespaced group: traefik.containo.us version: v1alpha1 names: kind: IngressRoute plural: ingressroutes ion metadata: name: ingressroutetcps.traefik.containo.us spec: scope: Namespaced group: traefik.containo.us version: v1alpha1 names: kind: IngressRouteTCP plural: ingressroutetcps CustomResourceDefinition metadata: name: middlewares.traefik.containo.us spec: scope: Namespaced group: traefik.containo.us version: v1alpha1 names: kind: Middleware plural: middlewares

Pod Node Pod Unified logging、monitoring、alert with PaaS Consistent data Node group of build nodes Node group of user applications Scheduling customization Cluster Resource Auto Scaling kubelet task can push metric to gateway if needed • Cluster autoscaler will add/remove node from build group for scaling • HA is guaranteed by cluster HA, k8s Job controller and cluster autoscaler, can also

User User group Namespace Deployment Registry 
 project CI/CD workspace Pod … resources CPU quota MEM quota Storage quota Device (GPU) quota …. quota Service Config group … k8s

Clients kubernetes- csi Orgs/Repos Orgs/Repos Orgs/Repos SIGs/WGs Special Interest Group Working Group https://github.com/kubernetes/community/blob/master/sig-list.md SIGs/WGs Communication

Active discussions regarding Kubernetes enhancements going on now in Resource Management Working Group – please join in • See Issue #49964 14 Using a NUMA aware hypervisor to solve issues now VM control plane starts first, and Prodsystems before others 27 The VMware SIG Charter Link to join group: https://groups.google.com/forum/#!forum/kubernetes-sig-vmware Link to join Slack: https://kubernetes