Kailun Qin, Ant Group Putting an Invisible Shield on Kubernetes Secrets Agenda • K8s Secrets: Overview • TEE-based K8s Secrets Protection: Solution • Production Experience @ Ant Group • Demo • Summary tokens • ssh keys etc. • Stored in etcd • distributed Key-Value data store • How about their security? • Default K8s setup • etcd contents not encrypted (only base64 encoded) • > K8s 1.7+ • at-rest scheme • DEK & KEK Motivation: K8s Secrets Protection • Performance & latency • Network • Security • DEK in the clear in memory • Secret in the clear in memory • kubeconfig in the clear in memory

“core”（由于没有明确的组名称，通常称为“legacy”）组，它的REST路径是 /api/v1 。例如 apiVersion: v1 。 2. 命名组是REST路径 /apis/$GROUP_NAME/$VERSION ，并使⽤ apiVersion: $GROUP_NAME/$VERSION （例如 apiVersion: batch/v1 ）。 ⽀持的API组的完整列表可详⻅：Kubernetes API reference Deployment描述，如果应⽤对该spec的更改，则Deployment Controller将以可控的速率，来将实际状态更改为期 望状态。（Deployment对象当前是 extensions API Group 的⼀部分。） 您可以操作Label进⾏调试。由于Kubernetes Replication Controller和Service使⽤Label来匹配Pod，因此可通过删 除相关Label来 describe nodes 命令检查Node的容量和数量。 例如： $ kubectl describe nodes e2e-test-minion-group-4lw4 Name: e2e-test-minion-group-4lw4 [ ... lines removed for clarity ...] Capacity: alpha.kubernetes.io/nvidia-gpu:

Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential 责任共担模型 Security in the Cloud Security of the Cloud © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Inspector AWS KMS AWS Secrets Manager AWS WAF AWS IAM Amazon GuardDuty Amazon Macie AWS Security Hub AWS CloudHSM AWS Certificate Manager AWS CloudTrail host container dependencies code Amazon Confidential Worker Worker Master Worker Worker Master Auto Scaling group AZ1 Region AZ2 Auto Scaling group CloudWatch Logs Elasticsearc h Kiban a Fluentd DaemonSet Kubectl logs

Pod Node Pod Unified logging、monitoring、alert with PaaS Consistent data Node group of build nodes Node group of user applications Scheduling customization Cluster Resource Auto Scaling kubelet task can push metric to gateway if needed • Cluster autoscaler will add/remove node from build group for scaling • HA is guaranteed by cluster HA, k8s Job controller and cluster autoscaler, can also get unexpected failure rate - Call Spinnaker API to start deployment pipeline serviceaccount Security policy for the build task deployment strategy Invoke Spinnaker web hook of specified deployment

Management - Lifecycle Management Infrastructure Services (Networking, Storage, DNS, Load Balancer, Security) master master api api © 2017 Rancher Labs, Inc. Kubernetes 1.7的扩展特性 • API aggregation(beta) project ‘your-domain’ would be like your private tenant name. • Then initialize your own resource group, version and kind. • Your API server could be build and run now • Build as an image and run in a

Gaps for spark Ø Dynamic Resource Allocation Ø Spark external shuffle service Ø Performance Ø Security p Kerberos support Ø … Gaps for Spark Ø Resource Management: p Queue p Hierarchical queue pod executor pod executor pod apiVersion：v1 kind: Pod metadata: annotations: scheduling.k8s.io/group-name: job-1574739729783- podgroup volcano.sh/task-spec: spark-driver createTimestamp: “2019-11-27T09:33:19Z”

隨便Google就可找到好幾卡車的Kubernetes安全最佳實務/指南.... 6 ©2019 VMware, Inc. Kubernetes安全最佳實務 Kubernetes Security Best Practices ©2019 VMware, Inc. 7  關閉公開存取 (Disable public access)  實施角色型存取權控管 (Implement (Keep your Kubernetes version up to date) Kubernetes Security Best Practices Kubernetes安全性的最佳實務指導 資料來源: https://blog.sqreen.com/kubernetes-security-best-practices/ ©2019 VMware, Inc. 8 NIST在容器安全指南中揭露了五種容器應用最應關注的風險 (Worker Node) 5. 政策 (Policies) ©2019 VMware, Inc. 10 Use Cases: Security Architecture Guidance / Replacement for Checklist / Security Training OWASP CSVS – 對Docker容器應用開發/調度平台的控制措施 組織面 基礎架構 容器 調度管理

the way down Turtles all the way down Alex Tcherniakhovski Security Engineer, Google Cloud Maya Kaczorowski Security PM, Google Cloud @MayaKaczorowski Protecting secrets What’s a secret combination of a safe. If a safe combination is known to an adversary, the strongest safe provides no security against penetration. Similarly, poor key management may easily compromise strong algorithms.” {SECRET}DEK + {DEK}KEK Envelope Source for crypto notation: https://en.wikipedia.org/wiki/Security_protocol_notation KMS 1.10 Envelope Encryption Sequence Master kube-apiserver etcd kms-plugin

集成第三方插件 2. Feature parity with kubectl 功能与kubectl保持一致 3. Multi-cluster management 多集群管理 4. Improved security 提高安全性 Top requested changes 1. Third-party plugins or integrations 集成第三方插件 Which third-party 有多重要？ https://github.com/kubernetes/dashboard/issues /3256#issuecomment-437199403 4. Improved security “During the week of June 1st, 2018, [researchers] discovered more than 21,000 publicly facing Kubernetes represented more than 78% of all open IP's.” → Lacework: Container Security Research 4. Improved security bit.ly/securing-dashboard Securely running Dashboard is possible! “We operate

KubeVirt RancherVM Kata Container Focus : deploy REAL vm (traditional vm app) Focus : container security Virtlet Virtlet is a Kubernetes runtime server which allows you to run VM workloads, based on scale. RancherVM Architecture RancherVM Networking Container Security gVisor NFV? Kata Container The speed of containers, the security of VMs https://github.com/kata-containers Kata Container Architecture