tokens • ssh keys etc. • Stored in etcd • distributed Key-Value data store • How about their security? • Default K8s setup • etcd contents not encrypted (only base64 encoded) • > K8s 1.7+ • at-rest Background: K8s Secrets • Encryption Keys stored on API Server • Secrets encrypted prior to storage in etcd • Secrets decrypted on API Server prior to use • Encryption keys stored in a remote KMS scheme • DEK & KEK Motivation: K8s Secrets Protection • Performance & latency • Network • Security • DEK in the clear in memory • Secret in the clear in memory • kubeconfig in the clear in memory

隨便Google就可找到好幾卡車的Kubernetes安全最佳實務/指南.... 6 ©2019 VMware, Inc. Kubernetes安全最佳實務 Kubernetes Security Best Practices ©2019 VMware, Inc. 7  關閉公開存取 (Disable public access)  實施角色型存取權控管 (Implement (Keep your Kubernetes version up to date) Kubernetes Security Best Practices Kubernetes安全性的最佳實務指導 資料來源: https://blog.sqreen.com/kubernetes-security-best-practices/ ©2019 VMware, Inc. 8 NIST在容器安全指南中揭露了五種容器應用最應關注的風險 (Worker Node) 5. 政策 (Policies) ©2019 VMware, Inc. 10 Use Cases: Security Architecture Guidance / Replacement for Checklist / Security Training OWASP CSVS – 對Docker容器應用開發/調度平台的控制措施 組織面 基礎架構 容器 調度管理

Master通常会部署在⼀个独⽴的服务器或虚拟机上，它是整个集群的⾸脑，如果Master宕机或不可⽤，那么我们所有的 控制命令都将会失效。 Master节点上运⾏着如下的关键进程： API Server：K8s⾥所有资源增删改查等操作的对外⼊⼝，也是集群控制的⼊⼝进程，它提供了HTTP RESTful API 接⼝给客户端以及其他组件调⽤。 Controller Manager：Controller 有Pod对象从apiserver中删除，并释放其名称。 Kubernetes 1.8引⼊了⼀个⾃动创建代表condition的 taints 功能（⽬前处于Alpha状态）。要启⽤此特性，请向API server、controller manager和scheduler传递标志 --feature-gates=...,TaintNodesByCondition=true 。⼀旦启⽤ TaintNodesByCondition 试向API server注册⾃⼰。这是⼤多数版本所使⽤ 的⾸选模式。 13-Node 38 对于⾃注册，kubelet会使⽤如下的选项启动： --kubeconfig ：凭证向apiserver进⾏身份验证的路径。 --cloud-provider ：如何与云提供商进⾏会话，从⽽获取⾃身的元数据。 --register-node ：⾃动向API server注册。 -

Kubernetes Architecture Operating Kubernetes ● Installation ● Upgrade ● Healing ● Scaling ● Security ● Monitoring ● ... Installation - SSH - Install kubelet - $pkgmanager install kubelet - Install kubelet Installation - master - SSH - Install scheduler - Install controller manager - Install API server - Config them correctly - Start them Installation - etcd - SSH - Install etcd - Config them correctly etcd Kubelet Bootkube API Server Scheduler Controller Manager etcd Kubelet Bootkube API Server Scheduler Controller Manager etcd Kubelet Bootkube API Server Scheduler Controller Manager

the way down Turtles all the way down Alex Tcherniakhovski Security Engineer, Google Cloud Maya Kaczorowski Security PM, Google Cloud @MayaKaczorowski Protecting secrets What’s a secret combination of a safe. If a safe combination is known to an adversary, the strongest safe provides no security against penetration. Similarly, poor key management may easily compromise strong algorithms.” {SECRET}DEK + {DEK}KEK Envelope Source for crypto notation: https://en.wikipedia.org/wiki/Security_protocol_notation KMS 1.10 Envelope Encryption Sequence Master kube-apiserver etcd kms-plugin

Container Focus : deploy REAL vm (traditional vm app) Focus : container security Virtlet Virtlet is a Kubernetes runtime server which allows you to run VM workloads, based on QCOW2 images. https://github scale. RancherVM Architecture RancherVM Networking Container Security gVisor NFV? Kata Container The speed of containers, the security of VMs https://github.com/kata-containers Kata Container Architecture

的Cupertino，同时在亚利桑那州的Phoenix和 中国设立研发中心 • 核心团队曾创立Cloud.com，并推出了 CloudStack，历经从VM到容器的完整技术演 进过程 • Rancher Server和Agent镜像在Docker Hub上 的下载次数已经超过4000万次，全球Rancher 的活动部署超过10,000个 GA March 2016 >20 million downloads Management - Lifecycle Management Infrastructure Services (Networking, Storage, DNS, Load Balancer, Security) master master api api © 2017 Rancher Labs, Inc. Kubernetes 1.7的扩展特性 • API aggregation(beta) extend managed resource into a current Kubernetes cluster • Auto-generated API in Kubernetes API server • Customized resource controller to implement your business logic of managed resource • Natural

Confidential ALB Ingress controller AWS Resources Kubernetes Cluster Node Node Kubernetes API Server ALB Ingress Controller Node HTTP Listener HTTPS Listener Rule: /cheeses Rule: /charcuterie Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential 责任共担模型 Security in the Cloud Security of the Cloud © 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved Inspector AWS KMS AWS Secrets Manager AWS WAF AWS IAM Amazon GuardDuty Amazon Macie AWS Security Hub AWS CloudHSM AWS Certificate Manager AWS CloudTrail host container dependencies code

备份策略、备份方式、恢复方式、备份管理等等。 Disaster Recovery & High Availability Failover/Switchover、多可用区、数据恢复等等。 Security & Compliance 访问控制、审计、安全链接、加密存储等等。 Patching & Upgrades 小版本升级、大版本升级、安全漏洞修复等等。 Data Migrations 全量 objects。list 可以简单理解为一个 HTTP GET 请求，watch 为一 个 HTTP/2 长连接 Cache 如何保持与 API Server 一致性 list & watch 机制中，list 获取 API Server 中数据的一份快照，并记 录 ResourceVersion 版本信息，watch 从 ResourceVersion 开始，获取后 续的增量数据。 watch watch 通过网络异步（asynchronous）获取增量数据，所以 cache 提供 的是最终一致性（eventual consistency）。 期间遇到网络、API Server 报错等异常时，会有重试机制 Controller-runtime 的 Informer 增加一段逻辑：如果上层 GET 某个 object 没有对应的 informer， controller-runtime 会马上为其增加

Zone ASK-Scheduler K8S API Server Metrics API CloudMonitor, Prometheus HPA Controller Cloud Controller Manager Serverless Scheduler ECI ASK-Scheduler K8S API Server • Nodeless：最佳弹性和扩展性，无传统调度器的复杂调度逻辑，最优的调度效率。 ASK-Scheduler K8S API Server • Pod（N） : Node（1） • 单集群支持1万Pod • 大量Pod场景中的Kube-proxy风暴问题 ECI ECI ECI ECI ECI Controllers - service discovery, ingress ECI ASK-Scheduler K8S API Server • 基于云产品控制器降低Kubernetes集群的复杂度 agent Container Container ECI ECI ECI ECI ECI ECI ECI ECI ECI ECI ECI ECI ECI ECI • Bulti-in Security • Light-Weight Virtualization • High Density • Scale and efficiency • Optimization Alinux2