"annotations": { "key1" : "value1", "key2" : "value2" } 类似以下信息可记录到Annotation中： 由declarative configuration layer管理的字段。将这些字段附加为Annotation，可将它们与客户端或服务器设置的默 认值、⾃动⽣成的字段或以及auto-sizing或auto-scaling的系统所设置的字段区分开。 原⽂ https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/ 18-Daemon Set 78 Configuration Best Practices 本⽂档强调并整合了整个⽤户指南、⼊⻔⽂档和示例中引⼊的配置最佳实践。 这是⼀个“活”的⽂件。如果你想到的东⻄不在这个名单上，但可能对他⼈有⽤，请不要犹豫，提交issue或提交PR（pull 快速创建和暴露单个容器部署。有关示例，请参阅quick start guide 。 19-配置最佳实践 80 原⽂ https://kubernetes.io/docs/concepts/configuration/overview/ 19-配置最佳实践 81 Managing Compute Resources for Containers（管理容 器的计算资源） 译者按：本节中，笔者将r

隨便Google就可找到好幾卡車的Kubernetes安全最佳實務/指南.... 6 ©2019 VMware, Inc. Kubernetes安全最佳實務 Kubernetes Security Best Practices ©2019 VMware, Inc. 7  關閉公開存取 (Disable public access)  實施角色型存取權控管 (Implement (Keep your Kubernetes version up to date) Kubernetes Security Best Practices Kubernetes安全性的最佳實務指導 資料來源: https://blog.sqreen.com/kubernetes-security-best-practices/ ©2019 VMware, Inc. 8 NIST在容器安全指南中揭露了五種容器應用最應關注的風險 控制平面設置 (Control Plane Configuration) 4. 工作節點 (Worker Node) 5. 政策 (Policies) ©2019 VMware, Inc. 10 Use Cases: Security Architecture Guidance / Replacement for Checklist / Security Training OWASP CSVS

tokens • ssh keys etc. • Stored in etcd • distributed Key-Value data store • How about their security? • Default K8s setup • etcd contents not encrypted (only base64 encoded) • > K8s 1.7+ • at-rest scheme • DEK & KEK Motivation: K8s Secrets Protection • Performance & latency • Network • Security • DEK in the clear in memory • Secret in the clear in memory • kubeconfig in the clear in memory User access management => raw and extensive! ü Secrets management => crucial! • Financial-grade security [1] KubeCon China 2018: Node Operator: Kubernetes Node Management Made Simple - Joe Chen, Ant Financial

the way down Turtles all the way down Alex Tcherniakhovski Security Engineer, Google Cloud Maya Kaczorowski Security PM, Google Cloud @MayaKaczorowski Protecting secrets What’s a secret combination of a safe. If a safe combination is known to an adversary, the strongest safe provides no security against penetration. Similarly, poor key management may easily compromise strong algorithms.” {SECRET}DEK + {DEK}KEK Envelope Source for crypto notation: https://en.wikipedia.org/wiki/Security_protocol_notation KMS 1.10 Envelope Encryption Sequence Master kube-apiserver etcd kms-plugin

clean up finished build tasks after if finishes CronJob Spec schedule Cron style scheduler configuration concurrencyPolicy Job Template Concurrency policy of CronJob suspend Whether suspend latter Operator DevOps Service DevOps Manager CronJob k8s API MySQL k8s API MySQL MySQL • Pipeline configuration and history in MySQL • Logging in central logging service - ElasticSearch • Metric data in Update jobs status to buildjob Submit buildjob List/Watch buildjob Pod Pod Pod Pod Build task configuration - map to k8s Job, can also be a raw k8s job Job / Pod / Node info BuildJob / Job status Pipeline

block of the Custom Resource Configuration of the workload • Operator provides configuration via the spec section of the Custom Resource • Operator reconciles configuration and updates to it with the Operand • Operator is able to restore a backup of an Operand • Operator orchestrates complex re- configuration flows on the Operand • Operator implements fail-over and fail-back of clustered Operands • 备份策略、备份方式、恢复方式、备份管理等等。 Disaster Recovery & High Availability Failover/Switchover、多可用区、数据恢复等等。 Security & Compliance 访问控制、审计、安全链接、加密存储等等。 Patching & Upgrades 小版本升级、大版本升级、安全漏洞修复等等。 Data Migrations

using kubernetes native approach. • Adapt to various internal systems like Route System, CMDB, CI, Security Platform, etc. • Declarative application lifecycle management. • Support big data and AI jobs without restarting container Ø High-performance Ø Safe autoscaling decisions Ø Personalized configuration of VWA objects Ø Cooperate with HPA through events Vertical Workload AutoScaler (VWA) Recommender Autoscaler ) Ø Deploy HPAPlus-Controller independently. Ø High Performance. Ø Personalized configuration of HPA objects. Ø Calculate replicas based on pod resource request or limit. Ø Cooperate with

Services (w/API) • Node = Container Host w/agent called “Kubelet” • Application Deployment File = Configuration File of desired state • Container Image = Runs in a Pod (~1:1) • Replicas = QTY of Pods that compatible with GKE Built for Day 2 Operations PKS simplifies Day 2 operations with built-in network security—powered by NSX, high availability, logging, monitoring, analytics, and automated health checks Structured Data Metrics Alerts Events VMware vRealize Operations Capacity, Performance and Configuration Management Events Launch in Context Unstructured Data Logs Messages VMware vRealize Log

集成第三方插件 2. Feature parity with kubectl 功能与kubectl保持一致 3. Multi-cluster management 多集群管理 4. Improved security 提高安全性 Top requested changes 1. Third-party plugins or integrations 集成第三方插件 Which third-party 有多重要？ https://github.com/kubernetes/dashboard/issues /3256#issuecomment-437199403 4. Improved security “During the week of June 1st, 2018, [researchers] discovered more than 21,000 publicly facing Kubernetes represented more than 78% of all open IP's.” → Lacework: Container Security Research 4. Improved security bit.ly/securing-dashboard Securely running Dashboard is possible! “We operate