Kailun Qin, Ant Group Putting an Invisible Shield on Kubernetes Secrets Agenda • K8s Secrets: Overview • TEE-based K8s Secrets Protection: Solution • Production Experience @ Ant Group • Demo • Summary security [1] KubeCon China 2018: Node Operator: Kubernetes Node Management Made Simple - Joe Chen, Ant Financial TEE-based Secrets Protection: Solution Confidential Computing A Trusted Execution Environment Secrets • Introducing mutual (remote / local) attestations between entities Production Experience @ Ant Group KMS Plugin • Workflow • Encryption • Decryption • Engineering decisions • apiserver is responsible

Node Operator: Kubernetes Node Management Made Simple 陈俊(Joe), Ant Financial Agenda • Background and Motivation • Introduction of Operators • Node-Operator • Advanced Topic:

constraints - included in published deck for reference 23 Configuring VM affinity rules Quorum dictates design VM Fault Domain A VM VM VM VM VM VM VM VM K8S Prod VM K8S Prod V K8S Prod VM K8S Prod failed host are restarted on alternate hosts. When running on hardware that supports health reporting, Pro-active failure avoidance can also be engaged. Example loss of a system cooling fan, degraded storage

deployed across Zones with zone-local Storage #67703 24 Configuring VM affinity rules Quorum dictates design VM Fault Domain A VM VM VM VM VM VM VM VM K8S Prod VM K8S Prod V K8S Prod VM K8S Prod failed host are restarted on alternate hosts. When running on hardware that supports health reporting, Pro-active failure avoidance can also be engaged. Example loss of a system cooling fan, degraded storage

A lauda Cloud Enterprise (A CE) Un i f i ed A P Is M u l ti -Cluster M a na gement M u l ti -Ten ant M a nagement K u bernetes In tegration API S e r ver C o r e /C ustom R e s ourc es API S e

SmartEdge on the modern battlefield, “At the tactical edge, time is a weapon. With edge computing and pro- cessing at the point of data collection, we will give warfighters access to real-time, data-driven biometrics that are tracking the state of an individual’s health, all these sensors capture and pro- cess data locally, supplying immediate in- sight which results in faster decision-mak- ing. Pisano

Capabilities/Advantages to Build DevOps Solution • Architecture and Features • CRD and operator design • Pipeline / Stage/ Task / Task Template / Version Control • Logging, monitoring, autoscaling Capabilities and Advantages to Build DevOps Solution • Architecture and Features • CRD and operator design • Pipeline/Stage/Task/Task Template/Version Control/UI generation/Volume... • Logging, monitoring alert and related actions docker registry Kubernetes Cluster Kubernetes Cluster CRD and Operator Design BuildJob DevOps Operator Job Job Job Job BuildJob BuildJob BuildJob MySQL MySQL MySQL DevOps

与开发⼈员和⽤户可⽤的API相同。⽤户可以使⽤ their own API 编写⾃⼰ 的控制器，例如 scheduler ，这些API可由通⽤ command-line tool 定位。 这种 design 使得许多其他系统可以构建在Kubernetes上。 Kubernetes不是什么？ Kubernetes不是⼀个传统的，全⾯的PaaS系统。 它保留了⽤户的重要选择。 Kubernetes： 使⽤v1.2规范。OpenAPI规范在Kubernetes 1.5中，进⼊Beta 阶段。 Kubernetes为主要⽤于集群内通信的API实现了另⼀种基于Protobuf的序列化格式，在 design proposal 有记录，每个 schema的IDL⽂件都存放在定义该API对象的Go语⾔包中。 API版本 为了更容易地消除字段或重组资源表示，Kubernetes⽀持多种API版本，每种API版本都有不同的API路径，例 。按照惯例，Kubernetes资源的名称最多可达253个字符，由⼩写字⺟、数字、 - 和 . 组 成，但某些资源有更具体的限制。有关名称的精确语法规则，详⻅： identifiers design doc 。 UID UID由Kubernetes⽣成。 在Kubernetes集群的整个⽣命周期中创建的每个对象都有不同的UID（即：它们在空间和时间 上是唯⼀的）。 07-Name

of Kubernetes (Controller, codegen etc) • Write your own Controller • gPRC based interface design in Kubernetes (CRI as example) • For Kubernetes users: • Effective pattern of programming based --token $token ${master_ip:port} • Done! But that’s only one part … • Kubernetes is also about design pattern in container world • decoupling containers • re-use images • well-designed architecture

Start with a Bang! Multiple Approaches Kubeadm Improve Project X Focus on Kubernetes only Design better abstractions Work with small number of teams at a time • Setting up the whole Go-Viet infrastructure