Node Operator: Kubernetes Node Management Made Simple 陈俊(Joe), Ant Financial Agenda • Background and Motivation • Introduction of Operators • Node-Operator • Advanced Topic: Teardown Cluster fast and convenient • Add & delete Node at any time • Upgrade Master & Node Components reliably • Canary Rollout • Master & Node Component Versions Management Motivation: Work Order Order Deployment Worker Order • Upgrade Nodes Versions • Upgrade Node 10.10.10.1 • Upgrade docker • Upgrade kubelet • Upgrade Node 10.10.10.2 • Upgrade docker • Upgrade kubelet …. Motivation: Work Order

Spark on Kubernetes Kubernetes extends beyond container orchestration, it has been expanded to support for data-intensive and stateful apps. Benefit: l Autoscaling in Cloud l Consolidate online service goal is to bring native support for Spark to use Kubernetes as a cluster manager like YARN, or Mesos. l Spark 2.3 added native support for Kubernetes. l Spark 2.4 added support for client mode, R, python python etc. l Spark 3.0 will add support for dynamic resource allocation, external shuffle service, Kerberos etc. How it works Spark on Kubernetes Spark-operator Gaps for spark Ø Dynamic Resource

of cluster nodes ？ Dynamic-Scheduler Node1 Node2 Kube-scheduler Pod Request Load Level Request Load Level Real Load Level Real Load Level Assigned to Node2 The native K8S scheduling is based on Dynamic-Scheduler Node1 Node2 Kube-scheduler Pod Request Load Level Request Load Level Real Load Level Real Load Level Assigned to Node1 Dynamic-scheduler Node1 has a lower load Dynamic-Scheduler Node1 Node2 Dynamic-scheduler- node-annotator 5m Load Prometheus 1h Load 1d Load 5m Load 1h Load 1d Load telegraf Record to node annotation telegraf Dynamic-Scheduler Predicate Node2 Node3 Node1 5m Load

management. • Support big data and AI jobs. • Optimize the isolation of resources, and improve resource utilization using hybrid deployment of online and offline services. • Support Service Mesh. Features: Ø Manual/Auto Batch Gray Release Ø Multi-Batch Rollback Ø Multi-Batch InPlaceUpdate Ø Support HPA, CronHPA, VWA (Vertical Workload Autoscaler) Ø Keep share memory during Pod upgrade Ø Scaled Up with LGV (Last Good Version) Ø Per Pod Per PV Ø Per Workload Per PV Ø Pod Auto Migrate when Node Abnormal Ø Gray Release for ConfigMap kube-apiserver StatefulSetPlus ListWatch PodConditionsChecker

continuously pull pods off the queue, evaluates the pod’s requirements, and assigns it to a worker node. 6 Kubenetes scheduling What does the scheduler do: As pod are created, they are place in a queue continuously pull pods off the queue, evaluates the pod’s requirements, and assigns it to a worker node. Placement Decision Stages: 1. Filter out impossible worker nodes a. Filters are called predicates continuously pull pods off the queue, evaluates the pod’s requirements, and assigns it to a worker node. Placement Decision Stages: 1. Filter out impossible worker nodes a. Filters are called predicates

Secrets management => crucial! • Financial-grade security [1] KubeCon China 2018: Node Operator: Kubernetes Node Management Made Simple - Joe Chen, Ant Financial TEE-based Secrets Protection: Solution receiving from malicious software entity (logic) TEE-based Kubelet • Address security threats • Node (kubelet) compromise • leak secrets on consumption TEE-based Secrets Protection TEE-based Secrets Decryption Workflow KMS Plugin (cont.) • Deployment Modes • One kms-plugin container per Master Node: sidecar to apiserver • Use Annotation to enable encrypted secret read / write • LivenessProbe for

10-Annotation 11-K8s架构及基本概念 12-Master与Node的通信 13-Node 14-Pod 15-Replica Set 16-Deployment 17-StatefulSet 18-Daemon Set 19-配置最佳实践 20-管理容器的计算资源 21-Kubernetes资源分配 22-将Pod分配到Node 23-容忍与污点 24-Secret 25-Pod优先级和抢占 主机规划 IP 作⽤ 172.20.0.87 ansible-client 172.20.0.88 master,node 172.20.0.89 master,node 172.20.0.90 node 172.20.0.91 node 172.20.0.92 node 准备⼯作 关闭selinux 所有机器都必须关闭selinux，执⾏如下命令即可。 ~]# setenforce /proc/sys/net/bridge/bridge-nf-call-iptables ~]# sysctl -w net.ipv4.ip_forward=1 如果关闭了防⽕墙，则只需执⾏最下⾯三⾏。 在node机器上 ~]# firewall-cmd --permanent --add-port=10250/tcp ~]# firewall-cmd --permanent --add-port=10255/tcp

Kubernetes Contributors opensource.google.com A strong community with corporate and independent support. Independent Google Red Hat Huawei ZTE Corp FathomDB IBM Microsoft HP others 在地高可靠度資料中心 for Alpha ● Simple CLI installation ● Online and Offline installation ● Private container registry support ● Latest 3 versions of k8s ● High-availability control plane ● Auto-repair Installation and Configuration GCP Services Google Kubernetes Engine Node Control Plane Node “Bring-your-own” Kubernetes Node Control Plane Node GKE On-Prem Node Control Plane Node Hybrid Use Cases Legacy Software Local

rights reserved. Amazon Confidential Amazon EKS 服务路线图摘要 已发布 - Amazon EKS control plane logs - Support for public IP space in VPC - Amazon EKS: Deep Learning Benchmarking Utility - New Amazon EKS Mumbai - CNI v1.5.0 - New Regions: Hong Kong 即将发布 - Service linked role for Amazon EKS - EKS Support for K8s version 1.13 + ECR AWS PrivateLink - EKS-optimized AMI metadata SSM parameter - IAM for 客户网 关 公司数 据中心 On-premises 10.1.0.0/16 VPN / DX Pod Outbound Traffic SNAT EKS worker node Primary elastic network interface Pod Secondary elastic network interface Pod – 100.64

Scheduling, provisioning, and resource management of multiple containers – Docker, Mesos à Kubernetes Support – AWS, Azure, Google à Kubernetes Services $docker run container1 $docker run container2 $docker Level • Container Cluster = “Desired State Management” – Kubernetes Cluster Services (w/API) • Node = Container Host w/agent called “Kubelet” • Application Deployment File = Configuration File of desired Runs in a Pod (~1:1) • Replicas = QTY of Pods that must be running Worker Node Worker Node Worker Node Kubernetes Master Node (Master & etcd nodes) API K K K App_Y.yaml ContainerImage1 Replicas: