A Day in the Life of a Data Scientist Conquer Machine Learning Lifecycle on Kubernetes Brian Redmond • Cloud Architect @ Microsoft (18 years) • Azure Global Black Belt Team • Live in Pittsburgh, PA

Node Operator: Kubernetes Node Management Made Simple 陈俊(Joe), Ant Financial Agenda • Background and Motivation • Introduction of Operators • Node-Operator • Advanced Topic: Topic: Kube-on-Kube-Operator • Achievement • Q&A Background: DC/OS From Sigma 2.0(Swarm) to Sigma 3.1(Kubernetes) Background: Cluster Scale • Production environment: • Dozens of Cluster • 5k+ Nodes / Cluster architecture Work order deployment system can not meet the requirements of resource management. Operator Observe Action Analyze • Observe: watch desired resource and actual resource • Analyze: difference

Operator Pattern：用 Go 扩展 K8s 的最佳实践 吴学强 ApeCloud KubeBlocks Maintainer & 研发总监 目 录 认识我们 00 什么是 Operator 01 Operator 基础模型 02 Operator 最佳实践 03 我们是谁 云猿生（ApeCloud）是一家提供数据库内核与管理平台的基 础软件开发商. KubeBlocks 从被收购到卷王（si） 回到初（qi）心（dian） KubeBlocks Maintainer & 研发总监 free6om 什么是 Operator 第一部分 Operator 前世今生 TPR Operator CRD Operator Pattern 2015.11 2016.12 2017.12 Now K8s 1.1 版本中正式推出 TPR （ThirdPartyResource），首次尝 K8s API 的扩展性问题， 但存在诸多问题，Alpha 阶段既 夭折 CoreOS 提出 Operator 概念，用 于管理和运行基于应用程序领 域的复杂有状态应用程序。 给出了用 TPR + controller- runtime 早期版本的 sample： etcd operator K8s 1.9 版本发布，CRD进入 beta 阶段并正式取代 TPR； controller-runtime

Kubernetes Capabilities/Advantages to Build DevOps Solution • Architecture and Features • CRD and operator design • Pipeline / Stage/ Task / Task Template / Version Control • Logging, monitoring, autoscaling persisted [] Resources - Resource requirement ActiveDeadlineSeconds Timeout of build task Lifecycle - Actions defined for postStart/preStop Kubernetes Capabilities/Advantages to Build Kubernetes Capabilities and Advantages to Build DevOps Solution • Architecture and Features • CRD and operator design • Pipeline/Stage/Task/Task Template/Version Control/UI generation/Volume... • Logging

yellow.mycluster Select color = yellow Load balancing Controller manager 5 Healing Controller manager 5 Healing Controller manager 5 Healing People love automation! I hate Kubernetes! I hate to - Systemctl start kubelet Installation - master - SSH - Install scheduler - Install controller manager - Install API server - Config them correctly - Start them Installation - etcd - SSH - Install etcd apply -f kube-scheduler.yaml $ kubectl apply -f kube-controller-manager.yaml $ kubectl apply -f kube-proxy.yaml Simplify k8s lifecycle management Manage your cluster with only kubectl Upgrading a self-hosted

internal systems like Route System, CMDB, CI, Security Platform, etc. • Declarative application lifecycle management. • Support big data and AI jobs. • Optimize the isolation of resources, and improve Schedule Ceres Job Queue Manager Spark-Operator OfflineJobs Scheduler Kubeflow Hybrid Deploy StatefulSetPlus-Operator Tencent Cloud Mesh MultiCluster-Route-Manager Application & Route Management kube-apiserver StatefulSetPlus ListWatch PodConditionsChecker NodeConditionsChecker StatefulSetPlus Operator RollingUpdate OnDelete ManualBatchDeploy AutoBatchDeploy InPlaceUpdate HPAPlus Controller

The Kubernetes Master Node Basic Components Master Node ETCD kube-apiserver kube-controller-manager kube-scheduler • Key/Value Store • Leader based clustering • Can be clustered across Master Nodes The Kubernetes Worker Node Basic Components Master Node ETCD kube-apiserver kube-controller-manager kube-Scheduler Worker Node CRI-containerd Kubeproxy Kubelet • Container Runtime Interface selector to provide a LB and Service DNS ReplicaSets A cluster wide Pod manager providing Pod scaling DaemonSets A Pod manager to ensure a Pod is scheduled across a Cluster Node set StatefulSets Replicated

Data Analysis Failures/Machine Failures/Reason Report Lifecycle of Pod Failure Reason Target Kubelet Apiserver Scheduler Operator Runtime Daemonset Alert Gray Scale Bug Fix Success Rate Trace Result: We can get: It is failed to deliver the pod，and the fail reason is FailedMount. Pod Lifecycle FailureReason Trace System: Node Metrics node metrics kubelet metrics daemonset metrics

统⼀一编程框架 统⼀一通讯协议 统⼀一部署环境 计算平台/KUN 公共服务 存储平台 Think in Cloud . 北北京 基于RBAC实现 账号管理理隔离 01 IPv6 02 Operator管理理有 状态的服务 03 监控 04 Think in Cloud . 北北京 • K8S提供了了多种身份认证策略略，具体如何实施？ • K8S的有两种⽤用户：服务账号(SA masquerade-all=true BGP(IPv6) BGP(IPv6) Service Gateway Service Gateway Think in Cloud . 北北京 Operator管理理有状态的服务 StatefulSet • 直接管理理的 Pod 的 hostname、名字等都是携带了了编号，Pod 的 创建，也是严格按照编号顺序进⾏行行 • 通过 Headless Volume Operator • ⾸首先在k8s中注册CRD • Operator 于 API server 交互，Watch 全部的 Namespace 或者特 定Namespace中对CR的创建、更更新、删除事件 • Operator 处理理这些事件，可以使⽤用 k8s 中的pod、deployment、 statefulset 对象构建应⽤用 Operator⼯工作原理理

etcd ⽤作Kubernetes的后端存储。集群的所有数据都存储在此。请为你Kubernetes集群的etcd数据提供备份计划。 kube-controller-manager kube-controller-manager 运⾏Controller，它们是处理集群中常规任务的后台线程。逻辑上来讲，每个Controller都是⼀ 个单独的进程，但为了降低复杂性，它们都被编译成独⽴的⼆进制⽂件并运⾏在⼀个进程中。 cloud-controller-manager cloud-controller-manager运⾏着与底层云提供商交互的Controller。cloud-controller-manager是在Kubernetes 1.6版中 引⼊的，处于Alpha阶段。 cloud-controller-manager仅运⾏云提供商特定的Controller循环。您必须在kube-controller-manager中禁⽤这些 Controller循环。可在启动kube-controller-manager时将 --cloud-provider 标志设为 external 来禁⽤控制器循环。 cloud-controller-manager允许云供应商代码和Kubernetes内核独⽴发展。在以前的版本中，核⼼的Kubernetes代码依 赖于特定云提供商的功能代码。在未来的版本中，云供应商的特定代码应由云