Node Operator: Kubernetes Node Management Made Simple 陈俊(Joe), Ant Financial Agenda • Background and Motivation • Introduction of Operators • Node-Operator • Advanced Topic: Topic: Kube-on-Kube-Operator • Achievement • Q&A Background: DC/OS From Sigma 2.0(Swarm) to Sigma 3.1(Kubernetes) Background: Cluster Scale • Production environment: • Dozens of Cluster • 5k+ Nodes / Cluster architecture Work order deployment system can not meet the requirements of resource management. Operator Observe Action Analyze • Observe: watch desired resource and actual resource • Analyze: difference

Operator Pattern：用 Go 扩展 K8s 的最佳实践 吴学强 ApeCloud KubeBlocks Maintainer & 研发总监 目 录 认识我们 00 什么是 Operator 01 Operator 基础模型 02 Operator 最佳实践 03 我们是谁 云猿生（ApeCloud）是一家提供数据库内核与管理平台的基 础软件开发商. KubeBlocks 从被收购到卷王（si） 回到初（qi）心（dian） KubeBlocks Maintainer & 研发总监 free6om 什么是 Operator 第一部分 Operator 前世今生 TPR Operator CRD Operator Pattern 2015.11 2016.12 2017.12 Now K8s 1.1 版本中正式推出 TPR （ThirdPartyResource），首次尝 K8s API 的扩展性问题， 但存在诸多问题，Alpha 阶段既 夭折 CoreOS 提出 Operator 概念，用 于管理和运行基于应用程序领 域的复杂有状态应用程序。 给出了用 TPR + controller- runtime 早期版本的 sample： etcd operator K8s 1.9 版本发布，CRD进入 beta 阶段并正式取代 TPR； controller-runtime

★配置docker服务使用systemd去管理（以及信任本地镜像仓库） # vi /etc/docker/daemon.json { "data-root": "/docker_data", "registry-mirrors": [ "h�ps://cof-lee.com:5443" ], "insecure-registries": [ "cof-lee.com:5443" ], "exec-opts": "bridge": "none", "iptables": false, "experimental": true, "data-root": "/docker_data", "registry-mirrors": [ "h�ps://cof-lee.com:5443" ], "insecure-registries": [ "cof-lee.com:5443" ], "exec-opts": io/etcd:3.4.13-0 k8s.gcr.io/coredns:1.7.0 #可以先下载以上7个镜像，传到每台k8s服务器上，再docker load导入；或者使 用内部registry仓库（内部registry镜像仓库里要有以上7个镜像） ★直接使用命令行方式初始化集群 （以下是非HA模式的master初始化，如果要部署高可用集群，则参考第4章） kubeadm init --kubernetes-

Kubernetes Capabilities/Advantages to Build DevOps Solution • Architecture and Features • CRD and operator design • Pipeline / Stage/ Task / Task Template / Version Control • Logging, monitoring, autoscaling environments（sidecar） Share files between containers, or cache build files Container Image - Image of build / dependent environment [] Command - Command to execute [] Args Kubernetes Capabilities and Advantages to Build DevOps Solution • Architecture and Features • CRD and operator design • Pipeline/Stage/Task/Task Template/Version Control/UI generation/Volume... • Logging

构建云原生应用管理平台 @ 阿里巴巴 Kubernetes 工程师 PaaS 工程师 基础设施运维工程师 … YAML 工程师 我们是如何构建的？ PaaS Serverless Operator Platform 基于 Kubernetes 我们构建了多种多样的应用管理平台： 电商 PaaS Kubernetes 用户 ( 应用开发者和运维人员 ) 我所在的团队 为什么 我们需要在 在服务端做抽象并不简单 WebService - image - replicas - port 抽象 Deployment - image - replicas Service - port 原始 k8s API 资源 Workload - image - replicas Rollout - canary ArgoRollout - image - replicas - rollout rollout Deployment - image - env - labelSelectors Knative Revision - image - env 组合 拆解 转换 做抽象容易形成“谷仓” • 一个抽象满足不了所有场景，所以… 有状态应用 PaaS 无状态应用 PaaS Serverless PaaS 用户 Kubernetes Cert Manager

com/kubernetes-security-best-practices/ ©2019 VMware, Inc. 8 NIST在容器安全指南中揭露了五種容器應用最應關注的風險 映像風險 Image Risk 登錄風險 Registry Risk 容器調度平台風險 Orchestrator Risk 容器風險 Container Risk 實體作業系統風險 Host OS Risk Consumes PKS API/CLI Day 1 & Day 2 for k8s clusters Manages access to k8s API for developers IT Operator IaaS Management Internet User Application User Trust Boundary Trust Boundary Trust Boundary CredHub服務安全地自動化產生與 保存帳號權限 • 這幾項服務可以針對多個 Kubernetes 叢集個別指派授權 Centralized Authentication with RBAC Operator admin Developers Developers Kubernetes NameSpace NameSpace NameSpace NameSpace Kubernetes

基础设施能力还如何演进和透出？ 研发自己的诉求如何传达给运维和基础设施？ K8s 扩展能力的真实情况 我的 Zookeeper 该用 哪种K8s Workload 接入？ 你恐怕得写个 Operator…… Operator是啥？ CRD Controller Informer Reflector Event Handler Loop … 我们业务压力大. 一定要学这些么…… 那好吧, 我们帮你写… 被同时安装给了同一个应用 • K8s 扩展能力之间的冲突关系，如何有效管理？如何有效的对运维透出？ K8s 如何管理描述云资源？ 太好了，我还需要启动一个 RDS, 能跟 helm一起打包吗？ Operator 写好了,用 helm 打包部署吧，美滋滋.. 这样啊，你们PaaS 平台的 体验好割裂… 好吧，我们试试 …… 这你恐怕得单独去RDS 界面创建… 业务方运维 K8s 团队 CRD interface for our nginx deployment. spec: workloadType: Server osType: linux containers: - name: nginx image: name: nginx:1.7.9 digest: env: - name: initReplicas value: 3 - name: worker_connections

kubeadm.k8s.io/v1beta2 kind: ClusterConfiguration kubernetesVersion: v1.18.2 imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers controlPlaneEndpoint: "k8s-master:6443" networking: traefik-ingress-controller terminationGracePeriodSeconds: 1 containers: - image: traefik:v2.2.8 name: traefik-ingress-lb ports: - name: web name: traefik-config tolerations: #设置容忍所有污点，防止节点被设置污点 - operator: "Exists" nodeSelector: #设置 node 筛选器，在特定 label 的节点上启动 IngressProxy:

called “Kubelet” • Application Deployment File = Configuration File of desired state • Container Image = Runs in a Pod (~1:1) • Replicas = QTY of Pods that must be running Worker Node Worker Node Kubernetes Master API Cluster A Worker Worker Worker Kubernetes Master API Cluster B Operator Developer Architecture PKS Controller IaaS Worker Worker Kubernetes Cluster Services API Kubernetes (As a Bosh Release) BOSH (Deploys/Manages VMs) CPI CNI Harbor Private Container Registry Worker Worker Worker Kubernetes Cluster Services API Worker Worker Worker Kubernetes

Failures/Reason Report Lifecycle of Pod Failure Reason Target Kubelet Apiserver Scheduler Operator Runtime Daemonset Alert Gray Scale Bug Fix Success Rate SLO The unhealthy node Cluster Healthy Daily Report Tips on increasing SLO Case 1: Image Download Image lazyload technology provides the ability to run a container without downloading image. Case 2: Retry Pod should be recreate when the Critical Deamonset Node should be tainted when critical Daemonset is unhealthy. Case 4: Plugin registry Registration of plugin such as CSI plugin should be checked. Case 5: Capacity The QPS Limit and