组件异常 • API Server Load Balancer 异常 • API Server Pod 异常 影响 • 通过 API Server 访问集群概率失败 • 升级集群失败 Load Balancer Service Master API Server Pod Master API Server Pod Master API Server Pod Kubernetes /Standalong） 问题检测 • 硬件（CPU、内存、磁盘） • 操作系统（ NTP、内核死锁、文件系统异常） • Container Runtime（无响应） 问题上报 • API server • Prometheus node-problem-detector Sonobuoy 运行模式 • 集群节点（Collector Pod + DaemonSet/One Shot）

"annotations": { "key1" : "value1", "key2" : "value2" } 类似以下信息可记录到Annotation中： 由declarative configuration layer管理的字段。将这些字段附加为Annotation，可将它们与客户端或服务器设置的默 认值、⾃动⽣成的字段或以及auto-sizing或auto-scaling的系统所设置的字段区分开。 Master通常会部署在⼀个独⽴的服务器或虚拟机上，它是整个集群的⾸脑，如果Master宕机或不可⽤，那么我们所有的 控制命令都将会失效。 Master节点上运⾏着如下的关键进程： API Server：K8s⾥所有资源增删改查等操作的对外⼊⼝，也是集群控制的⼊⼝进程，它提供了HTTP RESTful API 接⼝给客户端以及其他组件调⽤。 Controller Manager：Controller 有Pod对象从apiserver中删除，并释放其名称。 Kubernetes 1.8引⼊了⼀个⾃动创建代表condition的 taints 功能（⽬前处于Alpha状态）。要启⽤此特性，请向API server、controller manager和scheduler传递标志 --feature-gates=...,TaintNodesByCondition=true 。⼀旦启⽤ TaintNodesByCondition

block of the Custom Resource Configuration of the workload • Operator provides configuration via the spec section of the Custom Resource • Operator reconciles configuration and updates to it with the Operand • Operator is able to restore a backup of an Operand • Operator orchestrates complex re- configuration flows on the Operand • Operator implements fail-over and fail-back of clustered Operands • 全量 objects。list 可以简单理解为一个 HTTP GET 请求，watch 为一 个 HTTP/2 长连接 Cache 如何保持与 API Server 一致性 list & watch 机制中，list 获取 API Server 中数据的一份快照，并记 录 ResourceVersion 版本信息，watch 从 ResourceVersion 开始，获取后 续的增量数据。 watch

"clickhouse.altinity.com/v1" kind: "ClickHouseInstallation" metadata: name: "demo-01" spec: configuration: clusters: - name: "demo" Здесь нет storage Еще вернемся к этому kubectl – наше всё "clickhouse.altinity.com/v1" kind: "ClickHouseInstallation" metadata: name: "demo-01" spec: configuration: clusters: - name: "demo" layout: shardsCount: 2 replicasCount: "clickhouse.altinity.com/v1" kind: "ClickHouseInstallation" metadata: name: "demo-01" spec: configuration: users: demo/password: secret demo/profile: default demo/networks/ip: "::/0"

Services (w/API) • Node = Container Host w/agent called “Kubelet” • Application Deployment File = Configuration File of desired state • Container Image = Runs in a Pod (~1:1) • Replicas = QTY of Pods that Kubernetes Master etcd API-Server Scheduler NSX Container Plugin (NCP) NSX Infra NSX Manager API Client Kubernetes Adapter Kubernetes Master etcd API-Server Scheduler Architecture • NSX-T ‘watch’ on K8s API for any Namespace events 2. A user creates a new K8s Namespace 3. The K8s API Server notifies NCP of the change (addition) of Namespaces 4. NCP creates the network topology for the

选择集群：选择配置存储类到哪个容器集群 存储类型：选择所⽀持的存储类型，⽬前仅⽀持NAS⽂件存储 存储对象：已创建并且挂在到该集群的NAS盘 驱动提供者：⽂件NAS存储选择 cds/nas server-ip: 系统会⾃动读取NAS盘的挂载点IP path：NAS盘对应的远端挂在⽬录，默认为 /nfsshare mode: pv使⽤⽂件夹的mode，⼀般选择 755 或 777 存储类名 选择集群：选择配置存储类到哪个容器集群 存储类型：选择所⽀持的存储类型，⽬前仅⽀持NAS⽂件存储 存储对象：已创建并且挂在到该集群的NAS盘 存储驱动：请选择CSI（Flexvolume今后会停⽌⽀持） server-ip: 系统会⾃动读取NAS盘的挂载点IP path：NAS盘对应的远端挂在⽬录，默认为 /nfsshare mode: pv使⽤⽂件夹的mode，⼀般选择 755 或 777 存储卷名 readiness-probes。 24 请求类型 配置说明 HTTP/HTTPS 即向容器发送⼀个 HTTPget 请求，⽀持的参数包括： 路径：访问 HTTP server 的路径。 端⼝：容器暴露的访问端⼝或端⼝名，端⼝号必须介于 1~65535。 HTTP 头：即 HTTPHeaders，HTTP 请求中⾃定义的请求头，HTTP 允许重复的 header。

Serverless＝ Backend as a Service Functions as a Service • Zero server ops – No provisioning, updating, and managing server infrastructure. – Flexible Scalability • No compute cost when idle production- grade container orchestration platform § Declarative management of objects using configuration files. § More introductions, go to • K8s official document http://kubernetes.io • Open Zookeeper – Redis Other objects used in OW charts • ConfigMap: like nginx deployment configuration • Secrets: like DB access credentials • Ingress Component Launch Sequence • In Kubernetes

without restarting container Ø High-performance Ø Safe autoscaling decisions Ø Personalized configuration of VWA objects Ø Cooperate with HPA through events Vertical Workload AutoScaler (VWA) Recommender Autoscaler ) Ø Deploy HPAPlus-Controller independently. Ø High Performance. Ø Personalized configuration of HPA objects. Ø Calculate replicas based on pod resource request or limit. Ø Cooperate with Kube-ApiServer Aggregator Prometheus Adaptor Prometheus Metrics Server Kubelet/cAdvisor External Metric Adaptor 3rd Monitor Server � Exportors CronHPA-Controller cooperate VWA-Controller Event

Google does GKE On-Prem ● Turn-key, production-grade, conformant Kubernetes with best-practice configuration ● Easy upgrade path to the latest Kubernetes releases that have been validated and tested On-Prem ● Cluster environments are consistent (k8s version, OS image, plug-ins, components configuration) Orchestrate and manage on-prem containers just like GKE in the cloud Consistent operating Installation and Configuration $ gke-on-prem create cluster --dry-run Welcome! This command will take you through the installation of a cluster. --dry-run saves your configuration to a YAML file. Please

Background: K8s Secrets • Encryption Keys stored on API Server • Secrets encrypted prior to storage in etcd • Secrets decrypted on API Server prior to use • Encryption keys stored in a remote KMS write • LivenessProbe for health check • Configurations • kms-plugin • apiserver • Caching • API server • Set up Encrypted(DEK) => DEK mapping • KMS plugin • Set up SecretKeyName:SecretKeyVersion => SecretKeyData Emergency management • High Availability guarantee • KMS • API server & kms-plugin • Cron job backup for KEKs (from KMS) • Static key configuration support in kms-plugin • One click decryption • Key force