custom resource definition(CRD) ü custom controller Operator 是什么 • Kubernetes 中一切都可视为资源 • 默认资源类型：如 Pod、Service、Volume 等 • Kubernetes 1.7 之后增加了 CRD 自定义资源 • 二次开发扩展 Kubernetes API CRD 的基本原理 ① 观察资源的当前状态 mysql pod-5 mysql pod-6 … … mysql pod-M Node1 kubelet kube-proxy docker mysql pod-0 mysql pod-1 … … mysql pod-N NodeN Mysql Operator pod … … mysql Pod-T 0. 创建 CRD 3. 集群管理 2. 调度 pod Ceph Reflector DeltaFIFO Local Storage Callbacks OnAdd OnUpdate OnDelete WorkQueue List/Watch - CRD - Pod Write Informer Worker ReadOnly • Informer：监听事件并触发回调函数的二级缓存工具包 • WorkQueue：事件合并、过滤、延时、限速 Operator

Kubernetes 计算资源 Pod ● Pod ○ 计算任务 → 容器 → Pod ○ 资源分配：CPU，内存，磁盘 ○ 资源调度：Pod → Node ● Pod管理 ○ 无状态计算资源组：Deployment ○ 有状态计算资源组：StatefulSet ● Pod持久存储 ○ 通过PVC申请PV存储资源 ● StatefulSet ○ Pod网络地址不变 ○ Pod与PV映射关系不变 Kubernetes Network Interconnect Standby Pod Master Pod Segment Pod Segment Instance Segment Pod Segment Instance Segment Pod Segment Instance Segment Pod Segment Instance Master StatefulSet (2) Segment节点示例 Greenplum on Kubernetes Greenplum on Kubernetes ● 存储计算分离 ○ PV持久化存储资源 ○ StatefulSet/Pod弹性扩展计算资源 ● 数据库服务层 ○ Service统一Master & Standby Master地址 ● 服务发现机制 ○ 所有节点地址名不变 ● 跨云能力 ○ 容器应用对基础设施透明

Kubernetes Master Pod kubelet kube-proxy docker Node Pod kubelet kube-proxy docker Node Greenplum on Kubernetes 101 Kubernetes Master Pod kubelet kube-proxy docker Node Pod kubelet kube-proxy Node Storage volumes Greenplum on Kubernetes 101 Kubernetes Master Pod Postgres Container kubelet kube-proxy docker Node Pod Postgres Container kubelet kube-proxy docker Node Storage volumes Greenplum Service Pod Postgres container kubelet kube-proxy docker Node Pod Postgres Container kubelet kube-proxy docker Node Storage volumes Greenplum on Kubernetes Node Pod segment-b-0 kubelet

StatefulSet を使って tidb や tikv のコンポー ネントを作成します StatefulSet は Pod を指定された数だけシーケンシャルに作成します (mypod-0, mypod-1 ,,, mypod-N) Pod が増える時は Pod の名前の末尾の数字が増えていき、Pod が減る時 は末尾の数字が大きいものから減っていきます Open Source Conference StatefulSet を使う つまり、通常の StatefulSet だとこういう状態にすることはできません mypod-0 mypod-4 mypod-2 mypod-3 特定の番号の Pod (今回の例だと 1 ) だけ欠番にして通常通り運用したい・・・ そんな願いを叶えてくれるのが Advanced StatefulSet になります ... Open Source Conference Open Source Conference 2022 Online/Spring 58 Advanced StatefulSet を使う Advanced StatefulSet は消したい Pod を annotation で記述します mypod-0 mypod-1 mypod-2 mypod-3 mypod-0 mypod-1 mypod-2 mypod-3 mypod-1 を消したい！

com - Kubernetes has pod, deployment, statefulset, etc. - But it does not know how to operate a TiDB cluster... Kubernetes Pattern - Declarative Model PingCAP.com - Kubernetes has pod, deployment, statefulset PingCAP.com Scheduler Extender kube-scheduler kube-scheduler tidb-scheduler other Pod other Pod other Pod PD Pod TiKV Pod TiDB Pod PingCAP.com Scheduler Extender 3 TiKV ❌ Node-A Node-B Node-C TiKV TiKV

RESTARTS AGE pod/commerce -apply-schema-initial -vlc6k 0/1 Completed 0 2m42s pod/commerce -apply-vschema -initial -9wb2k 0/1 Completed 0 2m42s pod/vtctld -58bd955948 -pgz7k 1/1 Running 0 2m43s pod/vtgate Running 3 2m43s pod/zone1-commerce -0-init-shard-master-gshz9 0/1 Completed 0 2m42s pod/zone1-commerce -0-replica -0 2/2 Running 0 2m42s pod/zone1-commerce -0-replica -1 2/2 Running 0 2m42s pod/zone1-commerce warning >}} The port-forward will only forward to a specific pod. Currently, kubectl does not automatically terminate a port-forward as the pod dissapears due to apply/upgrade operations. You will need to

RESTARTS AGE pod/commerce -apply-schema-initial -2pbzn 0/1 Completed 0 2m44s pod/commerce -apply-vschema -initial -mfhvl 0/1 Completed 0 2m44s pod/vtctld -6f955957bb -67bq7 1/1 Running 0 2m44s pod/vtgate Running 3 2m44s pod/zone1-commerce -0-init-shard-master-dh727 0/1 Completed 0 2m44s pod/zone1-commerce -0-replica -0 5/6 Running 0 2m44s pod/zone1-commerce -0-replica -1 5/6 Running 0 2m44s pod/zone1-commerce RESTARTS AGE pod/vtctld -6f955957bb -jp2t2 1/1 Running 0 18m pod/vtgate -zone1 -86b7cb87d6 -nsmw4 1/1 Running 3 18m pod/zone1-commerce -0-init-shard-master-d5vj4 0/1 Completed 0 18m 189 pod/zone1-commerce

Service Replica Service Replica Service User Config Map Common Config Map Stateful Set Pod Persistent Volume Claim Persistent Volume Per-replica Config Map Altinity ClickHouse Operator created $ kubectl get all -n demo NAME READY STATUS RESTARTS AGE pod/chi-demo-01-demo-0-0-0 1/1 Running 0 37s NAME TYPE NAME READY STATUS RESTARTS AGE pod/chi-demo-01-demo-0-0-0 1/1 Running 0 2m pod/chi-demo-01-demo-1-0-0 1/1 Running 0 33s NAME

Service Replica Service Replica Service User Config Map Common Config Map Stateful Set Pod Persistent Volume Claim Persistent Volume Per-replica Config Map Challenges running ClickHouse created $ kubectl get all -n demo NAME READY STATUS RESTARTS AGE pod/chi-a82946-2946-0-0-0 1/1 Running 0 52s NAME TYPE test # Connect to specific node clickhouse-client --host chi-a82946-2946-0-0.test # Connect via pod entry kubectl exec -it chi-a82946-2946-0-0-0 -n demo -- clickhouse-client Connect from outside

RESTARTS AGE pod/commerce -apply-schema-initial -vlc6k 0/1 Completed 0 2m42s pod/commerce -apply-vschema -initial -9wb2k 0/1 Completed 0 2m42s pod/vtctld -58bd955948 -pgz7k 1/1 Running 0 2m43s pod/vtgate Running 3 2m43s pod/zone1-commerce -0-init-shard-master-gshz9 0/1 Completed 0 2m42s pod/zone1-commerce -0-replica -0 2/2 Running 0 2m42s pod/zone1-commerce -0-replica -1 2/2 Running 0 2m42s pod/zone1-commerce warning >}} The port-forward will only forward to a specific pod. Currently, kubectl does not automatically terminate a port-forward as the pod dissapears due to apply/upgrade operations. You will need to