VMware SIG Deep Dive into Kubernetes Scheduling Performance and high availability options for vSphere Steve Wong, Michael Gasch KubeCon North America December 13, 2018 2 Open Source Community Relations options, for both control plane and worker nodes. 2 levels of scheduling and resource management are active. Currently no automatic scheduling integration occurs, that is, Kubernetes is not aware of the distribution running on the vSphere stack. Agenda 4 Kubernetes default scheduling How it works Utilizing Zones to improve scheduling Using vSphere tags to define regions and zones – add cloud provider

VMware SIG Deep Dive into Kubernetes Scheduling Performance and high availability options for vSphere Steve Wong, Hui Luo VMware Cloud Native Applications Business Unit November 12, 2018 2 Open options, for both control plane and worker nodes. 2 levels of scheduling and resource management are active. ​Currently no automatic scheduling integration occurs, that is, Kubernetes is not aware of the distribution running on the vSphere stack. Agenda 4 ​Kubernetes default scheduling ​How it works ​Utilizing Zones to improve scheduling ​Using vSphere tags to define regions and zones – add cloud provider

batch system (Volcano) development • IBM spectrum computing - Cluster resource and workload scheduling platform development l Gaps for Spark • Agenda l Why Spark on Kubernetes l Volcano solution … Ø Scheduler p Job preemption p Fair-share scheduling p Queue scheduling p Resource reservation p Binpack p Task topology p Zone aware scheduling p … Volcano: A Kubernetes native batch system pod executor pod executor pod executor pod apiVersion：v1 kind: Pod metadata: annotations: scheduling.k8s.io/group-name: job-1574739729783- podgroup volcano.sh/task-spec: spark-driver createTimestamp:

management system ● Automation ○ Scheduling ○ Deployment ○ Healing ○ Discovery/load balancing ○ Scaling Scheduling Scheduler Scheduler Scheduling Scheduler Scheduling color=yellow Discovery Select

Co-located�40% Seperated �10% 30% ������ �� ������ ������� ������ �� ��� ���� �������� �� ����������� Scheduling Isolation Efficient placement of service container and tasks When placed together, don’t affect running service. Driving current state towards desired state with control loops YARN Focus on scheduling jobs ���������� ���� Kubernetes Container centric – bottom up. Container is the primitive. Other such as replicaset, deployment are built around containers. YARN Application centric: top down. Scheduling sequence: Queue -> user -> application -> container request ������������� kubernetes Based on

in popular Linux distributions • Cons • O(N^2) in control plane / O(N) in data plane • Poor in scheduling algorithm • Iptables rules are difficult to debug IPVS mode • Services are organized in hash Pros • O(1) time complexity in control/data plane • Stably runs for two decades • Support rich scheduling algorithm • Cons • Performance cost caused by conntrack • Some bugs How to optimize • Guidelines del refer • V.S service in pure eBPF • Reuse the IPVS’s functionality and maturity • Many scheduling algorithms • Reuse IPVS timer to reclaim eBPF map • More efficient and secure • V.S Taobao IPVS

with powerful extensibility Understand Kubernetes in 1 min Container Pod Deployment I need co-scheduling I have many Pod replicas Service I want to proxy my Pod replicas Ingress I want to expose makeChanges(desired, current) } • Write your own controller! Why DIY? Container Pod Deployment I need co-scheduling I have many Pod replicas Service I want to proxy my Pod replicas Ingress I want to expose gogoprotobuf based protobuf gen CRI Management kubelet Workloads Orchestration kubelet SyncLoop Scheduling api-server Etcd bind pod, node list pod GenericRuntime SyncPod CRI grpc dockershim remote

create and run single containers – Very manual, no fault tolerance, hard to scale, etc • Scheduling, provisioning, and resource management of multiple containers – Docker, Mesos à Kubernetes Support Source Project – Google/Pivotal/VMware 21 Container scheduling, scale, resiliency, and Day 2 Desired state of Kubernetes Kubernetes cluster scheduling, scale, resiliency, and Day 2 VMware PKS Value

（硬性选择） • 给Node打分 （软性选择） 绑定周期 • 向binding接⼝发送Post请求，把结果告诉k8s
 基于CPU实时使⽤率的调度器 借助Scheduling Framework可以实现⾃定义的调度器 score=(1 - cpu_used_seconds / cpu_total ) * Maxscore 未来规划 垂涎已久的架构 ServiceMesh

with Kubernetes Ø How to provide a fast & reliable way to release stateful service Ø Advanced scheduling to improve service stability Ø Quota management to optimize resource orchestration efficiency