VMware SIG Deep Dive into Kubernetes Scheduling Performance and high availability options for vSphere Steve Wong, Hui Luo VMware Cloud Native Applications Business Unit November 12, 2018 2 Open (sites, affinity groups, NUMA, etc.). ​This session will explain the options to gain better performance, resource optimization and availability through tuning of vSphere, and Kubernetes configuration the thread runs on, but can potentially come from other nodes with broad performance implications. Unpredictable performance? Swapping? This basically comes down to a choice of whether you would rather

eBPF Agenda 目录 01 Problems with K8s Service How to optimize 02 Comparison with industry Performance measurement 03 04 Future work 05 06 Lessons from eBPF What is K8s Service • It exposes a set control/data plane • Stably runs for two decades • Support rich scheduling algorithm • Cons • Performance cost caused by conntrack • Some bugs How to optimize • Guidelines • Use less CPU instructions Less modification to kernel Comparison with industry • Pitfalls • Performance of clusters of the same configure may differ • Performance of a cluster in different time slot may differ • Due to CPU oversold

service stability Ø Quota management to optimize resource orchestration efficiency Ø High performance and comprehensive autoscaling What is TKEx Ø Based on TKE (Tencent Kubernetes Engine) & EKS (Tencent deployment of online and offline services. • Support Service Mesh. • Large-scale and high-performance autoscaling capabilities. • Multi-tenant and quota management. • etc. TKEx Architecture EKS quota. Ø ValidatingWebhook to validate pod add request. DynamicQuota Large-scale and high-performance Autoscaling Motivation: Ø Reduce container OOM from bursted traffic. Ø Improve cluster resource

VMware SIG Deep Dive into Kubernetes Scheduling Performance and high availability options for vSphere Steve Wong, Michael Gasch KubeCon North America December 13, 2018 2 Open Source Community Relations (sites, affinity groups, NUMA, etc.). This session will explain the options to gain better performance, resource optimization and availability through tuning of vSphere, and Kubernetes configuration come from this node the thread runs on, but can potentially come from other nodes with broad performance implications. This basically comes down to a choice of whether you would rather have a fast

remote KMS • Use envelope encryption scheme • DEK & KEK Motivation: K8s Secrets Protection • Performance & latency • Network • Security • DEK in the clear in memory • Secret in the clear in memory isolation • Encrypted memory • SW/HW attacks prevented TEE-based KMS Plugin [1] • Address performance & latency concerns • Reduce / minimize remote KMS interactions w/o compromising security • Address kms-plugin-tools KMS Plugin as a Service • Motivation • SGX physical servers do not meet API servers’ performance requirements • Solution • Same TEE-based KMS-plugin runtime • Deployment modes • N (>=3) SGX

technologies Best of breed-technologies Right tool for the right job Easy prototyping with php - good performance with GoLang Scaling and orchestration Containerized development and production environment cluster Monitoring Prometheus autodiscovery Grafana dashboards Graylogs search and dashboards Performance test Performancetest: LOCUST ● Test as code ● Highly scalable ● Less hardware consuming

to desired • Policy-based workload scheduler • Topology aware • Assists with availability, performance and capacity • Affinity/Anti-Affinity Capable The Kubernetes Worker Node Basic Components Master ation Developer Structured Data Metrics Alerts Events VMware vRealize Operations Capacity, Performance and Configuration Management Events Launch in Context Unstructured Data Logs Messages VMware

Operand Monitoring • Operator exposing metrics about its health • Operator exposes health and performance metrics about the Operand Alerting and Events • Operand sends useful alerts • Custom Resources onto best suited nodes Abnormality detection • Operator determines deviations from a standard performance profile Observerbility 日志、系统指标等采集、分析；监控配置与报警；性能 指标收集与分析等等。 Backup & Restore 备份策略、备份方式、恢复方式、备份管理等等。

переезда в k8s ● Есть сетевая FS (MooseFS) ● Уже есть небольшой Clickhouse cluster для хранения performance logs Взгляд с практической стороны Устанавливаем оператор ● https://github.com/Altinity/clickhouse-operator/

Spark-operator Gaps for spark Ø Dynamic Resource Allocation Ø Spark external shuffle service Ø Performance Ø Security p Kerberos support Ø … Gaps for Spark Ø Resource Management: p Queue p Hierarchical