jobs Category Online shopping web apps, payment service MR, spark, flink Latency Sensitive Insensitive Priority high low Traffic pattern Peak at day time Peak at night time Fault tolerance should together, don’t affect each other Resource contention ���� ������ ���������� - Online workload low 1:00am – 6:00am - Offline jobs scale up while online workload remains idle - Offline jobs scale down

measurement Test topology Test result Service type Short connection cps Short connection P99 latency Long connection pps ClusterIP +40% -31% not available NodePort +64% -47% +22% Test result • com/product/tke • Jobs https://careers.tencent.com/home.html Bugs solved – 1/2 • IPVS conn_reuse_mode=1 low cps Ip_vs_conn nf_conn New ip_vs_conn Bugs solved – 2/2 • DNS resolution delays for 5s Iptables

Use envelope encryption scheme • DEK & KEK Motivation: K8s Secrets Protection • Performance & latency • Network • Security • DEK in the clear in memory • Secret in the clear in memory • kubeconfig • Encrypted memory • SW/HW attacks prevented TEE-based KMS Plugin [1] • Address performance & latency concerns • Reduce / minimize remote KMS interactions w/o compromising security • Address security force update • Liveness probe • Monitoring • Integration w/ Prometheus • Metrics including • latency of en/decryption • failure times of en/decryption • KMS health check • Ops tooling • kms-plugin-tools

强大的工具箱 ● PodChaos: kill / fail / ... ● NetworkChaos: delay / lose / dup / partition / … ● IOChaos: latency / fault / … ● TimeChaos: clock skew ● KernelChaos: kernel fault injection ● StressChaos: burn

function and process — to provide the best opportunity for service recipient success. — Gartner Latency SLI Availability QPS Correctness SLO …… Punishment SLA SLI defines an indicator, which can

request of Pod. However, in many cases, some nodes have low resource requests but high load, while some nodes have high resource requests but low load. Dynamic-Scheduler Node1 Node2 Kube-scheduler

RDS IOPS Pods HPA Evaluation & conclusions Evaluation Kubernetes and docker for fast scaling Low price tag on operations No bottlenecks throughout the season 17% Record breaking Second screen