Spark on Kubernetes Kubernetes extends beyond container orchestration, it has been expanded to support for data-intensive and stateful apps. Benefit: l Autoscaling in Cloud l Consolidate online service goal is to bring native support for Spark to use Kubernetes as a cluster manager like YARN, or Mesos. l Spark 2.3 added native support for Kubernetes. l Spark 2.4 added support for client mode, R, python python etc. l Spark 3.0 will add support for dynamic resource allocation, external shuffle service, Kerberos etc. How it works Spark on Kubernetes Spark-operator Gaps for spark Ø Dynamic Resource

conntrack/iptables SNAT • Pros • O(1) time complexity in control/data plane • Stably runs for two decades • Support rich scheduling algorithm • Cons • Performance cost caused by conntrack • Some bugs How to • No loop support in eBPF verifier (Linux 4.14) • #param unroll • Size limitation of BPF program <= 4096 • Move SNAT allocate port loop into IPVS kernel module • Bounded loop support in Linux 5.3 • https://github.com/Tencent/TencentOS-kernel/ • More components will be open source later • Support more Linux distributions • Build IPVS kernel modules in Ubuntu, Centos • IPVS-eBPF next generation

is not a shared VPC # (There is more than one availability zone for this cluster) # # Also add support for us-east-1 # --------------------------------------------------------------- {{ if not SharedVPC is not a shared VPC # (There is more than one availability zone for this cluster) # # Also add support for us-east-1 # --------------------------------------------------------------- {{ if not SharedVPC is not a shared VPC # (There is more than one availability zone for this cluster) # # Also add support for us-east-1 # --------------------------------------------------------------- {{ if not SharedVPC

KMS • API server & kms-plugin • Cron job backup for KEKs (from KMS) • Static key configuration support in kms-plugin • One click decryption • Key force update • Liveness probe • Monitoring • Integration Version-based key synchronization • Adaption • apiserver KMS provider endpoint to support https endpoint • KMS plugin to support https [1] https://github.com/AliyunContainerService/sgx-device-plugin Secure Interface Demo • The purpose of this demo is to • Demonstrate TEE Transparency w/ Occlum’s Golang support • Showcase the confidentiality guaranteed by TEE Demo Demo Summary & Plan Summary & Next Steps

since 2016 KubeCon update ● Exec into pod ● Global search ● Login mechanism ● Settings page ● Support for Cron Jobs ● Redesigned resource creation ● ...and much much more. github.com/kubernetes/dashboard/releases running Kubernetes in GCP and on-prem ● Custom Resource Definitions support ● Service topology view ● Mobile device support ● Cost estimates ● CI/CD pipelines ● ...and more! Additional feature

SIG VMware serves to bring together members of the VMware and Kubernetes community to maintain, support and provide guidance for running Kubernetes on VMware platforms. SIG VMware provides a forum for discussion related to new CRDs, plug-ins and KEPs that allow the vSphere platform to supplement and support Kubernetes How to Join 6 SIG Sponsored projects vSphere cloud provider (In-tree and Out-of-tree) across platforms • hosts portable apps in a standard way What it does The external cloud provider support has been added as Alpha in version 1.6, it is currently in Beta (as of version 1.13) and will graduate

management. • Support big data and AI jobs. • Optimize the isolation of resources, and improve resource utilization using hybrid deployment of online and offline services. • Support Service Mesh. Features: Ø Manual/Auto Batch Gray Release Ø Multi-Batch Rollback Ø Multi-Batch InPlaceUpdate Ø Support HPA, CronHPA, VWA (Vertical Workload Autoscaler) Ø Keep share memory during Pod upgrade Ø Scaled

HPA can be turned on/off separately 2. Support for separate configuration of compute cycles 3. Support configuring tolerance for each HPA separately 4. Support Cron HPA An HPA corresponds to a gorountine

different deployment strategies and operate our edge devices in a clustered fashion. It really does support distributed processing across devices.” Ben Reif Lead Developer Booz Allen Hamilton 4 www.susergs different deployment strategies and operate our edge devices in a clus- tered fashion. It really does support distrib- uted processing across devices.” Pioneering a New Frontier in Military Tactics As the project might run in AWS, Azure or GCP (or a mixture), and so the SmartEdge infrastructure had to support multiple architectures in a variety of flavors. In contrast to commodity, cloud-based re- sources

+ kata not easy kubernetes(dockershim) does not support to choose OCI runtime k8s + docker + kata not easy kata container network hotplug (support now) Dockershim / Docker Containerd Cri-o a.create