Scaling a Multi-Tenant k8s Cluster in a Telco Pablo Moncada October 28, 2020 About MasMovil group ● 4th telecom company in Spain ● Provides voice and broadband services to +12M customers ● Several Services +3k CPU +2k Mem +5TB Nodes +300 kube-proxy replacement NetworkPolicy logging Multi-cluster DNS Aware NetworkPolicy Increased Istio security External Services TLS visibility Performance

Advanced Networking Cluster Mesh Operations Istio Concepts Component Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy Troubleshooting etcd (kvstore) Cluster Mesh Troubleshooting Symptom clusters? What is the 95th and 99th percentile latency between HTTP requests and responses in my cluster? Which services are performing the worst? What is the latency between two services? Security observability

Advanced Networking Cluster Mesh Operations Istio Concepts Component Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy Troubleshooting etcd (kvstore) Cluster Mesh Troubleshooting Symptom clusters? What is the 95th and 99th percentile latency between HTTP requests and responses in my cluster? Which services are performing the worst? What is the latency between two services? Security observability

Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs Operations Scalability report Performance Evaluation Setup Evaluation Results Tuning Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy clusters? What is the 95th and 99th percentile latency between HTTP requests and responses in my cluster? Which services are performing the worst? What is the latency between two services? Security observability

Overview Terminology Networking Network Security eBPF Datapath Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Security Bugs Operations System Requirements Summary Scalability report Performance Evaluation Setup Evaluation Results Tuning Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy clusters? What is the 95th and 99th percentile latency between HTTP requests and responses in my cluster? Which services are performing the worst? What is the latency between two services? Security observability

scheduling and management software to build a solid cloud base. • High availability (HA) cluster solution: The HA cluster solution implemented by KylinSoft enables failover within seconds. Flourishing community IDE Auto-tuning tool A-Tune Test platform Compass-CI Toolchain OpenStack Kubernetes Kylin HA Cluster scheduling and management CPU: x86, ARM, RISC-V GPU NPU Chips Apps Virtualization Containers community in 2020, including multiple modules such as computing, storage, network, PaaS, security, and cluster management. Among these modules, Nova, Keystone, Neutron, Glance, Ironic, and Horizon have been

PilotGo 088 SysCare 090 5 Developer Support 092 Infrastructure 093 Compass-CI 093 CVE Manager 095 EUR 097 oepkgs 098 openEuler Software Package Contribution Platform 100 Signatrust 101 Maximized node performance Application cluster Storage cluster (multi-node/multi-controller) /A /A /B /C /D /E /F Multipath NFS/RDMA or NFS/TCP Maximized cluster performance in client-server model improving the application performance. Repositories https://gitee.com/openeuler/wayca-scheduler • Cluster and NUMA scheduling domains are established based on the hardware topology, and the scheduler supports

Troubleshooting Monitoring & Metrics Installation cilium-agent cilium-operator Troubleshooting Component & Cluster Health Connectivity Problems Policy Troubleshooting Symptom Library Useful Scripts Reporting a problem requires the firewalls on all servers to be manipulated whenever a container is started anywhere in the cluster. In order to avoid this situation which limits scale, Cilium assigns a security identity to groups store. Secure access to and from external services Label based security is the tool of choice for cluster internal access control. In order to secure access to and from external services, traditional CIDR

your datacentre, public or private. Whether you want to deploy an OpenStack cloud, a Kubernetes cluster or a 50,000-node render farm, Ubuntu Server delivers the best value scale-out performance available virtualization-multipass Multipass 4 virtualization-uvt UVtool 4 virtualization-virt-tools Virt-manager 4 virtualization-libvirt libvirt and virsh 3 Containers 2 Level Path Navlink 4 containers-lxc partitions, or even be used as part of another RAID device or LVM volume group. 25 Logical Volume Manager (LVM) The LVM is a system of managing logical volumes, or filesystems, that is much more advanced

Exported Metrics Cilium as a Kubernetes pod Cilium as a host-agent on a node Troubleshoo�ng Component & Cluster Health Connec�vity Problems Policy Troubleshoo�ng Automa�c Diagnosis Symptom Library Useful Scripts requires the firewalls on all servers to be manipulated whenever a container is started anywhere in the cluster. In order to avoid this situa�on which limits scale, Cilium assigns a security iden�ty to groups store. Secure access to and from external services Label based security is the tool of choice for cluster internal access control. In order to secure access to and from external services, tradi�onal CIDR