Scaling a Multi-Tenant k8s Cluster in a Telco Pablo Moncada October 28, 2020 About MasMovil group ● 4th telecom company in Spain ● Provides voice and broadband services to +12M customers ● Several Services +3k CPU +2k Mem +5TB Nodes +300 kube-proxy replacement NetworkPolicy logging Multi-cluster DNS Aware NetworkPolicy Increased Istio security External Services TLS visibility Performance

Advanced Networking Cluster Mesh Operations Istio Concepts Component Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy Troubleshooting etcd (kvstore) Cluster Mesh Troubleshooting Symptom Guarantees API Reference Internals Hubble internals Hubble Architecture Cilium Operator Highly Available Cilium Operator CRD Registration IPAM KVStore operations Identity garbage collection CiliumEndpoint

Advanced Networking Cluster Mesh Operations Istio Concepts Component Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy Troubleshooting etcd (kvstore) Cluster Mesh Troubleshooting Symptom Guarantees API Reference Internals Hubble internals Hubble Architecture Cilium Operator Highly Available Cilium Operator CRD Registration IPAM KVStore operations Identity garbage collection CiliumEndpoint

Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs Operations Scalability report Performance Evaluation Setup Evaluation Results Tuning Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy examples: Command Reference cilium-agent cilium cilium-health cilium-operator cilium-operator-aws cilium-operator-azure cilium-operator-generic Key-Value Store Key-Value Store Layout Leases Debugging

Overview Terminology Networking Network Security eBPF Datapath Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Security Bugs Operations System Requirements Summary Scalability report Performance Evaluation Setup Evaluation Results Tuning Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy examples: Command Reference cilium-agent cilium cilium-health cilium-operator cilium-operator-aws cilium-operator-azure cilium-operator-generic Key-Value Store Key-Value Store Layout Leases Debugging

Configuration Core Agent Monitoring & Metrics Installation cilium-agent cilium-operator Troubleshooting Component & Cluster Health Connectivity Problems Policy Troubleshooting Symptom Library Useful Scripts Command examples: Kubernetes examples: Command Reference cilium-agent cilium cilium-health cilium-operator Key-Value Store Key-Value Store Layout Leases Debugging Further Reading Related Material Presentations requires the firewalls on all servers to be manipulated whenever a container is started anywhere in the cluster. In order to avoid this situation which limits scale, Cilium assigns a security identity to groups

Lifecycle Troubleshooting Monitoring & Metrics Installation cilium-agent cilium-operator Troubleshooting Component & Cluster Health Connectivity Problems Policy Troubleshooting Symptom Library Useful Scripts Command examples: Kubernetes examples: Command Reference cilium-agent cilium cilium-health cilium-operator Key-Value Store Key-Value Store Layout Leases Debugging Further Reading Related Material Presentations requires the firewalls on all servers to be manipulated whenever a container is started anywhere in the cluster. In order to avoid this situation which limits scale, Cilium assigns a security identity to groups

Exported Metrics Cilium as a Kubernetes pod Cilium as a host-agent on a node Troubleshoo�ng Component & Cluster Health Connec�vity Problems Policy Troubleshoo�ng Automa�c Diagnosis Symptom Library Useful Scripts requires the firewalls on all servers to be manipulated whenever a container is started anywhere in the cluster. In order to avoid this situa�on which limits scale, Cilium assigns a security iden�ty to groups store. Secure access to and from external services Label based security is the tool of choice for cluster internal access control. In order to secure access to and from external services, tradi�onal CIDR

Containers Change the Game Argo CD Flux Open Cluster Management Other GitOps Tools PipeCD Keptn Pulumi Kubernetes Operator Summary 18 Chapter 3–Templating Everything refined way to manage configurations and declare the state of everything in a given Kubernetes cluster called GitOps is something that Christian embraced wholeheartedly. Fast forward again to 2020, on a source of truth. Based on a reconciliation loop, the GitOps controller makes changes to the cluster by deploying new instances, once those changes have been committed to the state store. How a deployment

count_foreach functions. • The count and count_foreach aggregate functions support optional parameters operator and pattern, which can be used to fine-tune item filtering and only count values that match given connector - statistics for connectors with the largest queue ha_status Log high availability (HA) cluster status. ha_remove_node=target Remove the high availability (HA) node specified by its name or ID Zabbix high availability mode multiple Zabbix servers are run as nodes in a cluster. While one Zabbix server in the cluster is active, others are on standby, ready to take over if necessary. 24 Switching