Scaling a Multi-Tenant k8s Cluster in a Telco Pablo Moncada October 28, 2020 About MasMovil group ● 4th telecom company in Spain ● Provides voice and broadband services to +12M customers ● Several Services +3k CPU +2k Mem +5TB Nodes +300 kube-proxy replacement NetworkPolicy logging Multi-cluster DNS Aware NetworkPolicy Increased Istio security External Services TLS visibility Performance

Advanced Networking Cluster Mesh Operations Istio Concepts Component Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy Troubleshooting etcd (kvstore) Cluster Mesh Troubleshooting Symptom clusters? What is the 95th and 99th percentile latency between HTTP requests and responses in my cluster? Which services are performing the worst? What is the latency between two services? Security observability

Advanced Networking Cluster Mesh Operations Istio Concepts Component Overview Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy Troubleshooting etcd (kvstore) Cluster Mesh Troubleshooting Symptom clusters? What is the 95th and 99th percentile latency between HTTP requests and responses in my cluster? Which services are performing the worst? What is the latency between two services? Security observability

Terminology Networking Network Security eBPF Datapath Observability Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Training Enterprise support Security Bugs Operations Scalability report Performance Evaluation Setup Evaluation Results Tuning Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy clusters? What is the 95th and 99th percentile latency between HTTP requests and responses in my cluster? Which services are performing the worst? What is the latency between two services? Security observability

Overview Terminology Networking Network Security eBPF Datapath Kubernetes Integration Multi-Cluster (Cluster Mesh) Getting Help FAQ Slack GitHub Security Bugs Operations System Requirements Summary Scalability report Performance Evaluation Setup Evaluation Results Tuning Troubleshooting Component & Cluster Health Observing Flows with Hubble Observing flows with Hubble Relay Connectivity Problems Policy clusters? What is the 95th and 99th percentile latency between HTTP requests and responses in my cluster? Which services are performing the worst? What is the latency between two services? Security observability

Agent Monitoring & Metrics Installation cilium-agent cilium-operator Troubleshooting Component & Cluster Health Connectivity Problems Policy Troubleshooting Symptom Library Useful Scripts Reporting a problem microservices. Traditional Linux network security approaches (e.g., iptables) filter on IP address and TCP/UDP ports, but IP addresses frequently churn in dynamic microservices environments. The highly volatile additional challenge is the ability to provide accurate visibility as traditional systems are using IP addresses as primary identification vehicle which may have a drastically reduced lifetime of just a

Troubleshooting Monitoring & Metrics Installation cilium-agent cilium-operator Troubleshooting Component & Cluster Health Connectivity Problems Policy Troubleshooting Symptom Library Useful Scripts Reporting a problem microservices. Traditional Linux network security approaches (e.g., iptables) filter on IP address and TCP/UDP ports, but IP addresses frequently churn in dynamic microservices environments. The highly volatile additional challenge is the ability to provide accurate visibility as traditional systems are using IP addresses as primary identification vehicle which may have a drastically reduced lifetime of just a

Exported Metrics Cilium as a Kubernetes pod Cilium as a host-agent on a node Troubleshoo�ng Component & Cluster Health Connec�vity Problems Policy Troubleshoo�ng Automa�c Diagnosis Symptom Library Useful Scripts microservices. Tradi�onal Linux network security approaches (e.g., iptables) filter on IP address and TCP/UDP ports, but IP addresses frequently churn in dynamic microservices environments. The highly vola�le An addi�onal challenge is the ability to provide accurate visibility as tradi�onal systems are using IP addresses as primary iden�fica�on vehicle which may have a dras�cally reduced life�me of just a few

What’s new in Zabbix 6.4.5 Templates New templates are available: • AWS ECS Cluster by HTTP (along with its Serverless Cluster version) • Cisco SD-WAN by HTTP • OpenStack by HTTP, which includes OpenStack manually into Zabbix. 22 Updated templates Integration with OpenShift has been added to Kubernetes cluster state by HTTP template. 15 What’s new in Zabbix 6.4.10 TimescaleDB 2.13 support Support for TimescaleDB features such as Failover Cluster Instances (FCI) and Availability Groups (AG). It is now possible to use the template for monitoring a host in cluster, standalone host and host by cluster name. A macro for

checks are picked up and performed by the discovery workers. Only those checks that have the same IP and port are scheduled sequentially because some devices may not allow concurrent connections on the connector - statistics for connectors with the largest queue ha_status Log high availability (HA) cluster status. ha_remove_node=target Remove the high availability (HA) node specified by its name or ID Zabbix high availability mode multiple Zabbix servers are run as nodes in a cluster. While one Zabbix server in the cluster is active, others are on standby, ready to take over if necessary. 24 Switching