Scaling a Multi-Tenant k8s Cluster in a Telco Pablo Moncada October 28, 2020 About MasMovil group ● 4th telecom company in Spain ● Provides voice and broadband services to +12M customers ● Several

Hubble can answer questions such as: Service dependencies & communication map What services are communicating with each other? How frequently? What does the service dependency graph look like? What HTTP HTTP calls are being made? What Kafka topics does a service consume from or produce to? Network monitoring & alerting Is any network communication failing? Why is communication failing? Is it DNS? Is it requests? Application monitoring What is the rate of 5xx or 4xx HTTP response codes for a particular service or across all clusters? What is the 95th and 99th percentile latency between HTTP requests and responses

Hubble can answer questions such as: Service dependencies & communication map What services are communicating with each other? How frequently? What does the service dependency graph look like? What HTTP HTTP calls are being made? What Kafka topics does a service consume from or produce to? Network monitoring & alerting Is any network communication failing? Why is communication failing? Is it DNS? Is it requests? Application monitoring What is the rate of 5xx or 4xx HTTP response codes for a particular service or across all clusters? What is the 95th and 99th percentile latency between HTTP requests and responses

container configura�on. Why Cilium? The development of modern datacenter applica�ons has shi�ed to a service- oriented architecture o�en referred to as microservices, wherein a large applica�on is split into to transparently insert security visibility + enforcement, but does so in a way that is based on service / pod / container iden�ty (in contrast to IP address iden�fica�on in tradi�onal systems) and can requests with method GET and path /public/.* . Deny all other requests. Allow service1 to produce on Ka�a topic topic1 and service2 to consume on topic1 . Reject all other Ka�a messages. Require the HTTP

Hubble can answer questions such as: Service dependencies & communication map What services are communicating with each other? How frequently? What does the service dependency graph look like? What HTTP HTTP calls are being made? What Kafka topics does a service consume from or produce to? Network monitoring & alerting Is any network communication failing? Why is communication failing? Is it DNS? Is it requests? Application monitoring What is the rate of 5xx or 4xx HTTP response codes for a particular service or across all clusters? What is the 95th and 99th percentile latency between HTTP requests and responses

configuration. Why Cilium? The development of modern datacenter applications has shifted to a service-oriented architecture often referred to as microservices, wherein a large application is split into to transparently insert security visibility + enforcement, but does so in a way that is based on service / pod / container identity (in contrast to IP address identification in traditional systems) and requests with method GET and path /public/.*. Deny all other requests. Allow service1 to produce on Kafka topic topic1 and service2 to consume on topic1. Reject all other Kafka messages. Require the HTTP header

Hubble can answer questions such as: Service dependencies & communication map What services are communicating with each other? How frequently? What does the service dependency graph look like? What HTTP HTTP calls are being made? What Kafka topics does a service consume from or produce to? Network monitoring & alerting Is any network communication failing? Why is communication failing? Is it DNS? Is it requests? Application monitoring What is the rate of 5xx or 4xx HTTP response codes for a particular service or across all clusters? What is the 95th and 99th percentile latency between HTTP requests and responses

configuration. Why Cilium? The development of modern datacenter applications has shifted to a service-oriented architecture often referred to as microservices, wherein a large application is split into to transparently insert security visibility + enforcement, but does so in a way that is based on service / pod / container identity (in contrast to IP address identification in traditional systems) and requests with method GET and path /public/.*. Deny all other requests. Allow service1 to produce on Kafka topic topic1 and service2 to consume on topic1. Reject all other Kafka messages. Require the HTTP header

Private Cloud Service IaaS LB NAT … PaaS FaaS … Verda and XDP Based L4 Load Balancer Service • Part of our private cloud service since 2017 • 5100 private, 760 public VIPs • k8s CCM integration Routers Advertise VIP with eBGP Configure with RPC Health check daemon etc… Service Discovery Per-flow ECMP k8s CCM Frontend (dash board) To Backends User For More Information • Our motivation

请求和下游依赖，以及自身服务实例的 运行情况，进一步提升问题定位能力，通常在已经定位到某个异常节点后使用。 实例 全栈数据源，70+个告警模板开箱即用： 应用级别：Pod/Service/Deployment K8S控制面：apiserver/ETCD/Scheduler 基础设施：节点、网络、存储 云服务界别：Kafka/MySQL/Redis/ 告警 拓扑图排查 根因定位 修复 告警收敛，幸福感UP 析 面向失败、高可用设计 优化告警 主动发现 智能降噪、去重 系统性解决 系统性解决 关闭 智能告警 全栈数据源，70+个告警模板开箱即用： 应用级别：Pod/Service/Deployment K8S控制面：apiserver/ETCD/Scheduler 基础设施：节点、网络、存储 云服务界别：Kafka/MySQL/Redis/ 告警 拓扑图排查 根因定位 修复 告警收敛，幸福感UP