Scaling a Multi-Tenant k8s Cluster in a Telco Pablo Moncada October 28, 2020 About MasMovil group ● 4th telecom company in Spain ● Provides voice and broadband services to +12M customers ● Several

KCM， etcd，api-server, coredns… 系统调用异常：网络请 求，内存申请，文件操 作，CGroup… 内核异常：进程调度， 内存管理，文件管理， 夯机宕机，资源异 常… 应用组件异常：线程池满，数据库连接无法获取， OOM，文件读取错误… 无法自顶向下端到端 串联导致棘手问题频 发。 Kubernetes下的可观测 Golang + eBPF实现数据采 ✅ perf ✅ … eBPF的编程实践 bcc libbpf + bpf + core 编程  bcc 依靠运行时汇编，将整个大型LLVM/Clang 库带入并嵌入其中  编译过程中资源用量大，对Cpu、Mem有要求  依赖内核的头包  bpf 程序跟其他的用户空间的程序没有太大区别  编译成二进制文件，可以适应不同运行环境  libbpf 扮演bpf程序装载机角色 自身服务实例的 运行情况，进一步提升问题定位能力，通常在已经定位到某个异常节点后使用。 实例 全栈数据源，70+个告警模板开箱即用： 应用级别：Pod/Service/Deployment K8S控制面：apiserver/ETCD/Scheduler 基础设施：节点、网络、存储 云服务界别：Kafka/MySQL/Redis/ 告警 拓扑图排查 根因定位 修复 告警收敛，幸福感UP

10 ，cilium github 项目已有 9.3K star，Contributors 316位 cilium的特色功能： • 网络功能 • 负载均衡 • 网络安全 • 可观察性 • 多集群连通 注：本 PPT 基于 cilium v1.10.4 进行分析 ��������������� ��������������� �������������������� �������������������� 、 sendmsg、 recvmsg 、getpeername 、 bind等系统调用，实现 service 的地址解 析，并且伪装通信目的地址，让上层应用 无感知 DNAT 的发生 效果： • 集群内访问nodePort、LoadBalancer 的service时，能够减少数据包转发跳 数，极大提高网络性能 • 相比传统 iptables 等 技术，降低了访 问延时。例如在相同环境下，service nodePort 访问 cilium 借助 eBPF 程序 ，能快速完 成 nodePort 、 LoadBalancer service 的解析和转发，其转发性能能比肩 DPDK 技术，且能节省大量CPU资源 当 PPS 压力越大，提升效果越发显 著，相比 kube-proxy，测量得出以下 效果： 1. TC 转发方式，在10Mpps input压 力下提升 1 倍的吞吐量，在2Mpps 压力下，节省了30%的CPU利用率

contact k8s api-server In the Cilium agent logs you will see: level=info msg="Establishing connection to apiserver" host="https://10.96.0.1:443" subsys=k8s level=error msg="Unable to contact k8s api-server" ipAddr="https://10.96.0.1:443" subsys=k8s level=fatal msg="Unable to initialize Kubernetes subsystem" error="unable to create k8s client: unable to create k8s client: Get https://10.96.0.1:443/api/ Important Please ensure that you are running version 1.7.9 [https://github.com/aws/amazon-vpc-cni- k8s/releases/tag/v1.7.9] or newer of the AWS VPC CNI plugin to guarantee compatibility with Cilium. The

The DaemonSet will automa�cally install itself as Kubernetes CNI plugin. K8s 1.15 K8s 1.14 K8s 1.13 K8s 1.12 K8s 1.11 K8s 1.10 kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1 The DaemonSet will automa�cally install itself as Kubernetes CNI plugin. K8s 1.15 K8s 1.14 K8s 1.13 K8s 1.12 K8s 1.11 K8s 1.10 kubectl create -f https://raw.githubusercontent.com/cilium/cilium/v1 during the bootstrapping phase of Cilium. For Docker as container runtime: K8s 1.15 K8s 1.14 K8s 1.13 K8s 1.12 K8s 1.11 K8s 1.10 kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/v1

contact k8s api-server In the Cilum agent logs you will see: level=info msg="Establishing connection to apiserver" host="https://10.96.0.1:443" subsys=k8s level=error msg="Unable to contact k8s api-server" ipAddr="https://10.96.0.1:443" subsys=k8s level=fatal msg="Unable to initialize Kubernetes subsystem" error="unable to create k8s client: unable to create k8s client: Get https://10.96.0.1:443/api/ "ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"

contact k8s api-server In the Cilum agent logs you will see: level=info msg="Establishing connection to apiserver" host="https://10.96.0.1:443" subsys=k8s level=error msg="Unable to contact k8s api-server" ipAddr="https://10.96.0.1:443" subsys=k8s level=fatal msg="Unable to initialize Kubernetes subsystem" error="unable to create k8s client: unable to create k8s client: Get https://10.96.0.1:443/api/ "ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"

--kubernetes-version 1.10.3 : Kubernetes version that is to be installed. Please note [Kops 1.9 officially supports k8s version 1.9] --cloud-labels "Team=Dev,Owner=Admin" : Labels for your cluster ${NAME} : Name of the "ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig" Disabled 104 k8s:io.cilium.k8s.policy.cluster=default 10.15.233.139 ready k8s:io.cilium.k8s.policy.serviceaccount=coredns

contact k8s api-server In the Cilum agent logs you will see: level=info msg="Establishing connection to apiserver" host="https://10.96.0.1:443" subsys=k8s level=error msg="Unable to contact k8s api-server" ipAddr="https://10.96.0.1:443" subsys=k8s level=fatal msg="Unable to initialize Kubernetes subsystem" error="unable to create k8s client: unable to create k8s client: Get https://10.96.0.1:443/api/ "ipam": { "type": "calico-ipam" }, "policy": { "type": "k8s" }, "kubernetes": { "kubeconfig": "/etc/cni/net.d/calico-kubeconfig"

contact k8s api-server In the Cilium agent logs you will see: level=info msg="Establishing connection to apiserver" host="https://10.96.0.1:443" subsys=k8s level=error msg="Unable to contact k8s api-server" ipAddr="https://10.96.0.1:443" subsys=k8s level=fatal msg="Unable to initialize Kubernetes subsystem" error="unable to create k8s client: unable to create k8s client: Get https://10.96.0.1:443/api/ Important Please ensure that you are running version 1.7.9 [https://github.com/aws/amazon-vpc-cni- k8s/releases/tag/v1.7.9] or newer of the AWS VPC CNI plugin to guarantee compatibility with Cilium. $