LINE’s Private Cloud Yutaro Hayakawa October 28, 2020 • Messaging & many family services • 185 million global MAU • 3Tbps+ network traffic in total LINE Verda: LINE’s Private Cloud Service IaaS LB LB NAT … PaaS FaaS … Verda and XDP Based L4 Load Balancer Service • Part of our private cloud service since 2017 • 5100 private, 760 public VIPs • k8s CCM integration (Type: LoadBalancer) L4LB

commit access Granting Commit Access Revoking Commit Access Changing the Policy Voting Company Block Vote Limit Templates Nomination to Grant Commit Access Vote to Grant Commit Access Vote Results underlying networking infrastructure. This mode works well with: Native IPv6 networks In conjunction with cloud network routers If you are already running routing daemons Load Balancing Cilium implements distributed Kubernetes cluster using Google Kubernetes Engine [https://cloud.google.com/kubernetes-engine]. See Installing Google Cloud SDK [https://cloud.google.com/sdk/install] for instructions on how to install

underlying networking infrastructure. This mode works well with: Native IPv6 networks In conjunction with cloud network routers If you are already running routing daemons Load balancing Distributed load balancing ConfigMap and validate that in-addr.arpa and ip6.arpa are listed as wildcards for the kubernetes block like this: kubectl -n kube-system edit cm coredns [...] apiVersion: v1 data: Corefile: | .:53 as you scale up your clusters: etcd nodes operated by the etcd-operator will not use persistent storage. Once the etcd cluster looses quorum, the etcd cluster is automatically re-created by the cilium-etcd-operator

underlying networking infrastructure. This mode works well with: Na�ve IPv6 networks In conjunc�on with cloud network routers If you are already running rou�ng daemons Load balancing Distributed load balancing ConfigMap and validate that in-addr.arpa and ip6.arpa are listed as wildcards for the kubernetes block like this: kubectl -n kube-system edit cm coredns [...] apiVersion: v1 data: Corefile: | as you scale up your clusters: etcd nodes operated by the etcd-operator will not use persistent storage. Once the etcd cluster looses quorum, the etcd cluster is automa�cally re- created by the cilium-etcd-operator

commit access Granting Commit Access Revoking Commit Access Changing the Policy Voting Company Block Vote Limit Templates Nomination to Grant Commit Access Vote to Grant Commit Access Vote Results underlying networking infrastructure. This mode works well with: Native IPv6 networks In conjunction with cloud network routers If you are already running routing daemons Load Balancing Cilium implements distributed Kubernetes cluster using Google Kubernetes Engine [https://cloud.google.com/kubernetes-engine]. See Installing Google Cloud SDK [https://cloud.google.com/sdk/install] for instructions on how to install

underlying networking infrastructure. This mode works well with: Native IPv6 networks In conjunction with cloud network routers If you are already running routing daemons Load balancing Distributed load balancing Install Hubble Hubble is a fully distributed networking and security observability platform for cloud native workloads. It is built on top of Cilium and eBPF to enable deep visibility into the communication Install Hubble Hubble is a fully distributed networking and security observability platform for cloud native workloads. It is built on top of Cilium and eBPF to enable deep visibility into the communication

underlying networking infrastructure. This mode works well with: Native IPv6 networks In conjunction with cloud network routers If you are already running routing daemons Load Balancing Cilium implements distributed ConfigMap and validate that in-addr.arpa and ip6.arpa are listed as wildcards for the kubernetes block like this: Kubernetes 1.16+ kubectl -n kube-system edit cm coredns [...] apiVersion: v1 data: as you scale up your clusters: etcd nodes operated by the etcd-operator will not use persistent storage. Once the etcd cluster looses quorum, the etcd cluster is automatically re-created by the cilium-etcd-operator

underlying networking infrastructure. This mode works well with: Native IPv6 networks In conjunction with cloud network routers If you are already running routing daemons Load balancing Distributed load balancing Enable Hubble Hubble is a fully distributed networking and security observability platform for cloud native workloads. It is built on top of Cilium and eBPF to enable deep visibility into the communication Installation Installation with managed etcd Installation with external etcd Installation on Microsoft Azure Cloud (beta) Installation on OpenShift OKD Quick Installation This guides takes you through the quick

2020 1 The data deluge on modern storage 2 Compute node CPU Network Storage node Flash The data deluge on modern storage 3 Compute node 3 CPU Network Storage node Flash 16-lane PCIe, 16GB/s throughput gap https://cacm.acm.org/magazines/2019/6/237002-programmable-solid-state-stora ge-in-future-cloud-datacenters/fulltext?mobile=false whoami ● Giulia Frascaria ● PhD student in Vrije Universiteit research group ● Work with eBPF for storage! 5 eBPF and DoS 6 Compute node CPU Network Storage node Flash eBPF and DoS 7 Compute node CPU Network Storage node Flash DoS eBPF and DoS 8

to the motors. The engines were smokey and some bees were afraid To damage the engines or just to block a cog. “Fear not” said Captain Tux, “I will come to your aid”: He trained them and made sure they the networking stack. This enhances network performance and flexibility making it ready for the cloud-native world. Projects using eBPF for networking include Cilium and Katran, for example. System Phippy (the giraffe), and Tai (the elephant) are copy- right The Linux Foundation, on behalf of the Cloud Native Computing Foundation. They are licensed under Creative Commons Attribution 4.0 International