Sebastian Toader & Zsolt Varga 2021-Feb-26 Apache Kafka with Istio on K8s 2 • Scalability • Resiliency • Security • Observability • Disaster recovery Production grade Apache Kafka on Kubernetes certificate attached automatically by Istio Proxy sidecar container • Client certificate includes the K8s service account of the Kafka client application • SPIFE:///ns//sa/

0 码力 | 14 页 | 875.99 KB | 1 年前

3

External IPs #IstioCon V1.1 ServiceEntry #IstioCon V1.6-1.8 Better VM Workload Abstraction A K8s Service and Pods Two separate object with distinct lifecycles Before Workload Entry, a single Istio WorkloadEntry Component Deployment WorkloadGroup Service registry and discovery Service ServiceEntry K8s Pods labels: app: foo class: pod ServiceEntry selector: app: foo Istio Workload Entries labels: ■ based on a platform-specific identity ■ w/o a platform-specific identity ● using a short-lived K8s service account token ● Automatic certificate rotation ● Validation of the proxy’s status for VM-based

to connect, manage, and secure microservices. Istio项目 微服务角度看Istio: 治理形态的演变 Node 1 svc1 自身业务 SDK Sidecar 服务治理 Node 2 svc 2 自身业务 SDK Sidecar 服务治理 通信基础 服务发现 负载均衡 熔断容错 动态路由 … for (封装++) { 应用侵入--； 应用侵入--； 治理位置--； } 微服务角度看Istio: 服务网格 服务网格控制面 从基础设施(Kubernetes)看Istio: 服务访问 Node svca svcc svcb.ns svcc.ns svcb svcd svce svce.ns svcd.ns svcd.ns Kube-proxy Kube-APIServer ServiceIp Backend Istio & Kubernetes：架构结合 Kube-APIServer Etcd istioctl / kubectl Pilot Envoy SVC Pod Node Envoy SVC Pod Node Envoy SVC Pod list/watch (Service, Endpoints, Pod) 用户 Istio & Kubernetes：统一服务发现

connect, manage, and secure microservices.4 Istio项目5 微服务角度看Istio: 治理形态的演变 Node 1 svc1 自身业务 SDK Sidecar 服务治理 Node 2 svc 2 自身业务 SDK Sidecar 服务治理 通信基础 服务发现 负载均衡 熔断容错 动态路由 … for (封装++) { 应用侵入--； 应用侵入--； 治理位置--； }6 微服务角度看Istio: 服务网格 服务网格控制面7 从基础设施(Kubernetes)看Istio: 服务访问 Node svca svcc svcb.n s svcc.ns svcb svcd svce svce.n s svcd.n s svcd.n s Kube-proxy Kube-APIServer ServiceIp Istio & Kubernetes：架构结合 Kube-APIServer Etcd istioctl / kubectl Pilot Envoy SVC Pod Node Envoy SVC Pod Node Envoy SVC Pod list/watch (Service, Endpoints, Pod) 用户13 Istio & Kubernetes：统一服务发现

ü抢救机制触发 ü抢救令牌减少一个（总共10个） ü在2(n-1) * 200毫秒后执行（为什么不立即执行） ü失败再次触发抢救机制 ü10个令牌用完，没有抢救成功，放弃退出优雅关闭envoy ü K8s发送SIGTERM信号让容器优雅关闭 ü Pilot-Agent接收信号通过context关闭子服务，发送SIGKILL关闭envoy ü Envoy不支持优雅关闭，需要通过金丝雀或蓝绿部署方式实现 和Google强强联手 官方博客：The universal data plane API缓存Istio和k8s配置 ü一个小型的非持久性key/value数据库 ü借助k8s.io/client-go建立缓存 ü缓存Istio：route-rule，virtual-service，gateway等 ü缓存k8s：node，Service，Endpoints等触发配置生效方式 V2通过GRPC双向流，主动推送配置给envoy： u证书生成 u证书挂载 u证书过期证书生成 ü生成root-cert.pem ü生成cert-chain.pem ü生成key.pem证书挂载 üICA以Name为istio.default在k8s创建Secrets对象 ü应用服务获取Secrets对象证书，并挂载到/etc/certs • volumeMounts: • - mountPath: /etc/certs/ • name: istio-certs

platform has multiple shard k8s clusters, each cluster should support 1000 sequential (interval 5s) Knative service provisionings with route ready time <= 30s. Type Info K8s Cluster Capacity 12 nodes template. Mitigations: o When adding new worker node, make sure daemonset pod of istio CNI plugin is up and running before knative pods scheduling on the node. o Crontab job could help to detect whether

3 FuzzReadCACert istio.io/istio/security/pkg/ k8s/chiron https://github.com/istio/istio/blob/6 5478ea81272c0ceaab568974aff7 00aef907312/security/pkg/k8s/chir on/fuzz_test.go#L22 4 FuzzIstioCASign istio happened. https://github.com/is tio/istio/blob/a27511 3235b95a10ace56b 8bef5d69278513bcc 1/security/pkg/node agent/caclient/provi ders/google/client.g o#L124 func (cl *googleCAClient) Close() { if cl.conn } https://github.com/is tio/istio/blob/d0705cf 0ed5591cc26c0800 1f3faab0a880aec48/ security/pkg/k8s/chir on/utils.go#L168 conn, err := net.DialTimeout("tcp", addr, 1*time.Second) if err != nil { log

Application Deployment: Cloud Layout ● Multiple K8s Clusters in an AZ ○ Each K8s cluster ~ 200 - 5,000 nodes ○ Upto 100,000 Pods in a cluster ○ 10,000+ K8s services - including prod, pre-prod, staging worst-case scenario Region R1 AZ 1 AZ 2 AZ n Data Center DC1 K8s Cluster K8s Cluster K8s Cluster K8s Cluster K8s Cluster K8s Cluster Region Rn #IstioCon Application Specs Region services - Global IPAM, Access-control Policy store, etc. ● AZ Control Plane ○ Syncs specs to workload K8s clusters in the AZ ○ Shared-Nothing Architecture ■ Hosts services catering to the AZ, e.g., AZ

Prevention Certificate Authority K8s Network Policy K8s RBAC Audit Logging Image Verification Admission Control Workload Identity K8s RBAC K8s CNI AuthZ Policy Peer AuthN Policy access control Service 2 Service 1 1. Ensure traffic is natively encrypted, such as HTTPS 3. use k8s network policies to limit traffic bypassing sidecars Cluster security best practices: safely handle & authz policies Namespace bar 2. Enforce k8s RBAC policies: roles bound to namespace, only mesh admins are allowed to have ClusterRole. 1. Use k8s network policies to limit the traffic in & out

自动维护mTLS密钥 • Mixer: 在k8s中部署了两组Mixer • Policy提供授权、Quota等能力 • Telemetry提供监控数据收集能力 基本原理 • Istio从架构上可以分为4个板块： • Istio Proxy: Mesh的基础 • 网络安全：兼容Spiffe标准实现 • 配置管理：为C++实现的Proxy接 入k8s的动态配置管理 • Attribute 情： • Sidecar Injection: 注入initContainer, Sidecar, istio-certs volume • Citadel: 自动刷新secrets, k8s自动加 载istio-secrets volume • Pilot: 和Sidecar建立连接，管理动态配 置 • Mixer: 和Sidecar建立连接，管理授权 、Quota和审计数据 FreeWheel的Istio实践 • 通过在Sidecar中增加FreeWheel自定义认证支持，下游可以充分利用 Istio提供的授权、限流、监控接口，不过要注意Sidecar也有一些小坑 ： • Sidecar没有k8s自动注入的secret，也无法通过容器内环境变量自 动建立master连接，需要管理额外的kubeconfig • Sidecar内的服务流量默认是不被劫持的，如果需要劫持需要添加额 外的annotation