## Apache Kafka with Istio on K8s Sebastian Toader & Zsolt Varga 2021-Feb-26 ## Production grade Apache Kafka on Kubernetes • Scalability • Resiliency • Security • Observability • Disaster recovery authn/authz • Secure cross-cluster interaction between client apps and Kafka ## Challenges - Kafka broker SSL with client auth • Kafka brokers require private-key and certificate pairs • Private keys and container • Each Kafka client request gets a client certificate attached automatically by Istio Proxy sidecar container • Client certificate includes the K8s service account of the Kafka client application

Data Stream Processing and Analytics Spring 2020 ## 1 /30: Introduction to Apache Flink and Apache Kafka Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Apache Flink • An open-source, distributed data analysis analysis framework • True streaming at its core • Streaming & Batch API Event logs Kafka, RabbitMQ, ... Historic data HDFS, JDBC, ... ![Image](/uploads/documents/8/6/d/0/86d061c164744c6ce188357a43cc9699/p2_1 https://flink.apache.org/community.html#mailing-lists • Conference • http://flink-forward.org/ ## APACHE kafka $ ^{®} $ A distributed streaming platform A distributed and fault-tolerant publish-subscribe messaging

Kubernetes 的 Kafka 平台 探索和实践 知乎 白瑜庆 ## 自我介绍 知乎技术平台工程师 负责 Kafka 和数据库平台 曾在新浪和金山云负责镜像流量分析项目 ## 纲要 Kafka 在知乎的应用 为什么做基于 Kubernetes 的 Kafka 平台 基于 Kubernetes 的 Kafka 平台实践 ## Apache Kafka Producer Topic Topic Consumer ## Kafka 在知乎的应用 平台承载知乎业务日志、数据传输和消息队列服务 平台线上稳定运行 基于 Kubernetes 的 Kafka 集群 13 个，1000+ Topic 知乎技术平台重要的组件 ## 平台概览 · 多集群 · 高可用 平台 API Kafka 客户端 监控 多 Kafka 集群 Kubernetes 服务 注册 注册 ## 为什么采用 Kubernetes 问题驱动 • Kafka 资源规划不合理 • 单一集群造成系统单点 - 不区分集群和 Topic 等级，影响重要业务 • 业务与 Kafka 深度耦合 ## 资源规划 多 Kafka 集群方式 根据 Topic 类型划分集群 同一类型 Topic 的集群细分 - Topic 服务等级、容量和规模划分 ![Image](/uploads/d

[Image](/uploads/documents/0/1/a/3/01a3b6e6a4062a56456d19613708a3c2/p13_2.jpg) ## 交易系统架构设计-redis stream • Kafka-like • 一个订阅者的情况，消息有序 • Consumer group负载均衡（不保证消息顺序） • 可以批量处理消息 - 在同一个事务中存储处理结果和更新消费offset $ redis-cli

s/2/8/8/5/2885b429b889e942d6d827594bab5473/p5_1.jpg) Apache Flink: flink.apache.org Apache Kafka: kafka.apache.org  features and processing guarantees of streaming systems • be proficient in using Apache Flink and Kafka to build end-to-end, scalable, and reliable streaming applications • have a solid understanding of fundamentals||| |01/28|Stream ingestion and pub/sub systems||| |01/30|Introduction to Apache Flink and Apache Kafka||Assignment #1 available| |02/04|Streaming languages and operator semantics||Quiz #1| |02/06|Notions

Apache Kafka Support 344 10.3.1. Sending a Message 345 10.3.2. Receiving a Message 346 10.3.3. Kafka Streams 347 10.3.4. Additional Kafka Properties 348 10.3.5. Testing with Embedded Kafka 350 Repository Metrics ..... 454 RabbitMQ Metrics ..... 455 Spring Integration Metrics ..... 455 Kafka Metrics ..... 455 MongoDB Metrics ..... 455 Jetty Metrics ..... 458 @Timed Annotation Support and Receiving messages through JMS • AMQP: Auto-configuration for RabbitMQ • Kafka: Auto-configuration for Spring Kafka • RSocket: Auto-configuration for Spring Framework’s RSocket Support • Spring

359 10.3. Apache Kafka Support ..... 362 10.3.1. Sending a Message ..... 363 10.3.2. Receiving a Message ..... 364 10.3.3. Kafka Streams ..... 365 10.3.4. Additional Kafka Properties ..... 366 366 10.3.5. Testing with Embedded Kafka ..... 368 10.4. RSocket ..... 370 10.4.1. RSocket Strategies Auto-configuration ..... 370 10.4.2. RSocket server Auto-configuration ..... 370 10.4.3. Spring Repository Metrics ..... 473 RabbitMQ Metrics ..... 473 Spring Integration Metrics ..... 473 Kafka Metrics ..... 473 MongoDB Metrics ..... 473 Jetty Metrics ..... 477 @Timed Annotation Support

355 10.3. Apache Kafka Support ..... 358 10.3.1. Sending a Message ..... 359 10.3.2. Receiving a Message ..... 360 10.3.3. Kafka Streams ..... 361 10.3.4. Additional Kafka Properties ..... 362 362 10.3.5. Testing with Embedded Kafka ..... 364 10.4.RSocket 366 10.4.1.RSocket Strategies Auto-configuration 366 10.4.2.RSocket server Auto-configuration 366 10.4.3.Spring Messaging RSocket support Repository Metrics ..... 469 RabbitMQ Metrics ..... 469 Spring Integration Metrics ..... 469 Kafka Metrics ..... 469 MongoDB Metrics ..... 469 Jetty Metrics ..... 473 @Timed Annotation Support

353 10.3. Apache Kafka Support ..... 356 10.3.1. Sending a Message ..... 357 10.3.2. Receiving a Message ..... 358 10.3.3. Kafka Streams ..... 359 10.3.4. Additional Kafka Properties ..... 360 360 10.3.5. Testing with Embedded Kafka ..... 362 10.4. RSocket ..... 364 10.4.1. RSocket Strategies Auto-configuration ..... 364 10.4.2. RSocket server Auto-configuration ..... 364 10.4.3. Spring Repository Metrics ..... 468 RabbitMQ Metrics ..... 469 Spring Integration Metrics ..... 469 Kafka Metrics ..... 469 MongoDB Metrics ..... 469 Jetty Metrics ..... 472 @Timed Annotation Support

364 10.3. Apache Kafka Support ..... 367 10.3.1. Sending a Message ..... 368 10.3.2. Receiving a Message ..... 369 10.3.3. Kafka Streams ..... 370 10.3.4. Additional Kafka Properties ..... 371 371 10.3.5. Testing with Embedded Kafka ..... 373 10.4. RSocket ..... 375 10.4.1. RSocket Strategies Auto-configuration ..... 375 10.4.2. RSocket server Auto-configuration ..... 375 10.4.3. Spring Repository Metrics ..... 480 RabbitMQ Metrics ..... 480 Spring Integration Metrics ..... 480 Kafka Metrics ..... 481 MongoDB Metrics ..... 481 Jetty Metrics ..... 484 @Timed Annotation Support