# AMTHEADENSE ## TOPIC THROWBACK Vote on your favorites and those you would love to hear again!

0 码力 | 1 页 | 4.74 MB | 1 年前

3

Topic Throwback Vote Tally

Date	Speaker	Topic	Mark your vote here!!!

Date	Speaker	Topic	Nov 2015 Vote Tally
Feb 2014	Mike Ballou	Agile Principles	0
	Nick Tuck	XP r>	Jerry Koske	The Process of Innovation	2

|Speaker|Topic|Nov 2015 Vote Tally| |---|---|---| |Darin Plum|Agile Coaching|5| |Shawn Stumme|Continuous Delivery|4|

## 使用gRPC go实现 基于Topic的高效消息订阅发布模型  姓名：___ 张凯 中国电子云  ## 基于Topic的消息发布模型简介 service PubSub { message PublishRequest { string topic = 1; bytes payload = 2; } // SubscribeRequest is passed when subscribing message SubscribeRequest { string topic = 1; } // SubscribeResponse object message

Consumer 分布式的流式数据平台 高吞吐 容错性 Producer Topic Topic Topic Consumer ## Kafka 在知乎的应用 平台承载知乎业务日志、数据传输和消息队列服务 平台线上稳定运行 基于 Kubernetes 的 Kafka 集群 13 个，1000+ Topic 知乎技术平台重要的组件 ## 平台概览 · 多集群 · 高可用 平台 Kubernetes 问题驱动 • Kafka 资源规划不合理 • 单一集群造成系统单点 - 不区分集群和 Topic 等级，影响重要业务 • 业务与 Kafka 深度耦合 ## 资源规划 多 Kafka 集群方式 根据 Topic 类型划分集群 同一类型 Topic 的集群细分 - Topic 服务等级、容量和规模划分  ## 多集群问题 多变需求引发集群规模增长 • Broker, Topic 规模 服务器资源利用率 • 单机运行多 Broker 方式  Server

进入某个会话进行数据漫游 3，sync_inbox只存储最近30天数据 会话视图表 ## 电商消息系统隔离和流控 IM消息系统 群内扩散 顺序发送消息 单聊扩散 按用户分库分表 单个消息队列 topic队列=6144 0 1 2 3 4 5 队列路由选择 消息队列 1. 租户资源路由(例：千牛订单提醒敏感) 2. 业务场景优先级和流控(例：单聊和群聊分开) 3. 错误反馈调节消费速率 pageSize + index 未用令牌 未用令牌 使用令牌 使用令牌 Topic1 Topic2 消息队列集群-IM Queue1 Topic1 Queue2 QueueN Queue1 Queue2 Topic-N QueueN ## 消息队列集群-消息号 Queue1 Topic1 TopicN ## 独立和共享令牌机制 SUM 未用令牌 使用令牌 使用令牌 流控器 RateLimiter TopicN 处理消息 共享 数据处理 Sync同步应用 多topic消费 并发批量处理 租户+dataType 线程池 手淘 千牛 共享 ## 错误反馈流控 调节 调节流控速率 监测 反馈错误 处理错误反馈 ## 电商消息全链路稳定性保障 ## 电商消息全链路保障体系 Ali-Metric 多维度聚合 Tracing

The maximum number of separators / in the publish and subscribe topics is no more than 8, and the topic name can be up to 255 characters in length. • The maximum length of the message message is 32k. The used for topic authentication. If password is not empty, it will further check if the password is correct. The Hub supports simple topic routing, such as subscribing to a message with the topic t and publishing publishing it back with a new topic t/topic. If this module does not meet your requirements, you can also use a third-party MQTT Broker/Server to replace it. #### 2.4.3 baetyl-function-manager The b

The maximum number of separators / in the publish and subscribe topics is no more than 8, and the topic name can be up to 255 characters in length. - The maximum length of the message message is 32k. The used for topic authentication. If password is not empty, it will further check if the password is correct. The Hub supports simple topic routing, such as subscribing to a message with the topic t and publishing publishing it back with a new topic t/topic. If this module does not meet your requirements, you can also use a third-party MQTT Broker/Server to replace it. ## baetyl-function-manager The baetyl-function-manager

The maximum number of separators / in the publish and subscribe topics is no more than 8, and the topic name can be up to 255 characters in length. • The maximum length of the message message is 32k. The used for topic authentication. If password is not empty, it will further check if the password is correct. The Hub supports simple topic routing, such as subscribing to a message with the topic t and publishing publishing it back with a new topic t/topic. If this module does not meet your requirements, you can also use a third-party MQTT Broker/Server to replace it. #### 2.4.3 baetyl-function-manager The b

The maximum number of separators / in the publish and subscribe topics is no more than 8, and the topic name can be up to 255 characters in length. - The maximum length of the message message is 32k. The used for topic authentication. If password is not empty, it will further check if the password is correct. The Hub supports simple topic routing, such as subscribing to a message with the topic t and publishing publishing it back with a new topic t/topic. If this module does not meet your requirements, you can also use a third-party MQTT Broker/Server to replace it. ## baetyl-function-manager The baetyl-function-manager

f0f3315cde95e34f073/p10_1.jpg) 在 RocketMQ 主要的组件如下: ### 1. Nameserver Nameserver 集群，topic 的路由注册中心，为客户端根据 Topic 提供路由服务，从而引导客户端向 Broker 发送消息。Nameserver 之间的节点不通信。路由信息在 Nameserver 集群中数据一致性采取的最终一致性。 ### 上所有的 topic 的路由信息。 ### 3. Client 消息客户端，包括 Producer(消息发送者)和 Consumer(消费消费者). 客户端在同一时间只会连接一台 nameserver，只有在连接出现异常时才会向尝试连接另外一台。客户端每隔 30s 向 Nameserver 发起 topic 的路由信息查询。 温馨提示: Nameserver 是在内存中存储 Topic 的路由信息 的路由信息, 持久化 Topic 路由信息的地方是在 Broker 中, 即$\{$ ROCKETMQ_HOME$\}$/store/config/topics.json。 在 RocketMQ4.5.0 版本后引入了多副本机制，即一个复制组（m-s）可以演变为基于 raft 协议的复制组，复制组内部使用 raft 协议保证 broker 节点数据的强一致性，该部署架构在金融行业用的比较多。 ## 二、