进程自动退出排查经验分享 78 1.8 RocketMQ 主题扩分片后遇到的坑 82 1.9 RocketMQ 消息发送 system busy、broker busy 原因分析与解决方案坑 91 1.10 再谈 RocketMQ broker busy 104 1.11 从年末生产故障解锁 RocketMQ 集群部署的最佳实践 108 1.12 RocketMQ 一行代码造成大量消息丢失 Nameserver Nameserver 集群，topic 的路由注册中心，为客户端根据 Topic 提供路由服务，从 而引导客户端向 Broker 发送消息。Nameserver 之间的节点不通信。路由信息在 Nameserver 集群中数据一致性采取的最终一致性。 2. Broker 消息存储服务器，分为两种角色：Master 与 Slave，上图中呈现的就是 2 主 2 从的部 署架构，在 RocketMQ RocketMQ 中，主服务承担读写操作，从服务器作为一个备份，当主服务器存 在压力时，从服务器可以承担读服务（消息消费）。所有 Broker，包含 Slave 服务器每隔 30s 会向 Nameserver 发送心跳包，心跳包中会包含存在在 Broker 上所有的 topic 的路 由信息。 本文来自『中间件兴趣圈』公众号，仅作技术交流，未授权任何商业行为。 11 > 1.1 RocketMQ

architecture� 典型应⽤用场景 ⾦金金融交易易 电⼦子商务 智能制造 分布式事务 异步解耦 IoT/IIoT 决策分析 实时计算 概念模型 Broker A Producer A Topic A Broker B Topic B ConsumerGroupA ConsumerGroupB Consumer 1 Consumer 2 Producer B Consumer Consumer 1 Consumer 2 MessageQueue1 MessageQueue2 MessageQueue1 MessageQueue2 部署模型 组件细节 Broker Netty RemotingServer Netty Remoting Client NameServer All Info In Memory except KV config info RocketMQ 存储设计 存储⽂文件 - - - - - 1G - - - - - 6M - - - - - 400M 配置和状态相关 消息存储和检索相关 存储设计 producer message(topic, queueId, tags, keys, msgBody) commitlog 8 bytes | 4 bytes | 8 bytes commitLogOffset

designed for B2B Trading and other scenarios MetaQ v1.0 Designed for ordered message, especially for message accumulation Aliware MQ v1.0 Notify v3.0 MetaQ v2.0 v3.0 RocketMQ v3.0 and Non-Ordered message  Broadcast message  Scheduled message  Transactional message  Batched message  One-way message  Traceable message  Retroactive message  Message Filtering  Massive e G r o u p Console Scheduled Message © 2 0 1 7 A l i b a b a M i d d l e w a r e G r o u p Transactional Message – Eventually Consistent © 2 0 1 7 A l i b a b a M i

吞吐、高可靠、低延迟方面的要求。 目前RocketMQ主要由NameServer、Broker、Producer以及Consumer四部分构成，如下图所示。 原文链接：Apache RocketMQ 介绍 NameServer以轻量级的方式提供服务发现和路由功能，每个NameServer存有全量的路由信息，提 对等的读写服务，支持快速扩缩容。 Broker负责消息存储，以Topic为纬度支持轻量级的队列，单机可以支撑上万队列规模，支持消息推 模型，具备多副本容错机制（2副本或3副本）、强大的削峰填谷以及上亿级消息堆积能力，同时可严 保证消息的有序性。除此之外，Broker还提供了同城异地容灾能力，丰富的Metrics统计以及告警机 。这些都是传统消息系统无法比拟的。 Producer由用户进行分布式部署，消息由Producer通过多种负载均衡模式发送到Broker集群，发送 延时，支持快速失败。 Consumer也由用户部署，支持PUSH和PULL两种消费 RocketMQ单机最高支持5万个队列，且负载不会发生明显递增。 ● RocketMQ支持消费失败重试，这个特性非常适合运用在充值方面的应用。 ● RocketMQ支持严格的意义上的消息顺序。即时在一台Broker宕机后，消息发送会失败，但是不会 序。 ● RocketMQ支持定时消息 ● RocketMQ支持分布式事务消息 ● RocketMQ支持根据消息标识或内容查询。 ● RocketMQ支

产环境平台和实践 中，进行快速的迭 代和更新 Application Transformation • 基于云原生的架构 • 支持应用层面的创新、扩展、 弹性和生态 RDBMS Message Broker Memory Cache massively parallel data warehouse Data Services Management On-demand AI 分布式内存计算网格 所需而变得的人工智能与数据科学 • 结对编程 • 敏捷交付 VMware Data Services Manager Control Plane RDBMS Message Broker Memory Cache massively parallel data warehouse Data Management On-demand AI w/ Data Scientists Data Science Services On-demand AI with Data Science Application Transformation RabbitMQ® Message Broker & Log Streaming Greenplum® Data Warehouse & Big Data Platform 数字化转型过程中的数据流转 GemFire® In-Memory

数据的根节点。Doris 将通过 json_root 抽取根节点的元素 进行解析。默认为空。 数据源的类型。当前支持： 支持如下数据源属性： 1. Kafka 的 broker 连接信息。格式为 ip:host。多个broker之间以逗号分隔。 2. 指定要订阅的 Kafka 的 topic。 3. 指定需要订阅的 kafka partition，以及对应的每个 partition data_source_properties data_source_properties kafka_broker_list kafka_broker_list "kafka_broker_list" = "broker1:9092,broker2:9092" "kafka_broker_list" = "broker1:9092,broker2:9092" kafka_topic kafka_topic "kafka_topic" FROM FROM KAFKA KAFKA (( "kafka_broker_list" "kafka_broker_list" == "broker1:9092,broker2:9092,broker3:9092" "broker1:9092,broker2:9092,broker3:9092",, "kafka_topic" "kafka_topic"

Maintain persistent state Machine. • Maintain ephemeral broker status. • Notification mechanism when there is a change. Controller provides: • Watch broker status’s change. • Handle state machine transition transition and push new state to ZK. Report status Change status Watch state machine Broker works as: 1.4万亿 RocketMQ 多副本高可靠 Variables Values & Description MTBF of Disk(Hours) 1200000, from Seagate(希捷) Latency Message Trace MQTT/REST Proxy Relay Service Transaction Message Kafka Service Apache RocketMQ SDK(Java/C++/.Net/PHP/Go) | Console | JMS | HTTP | Schedule Message Order Message File

Maintain persistent state Machine. • Maintain ephemeral broker status. • Notification mechanism when there is a change. Controller provides: • Watch broker status’s change. • Handle state machine transition transition and push new state to ZK. Report status Change status Watch state machine Broker works as: 1.4万亿 RocketMQ 多副本高可靠 Variables Values & Description MTBF of Disk(Hours) 1200000, from Seagate(希捷) Latency Message Trace MQTT/REST Proxy Relay Service Transaction Message Kafka Service Apache RocketMQ SDK(Java/C++/.Net/PHP/Go) | Console | JMS | HTTP | Schedule Message Order Message File

this page, most of which coming from the project source code (like context, comments or where the message is being used). Translation fields for any secondary languages translators select in the preferences contextual description provides related info about the current string. String attributes Things like message ID, context (msgctxt) or location in source code. 截图 Screenshots can be uploaded to Weblate to in a similar context and prove useful in keeping the translation consistent. 其它的出现位置 In case a message appears in multiple places (e.g. multiple components), this tab shows all of them if they are found

this page, most of which coming from the project source code (like context, comments or where the message is being used). Translation fields for any secondary languages translators select in the preferences contextual description provides related info about the current string. String attributes Things like message ID, context (msgctxt) or location in source code. 截图 Screenshots can be uploaded to Weblate to in a similar context and prove useful in keeping the translation consistent. 其它的出现位置 In case a message appears in multiple places (e.g. multiple components), this tab shows all of them if they are found