#IstioCon Moving large scale consumer e-commerce Infrastructure to Mesh Rajath Ramesh Principal Software Engineer @Carousell Harshad Rotithor Software Architect @Carousell #IstioCon About Carousell

............................................................................... 35 11.3.2 针对 Consumer ............................................................................................... ...................................................................................... 42 14 Consumer 最佳实践 .......................................................................................... 产生消息。  Consumer 消息消费者，负责消费消息，一般是后台系统负责异步消费。  Push Consumer Consumer 的一种，应用通常吐 Consumer 对象注册一个 Listener 接口，一旦收到消息，Consumer 对象立 刻回调 Listener 接口方法。  Pull Consumer Consumer 的一种，应用通常主劢调用

本文来自『中间件兴趣圈』公众号，仅作技术交流，未授权任何商业行为。 11 > 1.1 RocketMQ 核心概念扫盲篇 3. Client 消息客户端，包括 Producer(消息发送者)和 Consumer(消费消费者)．客户端在同一 时间只会连接一台 nameserver，只有在连接出现异常时才会向尝试连接另外一台。客户 端每隔 30s 向 Nameserver 发起 topic 的路由信息查询。 order_consumer 来订阅 order_topic，并且基于分布式部署， order-service-app 的部署情况如下： 本文来自『中间件兴趣圈』公众号，仅作技术交流，未授权任何商业行为。 1.1 RocketMQ 核心概念扫盲篇 < 12 即 order-service-app 部署了 3 台服务器，每一个 jvm 进程可以看做是消费组 order_consumer 消费组的其中一个消费者。 ConsumerGroup: consumer-grouptest The consumerId: c onsumer-client-id-clusterA-192.168.3.122@21932 not in cidAll: [consumer-client-id-clust erA-192.168.3.123@22164] 上面的日志显示在队列负载时候，当前节点竟然不属于 consumer-grouptest

................ 12 第二章 consumer .......................................................................................................................... 14 一：consumer 启动流程 ..................... 第二章 consumer 有别于其他消息中间件由 broker 做负载均衡并主动向 consumer 投递消息，RocketMq 是基于 拉模式拉取消息，consumer 做负载均衡并通过长轮询向 broker 拉消息。 Consumer 消费拉取的消息的方式有两种 1. Push 方式：rocketmq 已经提供了很全面的实现， consumer 通过长轮询拉取消息后回调 因为绝大多数是由 push 消费方式来使用 rocketmq 的。 一：consumer 启动流程 指定 group 订阅 topic 注册消息监听处理器，当消息到来时消费消息 消费端 Start 复制订阅关系 初始化 rebalance 变量 构建

Indirectly • Producer writes to a file or database • Consumer periodically polls and retrieves new data • polling overhead, latency? • Consumer receives a notification when new data is available • Direct messaging • Direct network communication, UDP multicast, TCP • HTTP or RPC if the consumer exposes a service on the network • Failure handling: application needs to be aware of message loss message is processed only once, by a single consumer • Event retrieval is not defined by content / structure but its order • FIFO, priority producer consumer queue 6 Message brokers Message broker:

key, a value, and a timestamp. A producer publishes a stream of records to a Kafka topic and a consumer subscribes to one or more topics and processes the stream of records published in them. Topics label themselves with a consumer group name, and each record published to a topic is delivered to one consumer instance within each subscribing consumer group. Consumer instances can be in separate all the consumer instances have the same consumer group, then the records will effectively be load balanced over the consumer instances. If all the consumer instances have different consumer groups

Producer A Topic A Broker B Topic B ConsumerGroupA ConsumerGroupB Consumer 1 Consumer 2 Producer B Consumer 1 Consumer 2 MessageQueue1 MessageQueue2 MessageQueue1 MessageQueue2 部署模型 组件细节 In Memory except KV config info Netty RemotingServer topic register and heart beats Producer Consumer NRC BrokerAddrTlb send Msg topic route heatbeats MsgQueList callback/check NRC pull MsgExt | a5 - a4+1 | h3 doDispatch() Consumer config/consumerOffset.json commitOffset() ConsumerOffsetManager "offsetTable":{ "TopicTest@consumer-g1":{0:0,1:21,2:0}, ... } index

gz解压后的目录结构： . ├── LICENSE ├── NOTICE ├── README.md ├── benchmark │ ├── batchproducer.sh │ ├── consumer.sh │ ├── producer.sh │ ├── runclass.sh │ ├── shutdown.sh │ └── tproducer.sh ├── bin │ server是多台集群的话，就用分号分隔 namesrvAddr=192.168.166.20:9876;192.168.166.206:9876 # 每个topic对应队列的数量，默认为4，实际参考consumer实例的数量，值过小不利于consumer负 均衡 defaultTopicQueueNums=8 # 是否允许broker自动创建Topic，生产建议关闭 autoCreateTopicEnable=true # server是多台集群的话，就用分号分隔 namesrvAddr=192.168.166.20:9876;192.168.166.206:9876 # 每个topic对应队列的数量，默认为4，实际参考consumer实例的数量，值过小不利于consumer负 均衡 defaultTopicQueueNums=8 # 是否允许broker自动创建Topic，生产建议关闭 autoCreateTopicEnable=true #

producer, handler, consume, and synchronizer 5.4. Synchronizers and sync policy 5.5. Producer, consumer, and handlers 5.6. Listeners 6. Clustered resources 7. Cellar groups 7.1. New group 7.2. Clustered "managed" nodes). Like this, only the "manager" will send event to the nodes (which are able to consumer and handle), but no event can be produced on the nodes. 2. Installation This chapter describes -o (--obr) for OBR URLs or a given cluster group using the -g (--group) option. 5.5. Producer, consumer, and handlers To notify the other nodes in the cluster, Cellar produces a cluster event. For that

producer, handler, consume, and synchronizer 5.4. Synchronizers and sync policy 5.5. Producer, consumer, and handlers 5.6. Listeners 6. Clustered resources 7. Cellar groups 7.1. New group 7.2. Clustered "managed" nodes). Like this, only the "manager" will send event to the nodes (which are able to consumer and handle), but no event can be produced on the nodes. 2. Installation This chapter describes --obr ) for OBR URLs or a given cluster group using the -g ( --group ) option. 5.5. Producer, consumer, and handlers To notify the other nodes in the cluster, Cellar produces a cluster event. For that