�→ pd �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a Re- gion’s heart- beat to PD is trig- gered raftstore �→ . �→ pd �→ - �→ store �→ - �→ heartbeat �→ - �→ tick �→ "2019-03-21T14:09:32+08:00", # The starting timestamp. "last_heartbeat_ts": "2019-03-21T14:14:22.961171958+08:00", # The �→ timestamp of the last heartbeat. "uptime": "4m50.961171958s" } } ] 7.2.2 Use the sum(etcd_server_is_leader)by (instance)> 0) • Description: PD has not received a TiKV/TiFlash heartbeat for a long time (the default configura- tion is 30 minutes). • Solution: – Check whether the TiKV/TiFlash

long time, it is automatically set to a silent state, which reduces the system overhead of the heartbeat information between the Leader and the Follower. User document, #10266 38 2.2.2.4 Stability scheduling issues during scaling #3563 #10059 #10001 * Solve slow scheduling issues caused by heartbeat pressure due to traffic changes #3693 #3739 #3728 #3751 * Reduce the space discrepancies of large por- tion of tomb- stone re- quired to trig- ger RocksDB com- paction raftstore �→ . �→ pd �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a Re- gion’s heart- beat to PD is

por- tion of tomb- stone re- quired to trig- ger RocksDB com- paction raftstore �→ . �→ pd �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a Re- gion’s heart- beat to PD is is trig- gered 289 Configuration item Description raftstore �→ . �→ pd �→ - �→ store �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a store’s heart- beat to PD is trig- "2019-03-21T14:09:32+08:00", # The starting timestamp. "last_heartbeat_ts": "2019-03-21T14:14:22.961171958+08:00", # The �→ timestamp of the last heartbeat. "uptime": "4m50.961171958s" } } ] 7.2.2 Use the

�→ pd �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a Re- gion’s heart- beat to PD is trig- gered raftstore �→ . �→ pd �→ - �→ store �→ - �→ heartbeat �→ - �→ tick �→ "2019-03-21T14:09:32+08:00", # The starting timestamp. "last_heartbeat_ts": "2019-03-21T14:14:22.961171958+08:00", # The �→ timestamp of the last heartbeat. "uptime": "4m50.961171958s" } } ] 7.2.2 Use the sum(etcd_server_is_leader)by (instance)> 0) • Description: PD has not received a TiKV/TiFlash heartbeat for a long time (the default configura- tion is 30 minutes). • Solution: – Check whether the TiKV/TiFlash

�→ pd �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a Re- gion’s heart- beat to PD is trig- gered raftstore �→ . �→ pd �→ - �→ store �→ - �→ heartbeat �→ - �→ tick �→ retried multiple times. Usually, the estimated duration is 3 to 10 periods of store heartbeat (one period of store heartbeat is 10 seconds by default). After the recovery is completed, the last stage in the "2019-03-21T14:09:32+08:00", # The starting timestamp. "last_heartbeat_ts": "2019-03-21T14:14:22.961171958+08:00", # The �→ timestamp of the last heartbeat. "uptime": "4m50.961171958s" } } ] 9.2.2 Use the

�→ pd �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a Re- gion’s heart- beat to PD is trig- gered raftstore �→ . �→ pd �→ - �→ store �→ - �→ heartbeat �→ - �→ tick �→ retried multiple times. Usually, the estimated duration is 3 to 10 periods of store heartbeat (one period of store heartbeat is 10 seconds by default). After the recovery is completed, the last stage in the "2019-03-21T14:09:32+08:00", # The starting timestamp. "last_heartbeat_ts": "2019-03-21T14:14:22.961171958+08:00", # The �→ timestamp of the last heartbeat. "uptime": "4m50.961171958s" } } ] 9.2.2 Use the

#7249 @lhy1024 • Fix the issue that adding multiple TiKV nodes to a large cluster might cause TiKV heartbeat reporting to become slow or stuck #7248 @rleungx • TiFlash • Fix the issue that the UPPER() and por- tion of tomb- stone re- quired to trig- ger RocksDB com- paction raftstore �→ . �→ pd �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a Re- gion’s heart- beat to PD is is trig- gered 1006 Configuration item Description raftstore �→ . �→ pd �→ - �→ store �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a store’s heart- beat to PD is trig-

por- tion of tomb- stone re- quired to trig- ger RocksDB com- paction raftstore �→ . �→ pd �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a Re- gion’s heart- beat to PD is is trig- gered 1020 Configuration item Description raftstore �→ . �→ pd �→ - �→ store �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a store’s heart- beat to PD is trig- retried multiple times. Usually, the estimated duration is 3 to 10 periods of store heartbeat (one period of store heartbeat is 10 seconds by default). After the recovery is completed, the last stage in the

por- tion of tomb- stone re- quired to trig- ger RocksDB com- paction raftstore �→ . �→ pd �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a Re- gion’s heart- beat to PD is is trig- gered 1004 Configuration item Description raftstore �→ . �→ pd �→ - �→ store �→ - �→ heartbeat �→ - �→ tick �→ - �→ interval �→ The time inter- val at which a store’s heart- beat to PD is trig- retried multiple times. Usually, the estimated duration is 3 to 10 periods of store heartbeat (one period of store heartbeat is 10 seconds by default). After the recovery is completed, the last stage in the

determining the data loading behavior through write cf stats #16245 @Connor1996 • Check the latest heartbeat for nodes being deleted and voters being demoted during the Raft conf change process to ensure that availability when using etcd watch #7738 #7724 #7689 @lhy1024 • Add more monitoring metrics for heartbeat to better analyze performance bottle- necks #7868 @nolouch • Reduce the impact of the etcd leader issue that the entire Region becomes unavailable when one replica is offline, by checking the last heartbeat time of all replicas of the Region before taking a node offline #16465 @tonyxuqqi • Fix the issue