Async HBase Client ○ Implementation ○ Performance ● How do we tuning G1GC for HBase ○ CMS vs G1 ○ Tuning G1GC ○ G1GC in XiaoMi HBase Cluster Part-1 Async HBase Client Why Async HBase Client Stuck Availability: 66% Availability: 0% Why Async HBase Client ? ● Region Server / Master STW GC ● Slow RPC to HDFS ● Region Server Crash ● High Load ● Network Failure BTW: HBase may also suffer Part-2 HBase + G1GC Tuning CMS VS G1 OldGen GC (Before) OldGen GC (After) CMS Old Gen GC CMS VS G1 Mixed GC (Before) Mixed GC (After) G1 Old Gen GC CMS VS G1 ● STW Full GC ○ CMS can only compact

HBASE-21879 Read HFile ’s Block into ByteBuffer directly. 1. Background For reducing the Java GC impact to p99/p999 RPC latency, HBase 2.x has made an offheap read and write path. The KV are allocated Young GC​: Picture.1 QPS and latencies in 100% Get Case (5 Nodes) Picture.2 G1GC related metrics in 100% Get Case In above pictures, the p999 latency is almost the same as G1GC STW cost objects are allocated in Heap, which eventually leads to the raising Young GC pressure. 2. Basic Idea Idea to eliminate the Young GC is: ​read the block of HFile to offheap directly​. We didn’t accomplish

not exist in the set. 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 LruCache + onheap Before GC (G1) After GC (G1) Memory allocated on Java heap Free memory on Java heap BucketCache Bucket-1(2MB) Block-4 (3KB) Block-5 (3KB) BucketCache + OnHeap Before GC (G1) bucket-4KB (2MB) bucket-8KB (2MB) bucket-16KB (2MB) bucket-32KB (2MB) After GC (G1) bucket-16KB (2MB) bucket-8KB (2MB) bucket-4KB (2MB) time) , Less Mixed GC. ● JVM will never free BucketCache’s byte buffer. ● But JVM will still sweep the buffer and compact them. (old generation) BucketCache + OffHeap Before GC (G1) bucket-4KB (2MB)

到150个，集群粗略估计可按100个Region 为上限，具体视服务器能力（CPU内核） 及访问压力（每个Region的服务量）而定 • 过多Region的症状： – CPU线程切换频繁 – 内存使用过大，造成GC频繁，服务Timeout – 每个Region的Memstore太小，磁盘flush频繁， HFile文件过多小文件 Postgres Conference China 2016 中国用户大会 China 2016 中国用户大会 HBase的GC特点 • 由单个RPC带来的操作类垃圾对象是短期的 • Memstore是相对长期驻留的，按2MB为单位分配 • Blockcache是长期驻留的，按64KB为单位分配 • 如何有效的回收RPC操作带来的临时对象是HBase 的GC重点 • 不建议HBase的堆大小操作操过64GB，否则GC压 力大、执行时间太长 https://blogs org/hbase/entry/tuning_g1gc_for_your_hbase http://blog.cloudera.com/blog/2014/12/tuning-java-garbage-collection-for-hbase/ Postgres Conference China 2016 中国用户大会 G1GC -XX:+UseG1GC -XX:+UnlockExperimentalVMOptions