HBASE-21879 Read HFile ’s Block into ByteBuffer directly.

• 语言: 英语
• 格式: .pdf
• 来源: openinx.github.io

摘要

HBASE-21879 Read HFile ’s Block into ByteBuffer directly. 1. Background For reducing the Java GC impact to p99/p999 RPC latency, HBase 2.x has made an offheap read and write path. The KV are allocated from the JVM offheap, and the memory area in offheap won’t be garbage collected by JVM, so upstreams need to deallocate the memory explicitly by themself. On the write path, the request packet received from client will be allocated offheap and retained until those key values are successfully written to the WAL log and Memstore. The ConcurrentSkipListSet in Memstore does not directly store the Cell data, but reference to cells, which are encoded in multiple Chunks in MSLAB, this is easier to manage the offheap memory. Similarly, on the read path, we’ll try to read the BucketCache firstly, if the Cache misses, go to the HFile and read the corresponding block. The workflow: reading block from cache OR sending cells to client, is basically not involved in heap memory allocations. However, in our performance test at XiaoMi, we found that ​the 100% Get case is still seriously affected by Young GC​: Picture.1 QPS and latencies in 100% Get Case (5 Nodes) Picture.2 G1GC related metrics in 100% Get Case In above pictures, the p999 latency is almost the same as G1GC STW cost (~100ms). After HBASE-11425​ , almost all memory allocations should be in the offheap, there should be rarely heap allocation. However, we found that the process of reading the Block from HFile is still copied to the heap firstly, the heap block won’t free unless the WriterThread of BucketCache flushes the Block to offheap IOEngine successfully. In the pressure test, due to the large amount of data, the cache hit rate is not high (~70%), many blocks are read by disk IO, so a large number of young generation objects are allocated in Heap, which eventually leads to the raising Young GC pressure. 2. Basic Idea Idea to eliminate the Young GC is: ​read the block of HFile to offheap directly​. We didn’t accomp...