HBASE-21879 Read HFile ’s Block into ByteBuffer directly. 1. Background For reducing the Java GC impact to p99/p999 RPC latency, HBase 2.x has made an offheap read and write path. The KV are allocated Similarly, on the read path, we’ll try to read the BucketCache firstly, if the Cache misses, go to the HFile and read the corresponding block. The workflow: reading block from cache OR sending cells to client there should be rarely heap allocation. However, we found that the process of reading the Block from HFile is still copied to the heap firstly, the heap block won’t free unless the WriterThread of BucketCache

• 写缓存: Memstore • 写操作⽇日志: WAL • 数据⽂文件: HFile 系统组成 RegionServer 深⼊入RegionServer内部. 有两个Cache和两种⽂文件 • 先写WAL做故障恢复⽤用 • 写到Memstore中 • ⾜足够⼤大时flush到HFile • Compaction (Minor/Major) 系统组成 RegionServer写操作 就整个Region flush 这⾥里里每隔⼀一会就flush, 会⽣生成很多⼩小的HFile, HBase会执⾏行行两种compaction, minor com只是将⼩小的何为⼀一些⼤大的. major更更狠⼀一些, 合成⼀一个⽂文件 (all the HFiles in a region to one HFile per column family). 归并排序. • 先找BlockCache 根据index去HFile⾥里里找 • HBase如何在hdfs这种append- only⽂文件系统上实现, 修改/删除 操作的 系统组成 RegionServer读操作 这⾥里里需要提的⼀一点是, BlockCache⾥里里不不光对数据做了了缓存, 其实在RegionServer启动的时候, 会把所有region的索引信息加载进去. 系统组成 HFile • 整个HFile分为4⼤大部分

Abstract In Server Side ❏ High Efficiency ❏ Query Diversity ❏ Server Resource Protection HFile Response Result HFile Block-2 Block-1 Two Cases Put-1 Delete-1 Delete-2 Delete-999 …. …. Block-100 Put-2000 Response Result Put-1 Put-2000 scan.setFilter(new SingleColumnValueFilter(...)) Scan the HFile with so many deletes Read too many block data into memory, which may cause GC or OOM Server Side

Conference China 2016 中国用户大会 读写性能取舍* Postgres Conference China 2016 中国用户大会 性能优化目标 • 读 vs. 写？ – 读需要合并HFile，因此文件越少越好 – 写需要减少Compaction操作，因此文件越多越好 – 优化读或者写之一，而不是全部 • 顺序 vs. 随机？ • 参考值——每个RegionServer吞吐率>20MB/s 及访问压力（每个Region的服务量）而定 • 过多Region的症状： – CPU线程切换频繁 – 内存使用过大，造成GC频繁，服务Timeout – 每个Region的Memstore太小，磁盘flush频繁， HFile文件过多小文件 Postgres Conference China 2016 中国用户大会 Major Compaction • HBase根据时间来计划执行Major Compaction 写入非常慢，甚至慢上万倍，一般问题在服务器端： – 写入Memstore慢 • HLog写入超时——考虑HDFS及硬盘异常 • GC——考虑优化内存使用（GC参数及算法调优有限） – Flush慢 • HFile写入超时——考虑HDFS及硬盘异常 • Compaction被触发且运行时间长——优化高峰期Compaction 策略 Postgres Conference China 2016 中国用户大会

WIN32_FIND_DATA fd; // prepare srcFiles[]... bRet = TRUE; iSrcFiles = 0; hFile = FindFirstFile(SrcDir, &fd); while (hFile != INVALID_HANDLE_VALUE && bRet) { if (fd.dwFileAttributes == FILE_ATTRIBUTE_ARCHIVE) srcFiles[iSrcFiles].fd = fd; srcFiles[iSrcFiles].bIsNew = FALSE; iSrcFiles++; } 34 bRet = FindNextFile(hFile, &fd); } 再把DstDir 中的所有文件（不含子目录文件）搜寻一遍，储存在一个destFiles[ ] 数组 中，每个数组元素是一个我自定的DESTFILE 数据结构： typedef DESTFILE destFiles[FILEMAX]; WIN32_FIND_DATA fd; bRet = TRUE; iDestFiles = 0; hFile = FindFirstFile(DstDir, &fd); while (hFile != INVALID_HANDLE_VALUE && bRet) { if (fd.dwFileAttributes == FILE_ATTRIBUTE_ARCHIVE)

hour for each JVM option changed ● HBase configuration ○ 0.3 <= global memstore limit <= 0.45 ○ hfile.block.cache.size = 0.1 ○ hbase.hregion.memstore.flush.size = 256 MB ○ hbase.bucketcache.ioengine

root.u.value32 = pMap->root; data.handle.type = VMMDevHGCMParmType_64bit; data.handle.u.value64 = hFile; data.offset.type = VMMDevHGCMParmType_64bit; data.offset.u.value64 = offset; data.cb.type = VM

root.u.value32 = pMap->root; data.handle.type = VMMDevHGCMParmType_64bit; data.handle.u.value64 = hFile; data.offset.type = VMMDevHGCMParmType_64bit; data.offset.u.value64 = offset; data.cb.type = VM

root.u.value32 = pMap->root; data.handle.type = VMMDevHGCMParmType_64bit; data.handle.u.value64 = hFile; data.offset.type = VMMDevHGCMParmType_64bit; data.offset.u.value64 = offset; data.cb.type = VM

root.u.value32 = pMap->root; data.handle.type = VMMDevHGCMParmType_64bit; data.handle.u.value64 = hFile; data.offset.type = VMMDevHGCMParmType_64bit; data.offset.u.value64 = offset; data.cb.type = VM