## HBase Read Path h openi nx@apache.org ## Abstract ☐ A i ent Si de ☐ Server Si de Tuni ng ## Part-1 Client Si de ## A i ent Scanner  and shorter STWtime. ## End-to-end off heap on the read-path (HBASE-114)  branch-1.5

## HBase Practice At Xiaomi huzheng@xiaomi.com ## About This Talk • Async HBase Client ☐ WhyAsync HBase Client ○ Implementation ☐ Performance • How do we tuning G1GC for HBase ☐ CMS vs G1 o Tuning Tuning G1GC G1GC in XiaoMi HBase Cluster ## Part-1Async HBase Client ## WhyAsync HBase Client? Blocking Client (Single Thread) ![Image](/uploads/documents/4/4/8/2/4482a28355610ca65c1d5f9c0c49bd2b/p4_2 using blocking client ## WhyAsync HBase Client? • Region Server / Master STW GC • Slow RPC to HDFS • Region Server Crash • High Load • Network Failure BTW: HBase may also suffer from fault amplification

## HBase Practice At Xi ao tianj.y1990@mail.com openi nx@apache.org ## Part-1 Problems In Practice ## Probl en s in Xi aoM ☐ Problem 1. How to satisfy the regular demand of scanning table with affecting analysis need to scan a large number of data from hbase ☐ They are executed by map reduce or spark, that put a heavy burden ## Scan snapshot directly ☐ HBase already provides this table's snapshot Input Format (C i ent Si deRegi onScanner) ☐ Construct regions by snapshot files ☐ Read data without any HBase RPC requests ☐ Required READ access to reference files and HFiles ## Snapshot ACL ☐ HDFS ACL could

## HBase基本介绍 田志鹏 20190714 今天讲的内容比较基础, 而且偏理论, 因为我个人也没有太多实际使用经验, 纸上谈兵. 上次分位点估算当时没解决的两个问题已更新ppt. ## APACHE HBASE Apache HBase $ ^{™} $ is the Hadoop database, a distributed, scalable, big data store store. Use Apache HBase $ ^{™} $ when you need random, realtime read/write access to your Big Data. This project's goal is the hosting of very large tables -- billions of rows X millions of columns -- clusters of commodity hardware. Apache HBase is an open-source, distributed, versioned, non-relational database modeled after Google's Bigtable ... 先来一段HBase官网的自我介绍. blabla翻译一下 重点看其中的红字, 什么hadoop数据库

## cloudera ## HBase最佳实践及优化 陈飚 cb@cloudera.com Cloudera ## 关于我 陈飚 Cloudera售前技术经理、资深方案架构师 http://biaobean.pro  原Intel 年后开始Hadoop 产品开发及方案顾问，先后负责Hadoop 产品化、HBase 性能调优，以及行业解决方案顾问 ## HBase的历史 HBase是Google BigTable的开源实现 • BigTable利用GFS作为其文件存储系统 • HBase使用HDFS作为其文件存储系统 2007年2月建立了HBase的原型版本 2006年 Google发表了BigTable论文 2006年底由PowerSet的Chad 2006年底由PowerSet的Chad Walters和Jim Kellerman发起了HBase项目，依据BigTable的论文重构关系数据库 2007年10月建立了第一个可用的HBase版本 2008年成为Apache Hadoop的一个子项目 ## HBase的模型特性 Hadoop database and NoSQL database • 基本的数据库操作CRUD • 强一致性

## TiDB: HBase分布式事务与SQL实现 ## About me • TiDB & Codis founder • Golang expert • Distributed database developer • Currently, CEO and co-founder of PingCAP liuqi@pingcap.com https://github.com/pingcap/tidb com/pingcap/tidb weibo: @goroutine ## Agenda • HBase introduction • TiDB features • Google percolator and omid • Internals of TiDB over HBase ## Features of HBase • Linear and modular scalability. - Strictly server side Filters MVCC ## What did they say? “Nothing is hotter than SQL-on-Hadoop, and now SQL-on-HBase is fast approaching equal hotness status” Form HBaseCon 2015 We want more! SQL + Transaction(ACID)

### 1. Background For reducing the Java GC impact to p99/p999 RPC latency, HBase 2.x has made an off-heap read and write path. The KV are allocated from the JVM off-heap, and the memory area in off-heap In above pictures, the p999 latency is almost the same as G1GC STW cost ( $ \sim $ 100ms). After HBASE-11425, almost all memory allocations should be in the offheap, there should be rarely heap allocation ByteBuffAllocator.  In HBASE-11425, we have introduced an off-heap memory management policy as following: 1. Set a max memory

推进终端设备和传感器的智能化，构建互联、可管理的和安全的分布式架构 ## 软硬结合 ## I ntel Hadoop商业发行版 ## 优化的大数据处理软件栈 稳定的企业级hadoop发行版 利用硬件新技术进行优化 HBase改进和创新，为Hadoop提供实时数据处理能力。针对行业的功能增强，应对不同行业的大数据挑战。 Intel Hadoop Manager 2.2 安装、部署、配置、监控、告警和访问控制 Sqoop 日志收集工具 Mahout 0.6 数据挖掘 R 统计语言 Hive 0.9.0 交互式数据仓库 Pig 0.9.2 数据流处理语言 Map/Reduce 1.0.3 分布式计算框架 HBase 0.94.1 实时、分布式、高维数据库 HDFS 1.0.3 分布式文件系统 Zookeeper 3.4.4 分布式协作服务 # Intel Hadoop Manager - 安装、配置、管理、监控、告警 |---|---|---|---|---|---| |控制集群|||||| |控制集群|||||| |HDFS集群|||||| |HDFS集群|||||| |HBase集群|||||| |HBase集群|||||| |HBase集群|||||| |HBase集群|||||| |H|||||| ## HBase 统计 ## ZooKeeper 节点状况 状态 运行中 集群负载 : 11.00 bdqac1-node2 RegionServer

是一切應用的基礎 • 所有 Job 都得轉換成 MapReduce Other Projects Ambari, Avro, Cassandra, Oozie, Zookeeper, etc. HBase Non-relational Database PIG Script Hive Query HCatalog Metadata Services MapReduce Distributed Other YARN Frameworks 就有存在的可能了 Other Projects Ambari, Avro, Cassandra, Oozie, Zookeeper, etc. HBase Non-relational Database Other YARN Frameworks PIG Script Hive Query HCatalog Metadata Services Serialization System (2010-05 成為 Top-Level Project) • Mahout : Scalable Library for Machine Learning • HBase : Distributed Data Storage (2010-05 成為 Top-Level Project) • Pig : High Level Language for Data Analysis

/root/hive/conf/hive-env.sh #tez scp -r root@master_ip:/home/hadoop/tez /root/ #hbase scp -r root@master_ip:/home/hadoop/hbase /root/ #spark scp -r root@master_ip:/home/hadoop/spark /root/ #pig scp -r export HIVE_HOME=/root/hive export HIVE_CONF_DIR=$HIVE_HOME/conf # HBase export HBASE_HOME=/root/hbase export HBASE_CONF_DIR=$HBASE_HOME/conf # spark export SPARK_HOME=/root/spark export SPARK_CONF_ PIG_CLASSPATH=$HADOOP_HOME/conf export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$HIVE_HOME/bin:$HBASE_HOME/bin:$SPARK_HOME/bin:$PIG_HOME/bin:$PATH export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native:/us