# Hadoop 概述 ## 本章内容提要 • Hadoop 的组件 • HDFS、MapReduce、YARN、ZooKeeper 和 Hive 的角色 ● Hadoop 与其他系统的集成 ● 数据集成与 Hadoop Hadoop 是一种用于管理大数据的基本工具。这种工具满足了企业在大型数据库(在 Hadoop 中亦称为数据湖)管理方面日益增长的需求。当涉及数据时，企业中最大的需求 MapReduce 中包含的编程逻辑，它提供了在 Hadoop 群集上横跨多台服务器的可扩展性。为实现资源管理，可考虑将 Hadoop YARN 加入到软件栈中，它是面向大数据应用程序的分布式操作系统。 ZooKeeper 是另一个 Hadoop Stack 组件，它能通过共享层次名称空间的数据寄存器(称为 znode)，使得分布式进程相互协调工作。每个 znode 都由一个路径来标识，路径元素由斜杠(/)分隔。 网络I/O。 ### 1.2 ZooKeeper 是什么 ZooKeeper 是另一项 Hadoop 服务——分布式系统环境下的信息保管员。ZooKeeper 的集中管理解决方案用于维护分布式系统的配置。由于 ZooKeeper 用于维护信息，因此任何新节点一旦加入系统，将从 ZooKeeper 中获取最新的集中式配置。这也使得你只需要通过 ZooKeeper 的一个客户端改变集中式配置，便能改变分布式系统的状态。

High Availability Kyuubi provides both high availability and load balancing solutions based on Zookeeper. Usage Guide ## • Quick Start • 1. Getting Started with Apache Kyuubi • 2. Getting Started With Metastore for Spark SQL to connect| |Zookeeper|Service Discovery|Optional|Any zookeeper ensemble compatible with curator(2.12.0)|By default, Kyuubi provides an embedded Zookeeper server inside for non-production 0x177078469840000 with negotiated timeout 60000 for client /127.0.0.1:65320 2021-01-16 03:27:35.492 INFO zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x177078469840000

Hive distribution• An optional and external metadata store, whose version is decided by engines| |Zookeeper|HA|>=3.4.x|| |Disk|Storage|N/A|N/A| The other internal or external parts listed in the above you can use Kyuubi, Spark and Flink to build a streaming data warehouse. And then, you can use Zookeeper to enable the load balancing for high availability. The data could be stored in Hive, Apache Iceberg reliably utilized with a minimum amount of down-time. Kyuubi operates by using Apache ZooKeeper [https://zookeeper.apache.org/] to harness redundant service instances in groups that provide continuous

HIGH AVAILABILITY Kyuubi provides both high availability and load balancing solutions based on Zookeeper. ### 6.1 Quick Start In this section, you will learn how to setup and interact with kyuubi quickly that can be reliably utilized with a minimum amount of down-time. Kyuubi operates by using Apache ZooKeeper to harness redundant service instances in groups that provide continuous service when one or more aware of the below two things basically, • kyuubi.ha.zookeeper.quorum - the external zookeeper cluster address for deploy a k.i. • kyuubi.ha.zookeeper.namespace - the root directory, a.k.a. the ServerSpace

图数据库, 知识图谱 • GeoMesa: 时空位置数据库 • Kylin: OLAP, 用HBase存cube • Phoenix: Sql on HBase ZooKeeper ZooKeeper ZooKeeper ### 2. Architecture 系统模块架构啥样, 如何存取数据 HMaster Hmaster active Master servers  Slave servers 然后HBase的数据都要存放在hdfs上, 就要有node. 如图可以看出RegionServer和Datanode尽量在同一台机器上. zookeeper作为协调信息存储的地方，比如节点健康状态 如图有这么几个组成部分, 前两个是HBase的 Master是负责管理的, RegionServer是实际干活的 ## 系统组成 Region ColumnFamily

Hive distribution• An optional and external metadata store, whose version is decided by engines| |Zookeeper|HA|>=3.4.x|| |Disk|Storage|N/A|N/A| The other internal or external parts listed in the above you can use Kyuubi, Spark and Flink to build a streaming data warehouse. And then, you can use Zookeeper to enable the load balancing for high availability. The data could be stored in Hive, Apache Iceberg reliably utilized with a minimum amount of down-time. Kyuubi operates by using Apache ZooKeeper [https://zookeeper.apache.org/] to harness redundant service instances in groups that provide continuous

Hive distribution• An optional and external metadata store, whose version is decided by engines| |Zookeeper|HA|>=3.4.x|| |Disk|Storage|N/A|N/A| The other internal or external parts listed in the above you can use Kyuubi, Spark and Flink to build a streaming data warehouse. And then, you can use Zookeeper to enable the load balancing for high availability. The data could be stored in Hive, Apache Iceberg reliably utilized with a minimum amount of down-time. Kyuubi operates by using Apache ZooKeeper [https://zookeeper.apache.org/] to harness redundant service instances in groups that provide continuous

jpg) ## 满足生产环境的MaxQ集群： 1. 四层Proxy负载均衡 2. 集群中各Node通过grpc通信，publish、delivery、ack转发，HA消息同步 3. zookeeper存储元数据保证元数据一致性，Master queue选举 4. 集群管理维护所有 MaxQ 集群 ### 5. MaxQ相关特性 1. 消息可靠性 2. 容错性 3. 扩展性 4. 高并发 Slave Queue ## 容错性 zookeeper不可用  元数据已缓存在内存中，不会有任何影响，生产方和消费方仍可正常生产和消费 服务自动降级，元数据不可变更 • zookeeper恢复，服务自愈 ## 节点故障 ![I bdbaf3b54e04ecb40a1/p13_1.jpg) 通过zookeeper进行Master Queue选举： 1. NodeA和NodeB收到NodeC挂掉的事件，NodeA和NodeB成为Master queue的候选节点 2. NodeA 和 NodeB 各自上报同步的 offset 到 zookeeper 3. NodeA和NodeB各自决策，offset最新的NodeA选为Master

HIGH AVAILABILITY Kyuubi provides both high availability and load balancing solutions based on Zookeeper.  ### 6.1 Quick 1:65320 (continues on next page) (continued from previous page) 2021-01-16 03:27:35.492 INFO zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x177078469840000 ConnectionStateManager: State change: CONNECTED 2021-01-16 03:27:35.495 INFO client.ServiceDiscovery: Zookeeper client connection state changed to: CONNECTED 2021-01-16 03:27:36.516 INFO client.ServiceDiscovery:

HIGH AVAILABILITY Kyuubi provides both high availability and load balancing solutions based on Zookeeper.  ### 6.1 Quick 1:65320 (continues on next page) (continued from previous page) 2021-01-16 03:27:35.492 INFO zookeeper.ClientCnxn: Session establishment complete on server 127.0.0.1/127.0.0.1:2181, sessionid = 0x177078469840000 ConnectionStateManager: State change: CONNECTED 2021-01-16 03:27:35.495 INFO client.ServiceDiscovery: Zookeeper client connection state changed to: CONNECTED 2021-01-16 03:27:36.516 INFO client.ServiceDiscovery: