access data and metadata from a storage system, e.g. Apache Hadoop HDFS [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html], with permissions. Ease of Use You only need above By default Kyuubi is pre- built w/ a Apache Spark release inside at $KYUUBI_HOME/externals HDFS Distributed File System Optional referenced by Spark Hadoop Distributed File System is a part framework, used to store and process the datasets. You can interact with any Spark-compatible versions of HDFS. Components Role Optional Version Remarks Hive Metastore Optional referenced by Spark Hive Metastore

application; 2) a user account can only access data and metadata from a storage system, e.g. Apache Hadoop HDFS, with permissions. 3 Kyuubi, Release 1.3.0 4 Chapter 1. Multi-tenancy CHAPTER TWO EASE OF USE Spark with -Pyarn maven option • An active Apache Hadoop YARN cluster • An active Apache Hadoop HDFS cluster • Setup Hadoop client configurations at the machine the Kyuubi server locates Configurations configurations di- rectory, usually,$HADOOP_HOME/etc/hadoop If the HADOOP_CONF_DIR points the YARN and HDFS cluster correctly, you should be able to run the SparkPi example on YARN. $ HADOOP_CONF_DIR=/path/to/hadoop/conf

application; 2) a user account can only access data and metadata from a storage system, e.g. Apache Hadoop HDFS, with permissions. 3 Kyuubi, Release 1.3.0 4 Chapter 1. Multi-tenancy CHAPTER TWO EASE OF USE Spark with -Pyarn maven option • An active Apache Hadoop YARN cluster • An active Apache Hadoop HDFS cluster • Setup Hadoop client configurations at the machine the Kyuubi server locates Configurations configurations di- rectory, usually,$HADOOP_HOME/etc/hadoop If the HADOOP_CONF_DIR points the YARN and HDFS cluster correctly, you should be able to run the SparkPi example on YARN. $ HADOOP_CONF_DIR=/path/to/hadoop/conf

access data and metadata from a storage system, e.g. Apache Hadoop HDFS [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html], with permissions. Ease of Use You only need above By default Kyuubi is pre- built w/ a Apache Spark release inside at $KYUUBI_HOME/externals HDFS Distributed File System Optional referenced by Spark Hadoop Distributed File System is a part framework, used to store and process the datasets. You can interact with any Spark-compatible versions of HDFS. Components Role Optional Version Remarks Hive Metastore Optional referenced by Spark Hive Metastore

access data and metadata from a storage system, e.g. Apache Hadoop HDFS [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html], with permissions. Ease of Use You only need Required 3.0.0 and above By default Kyuubi binary release is delivered without a Spark tarball. HDFS Distributed File System Optional referenced by Spark Hadoop Distributed File System is a part framework, used to store and process the datasets. You can interact with any Spark-compatible versions of HDFS. Components Role Optional Version Remarks Hive Metastore Optional referenced by Spark Hive Metastore

access data and metadata from a storage system, e.g. Apache Hadoop HDFS [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html], with permissions. Ease of Use You only need Required 3.0.0 and above By default Kyuubi binary release is delivered without a Spark tarball. HDFS Distributed File System Optional referenced by Spark Hadoop Distributed File System is a part framework, used to store and process the datasets. You can interact with any Spark-compatible versions of HDFS. Components Role Optional Version Remarks Hive Metastore Optional referenced by Spark Hive Metastore

application; 2) a user account can only access data and metadata from a storage system, e.g. Apache Hadoop HDFS, with permissions. 3 Kyuubi, Release 1.4.1-incubating 4 Chapter 1. Multi-tenancy CHAPTER TWO Spark with -Pyarn maven option • An active Apache Hadoop YARN cluster • An active Apache Hadoop HDFS cluster • Setup Hadoop client configurations at the machine the Kyuubi server locates Configurations configurations di- rectory, usually, $HADOOP_HOME/etc/hadoop. If the HADOOP_CONF_DIR points the YARN and HDFS cluster correctly, you should be able to run the SparkPi example on YARN. $ HADOOP_CONF_DIR=/path/to/hadoop/conf

application; 2) a user account can only access data and metadata from a storage system, e.g. Apache Hadoop HDFS, with permissions. 3 Kyuubi, Release 1.3.0 4 Chapter 1. Multi-tenancy CHAPTER TWO EASE OF USE Spark with -Pyarn maven option • An active Apache Hadoop YARN cluster • An active Apache Hadoop HDFS cluster • Setup Hadoop client configurations at the machine the Kyuubi server locates Configurations configurations di- rectory, usually, $HADOOP_HOME/etc/hadoop. If the HADOOP_CONF_DIR points the YARN and HDFS cluster correctly, you should be able to run the SparkPi example on YARN. $ HADOOP_CONF_DIR=/path/to/hadoop/conf

data in various formats (Parquet, CSV, JSON, text) in your data lake in cloud storage or an on-prem HDFS cluster. • Lakehouse formation and analytics – Easily build an ACID table storage layer via Hudi possible to visit different Hive metastore server instance. Similarly, this works for other services like HDFS, YARN too. Limitation: As most Hive configurations are final and unmodifiable in Spark at runtime Spark with -Pyarn maven option • An active Apache Hadoop YARN cluster • An active Apache Hadoop HDFS cluster • Setup Hadoop client configurations at the machine the Kyuubi server locates 28 Chapter

access data and metadata from a storage system, e.g. Apache Hadoop HDFS [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop- hdfs/HdfsDesign.html], with permissions. Ease of Use You only need Optional 1.14.0 and above By default Kyuubi binary release is delivered without a Flink tarball. HDFS Distributed File System Optional referenced by Spark Hadoop Distributed File System is a part framework, used to store and process the datasets. You can interact with any Spark-compatible versions of HDFS. Components Role Optional Version Remarks Hive Metastore Optional referenced by Spark Hive Metastore