Ozone and how it scales • Why NVME is important for Ozone for scaling • Benefits of using NVME • Impala performance results from NVME clusters • Write path improvements results from NVME clusters • Summary load post hard restart (minimal caching) • Warp test to measure network saturation when using S3 • Impala TPCDS benchmark • Ratis streaming performance tests ## How much does disk read cost with NVME? org/apache/.. thread_native_entry(Thread*) ## I mpala TPCDS ## Why Impala and Ozone? • Data Warehouse is the most common use case. ($$$) - Impala historically optimized on HDFS -> what will it do on Ozone

## I MPALA + OZONE • Impala: SQL engine built to run in Hadoop clusters - Metadata stored in Hive Metastore – Data stored in Hadoop compatible storage • We will store Impala’s data in Ozone instead jpg) ## I MPALA-9400: IMPALA OZONE SUPPORT |Jira|Description| |---|---| |IMPALA-10212|ofs support in Impala| |IMPALA-9448|Test coverage for Ozone transparent data encryption| |IMPALA-10213|Support data data locality of Impala daemons on Ozone| |IMPALA-10214|Support file handle cache for Ozone| ## CHOOSING BUCKET TYPE • Impala has native support for HDFS - Some Impala operations will have poor performance

Cloudera Distribution for Hadoop ## CDH BATCH PROCESSING (MapReduce, Hive, Pig) ANALYTIC SQL (Impala) SEARCH ENGINE (Cloudera Search) MACHINE LEARNING (Spark, MapReduce, Mahout) STREAM PROCESSING . • Scribe User Interface : · ... • Hue Machine Learning : SQL on Hadoop : • Mahout … • Impala • Presto • Drill/Dremel/BigQuery · ... ## Tez ## Java 2014 TW • Hortonworks 主導 • A framework MapReduce，直接透過 In-Memory Process 來處理 • Compliant with ANSI-92 SQL Standard，所以透過 Cloudera ODBC Driver for Impala，就可以跟既有的 BI/DW 工具整合 

Hub组件)消息队列 Kafka| |流处理|Spark StreamingFlinkStorm|实时计算(原流计算)EMR(开源流计算组件)| |分析型数据存储|数据仓库:GreenPlum/Impala/Presto/HiveNoSQL:Hbase|数据仓库:MaxCompute/Hologres/分析型数据库NoSQL:云数据库 Hbase 版/表格存储| |分析与报表|BI工具Notebook|QuickBIPAI MRApache SparkMaxCompute Spark交互式分析ImpalaPrestoHawkGreenPlumImpala、Sqoop、Kylin、Flume） 机器配置（CPU 核数、内存大小） · 数据量及存储类型 作业量及作业类型（SQL 脚本上传） 调度系统及周期（Pipeline 配置上传） 已有数据应用（如血缘、监控、质量等）

计算 AI/ML HIVE/IMPALA/SPARK KAFKA / FLINK OTHER WORKLOADS  HDFS (300M FILES) 计算 AI/ML HIVE/IMPALA/SPARK KAFKA / FLINK 5/a/c/d/5acd8580f9ef16e1ab6780e04b6d6cb9/p14_2.jpg) 数据科学 数据仓库 S3 API S3 应用 计算 AI/ML HIVE/IMPALA/SPARK KAFKA / Flink OTHER WORKLOADS OZONE STORAGE ![Image](/uploads/documents/5/a/c/d/5acd85

(Safety Valve) for hue_safety_valve.ini Hue (Service-Wide) [desktop] app_blacklist=zookeeper.hbase.impala.search.sqoop.security use_new_editor=true [notebook] show_notebooks=false Refer following configuration configuration and tune it to fit your environment. [desktop] app_blacklist=zookeeper,hbase,impala,search,sqoop,security use_new_editor=true [[interpreters]] [[sparksql]] name=Spark SQL interface=hiveserver2 datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes

(Safety Valve) for hue_safety_valve.ini Hue (Service-Wide) [desktop] app_blacklist=zookeeper.hbase.impala.search.sqoop.security use_new_editor=true [notebook] show_notebooks=false Refer following configuration configuration and tune it to fit your environment. [desktop] app_blacklist=zookeeper,hbase,impala,search,sqoop,security use_new_editor=true [[interpreters]] [[sparksql]] name=Spark SQL interface=hiveserver2 datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes

(Safety Valve) for hue_safety_valve.ini Hue (Service-Wide) [desktop] app_blacklist=zookeeper.hbase.impala.search.sgoop.security use_new_editor=true [notebook] show_notebooks=false Refer following configuration configuration and tune it to fit your environment. [desktop] app_blacklist=zookeeper,hbase,impala,search,sqoop,security use_new_editor=true [[interpreters]] [[sparksql]] name=Spark SQL interface=hiveserver2 datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes

(Safety Valve) for hue_safety_valve.ini Hue (Service-Wide) [desktop] app_blacklist=zookeeper.hbase.impala.search.sqoop.security use_new_editor=true [notebook] show_notebooks=false Refer following configuration configuration and tune it to fit your environment. [desktop] app_blacklist=zookeeper,hbase,impala,search,sqoop,security use_new_editor=true [[interpreters]] [[sparksql]] name=Spark SQL interface=hiveserver2 datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes

Configuration Snippet (Safety Valve) for hue_safety_valve.ini [desktop] app_blacklist=zookeeper.hbase.impala.search.sgoop.security use_new_editor=true [notebook] show_notebooks=false Refer following configuration configuration and tune it to fit your environment. [desktop] app_blacklist=zookeeper,hbase,impala,search,sqoop,security use_new_editor=true [[interpreters]] [[sparksql]] name=Spark SQL interface=hiveserver2 datasets. Iceberg adds tables to compute engines including Spark, Trino, PrestoDB, Flink, Hive and Impala using a high-performance table format that works just like a SQL table. Tip: This article assumes