RUST CHINA CONF 2023 Build a lightweight logging and tracing tool with Apache Arrow, Parquet and DataFusion 朱霜 2023.06.18 6.17-6.18 @Shanghai ## Content 1. Introduction 2. Duo - Observability duet: • What is Duo? • How does it work? 3. Apache Arrow, Parquet and DataFusion • A brief introduction to Arrow, Parquet, and DataFusion • How does Duo store and query log, span data? 4. The vision of on_close(&self, _id: Id, _ctx: Context<'_, S>) { ... } } ## Apache Arrow, Parquet, and DataFusion ## APACHE ARROW Parquet DATA FUSION ## Apache Arrow • Created by Wes McKinney, creator of Pandas

7cb3c2c4ae15fb3c/p8_1.jpg) ### 1.3 基于 DataFusion 的高性能查询引擎  ### 1.4 基于DataFusion的高性能查询引擎 • 扩展数据源 • 扩展 SQL 语句 • 扩展流处理引擎 4. 反哺社区 ## 😄 ### 4.1 DataFusion 1. https://github.com/cnosdb/cnosdb/issues/852 2. https://github.com/cnosdb/cnosdb/issues/784 3. https://github.com/apache/arrow-datafusion/issues/4401 4. https://github com/apache/arrow-datafusion/issues/4843 复) 6. https://github.com/apache/arrow-datafusion/issues/4947 7. https://github.com/apache/arrow-datafusion/issues/3778 8. https://github.com/apache/arrow-datafusion/issues/4075

time MySQL [(none)]> ## 03 ## the Engine Which way to rewrite? Apache Arrow & DataFusion Diff between models ## REWRITE Around 1/7 of Prometheus is PromQL A straightforward 3-layer With some key infrastructure like WebAssembly waynexia / datafusion-playground ## DataFusion Playground Playground of Apache Arrow DataFusion. In the early experimental stage as my side project. It when it's ready. Known bug: backspace doesn't work, and error message won't display. DataFusion CLI v16.0.0 starting! select 'hello!'; > :echo: select 'hello!'; 1 row in

Ballista is a distributed SQL query engine powered by the Rust implementation of Apache Arrow and DataFusion. It's mainly for interactive queries of low latency. • Support DAG and fault tolerance • snowflake.pdf • Apache Arrow https://arrow.apache.org/ • Apache Arrow DataFusion https://github.com/apache/arrow-datafusion • Apache Arrow Ballista https://github.com/apache/arrow-ballista ## Thank