Duo Content Introduction • ID: Folyd • GitHub: @folyd • 博客: https://folyd.com • ⼯作：字节跳动 (⽕⼭引擎) Duo - Observability duet: Logging and Tracing https://github.com/duo-rs/duo Logging and Tracing Single instruction/multiple data (SIMD), vectorized processing, and vectorized querying • Adopt by OLAP and data warehouse systems • … Apache Arrow Apache Arrow • Field • Array • Schema • RecordBatch Free and open source file format • Language agnostic • Column-based format • Used for analytics (OLAP) use cases • Highly efficient data compression and decompression • Supports complex data types and

needs. However, OLAP mode has no limit to the number of rows returned. In order to change to this mode, you may issue the following command before executing your query: set workload='olap' You can also and reparent commands. The general convention is to send OLTP queries to REPLICA tablet types, and OLAP queries to RDONLY. Is there a list of supported/unsupported queries? Please see “SQL Syntax” under slightly stale data, the queries should be sent to REPLICA tablets for OLTP, and RDONLY tablets for OLAP workloads. This allows you to scale your read traffic more easily, and gives you the ability to distribute

needs. However, OLAP mode has no limit to the number of rows returned. In order to change to this mode, you may issue the following command before executing your query: set workload='olap' 23 You can and reparent commands. The general convention is to send OLTP queries to REPLICA tablet types, and OLAP queries to RDONLY. Is there a list of supported/unsupported queries? Please see “SQL Syntax” under slightly stale data, the queries should be sent to REPLICA tablets for OLTP, and RDONLY tablets for OLAP workloads. This allows you to scale your read traffic more easily, and gives you the ability to distribute

needs. However, OLAP mode has no limit to the number of rows returned. In order to change to this mode, you may issue the following command before executing your query: set workload='olap' You can also and reparent commands. The general convention is to send OLTP queries to REPLICA tablet types, and OLAP queries to RDONLY. Is there a list of supported/unsupported queries? Please see “SQL Syntax” under slightly stale data, the queries should be sent to REPLICA tablets for OLTP, and RDONLY tablets for OLAP workloads. This allows you to scale your read traffic more easily, and gives you the ability to distribute

needs. However, OLAP mode has no limit to the number of rows returned. In order to change to this mode, you may issue the following command before executing your query: set workload='olap' You can also and reparent commands. The general convention is to send OLTP queries to REPLICA tablet types, and OLAP queries to RDONLY. Is there a list of supported/unsupported queries? Please see “SQL Syntax” under slightly stale data, the queries should be sent to REPLICA tablets for OLTP, and RDONLY tablets for OLAP workloads. This allows you to scale your read traffic more easily, and gives you the ability to distribute

• Not end with a period 39 Consists of five parts: • Category – VTGate / MySQL compatibility – OLAP – System – VReplication – VTtablet – VTorc – PITR – Examples – Docs – Build – Other • Label – BugFix slightly stale data, the queries should be sent to REPLICA tablets for OLTP, and RDONLY tablets for OLAP workloads. This allows you to scale your read traffic more easily, and gives you the ability to distribute really well. • Not all cells need rdonly (or batch) instances. Only the cells that run batch jobs, or OLAP jobs, really need them. Note Vitess uses local-cell data first, and is very resilient to any cell

slightly stale data, the queries should be sent to REPLICA tablets for OLTP, and RDONLY tablets for OLAP workloads. This allows you to scale your read traffic more easily, and gives you the ability to distribute really well. • Not all cells need rdonly (or batch) instances. Only the cells that run batch jobs, or OLAP jobs, really need them. Note Vitess uses local-cell data first, and is very resilient to any cell recommended to design the VSchema in such a way that cross- shard modifications are not required. OLAP Workload By default, Vitess sets some intentional restrictions on the execution time and number of

• Not end with a period 40 Consists of five parts: • Category – VTGate / MySQL compatibility – OLAP – System – VReplication – VTtablet – VTorc – PITR – Examples – Docs – Build – Other • Label – BugFix slightly stale data, the queries should be sent to REPLICA tablets for OLTP, and RDONLY tablets for OLAP workloads. This allows you to scale your read traffic more easily, and gives you the ability to distribute really well. • Not all cells need rdonly (or batch) instances. Only the cells that run batch jobs, or OLAP jobs, really need them. Note Vitess uses local-cell data first, and is very resilient to any cell

organizations, 90+ repos, 600+ followers ClickHouse Contributor Begin Content Area = 16,30 4 OLAP in ML Systems Begin Content Area = 16,30 5 Begin Content Area = 16,30 6 Intensive Tasks in reports = Joining data + Summerizing data • ... The data processing scenario is very similar to OLAP Begin Content Area = 16,30 8 A Database is not Just a “Database” What an English Dictionary Tells product_id) FROM pageview • Matching behavior sequences within time window • Inspired by Analysys OLAP Challenge 2018 (Funnel Analysis) • Featuring built-in automata description DSL • It is implemented

DBMS (PostgreSQL, MySQL) › Coordination system (Zookeeper, etcd) › NoSQL DBMS (MongoDB, Couchbase) › OLAP Database (ClickHouse!) https://github.com/donnemartin/system-design-primer 8 / 97 ClickHouse in External External Data for OLAP › Data from external DBMS › Logs from Network file system › Messages from Message Queue › Backups in cold storage 45 / 97 External Data for OLAP › Data from external DBMS └──────────┴──────────┴─────────────┘ 3 rows in set. Elapsed: 9.797 sec. -- slow :( 73 / 97 External Data for OLAP DBMS Examples of data › Geo information › Goods names › Country Taxes › Anything else :) Properties