Vasiliki Kalavri | Boston University 2020 Types of Parallelism 7 B A C A B D A A B split Pipeline: A || B Task: B || C Data: A || A ??? Vasiliki Kalavri | Boston University 2020 8 Distributed computational steps • beneficial if it enables other optimizations, e.g. re-ordering • if the pipeline parallelism pays off Safety Profitability ??? Vasiliki Kalavri | Boston University 2020 24 • serialization and transport B A B ??? Vasiliki Kalavri | Boston University 2020 29 • removes pipeline parallelism but saves communication and serialization cost • if operators are separate, throughput

0x7fcd1ad0c0f0> Table Creation Table is a core component of the Python Table API. A Table object describes a pipeline of data transformations. It does not contain the data itself in any way. Instead, it describes how how to eventually write data to a table sink. The declared pipeline can be printed, optimized, and eventually executed in a cluster. The pipeline can work with bounded or unbounded streams which enables Creation DataStream is a core component of the Python DataStream API. A DataStream object describes a pipeline of data transformations. It does not contain the data itself in any way. Instead, it describes how

0x7fcd1ad0c0f0> Table Creation Table is a core component of the Python Table API. A Table object describes a pipeline of data transformations. It does not contain the data itself in any way. Instead, it describes how how to eventually write data to a table sink. The declared pipeline can be printed, optimized, and eventually executed in a cluster. The pipeline can work with bounded or unbounded streams which enables Creation DataStream is a core component of the Python DataStream API. A DataStream object describes a pipeline of data transformations. It does not contain the data itself in any way. Instead, it describes how

SQL extensions, CQL Java, Scala, Python, SQL Execution centralized distributed Parallelism pipeline pipeline, task, data State limited, in-memory partitioned, virtually unlimited, persisted to backends

SQL ⾼高吞吐 低延时 Hive vs. Spark vs. Flink Batch Hive/Hadoop Spark Flink 模型 MR MR(Memory/Disk) Pipeline 吞吐 TB-PB TB-PB 未经⼤大规模⽣生产验证 性能 ⼀一般(分钟⼩小时级别) 快(秒级) 优秀 x2 稳定性 好 ⼀一般 已在阿⾥里里内部验证 API 差(MR) 最丰富

a catalog of all IDs ever seen and checking it for de-duplication is expensive • In a healthy pipeline though, most records will not be duplicates • Each worker maintains a Bloom Filter of all IDs

channel or source Adjust processing rate of all operators to that of the slowest part of the pipeline ??? Vasiliki Kalavri | Boston University 2020 23 Progress is controlled though buffer availability

Apache Flink, which then writes the results to a database or calls a downstream system. In such a pipeline, latency can be introduced at each stage and for various reasons including the following: 1. It