. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.1.2.1 QuickStart: Table API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.1.2.2 QuickStart: DataStream 26 1.3.4.1 O1: Could not find any factory for identifier ‘xxx’ that implements ‘org.apache.flink.table.factories.DynamicTableFactory’ in the classpath . . . . . . . 26 1.3.4.2 O2: ClassNotFoundException: . . . 29 1.3.4.3 O3: NoSuchMethodError: org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/CatalogTable 30 1.3.5 Runtime issues . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.1.2.1 QuickStart: Table API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 1.1.2.2 QuickStart: DataStream 26 1.3.4.1 O1: Could not find any factory for identifier ‘xxx’ that implements ‘org.apache.flink.table.factories.DynamicTableFactory’ in the classpath . . . . . . . 26 1.3.4.2 O2: ClassNotFoundException: . . . 29 1.3.4.3 O3: NoSuchMethodError: org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/CatalogTable 30 1.3.5 Runtime issues . . . . . .

54 / 79 Structured Streaming 55 / 79 Structured Streaming ▶ Treating a live data stream as a table that is being continuously appended. ▶ Built on the Spark SQL engine. ▶ Perform database-like query Two main steps to develop a Spark stuctured streaming: ▶ 1. Defines a query on the input table, as a static table. • Spark automatically converts this batch-like query to a streaming execution plan. ▶ the input table), and incrementally updates the result. 57 / 79 Programming Model (1/2) ▶ Two main steps to develop a Spark stuctured streaming: ▶ 1. Defines a query on the input table, as a static

(derived) are defined as virtual views in SQL • Semantics are equivalent to having an append-only table to which new tuples are continuously added. 34 Vasiliki Kalavri | Boston University 2020 Example: start_price, start_time FROM OpenAuction WHERE start_price > 1000 Derived stream as an append- only table. 35 Vasiliki Kalavri | Boston University 2020 User-Defined Aggregates (UDAs) Constructs that allow Vasiliki Kalavri | Boston University 2020 Example: AVG UDA AGGREGATE myavg(Next Int): Real { TABLE state(tsum Int, cnt Int); INITIALIZE: { INSERT INTO state VALUES(Next, 1);

Example use-case: Distinct users visiting one or multiple webpages Naive solution: maintain a hash table ??? Vasiliki Kalavri | Boston University 2020 How can we count the number of distinct elements seen Example use-case: Distinct users visiting one or multiple webpages Naive solution: maintain a hash table Convert the stream into a multi-set of uniformly distributed random numbers using a hash function Example use-case: Distinct users visiting one or multiple webpages Naive solution: maintain a hash table The more different elements we encounter in the stream, the more different hash values we shall

Streaming Dataflow DataStream API Stream Processing DataSet API Batch Processing Table API & SQL Relational Table API & SQL Relational Local Single JVM Cloud GCE, EC2 Cluster Standalone, YARN Physical 统一 Operator 抽象 Pull-based operator Push-based operator 算子可自定义读取顺序 Table API & SQL 1.9 新特性 全新的 SQL 类型系统 DDL 初步支持 Table API 增强 统一的 Catalog API Blink Planner What’s new in Blink Planner 数据结构

步数RTransform I量h Apache Iceberg asic Data Metadata Database Table Partition Spec Manifest File TableMetadata Snapshot Current Table Version Pointer Apac2e Ice-er1 Bas3c Part3t354- f f3 Part3t354-2

w2 w1 w3 round-robin hash-based • Items are perfectly balanced among workers • No routing table required • Key semantics are not preserved: values of the same key might be routed to different different workers • Workers are responsible for roughly the same amount of keys • No routing table is required • Key semantics preserved: values of the same key are always processed by the same worker

是⼀一个分布式⼤大数据处理理引擎 * 可对有限数据流和⽆无限数据流进⾏行行有状态计算 * 可部署在各种集群环境 * 对各种⼤大⼩小的数据规模进⾏行行快速计算 为什什么Flink能做批处理理 Table Stream Bounded Data Unbounded Data SQL Runtime SQL ⾼高吞吐 低延时 Hive vs. Spark vs. Flink Batch

14 ??? Vasiliki Kalavri | Boston University 2020 Load Shedding Road Map (LSRM) • A pre-computed table that contains materialized load shedding plans ordered by how much load shedding they will cause