TableEnvironment Creation TableEnvironment is the entry point and central context for creating Table and SQL API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming User-defined function management: User-defined function registration, dropping, listing, etc. • Executing SQL queries • Job configuration • Python dependency management • Job submission For more details of [5]: root |-- id: BIGINT |-- data: STRING Create a Table from DDL statements [6]: table_env.execute_sql(""" CREATE TABLE random_source ( id TINYINT, data STRING ) WITH ( 'connector' = 'datagen', 'fields

TableEnvironment Creation TableEnvironment is the entry point and central context for creating Table and SQL API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming User-defined function management: User-defined function registration, dropping, listing, etc. • Executing SQL queries • Job configuration • Python dependency management • Job submission For more details of [5]: root |-- id: BIGINT |-- data: STRING Create a Table from DDL statements [6]: table_env.execute_sql(""" CREATE TABLE random_source ( id TINYINT, data STRING ) WITH ( 'connector' = 'datagen', 'fields

checkpoint("path/to/persistent/storage") 45 / 79 Stateful Stream Operations ▶ Spark API proposes two functions for statful processing: ▶ updateStateByKey • It is executed on the whole range of keys in DStream proportional to the size of the batch. 46 / 79 Stateful Stream Operations ▶ Spark API proposes two functions for statful processing: ▶ updateStateByKey • It is executed on the whole range of keys in DStream proportional to the size of the batch. 46 / 79 Stateful Stream Operations ▶ Spark API proposes two functions for statful processing: ▶ updateStateByKey • It is executed on the whole range of keys in DStream

Boston University 2020 Three classes of operators: • relation-to-relation: similar to standard SQL and define queries over tables. • stream-to-relation: define tables by selecting portions of a answers when it detects the end of its input. • NOT IN, set difference and division, traditional SQL aggregates • A Non-blocking query operator can produce answers incrementally as new input records where, for some S and j: [ ] ⊂ Gj(S) ⊂ G(Sj) 28 Vasiliki Kalavri | Boston University 2020 What functions on streams can be expressed using non-blocking operators? Proposition: A function F(S) on a sequence

disadvantages of each representation? 27 Vasiliki Kalavri | Boston University 2020 Reconstitution functions Insert (append-only): The reconstitution function ins starts with an empty bag and then inserts or more streams of possibly different type A series of transformations on streams in Stream SQL, Scala, Python, Rust, Java… 40 Vasiliki Kalavri | Boston University 2020 Stateful operators Logic Relational Dataflow Input in-order out-of-order Results approximate exact Language SQL extensions, CQL Java, Scala, Python, SQL Execution centralized distributed Parallelism pipeline pipeline, task, data

your cluster grows and shrinks • queryable: Flink state can be queried via a REST API Rich Functions • open(Configuration c) • close() • getRuntimeContext() DataStream> { out.collect(head); queue.remove(head); head = queue.peek(); } } SQL https://github.com/ververica/ sql-training Homework

可部署在各种集群环境 * 对各种⼤大⼩小的数据规模进⾏行行快速计算 为什什么Flink能做批处理理 Table Stream Bounded Data Unbounded Data SQL Runtime SQL ⾼高吞吐 低延时 Hive vs. Spark vs. Flink Batch Hive/Hadoop Spark Flink 模型 MR MR(Memory/Disk) Scala/Java SQL HiveSQL SparkSQL ANSI SQL 易易⽤用性 ⼀一般 易易⽤用 ⼀一般 ⼯工具/⽣生态 ⼀一般 丰富 ⼀一般 Flink Batch应⽤用 - 数据湖 Data Lake vs. Data Warehouse Flink Batch应⽤用 - 数据湖 Flink Batch应⽤用 - 数据湖 Blink SQL+UDF Queue

Schedule Task YARN RM K8S RM 增量 Checkpoint 时间 全量状态 增量状态 增量 snapshot 基于 credit 的流控机制 Streaming SQL ------------------------- | USER_SCORES | ------------------------- | User | Score | Time Dataflow DataStream API Stream Processing DataSet API Batch Processing Table API & SQL Relational Table API & SQL Relational Local Single JVM Cloud GCE, EC2 Cluster Standalone, YARN DataStream Physical 统一 Operator 抽象 Pull-based operator Push-based operator 算子可自定义读取顺序 Table API & SQL 1.9 新特性 全新的 SQL 类型系统 DDL 初步支持 Table API 增强 统一的 Catalog API Blink Planner What’s new in Blink Planner

or more streams of possibly different type A series of transformations on streams in Stream SQL, Scala, Python, Rust, Java… ??? Vasiliki Kalavri | Boston University 2020 Logic State

in the input • k independent and uniformly distributed hash functions, where k << n The Bloom filter n bits h1 h2 hk … k hash functions ??? Vasiliki Kalavri | Boston University 2020 25 for i=1 to filter Adding an element to the filter 0 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 n bits h1 h2 hk … k hash functions stream elements x The empty filter is initialized to all 0s ??? Vasiliki Kalavri | Boston Testing if an element is in the filter 0 0 0 0 0 1 1 0 0 1 0 0 0 0 1 1 n bits h1 h2 hk … k hash functions test element x If all bits are set, the element may exist in the set. If at least one element