Boston University 2020 Three classes of operators: • relation-to-relation: similar to standard SQL and define queries over tables. • stream-to-relation: define tables by selecting portions of a answers when it detects the end of its input. • NOT IN, set difference and division, traditional SQL aggregates • A Non-blocking query operator can produce answers incrementally as new input records continuous queries. 29 Vasiliki Kalavri | Boston University 2020 Non-blocking SQL Let NB-SQL be the non-blocking subset of SQL that excludes non- monotonic constructs: • EXCEPT, NOT EXIST, NOT IN and ALL

TableEnvironment Creation TableEnvironment is the entry point and central context for creating Table and SQL API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming User-defined function management: User-defined function registration, dropping, listing, etc. • Executing SQL queries • Job configuration • Python dependency management • Job submission For more details of [5]: root |-- id: BIGINT |-- data: STRING Create a Table from DDL statements [6]: table_env.execute_sql(""" CREATE TABLE random_source ( id TINYINT, data STRING ) WITH ( 'connector' = 'datagen', 'fields

TableEnvironment Creation TableEnvironment is the entry point and central context for creating Table and SQL API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming User-defined function management: User-defined function registration, dropping, listing, etc. • Executing SQL queries • Job configuration • Python dependency management • Job submission For more details of [5]: root |-- id: BIGINT |-- data: STRING Create a Table from DDL statements [6]: table_env.execute_sql(""" CREATE TABLE random_source ( id TINYINT, data STRING ) WITH ( 'connector' = 'datagen', 'fields

可部署在各种集群环境 * 对各种⼤大⼩小的数据规模进⾏行行快速计算 为什什么Flink能做批处理理 Table Stream Bounded Data Unbounded Data SQL Runtime SQL ⾼高吞吐 低延时 Hive vs. Spark vs. Flink Batch Hive/Hadoop Spark Flink 模型 MR MR(Memory/Disk) Scala/Java SQL HiveSQL SparkSQL ANSI SQL 易易⽤用性 ⼀一般 易易⽤用 ⼀一般 ⼯工具/⽣生态 ⼀一般 丰富 ⼀一般 Flink Batch应⽤用 - 数据湖 Data Lake vs. Data Warehouse Flink Batch应⽤用 - 数据湖 Flink Batch应⽤用 - 数据湖 Blink SQL+UDF Queue

Schedule Task YARN RM K8S RM 增量 Checkpoint 时间 全量状态 增量状态 增量 snapshot 基于 credit 的流控机制 Streaming SQL ------------------------- | USER_SCORES | ------------------------- | User | Score | Time Dataflow DataStream API Stream Processing DataSet API Batch Processing Table API & SQL Relational Table API & SQL Relational Local Single JVM Cloud GCE, EC2 Cluster Standalone, YARN DataStream Physical 统一 Operator 抽象 Pull-based operator Push-based operator 算子可自定义读取顺序 Table API & SQL 1.9 新特性 全新的 SQL 类型系统 DDL 初步支持 Table API 增强 统一的 Catalog API Blink Planner What’s new in Blink Planner

Treating a live data stream as a table that is being continuously appended. ▶ Built on the Spark SQL engine. ▶ Perform database-like query optimizations. 56 / 79 Programming Model (1/2) ▶ Two main -spark.html] 60 / 79 Structured Streaming Example (2/3) ▶ We could express it as the following SQL query. SELECT action, WINDOW(time, "1 hour"), COUNT * FROM events GROUP BY action, WINDOW(time, "1 groupBy("action") // using untyped API ds.groupByKey(_.action) // using typed API // SQL commands df.createOrReplaceTempView("dfView") spark.sql("select count(*) from dfView") // returns another streaming DF 63 / 79

or more streams of possibly different type A series of transformations on streams in Stream SQL, Scala, Python, Rust, Java… 40 Vasiliki Kalavri | Boston University 2020 Stateful operators Logic Relational Dataflow Input in-order out-of-order Results approximate exact Language SQL extensions, CQL Java, Scala, Python, SQL Execution centralized distributed Parallelism pipeline pipeline, task, data

{ out.collect(head); queue.remove(head); head = queue.peek(); } } SQL https://github.com/ververica/ sql-training Homework

or more streams of possibly different type A series of transformations on streams in Stream SQL, Scala, Python, Rust, Java… ??? Vasiliki Kalavri | Boston University 2020 Logic State