Summer School 2019 Apache Flink Based on https://training.ververica.com Maximilian Michels Software Engineer / Consultant
 Committer @ Apache Beam / Apache Flink @stadtlegende Dr Paris Apache Flink @SenorCarbone Contents • DataSet API • DataStream API • Concepts • Set up an environment to develop Flink programs • Implement streaming data processing pipelines • Flink managed managed state • Event time Streaming in Apache Flink • Streams are natural • Events of any type like sensors, click streams, logs • Batch processing as a subset of stream processing Processing Data

Scalable Stream Processing - Spark Streaming and Flink Amir H. Payberah payberah@kth.se 05/10/2018 The Course Web Page https://id2221kth.github.io 1 / 79 Where Are We? 2 / 79 Stream Processing Systems micro-batch processing ▶ Record-at-a-Time vs. declarative APIs 3 / 79 Outline ▶ Spark streaming ▶ Flink 4 / 79 Spark Streaming 5 / 79 Contribution ▶ Design issues • Continuous vs. micro-batch processing groupBy(window(col("timestamp"), "10 minutes", "5 minutes"), col("word")).count() 67 / 79 Flink 68 / 79 Flink ▶ Distributed data flow processing system ▶ Unified real-time stream and batch processing

Apache Flink的过去、现在和未来 杨克特（鲁尼） 阿里巴巴高级技术专家 过去 一切从2014年开始 2009 - 2014 2014 • 柏林工业大学博士生项目 • 基于流式 runtime 的批处理引擎 • 2014 年 8 月份 发布 Flink 0.6.0 Flink 0.7 Runtime Distributed Streaming Dataflow DataStream DataStream API Stream Processing DataSet API Batch Processing 2014 年 12 月份 发布 – 开始正式支持 DataStream Flink 0.9 Sink Source Offset Computation State Periodic Snapshots 2015 年 6 月份 发布 – 开始内置支持 State Global Part of Checkpoint N-1 • 吞吐和延迟不再是一个 tradeoff • 支持精准一次的语义，同时对性能的影响较低 基于 Chandy – Lamport 算法 Flink 1.0 版本基石 Checkpoint 基于 Chandy-Lamport 算法，实现了分布式一 致性快照，提供了一致 性的语义。 丰富的 State API。 ValueState

监控Apache Flink应用程序(入门) caolei Exported on 01/10/2020 caolei – 监控Apache Flink应用程序(入门) – 2 Table of Contents 1 Flink指标体系 ...................................................................... ........................................................................ 21 caolei – 监控Apache Flink应用程序(入门) – 3 4.13.2.1 Key Metrics ............................................................ 监控Apache Flink应用程序(入门) – 4 原文地址：https://www.ververica.com/blog/monitoring-apache-flink-applications-101 这篇博文介绍了Apache Flink内置的监控和度量系统，通过该系统，开发人员可以有效地监控他们的Flink作 业。通常，对于一个刚刚开始使用Apache Flink进行流处理的De

Flink 批处理理及其应⽤用 What is Apache Flink * Apache Flink 是⼀一个分布式⼤大数据处理理引擎 * 可对有限数据流和⽆无限数据流进⾏行行有状态计算 * 可部署在各种集群环境 * 对各种⼤大⼩小的数据规模进⾏行行快速计算 为什什么Flink能做批处理理 Table Stream Bounded Data Unbounded Data Data SQL Runtime SQL ⾼高吞吐 低延时 Hive vs. Spark vs. Flink Batch Hive/Hadoop Spark Flink 模型 MR MR(Memory/Disk) Pipeline 吞吐 TB-PB TB-PB 未经⼤大规模⽣生产验证 性能 ⼀一般(分钟⼩小时级别) 快(秒级) 优秀 x2 稳定性 好 ⼀一般 已在阿⾥里里内部验证 HiveSQL SparkSQL ANSI SQL 易易⽤用性 ⼀一般 易易⽤用 ⼀一般 ⼯工具/⽣生态 ⼀一般 丰富 ⼀一般 Flink Batch应⽤用 - 数据湖 Data Lake vs. Data Warehouse Flink Batch应⽤用 - 数据湖 Flink Batch应⽤用 - 数据湖 Blink SQL+UDF Queue 存储类 存储 计算 存储 Queue

Flink如何实时分析Iceberg数据湖的CDC数据 阿里巴巴 李/松/胡争 23选择 Flink Ic+b+1g #2 常DCCDC 分析方案 #1 如3实时写 4F取 ## 未来规划 #4 #见的CDC分析方案 #1 离线 HBase 集u分析 CDC 数a 、CDC记录实时写入HBase。高吞P + 低延迟。 2、小vSg询延迟低。 3、集u可拓展 ci评C 1、增量和全量表割p，时效性不足。 2、r计和l护额外hChang+ S+4表。 3、计算引擎并非原g支UCDC。 4、不支U实时U13+24。 缺点 为何选择 #+ink Iceberg ? #2 Flink 原生支持 C C 数据消费 ebezium 1lHLI W生支持 ./. 数据消费 -- BPDaRDs a mysOl BCB RaAlD sMSPBD .R0,T0 T,-L0 mysOl_AHLlMF =PP0R LamD), CDsBPHNRHML, UDHFGR 1RO6 mysOl_AHLlMF; FHRGSA.BMm/TDPTDPHBa/ElHLI-BCB-BMLLDBRMPs Flink 原生支持 Change Log Stream A C D E F G INSERT DELETE UPDATE INSERT DELETE UPDATE INSERT

vkalavri@bu.edu Spring 2020 1/30: Introduction to Apache Flink and Apache Kafka Vasiliki Kalavri | Boston University 2020 Apache Flink • An open-source, distributed data analysis framework • True Source Data Stream Operator Data Stream Sink Source Data Set Operator Data Set Sink Writing a Flink Program 1.Bootstrap Sources 2.Apply Operators 3.Output to Sinks 3 Vasiliki Kalavri | Boston University (let,1) (live,2) 4 Vasiliki Kalavri | Boston University 2020 Distributed architecture client Flink program JobManager web dashboard TaskManager TaskManager TaskManager 5 Vasiliki Kalavri |

Vasiliki (Vasia) Kalavri
 vkalavri@bu.edu Spring 2020 3/24: Exactly-once fault-tolerance in Apache Flink ??? Vasiliki Kalavri | Boston University 2020 Some slides in this lecture have been generously Output Logs 38 ??? Vasiliki Kalavri | Boston University 2020 Asynchronous checkpoints in Apache Flink 39 ??? Vasiliki Kalavri | Boston University 2020 40 • A source of increasing numbers partitioned Exactly-once state consistency (in Apache Flink) can be achieved only if all streaming sources are re-settable ??? Vasiliki Kalavri | Boston University 2020 44 • Flink checkpoints are initiated by the JobManager

Soon. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.2.2 PyFlink + Flink ML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 1.2.2.1 Coming . . 26 1.3.4.1 O1: Could not find any factory for identifier ‘xxx’ that implements ‘org.apache.flink.table.factories.DynamicTableFactory’ in the classpath . . . . . . . 26 1.3.4.2 O2: ClassNotFoundException: . . . . . . 29 1.3.4.3 O3: NoSuchMethodError: org.apache.flink.table.factories.DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/CatalogTable 30 1.3.5 Runtime issues . . .