Summer School 2019 ## Apache Flink Based on https://training.ververica.com Maximilian Michels apache.org> Software Engineer / Consultant Committer @ Apache Beam / Apache Flink Dr Paris Carbone Apache Flink @stadtlegende @SenorCarboneapache.org> ## Contents • DataSet API • DataStream API • Concepts • Set up an environment to develop Flink programs • Implement streaming data processing pipelines • Flink managed state • Event time ## Streaming in Apache Flink • Streams are natural • Events of any type like sensors, click streams, logs • Batch

# 监控Apache Flink应用程序(入门) caolei Exported on 01/10/2020 ## Table of Contents 1 Flink指标体系 ..... 5 1.1 Metrics ..... 5 1.2 MetricsReporters ..... 5 2 健康状况 ..... 6 3 监控 ..... 7 3.1 关键指标 ..... com/blog/monitoring-apache-flink-applications-101 这篇博文介绍了Apache Flink内置的监控和度量系统，通过该系统，开发人员可以有效地监控他们的Flink作业。通常，对于一个刚刚开始使用Apache Flink进行流处理的DevOps团队来说，选择对应的指标来监控Flink应用程序是非常艰巨的。在与许多大规模部署过Apache Flink的组织合作之后，我想与 随着越来越多的核心业务应用程序运行在Apache Flink上，性能监控在成功的生产环境部署中变得非常重要。它确保何故障或停机时间都可以被立即识别并尽快得到解决。 监控与观察相结合是故障诊断和性能调优的先决条件。如今，随着现代企业应用程序的复杂性和交付速度的加快，工程团队必须理解并在任何给定的时间点上对其应用程序的状态有一个完整的认识和概述。 ## 1 Flink指标体系 Flink作业监控的基础是它的度量系统，该系统由两个部分组成:

## 阿里云 ## Apache Flink的过去、现在和未来 杨克特（鲁尼） 阿里巴巴高级技术专家 ## 过去 ## 一 切从2014年开始  StratoSphere Above the Clouds  2009 - 2014 2014 • 柏林工业大学博士生项目 - 基于流式 runtime 的批处理引擎 • 2014 年 8 月份 发布 Flink 0.6.0 ## 阿里云 2019阿里云峰会·上海开发者大会 Flink 0.7 ## 2014 年 12 月份 发布 – 开始正式支持 DataStream DataStream API Stream Processing 8d2791cad3421/p4_1.jpg) DataSet API Batch Processing Runtime Distributed Streaming Dataflow Flink 0.9 ## 2015 年 6 月份 发布 – 开始内置支持 State ![Image](/uploads/documents/4/4/7/6/44768622b352d818cb18d2791cad3421/p5_1

Stream Processing with Apache RocketMQ and Apache Flink 王鑫 · The Apache Software Foundation Nov.4, 2018, Shanghai, Apache Flink China Meetup  ## X in Wang • Apache RocketMQ Committer • Apache Storm Committer & PMC member • Six years distributed system experience • Love open source & community • Focus on distributed ents/5/1/f/3/51f363b7213b31d1c40f613be3dd1945/p3_3.jpg) Practices of integrating RocketMQ with Flink  The trend of RocketMQ

Processing and Analytics Spring 2020 ## 1 /30: Introduction to Apache Flink and Apache Kafka Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Apache Flink • An open-source, distributed data analysis framework Data Set Operator Data Set Sink Source Data Stream Operator Data Stream Sink Writing a Flink Program 1. Bootstrap Sources 2. Apply Operators 3. Output to Sinks ## Streaming word count textStream keyBy(0) .sum(1) .print() (live,1) (and,1) (let,1) (live,2) ## Distributed architecture TaskManager Flink program web dashboard TaskManager client JobManager TaskManager ## DataStream API Basics ##

## Scalable Stream Processing - Spark Streaming and Flink Amir H. Payberah payberah@kth.se 05/10/2018 https://id2221kth.github.io ## Data Processing Graph Data Pregel, GraphLab, PowerGraph GraphX Spark SQL Machine Learning Mliib Tensorflow Streaming Data Storm, SEEP, Naiad, Spark Streaming, Flink, Millwheel, Google Dataflow ## Distributed File Systems ## Data Storage GFS, Flat FS NoSQL Databases Continuous vs. micro-batch processing Record-at-a-Time vs. declarative APIs ▶ Spark streaming ▶ Flink ## Spark Streaming ## ▶ Design issues • Continuous vs. micro-batch processing • Record-at-a-Time

K1: Data Stream Processing and Analytics Spring 2020 ## 3 /24: Exactly-once fault-tolerance in Apache Flink Vasiliki (Vasia) Kalavri vkalavri@bu.edu Go read his PhD thesis: http://kth.diva-portal.or [Image](/uploads/documents/0/a/a/4/0aa43070543cf30310bdd99235d1d629/p59_1.jpg) ## Asynchronous checkpoints in Apache Flink  • A source of to the position up to which they were consumed when the checkpoint was taken. - Event logs like Apache Kafka can provide records from a previous offset of the stream. ## Re-settable sources - All input

## Flink 批处理及其应用 ## What is Apache Flink $ ^{*} $ Apache Flink 是一个分布式大数据处理引擎 $ ^{*} $ 可对有限数据流和无限数据流进行有状态计算 * 可部署在各种集群环境 * 对各种大小的数据规模进行快速计算 ## 为什么Flink能做批处理  低延时 #### Hive vs. Spark vs. Flink Batch ||Hive/Hadoop|Spark|Flink| |---|---|---|---| |模型|MR|MR(Memory/Disk)|Pipeline| |吞吐|TB-PB|TB-PB|未经大规模生产验证| | |易用性|一般|易用|一般| |工具/生态|一般|丰富|一般| ## Flink Batch应用 - 数据湖 ### Data Lake vs. Data Warehouse  ## Flink Batch应用 – 数据湖 ![Image](/upl

## Flink如何实时分析Iceberg数据湖的CDC数据 阿里巴巴 李劲松/胡争 FLINK FORWARD #ASIA 2020 #1 #2 #3 #4 常见的CDC 为何选择 Flink 如何实时写 未来规划 分析方案 + Iceberg 入读取 FLINK FORWARD #ASIA 2020 ## #1 常见的CDC分析方案 ## 离线 HBase 集群分析 CDC 3、通过RegionServer定位HFile，Server的优化和缓存完全用不上。 4、数据格式绑定HFile，不方便拓展到Parquet、Avro、Orc等。 FLINK FORWARD #ASIA 2020 ## Apache Kudu 维护 CDC 数据集 ## MySQL ## 方案评估 优点 1、支持实时更新数据，时效性佳。 2、列存加速，适合OLAP分析。 ## 缺点 4、不支持增量拉取。 FLINK FORWARD #ASIA 2020 ## MySQL → GQOOP → HVE ## 方案评估 优点 1、流程能工作 2、Hive存量数据不受增量数据影响。 ## 缺点 1、数据不是实时写入； 2、每次数据导致都要 MERGE 存量数据。T+1 方式更新，时效性差。 3、不支持实时upsert。 FLINK FORWARD #ASIA 2020

JDBC interface for large-scale data processing and analytics, built on top of Apache Spark $ ^{™} $ [http://spark.apache.org/]. In general, the complete ecosystem of Kyuubi falls into the hierarchies each layer loosely coupled to the other. For example, you can use Kyuubi, Spark and Apache Iceberg [https://iceberg.apache.org/] to build and manage Data Lake with pure SQL for both data processing e.g. ETL why we want to create this project despite that the Spark Thrift JDBC/ODBC server [http://spark.apache.org/docs/latest/sql-distributed-sql-engine.html#running-the-thrift-jdbcodbc-server] already exists