Scalable Stream Processing - Spark Streaming and Flink Amir H. Payberah payberah@kth.se 05/10/2018 The Course Web Page https://id2221kth.github.io 1 / 79 Where Are We? 2 / 79 Stream Processing Systems Outline ▶ Spark streaming ▶ Flink 4 / 79 Spark Streaming 5 / 79 Contribution ▶ Design issues • Continuous vs. micro-batch processing • Record-at-a-Time vs. declarative APIs 6 / 79 Spark Streaming RDDs and processes them using RDD operations. • Discretized Stream Processing (DStream) 7 / 79 Spark Streaming ▶ Run a streaming computation as a series of very small, deterministic batch jobs. • Chops

Stream Bounded Data Unbounded Data SQL Runtime SQL ⾼高吞吐 低延时 Hive vs. Spark vs. Flink Batch Hive/Hadoop Spark Flink 模型 MR MR(Memory/Disk) Pipeline 吞吐 TB-PB TB-PB 未经⼤大规模⽣生产验证 性能 ⼀一般(分钟⼩小时级别)

Profitability A A’ Spark Streaming • Treat streaming computation as a series of deterministic batch computations on small time intervals • Keep intermediate state in memory • Use Spark's RDDs instead

Systems 2000 1992 2013 MapReduce 2004 Tapestry NiagaraCQ Aurora TelegraphCQ STREAM Naiad Spark Streaming Samza Flink Millwheel Storm S4 Google Dataflow Now Evolution of Stream Processing

3、kDCDC增量拉T相关Tab1e API接 口。 Iceberg内uAS 1、实现CDCmi自动合并和g动合并对 接； 、kDF1i3k增量拉TCDCmi的能力 。 F1i3k集成 1、Spark Strea2i3g 对接CDC写F链 路 、Presto等bl对接t询链路。 3、借助axA11uxioP速mit询。 I他生态集成 谢谢 谢谢 谢谢