## CS 591 K1: Data Stream Processing and Analytics Spring 2020 ## 3 /24: Exactly-once fault-tolerance in Apache Flink Vasiliki (Vasia) Kalavri vkalavri@bu.edu Go read his PhD thesis: http://kth.diva-portal /0aa43070543cf30310bdd99235d1d629/p66_1.jpg) What if the input stream comes from a socket? Exactly-once state consistency (in Apache Flink) can be achieved only if all streaming sources are re-settable [Image](/uploads/documents/0/a/a/4/0aa43070543cf30310bdd99235d1d629/p79_1.jpg) How can we ensure exactly-once output? ## Enabling and configuring checkpoints val env = StreamExecutionEnvironment.getExecutionEnvironment

High-availability and fault-tolerance in distributed stream processing • Recovery semantics and guarantees • Exactly-once processing in Apache Beam / Google Cloud Dataflow ## State in dataflow computations Any non-trivial • Precise recovery (exactly-once) • It hides the effects of a failure perfectly • Post-failure output is identical to no-failure ## Recovery types • Precise recovery (exactly-once) • It hides the effects duplicates • A backup needs to rebuild state of the failed node ## Recovery types • Precise recovery (exactly-once) • It hides the effects of a failure perfectly • Post-failure output is identical to no-failure

There are only two hard problemsin 0 distributed systems: 2. Exactly-once delivery 3 实时性 - 高性能 1. Guaranteed order of messages 2. Exactly-

parallelism • The state is automatically redistributed to the new set of parallel tasks • For exactly-once results, we need to prevent a checkpoint to complete after the savepoint! • Use the integrated

• JStorm • Flink Samza • Heron ## Core Storm or Storm Trident ? Storm Trident: • Batch • Exactly-once 选型考虑： - 性能 • 状态 ## Storm流式日志处理常见架构 ![Image](/uploads/documents/f/a/1/3/fa13f6fbe776421257

the three invocation types: • At-least-once, meaning that P will be invoked at least once; • Exactly-once, meaning that P will be invoked exactly once; • At-most-once, meaning that P will be invoked but no information is extracted from it. If the corresponding parameter P is introduced with exactly-once effect, this changes to: : calls-in-place effect with invocation kind “exactly-once” for its functional argument; - kotlin.check, kotlin.require (all overloads): returns-implies-condition

每一批次数据唯一且固定，保证 At-Most-Once 程序自身保证 At-Least-Once · 外部系统需要保证自身的 At-Least-Once，这样就可以保证导入流程的 Exactly-Once。 ## 多表原子性导入 • 每个表拆分多个任务，并下发BE • BE执行后汇报FE • FE 判断导入多数完成 publish 生效版本 • 后续查询规划时使用新的数据版本

management|shedding|backpressure, elasticity| |Fault tolerance|limited support, high availability|full support, exactly-once| ## Summary Today you learned: • stream representations, stream processing models • streaming

on consistent global snapshots (inspired by Chandy-Lamport). • Low runtime overhead, stateful exactly-once semantics. ## Fault Tolerance (2/2) ▶ Acks sequences of records instead of individual records

Database cluster for batch and streaming ETL operations. It requires Kafka version 0.11 or newer for exactly-once delivery assurance. Refer to the VMware Greenplum Streaming Server Documentation for more information