JobManager disappears. • Flink relies on Apache ZooKeeper for high-availability • coordination and consensus services, e.g. leader election • The JobManager writes the JobGraph and all required metadata | Boston University 2020 • Ensure result correctness • reconfiguration mechanism often relies on fault-tolerance mechanism • State re-partitioning and migration • minimize communication • keep correctness ??? Vasiliki Kalavri | Boston University 2020 Control: When and how much to adapt? Mechanism: How to apply the re-configuration? 12 • Detect environment changes: external workload and system

throughput ??? Vasiliki Kalavri | Boston University 2020 Control: When and how much to adapt? Mechanism: How to apply the re-configuration? 3 • Detect environment changes: external workload and system and configuration commands that cannot yet be applied Live state migration Can we apply this mechanism in Flink? ??? Vasiliki Kalavri | Boston University 2020 Lecture references • Vasiliki Kalavri

is an alternative to updateStateByKeys: • Update function (partial updates) • Built in timeout mechanism • Choose the return type • Initial state def mapWithState[StateType, MappedType](spec: StateSpec[K is an alternative to updateStateByKeys: • Update function (partial updates) • Built in timeout mechanism • Choose the return type • Initial state def mapWithState[StateType, MappedType](spec: StateSpec[K

Fault-tolerance trade-offs 12 Steady-state overhead • How is performance affected by the fault-tolerance mechanism under normal, failure- free operation? • How much memory or disk space is required to maintain

??? Vasiliki Kalavri | Boston University 2020 Remarks on buffer-based rate control • Simple mechanism:the buffer occupancy controls the data rate automatically. • The maximum throughput is limited

Keep intermediate state in memory • Use Spark's RDDs instead of replication • Parallel recovery mechanism in case of failures 44 input stream time-based micro-batches D-Streams • During an interval

Kalavri | Boston University 2020 End-to-end exactly once • Flink’s checkpointing and recovery mechanism only resets the internal state of a streaming application • Some result records might be emitted Kalavri | Boston University 2020 End-to-end exactly once • Flink’s checkpointing and recovery mechanism only resets the internal state of a streaming application • Some result records might be emitted