recovery semantics, and guarantees Vasiliki Kalavri | Boston University 2020 Today’s topics • High-availability and fault-tolerance in distributed stream processing • Recovery semantics and guarantees positive. Vasiliki Kalavri | Boston University 2020 Further resources • Jeong-Hyon Hwang et al. High-Availability Algorithms for Distributed Stream Processing. (ICDE ’05). • http://cs.brown.edu/research/aurora/hwang

Stream Processing and Analytics Vasiliki (Vasia) Kalavri
 vkalavri@bu.edu Spring 2020 3/31: High-availability & reconfiguration ??? Vasiliki Kalavri | Boston University 2020 • To recover from failures state 2 Checkpointing guards the state from failures, but what about process failure? High-availability ??? Vasiliki Kalavri | Boston University 2020 3 Flink processes ??? Vasiliki Kalavri | Boston keeps metadata about application execution, such as pointers to completed checkpoints. • A high-availability mode migrates the responsibility and metadata for a job to another JobManager in case the

Retractions & results amendment Reconfiguration & updates Debugging Fault-tolerance & high-availability Vasiliki Kalavri | Boston University 2020 actions, alerts continuous analytics … Building

the query is running might be impractical. • state accumulation and re-partitioning • high-availability and low latency requirements • scheduling overhead Challenges in streaming optimization