of reconfiguration • Ensure result correctness • reconfiguration mechanism often relies on fault-tolerance mechanism • State re-partitioning and migration • minimize communication • keep duration short

Exactly-once fault-tolerance in Apache Flink Vasiliki (Vasia) Kalavri vkalavri@bu.edu Go read his PhD thesis: http://kth.diva-portal.org/smash/get/diva2:1240814/FULLTEXT01.pdf ## Fault-tolerance approaches

guarantees Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Today's topics • High-availability and fault-tolerance in distributed stream processing • Recovery semantics and guarantees • Exactly-once processing you think of an operator that will diverge? ## Fault-tolerance trade-offs ## Steady-state overhead • How is performance affected by the fault-tolerance mechanism under normal, failure-free operation

Systems Architecture and design Scheduling and load management Scalability and elasticity Fault-tolerance and guarantees State management Operator semantics Window optimizations Filtering, counting updates Debugging Order Processing guarantees Retractions & results amendment Progress Fault-tolerance & high-availability ## Building a stream processor ![Image](/uploads/documents/2/8/8/5/

Re-delivery might cause re-ordering of messages • Re-delivery complicates stream processing and fault-tolerance • might process a message out-of-order or twice ![Image](/uploads/documents/f/2/9/f/f29f9b3

QPS metrics ## • Hypothesis TiDB uses Raft consensus algorithm to replicate data and provide fault-tolerance o Kill one TiDB instance, if the instance has a leader replica, the QPS may drop because the

streams, not synopses => results are typically exact • Challenges: computation progress, fault-tolerance and result guarantees, automatic scaling and state migration, out-of-order processing ## Distributed

distributability durability effectiveness efficiency reliability extensibility failure transparency fault-tolerance fidelity flexibility inspectability installability integrity interchangeability interoperability

[Image](/uploads/documents/a/8/4/0/a84004dbb3e345ee3fc74e487e8499c5/p11_1.jpg) High Availability Fault-tolerance for masters, workers, and etcd nodes ![Image](/uploads/documents/a/8/4/0/a84004dbb3e345ee3f

CONFIGURATION One node dedicated to context server, 2 nodes for elasticsearch storage with fault-tolerance ## Node A : node.data=false numberOfReplicas=1 monthlyIndex.numberOfReplicas=1 ## Node B :