automatic if there is a ResourceManager, e.g. in a YARN setup • A manual TaskManager re-start or a backup is required in standalone mode • The restart strategy determines how often the JobManager tries University 2020 • State is mapped into key-groups • Key-groups are mapped to subtasks as ranges • On restore, reads are sequential within each key-group, and often across multiple key-groups • The metadata

recovery (at-least-once) • It avoids information loss • The output may contain duplicates • A backup needs to rebuild state of the failed node 8 Vasiliki Kalavri | Boston University 2020 Recovery output may contain duplicates • A backup needs to rebuild state of the failed node • Gap recovery (at-most-once) • It drops data during failure • The backup starts from most recent information Can you see any disadvantage in this approach? Vasiliki Kalavri | Boston University 2020 Upstream Backup Upstream nodes act as backups for their downstream operators by logging tuples in their output

unique IDs. • Operators send acks when a record has been processed. • Records are dropped from the backup when the have been fully acknowledged. ▶ Fault tolerance in Flink • More coarse-grained approach unique IDs. • Operators send acks when a record has been processed. • Records are dropped from the backup when the have been fully acknowledged. ▶ Fault tolerance in Flink • More coarse-grained approach unique IDs. • Operators send acks when a record has been processed. • Records are dropped from the backup when the have been fully acknowledged. ▶ Fault tolerance in Flink • More coarse-grained approach

Filter out all compromised passwords? • Remove duplicate tuples on recovery when using upstream backup? The membership problem ??? Vasiliki Kalavri | Boston University 2020 22 What data structure Filter out all compromised passwords? • Remove duplicate tuples on recovery when using upstream backup? The membership problem A hash table requires O(logn) bits per element which might still be

checkpoint it, restore it, re- scale it Unmanaged Managed What are the advantages and disadvantages of each approach? Vasiliki Kalavri | Boston University 2020 • Copy, checkpoint, restore, merge, split

restoreInternal(StreamTask. ˓→java:687) at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task

restoreInternal(StreamTask. ˓→java:687) at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:654) at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task

2020 Fault-tolerance approaches recap 3 Vasiliki Kalavri | Boston University 2020 Upstream Backup Upstream nodes act as backups for their downstream operators by logging tuples in their output