local buffer and possibly update state 3. produce output What can go wrong: • lost events • duplicate or lost state updates • wrong result 5 mi mo Was mi fully processed? Was mo delivered downstream = Oe • Rollback recovery allows duplicate tuples downstream: • repeating: duplicate tuples are identical to those produced by the primary • convergent: duplicate tuples are different but eliminating eliminating them leads to output identical to an output without failure • divergent: duplicate tuples are different and eliminating them produces different output 9 Vasiliki Kalavri | Boston University 2020

Filter out all URLs that contain malware? • Filter out all compromised passwords? • Remove duplicate tuples on recovery when using upstream backup? The membership problem ??? Vasiliki Kalavri | Filter out all URLs that contain malware? • Filter out all compromised passwords? • Remove duplicate tuples on recovery when using upstream backup? The membership problem A hash table requires O(logn)

over a window 18 Vasiliki Kalavri | Boston University 2020 Flow Management Operators (II) • Duplicate/Copy Operator replicates a stream, commonly to be used as input to multiple downstream operators

fault-tolerant publish-subscribe messaging system and serves as the ingestion, storage, and messaging layer for large production streaming pipelines. Kafka is commonly deployed on a cluster of one or more

2020 System model: • No failures during snapshotting • FIFO reliable channels: no lost or duplicate messages • Strongly connected execution graph: each process can reach every other process in