University 2020 22 ??? Vasiliki Kalavri | Boston University 2020 22 Durably buffer events in a channel or source Adjust processing rate of all operators to that of the slowest part of the pipeline If tasks run on different worker nodes, the buffer can be recycled as soon as it is on the TCP channel. • If there is no buffer on the consumer side, reading from the TCP connection is interrupted virtual channels multiplexed over TCP connections: • In the presence of skew, a single overload channel can cause the slowdown of the entire dataflow… can we do better? 25 ??? Vasiliki Kalavri | Boston

that • Avoid deadlocks: if split cannot push data because one channel is full and merge cannot receive data because another channel is empty Operator fission Data parallelism, replication A A

the first time: 1. Records its own state. 2. Marks the channel that the marker came in on as empty. a. Future messages arriving on this channel will no be part of the snapshot. 3. Sends markers to its incoming channels except the one marked as empty. Otherwise: 1. It stops recording on the channel it received the marker from. 21 ??? Vasiliki Kalavri | Boston University 2020 Completing a snapshot local state and ll processes have received markers on all incoming channels and have recorded all channel states. We can then collect the locally recorded states and construct a global snapshot. 22

computation: load in terms of computation • communication: load in terms of flow size in the input channel of each parallel task • Partitioning function performance • space required to implement routing