perfectly balanced among workers • No routing table required • Key semantics are not preserved: values of the same key might be routed to different workers • Workers are responsible for roughly • Consider the problem of throwing n balls to n bins sequentially (balls -> records, bins -> workers) • Bins are selected uniformly at random • At the end of the process, the maximum load is Θ(ln • Choose one among n workers • check the load of each worker and send the item to the least loaded one • load checking for every item can be expensive • Choose two workers at random and send the

changes: external workload and system performance • Identify bottleneck operators, straggler workers, skew • Enumerate scaling actions, predict their effects, and decide which and when to apply changes: external workload and system performance • Identify bottleneck operators, straggler workers, skew • Enumerate scaling actions, predict their effects, and decide which and when to apply changes: external workload and system performance • Identify bottleneck operators, straggler workers, skew • Enumerate scaling actions, predict their effects, and decide which and when to apply

Balancing workloads in network clusters • tasks can be efficiently distributed among multiple workers, such as Google Compute Engine instances. • Distributing event notifications • a service that

size for the JobManager (coordinator). taskmanager.heap.size: JVM heap size for the TaskManagers (workers). parallelism.default: Default parallelism for jobs. You can override this option by using env

support cyclic dataflows and iterations on streams • Operators are data-parallel • distributed workers (threads) execute one parallel instance of one of more operators on disjoint data partitions 36

changes: external workload and system performance • Identify bottleneck operators, straggler workers, skew • Enumerate scaling actions, predict their effects, and decide which and when to apply