processes or release unused resources, safely terminate processes • Adjust dataflow channels and network connections • Re-partition and migrate state in a consistent manner • Block and unblock computations scaling controller detect symptoms decide whether to scale decide how much to scale metrics policy scaling action ??? Vasiliki Kalavri | Boston University 2020 Automatic scaling requirements 7 scaling controller detect symptoms decide whether to scale decide how much to scale metrics policy scaling action ??? Vasiliki Kalavri | Boston University 2020 Scaling approaches Metrics • service

for_row_format('/tmp/sink', Encoder.simple_string_encoder("UTF-8")) .with_rolling_policy(RollingPolicy.default_rolling_policy( part_size=1024 ** 3, rollover_interval=15 * 60 * 1000, inactivity_interval=5 *␣

for_row_format('/tmp/sink', Encoder.simple_string_encoder("UTF-8")) .with_rolling_policy(RollingPolicy.default_rolling_policy( part_size=1024 ** 3, rollover_interval=15 * 60 * 1000, inactivity_interval=5 *␣

records within a partition. It uniquely identifies records within each partition. The retention policy defines a time period after a record is published that it is available for consumption. Records

? Vasiliki Kalavri | Boston University 2020 Load shedding as an optimization problem N: query network I: set of input streams with known arrival rates C: system processing capacity H: headroom Load(N(I)): the load as a fraction of the total capacity C that network N(I) presents Uacc: the aggregate utility 6 Find a new network N' such that Load(N’(I))< H x C and Uacc(N(I)) - Uacc(N'I)) University 2020 Backpressure 20 ??? Vasiliki Kalavri | Boston University 2020 Rate control • In a network of consumers and producers such as a streaming execution graph with multiple operators, back-pressure

distributed • out-of-sync sources may produce out-of-order streams • can be connected to the network • latency and unpredictable delays • might be producing too fast • stream processor needs to implement triggers? • Direct messaging • Direct network communication, UDP multicast, TCP • HTTP or RPC if the consumer exposes a service on the network • Failure handling: application needs to be aware Cloud Pub/Sub Publishers and Subscribers are applications. 23 Use-cases • Balancing workloads in network clusters • tasks can be efficiently distributed among multiple workers, such as Google Compute

processes or release unused resources, safely terminate processes • Adjust dataflow channels and network connections • Re-partition and migrate state in a consistent manner • Block and unblock computations processes or release unused resources, safely terminate processes • Adjust dataflow channels and network connections • Re-partition and migrate state in a consistent manner • Block and unblock computations processes or release unused resources, safely terminate processes • Adjust dataflow channels and network connections • Re-partition and migrate state in a consistent manner • Block and unblock computations

functional reasons. 4. Each computation in your Flink topology (framework or user code), as well as each network shuffle, takes time and adds to latency. 5. If the application emits through a transactional sink 7 7 https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#configuring-the-network-buffers 8 https://www.da-platform.com/blog/manage-rocksdb-memory-size-apache-flink? __hstc=216506377 to 250 megabyte by default • The biggest driver of Direct memory is by far the number of Flink’s network buffers, which can be configured7. • Mapped memory is usually close to zero as Flink does not

would you compute… ??? Vasiliki Kalavri | Boston University 2020 51 • TaskManagers have a pool of network buffers to send and receive data. • If the sender and receiver run in separate processes, they task serializes the outgoing records into a byte buffer. • A TaskManager needs one dedicated network buffer for each receiving task that any of its tasks need to send data to. Batching in Apache Apache Flink • The TaskManagers ship data from sending tasks to receiving tasks. • The network component of a TaskManager collects records in buffers before they are shipped, i.e., records are not shipped

I’2 O’1 O’2 • The communication network ensures order-preserving, reliable message transport, e.g. TCP. • Failures are single-node and fail- stop, i.e. no network partitions or multiple simultaneous