constraints that can tolerate approximate results. Slow down the flow of data: • The system buffers excess data for later processing, once input rates stabilize. • Requires a persistent input consumed stream have managed buffer pools with bounded capacity. • A buffer pool is a set of buffers which are recycled after they have been consumed and can be re-used. ??? Vasiliki Kalavri | Boston soon as it is consumed. • The producer slows down according to the rate the consumer recycles buffers. Remote exchange: If tasks run on different worker nodes, the buffer can be recycled as soon as

https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/config.html#configuring-the-network-buffers 8 https://www.da-platform.com/blog/manage-rocksdb-memory-size-apache-flink? __hstc=216506377.c9d megabyte by default • The biggest driver of Direct memory is by far the number of Flink’s network buffers, which can be configured7. • Mapped memory is usually close to zero as Flink does not use memory-mapped

compute… ??? Vasiliki Kalavri | Boston University 2020 51 • TaskManagers have a pool of network buffers to send and receive data. • If the sender and receiver run in separate processes, they communicate sending tasks to receiving tasks. • The network component of a TaskManager collects records in buffers before they are shipped, i.e., records are not shipped one by one but batched. ??? Vasiliki Kalavri

} out.collect(new Tuple3<>(key, context.window().getEnd(), max)); } } Buffers all the events DataStream input = ... input .keyBy(x -> x.key) .window(T

‘org.apache.flink.table.factories.DynamicTableFactory’ in the classpath Exception Stack: py4j.protocol.Py4JJavaError: An error occurred while calling o13.execute. : org.apache.flink.table.api.ValidationException: official PyFlink documentation. 1.3.4.2 O2: ClassNotFoundException: com.mysql.cj.jdbc.Driver py4j.protocol.Py4JJavaError: An error occurred while calling o13.execute. : org.apache.flink.runtime.client.J occurred while calling z:org.apache.flink.client.python.PythonEnvUtils.resetCallbackClient py4j.protocol.Py4jError: An error occurred while calling z:org.apache.flink.client. ˓→python.PythonEnvUtils.resetCallbackClient

‘org.apache.flink.table.factories.DynamicTableFactory’ in the classpath Exception Stack: py4j.protocol.Py4JJavaError: An error occurred while calling o13.execute. : org.apache.flink.table.api.ValidationException: official PyFlink documentation. 1.3.4.2 O2: ClassNotFoundException: com.mysql.cj.jdbc.Driver py4j.protocol.Py4JJavaError: An error occurred while calling o13.execute. : org.apache.flink.runtime.client.J occurred while calling z:org.apache.flink.client.python.PythonEnvUtils.resetCallbackClient py4j.protocol.Py4jError: An error occurred while calling z:org.apache.flink.client. ˓→python.PythonEnvUtils.resetCallbackClient

Pre-Commit (snapshot) 1a 1c 2a 1b 1b 2b 2a Commit pre-committed 2b Mark Committed The Epoch Commit Protocol Output Logs 38 ??? Vasiliki Kalavri | Boston University 2020 Asynchronous checkpoints in Apache