University 2020 • Change parallelism • scale out to process increased load • scale in to save resources • Fix bugs or change business logic • Optimize execution plan • Change operator placement predict their effects, and decide which and when to apply • Allocate new resources, spawn new processes or release unused resources, safely terminate processes • Adjust dataflow channels and network connections predict their effects, and decide which and when to apply • Allocate new resources, spawn new processes or release unused resources, safely terminate processes • Adjust dataflow channels and network connections

minimize disk access • scheduling Objectives • optimize resource utilization or minimize resources • decrease latency, increase throughput • minimize monetary costs (if running in the cloud) Safety • Ensure resource kinds: all resources required by a fused operator should remain available. • Ensure resource amounts: the total amount of resources required by the fused operator must be | Boston University 2020 35 Safety • Ensure resource availability: the host must have enough resources for all assigned operators • Ensure security constraints: what are the trusted hosts for each

.............................................................................. 22 4.14 System Resources................................................................................................ decreasing the number of task slots per TaskManager (in case of a Standalone setup), by providing more resources to the TaskManager (in case of a containerized setup), or by providing more TaskManagers. In general ease-1.7/monitoring/metrics.html#system-resources 10 https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html 4.14 System Resources In addition to the JVM metrics above, it

file:///home/user/wordcount_out 19 Flink commands Vasiliki Kalavri | Boston University 2020 Resources • Documentation • https://flink.apache.org/ • Community • https://flink.apache.org/community failures without losing any records committed to the log. Vasiliki Kalavri | Boston University 2020 Resources • Documentation • https://kafka.apache.org/ • Community • https://kafka.apache.org/contact

Scale resource allocation: • Addresses the case of increased load and additionally ensures no resources are left idle when the input load decreases. ??? Vasiliki Kalavri | Boston University 2020 Load system processing capacity H: headroom factor, i.e. a conservative estimate of the percentage of resources required by the system at steady state Load(N(I)): the load as a fraction of the total capacity

predict their effects, and decide which and when to apply • Allocate new resources, spawn new processes or release unused resources, safely terminate processes • Adjust dataflow channels and network connections

1.1.1.4 YARN Apache Hadoop YARN is a cluster resource management framework for managing the resources and scheduling jobs in a Hadoop cluster. It’s supported to submit PyFlink jobs to YARN for execution talking with Kubernetes and allocating and de-allocating TaskManagers depending on the required resources. ./bin/flink run-application \ --target kubernetes-application \ --parallelism 8 \ -Dkubernetes

1.1.1.4 YARN Apache Hadoop YARN is a cluster resource management framework for managing the resources and scheduling jobs in a Hadoop cluster. It’s supported to submit PyFlink jobs to YARN for execution talking with Kubernetes and allocating and de-allocating TaskManagers depending on the required resources. ./bin/flink run-application \ --target kubernetes-application \ --parallelism 8 \ -Dkubernetes

joins • Examples • Real-time statistics, e.g. weather maps • Monitor conditions to adjust resources, e.g. power generation • Monitor energy consumption for billing purposes 22 Vasiliki Kalavri

Dive on Apache Flink State - Seth Wiesman: https:// www.youtube.com/watch?v=9GF8Hwqzwnk Further resources 24