4. Each computation in your Flink topology (framework or user code), as well as each network shuffle, takes time and adds to latency. 5. If the application emits through a transactional sink, the sink So far we have only looked at Flink-specific metrics. As long as latency & throughput of your application are in line with your expectations and it is checkpointing consistently, this is probably everything to the size of your application state (check the checkpointing metrics5 for an estimated size of the on-heap state). The possible reasons for growing state are very application-specific. Typically, an

PyFlink jobs for more details. 1.1.1.4 YARN Apache Hadoop YARN is a cluster resource management framework for managing the resources and scheduling jobs in a Hadoop cluster. It’s supported to submit PyFlink environments to use. ./bin/flink run-application -t yarn-application \ -Djobmanager.memory.process.size=1024m \ -Dtaskmanager.memory.process.size=1024m \ -Dyarn.application.name= \ -pyclientexec could not meet. ./bin/flink run-application -t yarn-application \ -Djobmanager.memory.process.size=1024m \ -Dtaskmanager.memory.process.size=1024m \ -Dyarn.application.name= \ -Dyarn

PyFlink jobs for more details. 1.1.1.4 YARN Apache Hadoop YARN is a cluster resource management framework for managing the resources and scheduling jobs in a Hadoop cluster. It’s supported to submit PyFlink environments to use. ./bin/flink run-application -t yarn-application \ -Djobmanager.memory.process.size=1024m \ -Dtaskmanager.memory.process.size=1024m \ -Dyarn.application.name= \ -pyclientexec could not meet. ./bin/flink run-application -t yarn-application \ -Djobmanager.memory.process.size=1024m \ -Dtaskmanager.memory.process.size=1024m \ -Dyarn.application.name= \ -Dyarn

Vasiliki Kalavri | Boston University 2020 Apache Flink • An open-source, distributed data analysis framework • True streaming at its core • Streaming & Batch API Historic data Kafka, RabbitMQ, ... HDFS Default parallelism for jobs. You can override this option by using env.setParallelism() in your application. taskmanager.numberOfTaskSlots: The number of parallel operator or user function instances that /bin/start-cluster.sh Stop Flink:
 ./bin/stop-cluster.sh Run an application with no arguments:
 ./bin/flink run ./examples/batch/WordCount.jar Run an application with input and output arguments:
 ./bin/flink run

2020 Grading Scheme (2) Final Project (50%): • A real-time monitoring and anomaly detection framework • To be implemented individually Deliverables • One (1) written report of maximum 5 pages Apache Flink and Kafka to build a real-time monitoring and anomaly detection framework for datacenters. Your framework will: • Detect “suspicious” event patterns • Raise alerts for abnormal system

output Redundancy elimination variations How can no-op or idempotent operators appear in an application? ??? Vasiliki Kalavri | Boston University 2020 23 Ensure the combination of A1, A2 is equivalent GET /dumprequest HTTP/1.1 Host: rve.org.uk Connection: keep-alive Accept: text/html,application/ xhtml+xml,application/ xml;q=0.9,*/*;q=0.8 User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.22 GET /dumprequest HTTP/1.1 Host: rve.org.uk Connection: keep-alive Accept: text/html,application/ xhtml+xml,application/ xml;q=0.9,*/*;q=0.8 User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.22

• To recover from failures, the system needs to • restart failed processes • restart the application and recover its state 2 Checkpointing guards the state from failures, but what about process sufficient number of processing slots in order to execute all tasks of an application. • The JobManager cannot restart the application until enough slots become available. • Restart is automatic if there standalone mode • The restart strategy determines how often the JobManager tries to restart the application and how long it waits between restart attempts. 4 TaskManager failures ??? Vasiliki Kalavri

hidden from the application developer Live state migration ??? Vasiliki Kalavri | Boston University 2020 35 control command Helper operators, hidden from the application developer Live state hidden from the application developer Live state migration ??? Vasiliki Kalavri | Boston University 2020 35 transfer state control command Helper operators, hidden from the application developer Live hidden from the application developer Live state migration ??? Vasiliki Kalavri | Boston University 2020 35 control command Helper operators, hidden from the application developer Helper operators

recorded in the snapshot but enforces the causal consistency. 3. Starts recording all data (application) messages it receives on all of its incoming channels. 20 ??? Vasiliki Kalavri | Boston University University 2020 Can we apply this algorithm to retrieve a consistent snapshot of a stream processing application? 31 ??? Vasiliki Kalavri | Boston University 2020 32 Epoch-Based Stream Execution Logged Input 1. Stop and restart the application. All operators have empty state. ??? Vasiliki Kalavri | Boston University 2020 42 Recovery process 1. Stop and restart the application. All operators have empty

multicast, TCP • HTTP or RPC if the consumer exposes a service on the network • Failure handling: application needs to be aware of message loss, producers and consumers always online 5 Message queues can subscribe to receive notifications of the event. • Refreshing distributed caches • an application can publish invalidation events to update the IDs of objects that have changed. • Logging to a residential sensor can stream data to backend servers hosted in the cloud. 24 A publisher application creates a topic and sends messages to the topic. Messages are persisted until they are