the docs python3 setup.py build_sphinx 4. Open the pyflink-docs/build/sphinx/html/index.html in the Browser 1.1 Getting Started This page summarizes the basic steps required to setup and get started environments to use. ./bin/flink run-application -t yarn-application \ -Djobmanager.memory.process.size=1024m \ -Dtaskmanager.memory.process.size=1024m \ -Dyarn.application.name= \ -pyclientexec could not meet. ./bin/flink run-application -t yarn-application \ -Djobmanager.memory.process.size=1024m \ -Dtaskmanager.memory.process.size=1024m \ -Dyarn.application.name= \ -Dyarn

the docs python3 setup.py build_sphinx 4. Open the pyflink-docs/build/sphinx/html/index.html in the Browser 1.1 Getting Started This page summarizes the basic steps required to setup and get started environments to use. ./bin/flink run-application -t yarn-application \ -Djobmanager.memory.process.size=1024m \ -Dtaskmanager.memory.process.size=1024m \ -Dyarn.application.name= \ -pyclientexec could not meet. ./bin/flink run-application -t yarn-application \ -Djobmanager.memory.process.size=1024m \ -Dtaskmanager.memory.process.size=1024m \ -Dyarn.application.name= \ -Dyarn

• To recover from failures, the system needs to • restart failed processes • restart the application and recover its state 2 Checkpointing guards the state from failures, but what about process tasks of an application. • The JobManager cannot restart the application until enough slots become available. • Restart is automatic if there is a ResourceManager, e.g. in a YARN setup • A manual standalone mode • The restart strategy determines how often the JobManager tries to restart the application and how long it waits between restart attempts. 4 TaskManager failures ??? Vasiliki Kalavri

(framework or user code), as well as each network shuffle, takes time and adds to latency. 5. If the application emits through a transactional sink, the sink will only commit and publish transactions upon successful So far we have only looked at Flink-specific metrics. As long as latency & throughput of your application are in line with your expectations and it is checkpointing consistently, this is probably everything to the size of your application state (check the checkpointing metrics5 for an estimated size of the on-heap state). The possible reasons for growing state are very application-specific. Typically, an

Vasiliki Kalavri | Boston University 2020 Software requirements • All assignments assume a UNIX-based setup. • If you are a Windows user, you are advised to use Windows subsystem for Linux (WSL), Cygwin

Integer age) { … }; } Person person = new Person("Fred Flintstone", 35); Setup • https://training.ververica.com/devEnvSetup.html • Datasets:
 wget http://training.ververica

output Redundancy elimination variations How can no-op or idempotent operators appear in an application? ??? Vasiliki Kalavri | Boston University 2020 23 Ensure the combination of A1, A2 is equivalent GET /dumprequest HTTP/1.1 Host: rve.org.uk Connection: keep-alive Accept: text/html,application/ xhtml+xml,application/ xml;q=0.9,*/*;q=0.8 User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.22 GET /dumprequest HTTP/1.1 Host: rve.org.uk Connection: keep-alive Accept: text/html,application/ xhtml+xml,application/ xml;q=0.9,*/*;q=0.8 User-Agent: Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.22

hidden from the application developer Live state migration ??? Vasiliki Kalavri | Boston University 2020 35 control command Helper operators, hidden from the application developer Live state hidden from the application developer Live state migration ??? Vasiliki Kalavri | Boston University 2020 35 transfer state control command Helper operators, hidden from the application developer Live hidden from the application developer Live state migration ??? Vasiliki Kalavri | Boston University 2020 35 control command Helper operators, hidden from the application developer Helper operators

recorded in the snapshot but enforces the causal consistency. 3. Starts recording all data (application) messages it receives on all of its incoming channels. 20 ??? Vasiliki Kalavri | Boston University University 2020 Can we apply this algorithm to retrieve a consistent snapshot of a stream processing application? 31 ??? Vasiliki Kalavri | Boston University 2020 32 Epoch-Based Stream Execution Logged Input 1. Stop and restart the application. All operators have empty state. ??? Vasiliki Kalavri | Boston University 2020 42 Recovery process 1. Stop and restart the application. All operators have empty

multicast, TCP • HTTP or RPC if the consumer exposes a service on the network • Failure handling: application needs to be aware of message loss, producers and consumers always online 5 Message queues can subscribe to receive notifications of the event. • Refreshing distributed caches • an application can publish invalidation events to update the IDs of objects that have changed. • Logging to a residential sensor can stream data to backend servers hosted in the cloud. 24 A publisher application creates a topic and sends messages to the topic. Messages are persisted until they are