local development to create a standalone Python environment and also useful when deploying a PyFlink job to production when there are massive Python dependencies. It’s supported to use Python virtual environment during job starting up. This is more flexible and useful when it’s not possible to set up the Python environments in advance on the cluster nodes or when there are some special requirements where the pre-installed environments on the cluster nodes of the standalone cluster and use custom Python virtual environment when there are some special requirements. Submit PyFlink jobs to a standalone Flink cluster You could

local development to create a standalone Python environment and also useful when deploying a PyFlink job to production when there are massive Python dependencies. It’s supported to use Python virtual environment during job starting up. This is more flexible and useful when it’s not possible to set up the Python environments in advance on the cluster nodes or when there are some special requirements where the pre-installed environments on the cluster nodes of the standalone cluster and use custom Python virtual environment when there are some special requirements. Submit PyFlink jobs to a standalone Flink cluster You could

the Receiver class. ▶ Implement onStart() and onStop(). ▶ Call store(data) to store received data inside Spark. 16 / 79 Input Operations - Custom Sources (2/3) class CustomReceiver(host: String, port: automatically converts this batch-like query to a streaming execution plan. ▶ 2. Specify triggers to control when to update the results. • Each time a trigger fires, Spark checks for new data (new row in the input automatically converts this batch-like query to a streaming execution plan. ▶ 2. Specify triggers to control when to update the results. • Each time a trigger fires, Spark checks for new data (new row in the input

distribution of each stage. In the rest of this section, we will only consider latency, which is introduced inside the Flink topology and cannot be attributed to transactional sinks or events being buffered for for functional reasons (4.). To this end, Flink comes with a feature called Latency Tracking4. When enabled, Flink will insert so-called latency markers periodically at all sources. For each sub-task, a most JVM applications - is the most volatile and important metric to watch. This is especially true when using Flink’s filesystem statebackend as it keeps all state objects on the JVM Heap. If the size

.flatMap(new TemperatureAlertFunction(1.7)) Using state in Flink 15 KeyedStream State access inside the flatMap will be scoped to the key being processed Vasiliki Kalavri | Boston University 2020 function’s operator. Keyed state can only be used by functions that are applied on a KeyedStream: • When the processing method of a function with keyed input is called, Flink’s runtime automatically puts ListCheckpointed interface • snapshotState() is invoked when Flink triggers a checkpoint of the stateful function. • restoreState() is always invoked when the job is started or in the case of a failure. Vasiliki

Kalavri | Boston University 2020 Outcomes At the end of the course, you will hopefully: • know when to use stream processing vs other technology • be able to comprehensively compare features and processing factors affect their performance • be aware of the challenges and trade-offs one needs to consider when designing and deploying streaming applications 6 Vasiliki Kalavri | Boston University 2020 Grading Kalavri | Boston University 2020 Consider a set of 1000 sensors deployed in different locations inside a forest. The sensors monitor temperature and smoke levels and generate a measurement every 5 seconds

University 2020 3 friend follows London Zurich Berlin “conservative” “liberal” If you like “Inside job” you might also like “The Bourne Identity” What’s the cheapest way to reach Zurich from London • Each vertex is also assigned a sign, (+) or (-) • Edge endpoints must have different signs • When merging components, if flipping all signs doesn’t work => the graph is not bipartite Bipartite

rate increase : input rate : throughput ??? Vasiliki Kalavri | Boston University 2020 Control: When and how much to adapt? Mechanism: How to apply the re-configuration? 3 • Detect environment changes: straggler workers, skew • Enumerate scaling actions, predict their effects, and decide which and when to apply • Allocate new resources, spawn new processes or release unused resources, safely terminate output • amounts to the time an operator instance runs for if executed in an ideal setting • when there is no waiting the useful time is equal to the observed time 18 Useful time Wu ??? Vasiliki

streams must flow through. • Pattern-based languages specify conditions and actions to be taken when conditions are met. • Conditions are commonly described as patterns that can match input stream (B(Y=10);[timespan:5] C(Z<5))[within:15] A, B, C are topics X, Y, Z are inner fields The rule fires when an item of type A having an attribute X > 0 enters the system and also an item of type B with satisfied when all items have been detected. • disjunction of items I1, I2, …, In is satisfied when at least one item has been detected. • repetition of an item I of degree (m, n) is satisfied when I is

handles and checkpoint locations 5 JobManager failures ??? Vasiliki Kalavri | Boston University 2020 When the JobManager fails all tasks are automatically cancelled. The new JobManager performs the following straggler workers, skew • Enumerate scaling actions, predict their effects, and decide which and when to apply • Allocate new resources, spawn new processes or release unused resources, safely terminate computations to ensure result correctness ??? Vasiliki Kalavri | Boston University 2020 Control: When and how much to adapt? 12 • Detect environment changes: external workload and system performance