Stream Processing and Analytics Vasiliki (Vasia) Kalavri
 vkalavri@bu.edu Spring 2020 2/11: Windows and Triggers Vasiliki Kalavri | Boston University 2020 • Practical way to perform operations on API, you can use the time characteristic to tell Flink how to define time when you are creating windows. The time characteristic is a property of the StreamExecutionEnvironment: Configuring a time characteristic be applied on a keyed or a non-keyed stream: • Window operators on keyed windows are evaluated in parallel • Non-keyed windows are processed in a single thread To create a window operator, you need

S4aps25t- S4aps25t-2 Meta Data 1NS/RT / UPDAT/ / D/2/T/ 写入 CR/AT/ TA,2/ D;ABl= ( id 1NT N5T NU22, d;E; 1NT N5T NU22, ( 1 (1,2 1 (1,2 D (1,2 1 (1,3 1 (1,2 D (1,2 1 (1,3 1 (3,5 1 (1,2

(I) • Time-based (logical) windows define their contents as a function of time. • average price of items bought within the last 5 minutes • Count-based (physical) windows define their contents according Boston University 2020 Window types (II) • Fixed windows have bound which don’t move • events received between 1/1/2019 and 12/1/2019 • Landmark windows have a fixed lower bound and the upper bound advances events since 1/1/2019 • Sliding windows have fixed size but both their bounds advance for new events • last 10 events or event in the last minute • Tumble windows are non-overlapping fixed-size

▶ Now instead, computation is kicked off explicitly by a call to the start() method. ▶ DStreams support many of the transformations available on normal Spark RDDs. 20 / 79 Transformations (2/4) ▶ map joinedStream = stream1.join(stream2) 27 / 79 Join Operation (2/3) ▶ Stream-stream joins ▶ Joins over windows of the streams. val windowedStream1 = stream1.window(Seconds(20)) val windowedStream2 = stream2 ▶ Use groupBy() and window() to express windowed aggregations. // count words within 10 minute windows, updating every 5 minutes. // streaming DataFrame of schema {time: Timestamp, word: String} val calls

6 Vasiliki Kalavri | Boston University 2020 1. Process events online without storing them 2. Support a high-level language (e.g. StreamSQL) 3. Handle missing, out-of-order, delayed data 4. Guarantee Combine batch (historical) and stream processing 6. Ensure availability despite failures 7. Support distribution and automatic elasticity 8. Offer low-latency 7 2005 Vasiliki Kalavri | Boston University 2020 Dataflow Systems Distributed execution Partitioned state Exact results Out-of-order support Single-node execution Synopses and sketches Approximate results In-order data processing Stream

ExecNodeBase. ˓→translateToPlan(ExecNodeBase.java:134) This is an issue around Java 17. It still doesn’t support Java 17 in Flink. You can refer to FLINK-15736 for more details. To solve this issue, you need to acquire(True, timeout) OverflowError: timeout value is too large This exception only occurs on Windows. It doesn’t affect the execution of PyFlink jobs and so you could ignore it usually. Besides, you

ExecNodeBase. ˓→translateToPlan(ExecNodeBase.java:134) This is an issue around Java 17. It still doesn’t support Java 17 in Flink. You can refer to FLINK-15736 for more details. To solve this issue, you need to acquire(True, timeout) OverflowError: timeout value is too large This exception only occurs on Windows. It doesn’t affect the execution of PyFlink jobs and so you could ignore it usually. Besides, you

extractTimestamp(MyEvent event) { return element.getCreationTime(); } } Windows (Not the OS) Global Vs Keyed Windows stream. .keyBy() .window() .reduce|a max)); } } Precombine Produce final result Lateness • By default, when using event-time windows, late events are dropped. stream. .keyBy(...) .window(...) .allowedLateness(Time.seconds(10))

properties 14 Vasiliki Kalavri | Boston University 2020 Watermarks are essential to both event-time windows and operators handling out-of-order events: • When an operator receives a watermark with time • It can then either trigger computation or order received events. 15 Evaluation of event-time windows Vasiliki Kalavri | Boston University 2020 16 http://streamingbook.net/fig/3-2 14 Vasiliki Kalavri

Software requirements • All assignments assume a UNIX-based setup. • If you are a Windows user, you are advised to use Windows subsystem for Linux (WSL), Cygwin, or a Linux virtual machine to run Flink in