different things • last 5 sec • last 10 events • last 1h every 10 min • last user session Window operators 2 Vasiliki Kalavri | Boston University 2020 object MaxSensorReadings { def main(args: 0)))
 .keyBy(_.id) .timeWindow(Time.minutes(1)) .max("temp")
 } } 3 Example: Window sensor readings Vasiliki Kalavri | Boston University 2020 In the DataStream API, you can use the or IngestionTime Vasiliki Kalavri | Boston University 2020 Window operators can be applied on a keyed or a non-keyed stream: • Window operators on keyed windows are evaluated in parallel • Non-keyed

Processing time • the time of the local clock where an event is being processed • a processing-time window wouldn’t account for game activity while the train is in the tunnel • results depend on the processing and aren’t deterministic • Event time • the time when an event actually happened • an event-time window would give you the extra life • results are deterministic and independent of the processing event-times of non-late data Watermark propagation 12 Vasiliki Kalavri | Boston University 2020 13 Event-time update Vasiliki Kalavri | Boston University 2020 1. Watermarks must be monotonically increasing

key MovingAverage average = averageState.value(); // create a new MovingAverage (with window size 2) if none exists for this key if (average == null) average = new MovingAverage(2); keyBy() .window(<window assigner>) .reduce|aggregate|process(<window function>) stream. .windowAll(<window assigner>) .reduce|aggregate|process(<window function>) ◦TumblingEventTimeWindows s.withGap(Time.minutes(30)) DataStream input = ... input .keyBy(“key”) .window(TumblingEventTimeWindows.of(Time.minutes(1))) .process(new MyWastefulMax()); public static class

FlinkKafkaConsumer The maximum lag in terms of the number of records for any partition in this window. An increasing value over time is your best indication that the consumer group is not keeping up section). 3. Some operators in a streaming topology need to buffer events for some time (e.g. in a time window) for functional reasons. 4. Each computation in your Flink topology (framework or user code), growing state are very application-specific. Typically, an increasing number of keys, a large event-time skew between different input streams or simply missing state cleanup may cause growing state.

DStream. 23 / 79 Window Operations (1/3) ▶ Spark provides a set of transformations that apply to a over a sliding window of data. ▶ A window is defined by two parameters: window length and slide interval interval. ▶ A tumbling window effect can be achieved by making slide interval = window length 24 / 79 Window Operations (2/3) ▶ window(windowLength, slideInterval) • Returns a new DStream which is computed based on windowed batches. ▶ countByWindow(windowLength, slideInterval) • Returns a sliding window count of elements in the stream. ▶ reduceByWindow(func, windowLength, slideInterval) • Returns

removable media . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.8.5 Resizing the machine’s window . . . . . . . . . . . . . . . . . . . . . . 24 1.8.6 Saving the state of the machine . . . . . Configure VM window menu entries . . . . . . . . . . . . . . . . . . . . 184 9.19.5 Configure VM window status bar entries . . . . . . . . . . . . . . . . . 187 9.19.6 Configure VM window visual modes shows you how VirtualBox, installed on a Mac computer, is running Windows 7 in a virtual machine window: In this User Manual, we’ll begin simply with a quick introduction to virtualization and how to

removable media . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.8.5 Resizing the machine’s window . . . . . . . . . . . . . . . . . . . . . . 24 1.8.6 Saving the state of the machine . . . . . Configure VM window menu entries . . . . . . . . . . . . . . . . . . . . 184 9.19.5 Configure VM window status bar entries . . . . . . . . . . . . . . . . . 187 9.19.6 Configure VM window visual modes shows you how VirtualBox, installed on a Mac computer, is running Windows 7 in a virtual machine window: In this User Manual, we’ll begin simply with a quick introduction to virtualization and how to

removable media . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.8.5 Resizing the machine’s window . . . . . . . . . . . . . . . . . . . . . . 24 1.8.6 Saving the state of the machine . . . . . Configure VM window menu entries . . . . . . . . . . . . . . . . . . . . 199 9.20.5 Configure VM window status bar entries . . . . . . . . . . . . . . . . . 202 9.20.6 Configure VM window visual modes shows you how VirtualBox, installed on a Mac computer, is running Windows 8 in a virtual machine window: In this User Manual, we’ll begin simply with a quick introduction to virtualization and how to

removable media . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.8.5 Resizing the machine’s window . . . . . . . . . . . . . . . . . . . . . . 24 1.8.6 Saving the state of the machine . . . . . Configure VM window menu entries . . . . . . . . . . . . . . . . . . . . 199 9.20.5 Configure VM window status bar entries . . . . . . . . . . . . . . . . . 202 9.20.6 Configure VM window visual modes shows you how VirtualBox, installed on a Mac computer, is running Windows 8 in a virtual machine window: In this User Manual, we’ll begin simply with a quick introduction to virtualization and how to

removable media . . . . . . . . . . . . . . . . . . . . . . . . . 24 1.8.5 Resizing the machine’s window . . . . . . . . . . . . . . . . . . . . . . 24 1.8.6 Saving the state of the machine . . . . . Configure VM window menu entries . . . . . . . . . . . . . . . . . . . . 212 9.20.5 Configure VM window status bar entries . . . . . . . . . . . . . . . . . 215 9.20.6 Configure VM window visual modes shows you how VirtualBox, installed on a Mac computer, is running Windows 8 in a virtual machine window: In this User Manual, we’ll begin simply with a quick introduction to virtualization and how to