ProcessingTIme or IngestionTime Vasiliki Kalavri | Boston University 2020 Window operators can be applied on a keyed or a non-keyed stream: • Window operators on keyed windows are evaluated in parallel window assigner produces a WindowedStream (or All WindowedStream if applied on a non-keyed DataStream). • A window function is applied on a WindowedStream (or AllWindowedStream) and processes the elements computation that is performed on the elements of a window • Incremental aggregation functions are applied when an element is added to a window: • They maintain a single value as window state and eventually

• Istream (for “insert stream”) applied to relation R contains a stream element whenever tuple s is in R(τ) − R(τ − 1). • Dstream (for “delete stream”) applied to relation R contains a stream stream element whenever tuple s is in R(τ − 1) − R(τ). • Rstream (for “relation stream”) applied to relation R contains a stream element whenever tuple s is in R at time τ. 6 Vasiliki (I) • Join operators merge two streams by matching elements satisfying a condition • commonly applied on windows • Union operators combine two or more streams without ordering guarantees • elements

maintain result and selectivity semantics • Dynamism: can the optimization be applied during runtime or does it have to be applied statically? When to optimize? ??? Vasiliki Kalavri | Boston University every data item is eventually processed • Ensure each worker is qualified: if load balancing is applied after fission, each instance must be capable of processing each item and have access to necessary

buffer data that cannot yet be safely routed and configuration commands that cannot yet be applied Live state migration ??? Vasiliki Kalavri | Boston University 2020 36 control command Helper buffer data that cannot yet be safely routed and configuration commands that cannot yet be applied Live state migration Can we apply this mechanism in Flink? ??? Vasiliki Kalavri | Boston University

parallel tasks of the function’s operator. Keyed state can only be used by functions that are applied on a KeyedStream: • When the processing method of a function with keyed input is called, Flink’s

DStream: sequence of RDDs representing a stream of data. 8 / 79 DStream (2/2) ▶ Any operation applied on a DStream translates to operations on the underlying RDDs. 9 / 79 StreamingContext ▶ StreamingContext