shipfiles/word_count.py Note: Per-Job mode has been deprecated since Flink 1.15 and may be dropped in the future releases. It’s suggested to use Application mode. See YARN Per-Job Mode for more details. It should FutureWarning: Schema passed to names= option, please pass␣ ˓→schema= explicitly. Will raise exception in future return pa.RecordBatch.from_arrays(arrays, schema) [5]: root |-- id: BIGINT |-- data: STRING Create FutureWarning: Schema passed to names= option, please pass␣ ˓→schema= explicitly. Will raise exception in future return pa.RecordBatch.from_arrays(arrays, schema) [21]: _c0 0 2 1 3 [ ]: # use the Python function

shipfiles/word_count.py Note: Per-Job mode has been deprecated since Flink 1.15 and may be dropped in the future releases. It’s suggested to use Application mode. See YARN Per-Job Mode for more details. It should FutureWarning: Schema passed to names= option, please pass␣ ˓→schema= explicitly. Will raise exception in future return pa.RecordBatch.from_arrays(arrays, schema) [5]: root |-- id: BIGINT |-- data: STRING Create FutureWarning: Schema passed to names= option, please pass␣ ˓→schema= explicitly. Will raise exception in future return pa.RecordBatch.from_arrays(arrays, schema) [21]: _c0 0 2 1 3 [ ]: # use the Python function

notify() subscribe() unsubscribe() subscribe notify unsubscribe advertise(): information reg. future events Publish/Subscribe Systems 17 Pub/Sub levels of de-coupling • Space: interacting parties

Kalavri | Boston University 2020 Stock trading • Discover correlations, identify trends, forecast future values Examples • Find all stocks priced between $20 and $200, where the spread between the

operators requires preserving the key semantics: • Existing state for a particular key and all future events with this key must be routed to the same parallel instance • Some kind of hashing is typically

record timestamps and watermarks • they can register timers that trigger at a specific time in the future ProcessFunction, KeyedProcessFunction, CoProcessFunction, ProcessJoinFunction, BroadcastProcessFunction

case how to define blocking operators, e.g. groupBy? • The data might be too large to store for future use. 33 Vasiliki Kalavri | Boston University 2020 ESL: Expressive Stream Language • Ad-hoc SQL

time: 1. Records its own state. 2. Marks the channel that the marker came in on as empty. a. Future messages arriving on this channel will no be part of the snapshot. 3. Sends markers to all its