jobs to YARN cluster It supports to execute PyFlink jobs in application mode, per-job mode and session mode in YARN deployment. You could execute PyFlink jobs in application mode as following: ./bin/flink the current shell environment. You could also execute PyFlink jobs in session mode as following: ./bin/flink run -t yarn-session \ -Djobmanager.memory.process.size=1024m \ -Dtaskmanager.memory.process /path/to/venv/bin/python3 \ -pyexec venv.zip/venv/bin/python3 \ -py shipfiles/word_count.py See Session Mode for more details. Note: Same as the per-job mode, the option -pyclientexec should point to

jobs to YARN cluster It supports to execute PyFlink jobs in application mode, per-job mode and session mode in YARN deployment. You could execute PyFlink jobs in application mode as following: ./bin/flink the current shell environment. You could also execute PyFlink jobs in session mode as following: ./bin/flink run -t yarn-session \ -Djobmanager.memory.process.size=1024m \ -Dtaskmanager.memory.process /path/to/venv/bin/python3 \ -pyexec venv.zip/venv/bin/python3 \ -py shipfiles/word_count.py See Session Mode for more details. Note: Same as the per-job mode, the option -pyclientexec should point to

might mean different things • last 5 sec • last 10 events • last 1h every 10 min • last user session Window operators 2 Vasiliki Kalavri | Boston University 2020 object MaxSensorReadings { def activity followed by a period of inactivity session gap key 3 key 2 key 1 Session windows 12 Vasiliki Kalavri | Boston University 2020 // event-time session windows assigner val sessionWindows = sensorData keyBy(_.id) // create event-time session windows with a 15 min gap .window(EventTimeSessionWindows.withGap(Time.minutes(15))) .process(…) 13 Session window example Window assigner Window

events? • clicks per user session? 49 t t+1 t+3 t+4 t+5 t+6 t+7 t+2 logged in logged out How would you compute… • the maximum every 100 events? • clicks per user session? • faster than the batch

built-in timeouts • Think what would happen in our example, if the event signaling the end of the user session was lost, or had not arrived for some reason. 48 / 79 mapWithState Operation ▶ mapWithState is different types of windows: • Tumbling windows (no overlap) • Sliding windows (with overlap) • Session windows (punctuated by a gap of inactivity) 71 / 79 Watermark and Late Elements ▶ It is possible