TableEnvironment. TableEnvironment is responsible for: • Table management: Table Creation, listing Tables, Conversion between Table and DataStream, etc. • User-defined function management: User-defined function registration +----+----------------------+--------------------------------+ 2 rows in set PyFlink Table also provides the conversion back to a pandas DataFrame to leverage pandas API. [14]: table.to_pandas() [14]: id data 0 1 n/beam/beam_operations_fast.pyx", line 158, in pyflink.fn_ ˓→execution.beam.beam_operations_fast.FunctionOperation.process File "pyflink/fn_execution/beam/beam_operations_fast.pyx", line 174, in pyflink

TableEnvironment. TableEnvironment is responsible for: • Table management: Table Creation, listing Tables, Conversion between Table and DataStream, etc. • User-defined function management: User-defined function registration +----+----------------------+--------------------------------+ 2 rows in set PyFlink Table also provides the conversion back to a pandas DataFrame to leverage pandas API. [14]: table.to_pandas() [14]: id data 0 1 n/beam/beam_operations_fast.pyx", line 158, in pyflink.fn_ ˓→execution.beam.beam_operations_fast.FunctionOperation.process File "pyflink/fn_execution/beam/beam_operations_fast.pyx", line 174, in pyflink

failures and guarantee correct results after recovery? • how can we ensure minimal downtime and fast recovery? • how can we hide recovery side-effects from downstream applications? Vasiliki Kalavri recovery? • How much input do we need to re-play? How expensive is it to re-construct the state? How fast can we de-duplicate output? Vasiliki Kalavri | Boston University 2020 Gap Recovery • Restart the

University 2020 DSMS with load shedder 8 Synopsis Maintenance Synopsis for S1 Synopsis for Sr … Fast approximate answers … S1 S2 Sr Input Manager Scheduler QoS Monitor Load Shedder Query caused by high congestion. • In the presence of bursty traffic, CFC causes backpressure to build up fast and propagate along congested VCs to their sources which can be throttled. • Essentially, CFC

propagation synchronized asynchronous Data historical recent and historical ETL process complex fast and light-weight ETL: Extract-Transform-Load
 e.g. unzipping compressed files, data cleaning and arrival order • Small space: memory footprint poly-logarithmic in the stream size • Low time: fast update and query times • Delete-proof: synopses can handle both insertions and deletions in an

7 ▸ Accuracy ▸ no over/under-provisioning ▸ Stability ▸ no oscillations ▸ Performance ▸ fast convergence scaling controller detect symptoms decide whether to scale decide how much Hoffmann, Desislava Dimitrova, Matthew Forshaw, and Timothy Roscoe. Three steps is all you need: fast, accurate, automatic scaling decisions for distributed streaming dataflows. (OSDI’18). • Moritz

University 2020 Mobile game application • input stream: user activity • output: rewards based on how fast the user meets goals • e.g. pop 500 bubbles within 1 minute and get extra life Vasiliki Kalavri

• can be connected to the network • latency and unpredictable delays • might be producing too fast • stream processor needs to keep up and not shed load • might be producing too slow or become

?? Vasiliki Kalavri | Boston University 2020 • Evenly distributes keys across parallel tasks • Fast to compute, no routing state • High migration cost • When a new node is added, state is shuffled