Machine Learning (ML) pipelines and ETL processes. If you’re already familiar with Python and libraries such as Pandas, then PyFlink makes it simpler to leverage the full capabilities of the Flink ecosystem Table Creation Table is a core component of the Python Table API. A Table object describes a pipeline of data transformations. It does not Getting Started 19 pyflink-docs, Release release-1.15 DataStream Creation DataStream is a core component of the Python DataStream API. A DataStream object describes a pipeline of data transformations.

Machine Learning (ML) pipelines and ETL processes. If you’re already familiar with Python and libraries such as Pandas, then PyFlink makes it simpler to leverage the full capabilities of the Flink ecosystem Table Creation Table is a core component of the Python Table API. A Table object describes a pipeline of data transformations. It does not Getting Started 19 pyflink-docs, Release release-1.16 DataStream Creation DataStream is a core component of the Python DataStream API. A DataStream object describes a pipeline of data transformations.

4 5 3 . . . 1 3, 4 2 1, 4 5 3 . . . ??? Vasiliki Kalavri | Boston University 2020 15 A component is a subgraph in which every vertex is reachable from all other vertices in the subgraph. Connected State: the graph and a component ID per vertex • initially equal to vertex ID • Iterative step: For each vertex • choose the min of neighbors’ component IDs and own component ID as the new ID • • if the component ID changed since the last iteration, notify neighbors 16 ??? Vasiliki Kalavri | Boston University 2020 1 4 3 2 5 i=0 Batch Connected Components 17 6 7 8 ??? Vasiliki Kalavri

Operator state Keyed state State types 6 Vasiliki Kalavri | Boston University 2020 A pluggable component that determines how state is stored, accessed, and maintained. State backends are responsible

. trigger evictor evaluation function result stream Custom windows 20 • Describe each component Vasiliki Kalavri | Boston University 2020 32 4 2 5 7 44 8 18 Window max over 5 last elements

Boston University 2020 Implementation • Load shedding is commonly implemented by a standalone component integrated with the stream processor • The load shedder continuously monitors input rates or

Apache Flink • The TaskManagers ship data from sending tasks to receiving tasks. • The network component of a TaskManager collects records in buffers before they are shipped, i.e., records are not shipped

2 Synopsis: a lossy, compact summary of the input stream input stream synopsis maintenance component user queries approximate results ??? Vasiliki Kalavri | Boston University 2020 A simple and