contains its own Python executable files and the installed Python packages. It is useful for local development to create a standalone Python environment and also useful when deploying a PyFlink job to production Local This page shows you how to set up PyFlink development environment in your local machine. This is usually used for local execution or development in an IDE. Set up Python environment It requires commonly used Python virtual environments on the cluster nodes of the standalone cluster and use custom Python virtual environment when there are some special requirements. Submit PyFlink jobs to a standalone

contains its own Python executable files and the installed Python packages. It is useful for local development to create a standalone Python environment and also useful when deploying a PyFlink job to production Local This page shows you how to set up PyFlink development environment in your local machine. This is usually used for local execution or development in an IDE. Set up Python environment It requires commonly used Python virtual environments on the cluster nodes of the standalone cluster and use custom Python virtual environment when there are some special requirements. Submit PyFlink jobs to a standalone

limitation on the stream: updates cannot change past entries in A. 11 Useful in theory for the development of streaming algorithms With limited practical value in distributed, real-world settings Vasiliki are data channels • operators can accumulate state, have multiple inputs, express event- time custom window-based logic • some systems, like Timely Dataflow support cyclic dataflows and iterations

Flink application. I highly recommend to start monitoring your Flink application early on in the development phase. This way you will be able to improve your dashboards and alerts over time and, more importantly importantly, observe the performance impact of the changes to your application throughout the development phase. By doing so, you can ask the right questions about the runtime behaviour of your application

file systems, socket connections. 2. Advanced sources, e.g., Kafka, Flume, Kinesis, Twitter. 3. Custom sources, e.g., user-provided sources. 13 / 79 Input Operations ▶ Every input DStream is associated file systems, socket connections. 2. Advanced sources, e.g., Kafka, Flume, Kinesis, Twitter. 3. Custom sources, e.g., user-provided sources. 13 / 79 Input Operations - Basic Sources ▶ Socket connection quorum], [consumer group id], [number of partitions]) 15 / 79 Input Operations - Custom Sources (1/3) ▶ To create a custom source: extend the Receiver class. ▶ Implement onStart() and onStop(). ▶ Call

2020 input stream window assigner ... trigger evictor evaluation function result stream Custom windows 20 • Describe each component Vasiliki Kalavri | Boston University 2020 32 4 2 5 7 44 on… Vasiliki Kalavri | Boston University 2020 Advanced transformation functions used to implement custom logic for which predefined windows and transformations might not be suitable: • they provide access

event in the last minute • Tumble windows are non-overlapping fixed-size • events every hour • Custom windows have neither fixed bounds nor fixed size • events in a period during which a user was active | Boston University 2020 User-Defined Aggregates (UDAs) Constructs that allow the definition of custom aggregations using three statement groups: • INITIALIZE: initialized local state. • ITERATE:

Java JDK. A Java JRE is not sufficient! • Apache Maven 3.x. • An IDE for Java and/or Scala development, such as IntelliJ IDEA (preferred), Eclipse, or Netbeans with appropriate plugins installed.

Checkpoints sent to JobManager's heap memory, i.e. the state is lost in case of failure • Use only for development and debugging purposes! FsStateBackend • Stores state on TaskManager’s heap but checkpoints it

selectivity > 1 • a filter operator typically has selectivity < 1 Is selectivity always known at development time? ??? Vasiliki Kalavri | Boston University 2020 Types of Parallelism 7 B A C A B D