even a simple counter might differ on a combined stream vs. on separate streams Redundancy elimination Eliminate redundant operations, aka subgraph sharing B A B C D A B C D ??? Vasiliki Kalavri on a single core, one with operators B and C, the other with operators B and D. Redundancy elimination B A B C D A B C D ??? Vasiliki Kalavri | Boston University 2020 22 • Multi-tenancy • several queries • when applications analyze data streams from a small set of sources • Operator elimination • remove a no-op, e.g. a projection that keeps all attributes • remove idempotent operations

Kalavri | Boston University 2020 Precise recovery To provide precise recovery, we need duplicate elimination methods: • In passive and active standby, the failover node must ask downstream operators for

their bounds advance for new events • last 10 events or event in the last minute • Tumble windows are non-overlapping fixed-size • events every hour • Custom windows have neither fixed bounds nor SELECT CustomerID,‘pattern123’ FROM state WHERE sno = 3; } } Initialize state to 0 Check next event Pattern failed Order matched Refund and cancel matched Output success!

for accuracy • Query results are approximate with either deterministic or probabilistic error bounds • There is no universal synopsis solution • They are purpose-built and query-specific • different

1.16 Python 3.6 to 3.9 PyFlink 1.15 Python 3.6 to 3.8 PyFlink 1.14 Python 3.6 to 3.8 You could check your Python version as following: 3 pyflink-docs, Release release-1.15 python3 --version Create apache-flink Installing from Source To install PyFlink from source, you could refer to Build PyFlink. Check the installed package You could then perform the following checks to make sure that the installed 1] # +I[be,--that, 1] # ... If there are any problems, you could perform the following checks. Check the logging messages in the log file to see if there are any problems: # Get the installation directory

1.16 Python 3.6 to 3.9 PyFlink 1.15 Python 3.6 to 3.8 PyFlink 1.14 Python 3.6 to 3.8 You could check your Python version as following: 3 pyflink-docs, Release release-1.16 python3 --version Create apache-flink Installing from Source To install PyFlink from source, you could refer to Build PyFlink. Check the installed package You could then perform the following checks to make sure that the installed 1] # +I[be,--that, 1] # ... If there are any problems, you could perform the following checks. Check the logging messages in the log file to see if there are any problems: # Get the installation directory

migration ??? Vasiliki Kalavri | Boston University 2020 36 control command Helper operators can check the frontier (watermark) at the output of the stateful operator to ensure only complete state migration ??? Vasiliki Kalavri | Boston University 2020 36 control command Helper operators can check the frontier (watermark) at the output of the stateful operator to ensure only complete state migration ??? Vasiliki Kalavri | Boston University 2020 36 control command Helper operators can check the frontier (watermark) at the output of the stateful operator to ensure only complete state

Heap increases significantly, this can usually be attributed to the size of your application state (check the checkpointing metrics5 for an estimated size of the on-heap state). The possible reasons for monitoring is disabled by default and requires additional dependencies on the classpath. Please check out the Flink system resource metrics documentation9 for additional guidance and details. System

is State? ▶ Accumulate and aggregate the results from the start of the streaming job. ▶ Need to check the previous state of the RDD in order to do something with the current RDD. ▶ Spark supports stateful is State? ▶ Accumulate and aggregate the results from the start of the streaming job. ▶ Need to check the previous state of the RDD in order to do something with the current RDD. ▶ Spark supports stateful

Periodic: periodically ask the user-defined function for the current watermark timestamp. Punctuated: check for a watermark in each passing record, e.g. if the stream contains special records that encode watermark