CDsBPHNRHML, UDHFGR 1RO6 mysOl_AHLlMF; FHRGSA.BMm/TDPTDPHBa/ElHLI-BCB-BMLLDBRMPs Flink 原生支持 Change Log Stream A C D E F G INSERT DELETE UPDATE INSERT DELETE UPDATE INSERT F3152 + Icebe7g

there are any problems, you could perform the following checks. Check the logging messages in the log file to see if there are any problems: # Get the installation directory of PyFlink python3 -c "import /path/to/python/site-packages/pyflink # Check the logging under the log directory ls -lh /path/to/python/site-packages/pyflink/log # You will see the log file as following: (continues on next page) 1.1. Getting previous page) # -rw-r--r-- 1 dianfu staff 45K 10 18 20:54 flink-dianfu-python-B-7174MD6R-1908. ˓→local.log Besides, you could also check if the files of the PyFlink package are consistent. It may happen that

there are any problems, you could perform the following checks. Check the logging messages in the log file to see if there are any problems: # Get the installation directory of PyFlink python3 -c "import /path/to/python/site-packages/pyflink # Check the logging under the log directory ls -lh /path/to/python/site-packages/pyflink/log # You will see the log file as following: (continues on next page) 1.1. Getting previous page) # -rw-r--r-- 1 dianfu staff 45K 10 18 20:54 flink-dianfu-python-B-7174MD6R-1908. ˓→local.log Besides, you could also check if the files of the PyFlink package are consistent. It may happen that

Kalavri | Boston University 2020 Let h be a hash function that maps each stream element into M = log2N bits, where N is the domain of input elements: For each element x, let rank(x) be the number maximum value of rank(.) seen so far. ̂n = 2R Claim: The maximum observed rank is a good estimate of log2n. In other words, the estimated number of distinct elements is equal to: ??? Vasiliki Kalavri | elements. We need a hash function that maps each input element to log2n bits. Then, each counter needs to be able to count up to log2(log2n) 0s. 13 ??? Vasiliki Kalavri | Boston University 2020 14 Combining

subscription's queue of messages. 25 Log-structured brokers Logs as message brokers • In typical message brokers, once a message is consumed it is deleted • Log-based message brokers take a different partitioned) log • A log is an append-only sequence of records on disk • a producer generates messages by simply appending them to the log and a consumer receives messages by reading the log sequentially sequentially 27 Partitions and offsets • A log can be partitioned, so that each partition can be read and written independently of others • a topic is a set of partitions • Within each partition, every

the Kafka cluster maintains a partitioned log. Each partition is an ordered, immutable sequence of records that is continually appended to—a structured commit log. An offset is a sequential id number time to free up disk space. 22 Vasiliki Kalavri | Boston University 2020 23 Partitions allow the log to scale beyond a size that will fit on a single server. Each individual partition must fit on will have a lower offset than M2 and appear earlier in the log. • A consumer instance sees records in the order they are stored in the log. • For a topic with replication factor N, we will tolerate

University 2020 • Change parallelism • scale out to process increased load • scale in to save resources • Fix bugs or change business logic • Optimize execution plan • Change operator placement | Boston University 2020 Streaming applications are long-running • Workload will change • Conditions might change • State is accumulated over time 10 events/s time rate decrease events/s time

p4 34 ??? Vasiliki Kalavri | Boston University 2020 • Assumptions: • DAG of tasks • Epoch change events triggered on each source task (⟨ep1⟩,⟨ep2⟩,…) • Issued by a coordinator or generated periodically 4y cle0= A epoch-complete consistent cut that includes events that 1. precede epoch change Epoch cuts p4 AB6XicbVBNS 8NAEJ3Ur1q/qh69L 4y cle0= A epoch-complete consistent cut that includes events that 1. precede epoch change 2. are produced by events in cut Epoch cuts p4 AB6XicbVBNS

| Boston University 2020 Streaming applications are long-running • Workload will change • Conditions might change • State is accumulated over time 2 events/s time rate decrease events/s time

identifiers of the last tuples they received. • In upstream backup, operators need to track and log tuple provenance / result lineage. Can such techniques be efficiently implemented? What if more