2 健康状况 caolei – 监控Apache Flink应用程序(入门) 监控 – 7 3 监控 您要监控的第一件事就是您的作业是否实际处于运行状态。此外，还可以监控重启的次数以及自上次重启之后 的时间。 通常来说，成功的检查点是应用程序总体健康状况的一个强大指示器。对于每个检查点，检查点屏障需要流经 Flink作业的整个拓扑结构，并且事件和屏障不能相互超越。因此，一个成功的检查点显示没有通道是完全拥挤 To this end, Flink comes with a feature called Latency Tracking4. When enabled, Flink will insert so-called latency markers periodically at all sources. For each sub-task, a latency distribution from Panel Figure 5: Latency distribution between a source and a single sink subtask. 4.13 JVM Metrics So far we have only looked at Flink-specific metrics. As long as latency & throughput of your application

D6l6t6Fil6-4 ApplA D6l6tion ApplA D6l6tion S质：拆分D5t5LD6l6t6 /5ni76sts，快速为TEFD5t5Fil6PIMN 的D6l6t6 Fil6列表D 文件级别并发读取 3 2 6 2 4 5 Bu3648- Bu3648-2 D282F574- D282F574-2 D47484F574-3 D282F574-6 D282F574-7

? Vasiliki Kalavri | Boston University 2020 How can we count the number of distinct elements seen so far in a stream? 3 Example use-case: Distinct users visiting one or multiple webpages ??? Vasiliki Vasiliki Kalavri | Boston University 2020 How can we count the number of distinct elements seen so far in a stream? 3 Example use-case: Distinct users visiting one or multiple webpages Naive solution: maintain ? Vasiliki Kalavri | Boston University 2020 How can we count the number of distinct elements seen so far in a stream? 3 Example use-case: Distinct users visiting one or multiple webpages Naive solution:

Solution #2: sampling users Sample 1/10th of the users instead 12 • Maintain a list of all users seen so far and a flag indicating whether they belong to the sample or not • When a query arrives: • Solution #2: sampling users Sample 1/10th of the users instead 12 • Maintain a list of all users seen so far and a flag indicating whether they belong to the sample or not • When a query arrives: • s elements. 14 How can we continuously maintain a representative fixed-size sample of the stream so far? ??? Vasiliki Kalavri | Boston University 2020 Instead of a fixed proportion, assume we can only

Someone steals your phone and sings in your banking app. The app allows transfers of up to €1000 and so the thief makes transfers of €1000 to a "fake account" until either you're out of money or the activity of integers and computes: 29 1. the maximum number seen so far 2. the average of all numbers seen so far 3. the median of all numbers seen so far Vasiliki Kalavri | Boston University 2020 [1, 4, 5

consumers (possibly implemented by several parallel physical processes) can subscribe to the same topic, so that the message broker delivers messages to all subscribed consumers in a broadcast fashion. 11 Compute Engine instance can write logs to the monitoring system, to a database for later querying, and so on. • Data streaming from various processes or devices • a residential sensor can stream data to receives messages by reading the log sequentially 27 Partitions and offsets • A log can be partitioned, so that each partition can be read and written independently of others • a topic is a set of partitions

following Dockerfile: FROM flink:1.15.2 # install python3: it has updated Python to 3.9 in Debian 11 and so install Python 3.7␣ ˓→from source # it currently only supports Python 3.6, 3.7 and 3.8 in PyFlink also Flink) distribution except the following connectors: blackhole, datagen, filesystem and print. So you need to specify the connector JAR package explicitly when executing PyFlink jobs: • The connector too large This exception only occurs on Windows. It doesn’t affect the execution of PyFlink jobs and so you could ignore it usually. Besides, you could also upgrade PyFlink versions to 1.12.8, 1.13.7, 1

following Dockerfile: FROM flink:1.15.2 # install python3: it has updated Python to 3.9 in Debian 11 and so install Python 3.7␣ ˓→from source # it currently only supports Python 3.6, 3.7 and 3.8 in PyFlink also Flink) distribution except the following connectors: blackhole, datagen, filesystem and print. So you need to specify the connector JAR package explicitly when executing PyFlink jobs: • The connector too large This exception only occurs on Windows. It doesn’t affect the execution of PyFlink jobs and so you could ignore it usually. Besides, you could also upgrade PyFlink versions to 1.12.8, 1.13.7, 1

maintains the state for this key • State access is automatically scoped to the key of the current record so that all records with the same key access the same state State management in Apache Flink 5 Vasiliki the name of the state and the data types of the state: • The state name is scoped to the operator so that a function can have more than one state object by registering multiple state descriptors.

Consider a sequence of length n, i.e., S = Sn. If G is a continuous sum, so that it returns the sum of all tuples seen so far: • what is Gj (S) for j < n? • for j = n? What if n = 5 and S = [3