..................................................................................... 14 4.12 Monitoring Latency....................................................................................... ............ 23 caolei – 监控Apache Flink应用程序(入门) – 4 原文地址：https://www.ververica.com/blog/monitoring-apache-flink-applications-101 这篇博文介绍了Apache Flink内置的监控和度量系统，通过该系统，开发人员可以有效地监控他们的Flink作 业。通常，对于一个刚刚开始使用Apache org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#registering-metrics 2 https://ci.apache.org/projects/flink/flink-docs-release-1.7/monitoring/metrics.html#reporter Flink指标体系 – 5 1 Flink指标体系

Vasiliki Kalavri | Boston University 2020 Grading Scheme (2) Final Project (50%): • A real-time monitoring and anomaly detection framework • To be implemented individually Deliverables • One (1) written Boston University 2020 Final Project You will use Apache Flink and Kafka to build a real-time monitoring and anomaly detection framework for datacenters. Your framework will: • Detect “suspicious” Online recommendations Vasiliki Kalavri | Boston University 2020 Sensor measurements analysis • Monitoring applications • Complex filtering and alarm activation • Aggregation of multiple sensors and joins

the jth update (k, c[j]), it must hold that c[j] ≥ 0. This can model insertion-only streams: • monitoring the total packets exchanged between two IP addresses • the collection of IP addresses accessing continuously inserted and deleted from the stream. It can model fully dynamic situations: • Monitoring active IP network connections is a Turnstile stream, as connections can be initiated or terminated

changed. • Logging to multiple systems • a Google Compute Engine instance can write logs to the monitoring system, to a database for later querying, and so on. • Data streaming from various processes

load imbalance • Resource management • utilization, isolation • Automation • continuous monitoring • bottleneck detection • stability, accuracy 11 Challenges of reconfiguration ??? Vasiliki

MapReduce combiners example: URL access frequency 26 map() reduce() GET /dumprequest HTTP/1.1 Host: rve.org.uk Connection: keep-alive Accept: text/html,application/ xhtml+xml,application/ xml;q=0.9 Accept-Language: en-US,en;q=0.8 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 GET /dumprequest HTTP/1.1 Host: rve.org.uk Connection: keep-alive Accept: text/html,application/ xhtml+xml,application/ xml;q=0.9 Accept-Language: en-US,en;q=0.8 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 GET /dumprequest HTTP/1.1 Host: rve.org.uk Connection: keep-alive Accept: text/html,application/ xhtml+xml,application/ xml;q=0.9

received data inside Spark. 16 / 79 Input Operations - Custom Sources (2/3) class CustomReceiver(host: String, port: Int) extends Receiver[String](StorageLevel.MEMORY_AND_DISK_2) with Logging { def onStart() run() { receive() }}.start() } def onStop() {} private def receive() { ... socket = new Socket(host, port) val reader = ... // read from the socket connection val userInput = reader.readLine() while( Operations - Custom Sources (3/3) val customReceiverStream = ssc.receiverStream(new CustomReceiver(host, port)) val words = customReceiverStream.flatMap(_.split(" ")) 18 / 79 Operations on DStreams

size that will fit on a single server. Each individual partition must fit on the servers that host it, but a topic may have many partitions so it can handle an arbitrary amount of data. The number