sample of an unbounded stream? • we want to ask queries and get statistically meaningful answers about the entire stream • we don’t necessarily know the queries in advance • we can store a fixed proportion statistically meaningful answers about the entire stream • we don’t necessarily know the queries in advance • we can store a fixed proportion of the stream, e.g. 1/10th 7 search engine timestamp> query stream Example use-case: Web search user behavior study Q: How many queries did users repeat last month? ??? Vasiliki Kalavri | Boston University 2020 Solution #1: uniform sampling

will use the Python environment of the current shell environment. See Session Mode for more details about session mode of Kubernetes. Execute PyFlink jobs with Flink Kubernetes Operator See PyFlink Example page in the official Flink documen- tation. For example, you can open the Kafka connector page and search keyword “SQL Client JAR” which is a fat JAR of Kafka connector. • It should be noted that you should

will use the Python environment of the current shell environment. See Session Mode for more details about session mode of Kubernetes. Execute PyFlink jobs with Flink Kubernetes Operator See PyFlink Example page in the official Flink documen- tation. For example, you can open the Kafka connector page and search keyword “SQL Client JAR” which is a fat JAR of Kafka connector. • It should be noted that you should

addition to the JVM metrics above, it is also possible to use Flink’s metrics system to gather insights about system resources, i.e. memory, CPU & network-related metrics for the whole machine as opposed to Flink’s metrics and monitoring system. You can utilise it as a starting point when you first think about how to successfully monitor your Flink application. I highly recommend to start monitoring your Flink development phase. By doing so, you can ask the right questions about the runtime behaviour of your application, and learn much more about Flink’s internals early on. Last but not least, this post only

If consumers are slow, throughput might degrade. • DBs support secondary indexes for efficient search while MBs only offer topic-based subscription. • DB query results depend on a snapshot and clients

way to reach Zurich from London through Berlin? These are the top-10 relevant results for the search term “graph” ??? Vasiliki Kalavri | Boston University 2020 Basics 1 5 4 3 2 “node” or “vertex”

• The number of distinct users who have visited a website? • The top-10 queries inserted in a search engine? • The connected components of accounts in a stream of financial transactions? What synopsis

assignment announcements & submissions 3 Vasiliki Kalavri | Boston University 2020 What is this course about? The design and architecture of modern distributed streaming 4 Fundamental for representing quizzes and announcements Vasiliki Kalavri | Boston University 2020 Guest Lectures • Learn about real-world use-cases of stream processing in industry • Learn from experts with decades of hands-on

the application and recover its state 2 Checkpointing guards the state from failures, but what about process failure? High-availability ??? Vasiliki Kalavri | Boston University 2020 3 Flink processes University 2020 • The JobManager is a single point of failure Flink applications • It keeps metadata about application execution, such as pointers to completed checkpoints. • A high-availability mode migrates

more delayed events will arrive. • Watermarks provide a logical clock which informs the system about the current event time. http://streamingbook.net/fig/2-9 Vasiliki Kalavri | Boston University 2020