historical data • batched updates during downtimes, e.g. every night Streaming Data Warehouse • low-latency materialized view updates • pre-aggregated, pre-processed streams and historical data Data append-only Update rates relatively low high, bursty Processing Model query-driven / pull-based data-driven / push-based Queries ad-hoc continuous Latency relatively high low 5 Vasiliki Kalavri | Boston Boston University 2020 Traditional DW vs. SDW Traditional DW SDW Update Frequency low high Update propagation synchronized asynchronous Data historical recent and historical ETL process complex fast and

• scheduling Objectives • optimize resource utilization or minimize resources • decrease latency, increase throughput • minimize monetary costs (if running in the cloud) Query optimization running might be impractical. • state accumulation and re-partitioning • high-availability and low latency requirements • scheduling overhead Challenges in streaming optimization ??? Vasiliki Kalavri constraints or QoS latency constraints. Batching Process multiple data elements in a single batch A A’ ??? Vasiliki Kalavri | Boston University 2020 43 • Batching trades throughput for latency • It improves

Watermarks provide a configurable trade-off between results confidence and latency: • Eager watermarks ensure low latency but provide lower confidence • Late events might arrive after the watermark watermark • Slow watermarks increase confidence but they might lead to higher processing latency. Trade-offs 17 Vasiliki Kalavri | Boston University 2020 Periodic: periodically ask the user-defined function

Kafka, RabbitMQ, ... HDFS, JDBC, ... Event logs ETL, Graphs,
 Machine Learning
 Relational, … Low latency,
 windowing, aggregations, ... 2 Vasiliki Kalavri | Boston University 2020 Basic API Concept

........................................................................... 14 4.12 Monitoring Latency................................................................................................. records-lag-max > threshold • millisBehindLatest > threshold 4.12 Monitoring Latency Generally speaking, latency is the delay between the creation of an event and the time at which results based which then writes the results to a database or calls a downstream system. In such a pipeline, latency can be introduced at each stage and for various reasons including the following: 1. It might take

trades-off result accuracy for sustainable performance. • Suitable for applications with strict latency constraints that can tolerate approximate results. Slow down the flow of data: • The system buffers plan • It detects overload and decides what actions to take in order to maintain acceptable latency and minimize result quality degradation. 7 ??? Vasiliki Kalavri | Boston University 2020 DSMS University 2020 Load shedding decisions • When to shed load? • detect overload quickly to avoid latency increase • monitor input rates • Where in the query plan? • dropping at the sources vs. dropping

• out-of-sync sources may produce out-of-order streams • can be connected to the network • latency and unpredictable delays • might be producing too fast • stream processor needs to keep up and to a file or database • Consumer periodically polls and retrieves new data • polling overhead, latency? • Consumer receives a notification when new data is available • how to implement triggers? messages, as each partition is read by a single thread What would you use when priority is: - latency but not ordering? - throughput and ordering? 31 How long to keep the log? • Log compaction: a

• CPU utilization, congestion, back pressure, throughput Policy • Queuing theory models: for latency objectives • Control theory models: e.g., PID controller • Rule-based models, e.g. if CPU utilization scaled and upstream channels • All-at-once • move state to be migrated in one operation • high latency during migration if the state is large • Progressive • move state to be migrated in smaller pieces Andrea Lattuada, Frank McSherry, Vasiliki Kalavri, John Liagouris, Timothy Roscoe. Megaphone: Latency-conscious state migration for distributed streaming dataflows. (VLDB 2019). 37

migration • minimize communication • keep duration short • minimize performance disruption, e.g. latency spikes • avoid introducing load imbalance • Resource management • utilization, isolation

coarse-grained approach than Storm. • Based on consistent global snapshots (inspired by Chandy-Lamport). • Low runtime overhead, stateful exactly-once semantics. 73 / 79 Fault Tolerance (1/2) ▶ Fault tolerance coarse-grained approach than Storm. • Based on consistent global snapshots (inspired by Chandy-Lamport). • Low runtime overhead, stateful exactly-once semantics. 73 / 79 Fault Tolerance (1/2) ▶ Fault tolerance coarse-grained approach than Storm. • Based on consistent global snapshots (inspired by Chandy-Lamport). • Low runtime overhead, stateful exactly-once semantics. 73 / 79 Fault Tolerance (2/2) ▶ Acks sequences