up an environment to develop Flink programs • Implement streaming data processing pipelines • Flink managed state • Event time Streaming in Apache Flink • Streams are natural • Events of any type

Scalable Stream Processing - Spark Streaming and Flink Amir H. Payberah payberah@kth.se 05/10/2018 The Course Web Page https://id2221kth.github.io 1 / 79 Where Are We? 2 / 79 Stream Processing Systems Spark streaming ▶ Flink 4 / 79 Spark Streaming 5 / 79 Contribution ▶ Design issues • Continuous vs. micro-batch processing • Record-at-a-Time vs. declarative APIs 6 / 79 Spark Streaming ▶ Run Run a streaming computation as a series of very small, deterministic batch jobs. • Chops up the live stream into batches of X seconds. • Treats each batch as RDDs and processes them using RDD operations

4/14: Stream processing optimizations ??? Vasiliki Kalavri | Boston University 2020 2 • Costs of streaming operator execution • state, parallelism, selectivity • Dataflow optimizations • plan translation ??? Vasiliki Kalavri | Boston University 2020 12 • What does efficient mean in the context of streaming? • queries run continuously • streams are unbounded • In traditional ad-hoc database queries the-fly. Different plans can be used for two consecutive executions of the same query. • A streaming dataflow is generated once and then scheduled for execution. • Changing execution strategy while

Processing and Analytics Vasiliki (Vasia) Kalavri
 vkalavri@bu.edu Spring 2020 4/28: Graph Streaming ??? Vasiliki Kalavri | Boston University 2020 Modeling the world as a graph 2 Social networks a vertex and all of its neighbors. Although this model can enable a theoretical analysis of streaming algorithms, it cannot adequately model real-world unbounded streams, as the neighbors cannot be continuously generated as a stream of edges? • How can we perform iterative computation in a streaming dataflow engine? How can we propagate watermarks? • Do we need to run the computation from scratch

Kalavri
 vkalavri@bu.edu CS 591 K1: Data Stream Processing and Analytics Spring 2020 2/04: Streaming languages and operator semantics Vasiliki Kalavri | Boston University 2020 Vasiliki Kalavri | Boston interval of 5–15 s) by an item of type C with Z < 5. 8 Vasiliki Kalavri | Boston University 2020 Streaming Operators 9 Vasiliki Kalavri | Boston University 2020 Operator types (I) • Single-Item Operators println!("seen: {:?}", x))
 .connect_loop(handle);
 }); t (t, l1) (t, (l1, l2)) Streaming Iteration Example Terminate after 100 iterations Create the feedback loop 13 Vasiliki Kalavri

used individually or all together. For example, you can use Kyuubi, Spark and Flink to build a streaming data warehouse. And then, you can use Zookeeper to enable the load balancing for high availability Release 1.7.0 Server Session Spnego Zookeeper Spark Configurations Via spark-defaults.conf Setting them in $SPARK_HOME/conf/spark-defaults.conf supplies with default values for SQL engine applica- found at Spark official online documentation for Spark Configurations Via kyuubi-defaults.conf Setting them in $KYUUBI_HOME/conf/kyuubi-defaults.conf supplies with default values for SQL engine appli-

used individually or all together. For example, you can use Kyuubi, Spark and Flink to build a streaming data warehouse. And then, you can use Zookeeper to enable the load balancing for high availability Metrics Operation Server Session Spnego Zookeeper Spark Configurations Via spark-defaults.conf Setting them in $SPARK_HOME/conf/spark-defaults.conf supplies with default values for SQL engine applica- found at Spark official online documentation for Spark Configurations Via kyuubi-defaults.conf Setting them in $KYUUBI_HOME/conf/kyuubi-defaults.conf supplies with default values for SQL engine appli-

used individually or all together. For example, you can use Kyuubi, Spark and Flink to build a streaming data warehouse. And then, you can use Zookeeper to enable the load balancing for high availability Metrics Operation Server Session Spnego Zookeeper Spark Configurations Via spark-defaults.conf Setting them in $SPARK_HOME/conf/spark-defaults.conf supplies with default values for SQL engine applica- found at Spark official online documentation for Spark Configurations Via kyuubi-defaults.conf Setting them in $KYUUBI_HOME/conf/kyuubi-defaults.conf supplies with default values for SQL engine appli-

used individually or all together. For example, you can use Kyuubi, Spark and Flink to build a streaming data warehouse. And then, you can use Zookeeper to enable the load balancing for high availability Metrics Operation Server Session Spnego Zookeeper Spark Configurations Via spark-defaults.conf Setting them in $SPARK_HOME/conf/spark-defaults.conf supplies with default values for SQL engine applica- found at Spark official online documentation for Spark Configurations Via kyuubi-defaults.conf Setting them in $KYUUBI_HOME/conf/kyuubi-defaults.conf supplies with default values for SQL engine appli-

used individually or all together. For example, you can use Kyuubi, Spark and Flink to build a streaming data warehouse. And then, you can use Zookeeper to enable the load balancing for high availability Metrics Operation Server Session Spnego Zookeeper Spark Configurations Via spark-defaults.conf Setting them in $SPARK_HOME/conf/spark-defaults.conf supplies with default values for SQL engine applica- found at Spark official online documentation for Spark Configurations Via kyuubi-defaults.conf Setting them in $KYUUBI_HOME/conf/kyuubi-defaults.conf supplies with default values for SQL engine appli-