up an environment to develop Flink programs • Implement streaming data processing pipelines • Flink managed state • Event time Streaming in Apache Flink • Streams are natural • Events of any type

Scalable Stream Processing - Spark Streaming and Flink Amir H. Payberah payberah@kth.se 05/10/2018 The Course Web Page https://id2221kth.github.io 1 / 79 Where Are We? 2 / 79 Stream Processing Systems Spark streaming ▶ Flink 4 / 79 Spark Streaming 5 / 79 Contribution ▶ Design issues • Continuous vs. micro-batch processing • Record-at-a-Time vs. declarative APIs 6 / 79 Spark Streaming ▶ Run Run a streaming computation as a series of very small, deterministic batch jobs. • Chops up the live stream into batches of X seconds. • Treats each batch as RDDs and processes them using RDD operations

4/14: Stream processing optimizations ??? Vasiliki Kalavri | Boston University 2020 2 • Costs of streaming operator execution • state, parallelism, selectivity • Dataflow optimizations • plan translation ??? Vasiliki Kalavri | Boston University 2020 12 • What does efficient mean in the context of streaming? • queries run continuously • streams are unbounded • In traditional ad-hoc database queries the-fly. Different plans can be used for two consecutive executions of the same query. • A streaming dataflow is generated once and then scheduled for execution. • Changing execution strategy while

Processing and Analytics Vasiliki (Vasia) Kalavri
 vkalavri@bu.edu Spring 2020 4/28: Graph Streaming ??? Vasiliki Kalavri | Boston University 2020 Modeling the world as a graph 2 Social networks a vertex and all of its neighbors. Although this model can enable a theoretical analysis of streaming algorithms, it cannot adequately model real-world unbounded streams, as the neighbors cannot be continuously generated as a stream of edges? • How can we perform iterative computation in a streaming dataflow engine? How can we propagate watermarks? • Do we need to run the computation from scratch

Kalavri
 vkalavri@bu.edu CS 591 K1: Data Stream Processing and Analytics Spring 2020 2/04: Streaming languages and operator semantics Vasiliki Kalavri | Boston University 2020 Vasiliki Kalavri | Boston interval of 5–15 s) by an item of type C with Z < 5. 8 Vasiliki Kalavri | Boston University 2020 Streaming Operators 9 Vasiliki Kalavri | Boston University 2020 Operator types (I) • Single-Item Operators println!("seen: {:?}", x))
 .connect_loop(handle);
 }); t (t, l1) (t, (l1, l2)) Streaming Iteration Example Terminate after 100 iterations Create the feedback loop 13 Vasiliki Kalavri

release-1.15 PyFlink is a Python API for Apache Flink that allows you to build scalable batch and streaming workloads, such as real-time data processing pipelines, large-scale exploratory data analysis, Machine context for creating Table and SQL API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming and batch API to create a TableEnvironment. TableEnvironment is responsible table_environment.TableEnvironment at 0x7fcd16342ac8> [2]: # Create a streaming TableEnvironment env_settings = EnvironmentSettings.in_streaming_mode() table_env = TableEnvironment.create(env_settings) table_env

release-1.16 PyFlink is a Python API for Apache Flink that allows you to build scalable batch and streaming workloads, such as real-time data processing pipelines, large-scale exploratory data analysis, Machine context for creating Table and SQL API programs. Flink is an unified streaming and batch computing engine, which provides unified streaming and batch API to create a TableEnvironment. TableEnvironment is responsible table_environment.TableEnvironment at 0x7fcd16342ac8> [2]: # Create a streaming TableEnvironment env_settings = EnvironmentSettings.in_streaming_mode() table_env = TableEnvironment.create(env_settings) table_env

内存存储不是持久性的，这意味着如果分布式追踪平台实例关闭、重启或被 替换，您的 trace 数据将会丢失。此外，内存存储无法扩展，因为每个 Pod 都有自己的内存。对于持久性存储，您必须使用 production 或 streaming 策略，这些策略使用 Elasticsearch 作为默认存储。 production - production 策略主要用于生产环境，在生产环境中，对 trace 数据进行长期存 储非常重 注入。Query 和 Collector 服务被配置为使用一个 受支持的存储类型 - 当前为 Elasticsearch。可以根据性能和恢复能力的需要提供每个组件的 多个实例。 streaming - streaming 策略旨在提供在 Collector 和 Elasticsearch 后端存储之间有效发挥 作用的流传输功能，以此增强 production 策略。这样做的好处是在高负载情况下降低后端存 储压力，并允许其他 trace 后处理功能直接从流传输平台 (AMQ Streams/ Kafka) 中利用实 时 span 数据。 注意 注意 streaming 策略需要额外的 AMQ Streams 订阅。 注意 注意 目前 IBM Z 不支持 streaming 部署策略。 注意 注意 有两种方法可用来安装和使用 Red Hat OpenShift distributed tracing，作为服务网格的一

台(Jaeger)实例关闭、重启或被替 换，您的 trace 数据将会丢失。此外，内存存储无法扩展，因为每个 Pod 都有自己的内 存。对于持久性存储，您必须使用 production 或 streaming 策略，这些策略使用 Elasticsearch 作为默认存储。 production production 策略主要用于生产环境，在生产环境中，对 trace 数据进行长期存储非常重要，同时需要 。 streaming streaming 策略旨在通过提供在 Collector 和 Elasticsearch 后端存储之间有效处的流传输功能来增强 production 策略。这样做的好处是在高负载情况下降低后端存储压力，并允许其他 trace 后处理功能 直接从流传输平台 (AMQ Streams/ Kafka) 中利用实时 span 数据。 注意 注意 streaming 策略需要额外的 策略需要额外的 AMQ Streams 订阅。 目前 IBM Z® 不支持 streaming 部署策略。 3.2.2. 从 Web 控制台部署分布式追踪平台默认策略 自定义资源定义(CRD)定义部署 Red Hat OpenShift distributed tracing 平台实例时使用的配置。默认 CR 名为 jaeger-all-in-one-inmemory，它配置为使用最少资源，以确保您可以在默认的

relatively static and historical data • batched updates during downtimes, e.g. every night Streaming Data Warehouse • low-latency materialized view updates • pre-aggregated, pre-processed streams streams and historical data Data Management Approaches 4 storage analytics static data streaming data Vasiliki Kalavri | Boston University 2020 DBMS vs. DSMS DBMS DSMS Data persistent relations stream can be viewed as a massive, dynamic, one-dimensional vector A[1…N]. The size N of the streaming vector is defined as the product of the attribute domain size(s). Note that N might be unknown