DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/CatalogTable 30 1.3.5 Runtime issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 1.3 flink-table-planner-loader-1.15.2.jar # -rw-r--r-- 1 dianfu staff 2.9M 10 18 20:43 flink-table-runtime-1.15.2.jar # -rw-r--r-- 1 dianfu staff 203K 10 18 20:43 log4j-1.2-api-2.17.1.jar # -rw-r--r-- 1 StreamExecutionEnvironment.get_execution_environment() # Config the Program run in Streaming Mode env.set_runtime_mode(RuntimeExecutionMode.STREAMING) env [1]:

0 码力 | 36 页 | 266.77 KB | 1 年前

3

DynamicTableFactory$Context.getCatalogTable()Lorg/apache/flink/table/catalog/CatalogTable 30 1.3.5 Runtime issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 1.3 flink-table-planner-loader-1.15.2.jar # -rw-r--r-- 1 dianfu staff 2.9M 10 18 20:43 flink-table-runtime-1.15.2.jar # -rw-r--r-- 1 dianfu staff 203K 10 18 20:43 log4j-1.2-api-2.17.1.jar # -rw-r--r-- 1 StreamExecutionEnvironment.get_execution_environment() # Config the Program run in Streaming Mode env.set_runtime_mode(RuntimeExecutionMode.STREAMING) env [1]:

0 码力 | 36 页 | 266.80 KB | 1 年前

3

阿里巴巴高级技术专家 过去 一切从2014年开始 2009 - 2014 2014 • 柏林工业大学博士生项目 • 基于流式 runtime 的批处理引擎 • 2014 年 8 月份 发布 Flink 0.6.0 Flink 0.7 Runtime Distributed Streaming Dataflow DataStream API Stream Processing Applications ✔ 现在 Flink 1.9 的架构变化 Runtime Distributed Streaming Dataflow Query Processor DAG & StreamOperator Local Single JVM Cloud GCE, EC2 Cluster Standalone, YARN Runtime Distributed Streaming Dataflow

state, parallelism, selectivity • Dataflow optimizations • plan translation alternatives • Runtime optimizations • load management, scheduling, state management • Optimization semantics, correctness properties • How can we estimate the cost of different strategies? • before execution or during runtime Query optimization (I) ??? Vasiliki Kalavri | Boston University 2020 10 Optimization strategies • maintain result and selectivity semantics • Dynamism: can the optimization be applied during runtime or does it have to be applied statically? When to optimize? ??? Vasiliki Kalavri | Boston University

lost events depends on • failure detection delay • stream input rates • state size • No runtime overhead 13 Vasiliki Kalavri | Boston University 2020 Passive Standby • Each primary periodically consists of • input queues • operator state • output queues • Short recovery time • High runtime overhead • The checkpoint interval determines the trade-off 14 Ni primary secondary I1 O1 consists of • input queues • operator state • output queues • Short recovery time • High runtime overhead • The checkpoint interval determines the trade-off 14 Ni primary secondary I1 O1

approach than Storm. • Based on consistent global snapshots (inspired by Chandy-Lamport). • Low runtime overhead, stateful exactly-once semantics. 73 / 79 Fault Tolerance (1/2) ▶ Fault tolerance in approach than Storm. • Based on consistent global snapshots (inspired by Chandy-Lamport). • Low runtime overhead, stateful exactly-once semantics. 73 / 79 Fault Tolerance (1/2) ▶ Fault tolerance in approach than Storm. • Based on consistent global snapshots (inspired by Chandy-Lamport). • Low runtime overhead, stateful exactly-once semantics. 73 / 79 Fault Tolerance (2/2) ▶ Acks sequences of records

University 2020 • To create a state object, we have to register a StateDescriptor with Flink’s runtime via the RuntimeContext, which is exposed by RichFunctions (RichFlatMapFunction, RichMapFunction a KeyedStream: • When the processing method of a function with keyed input is called, Flink’s runtime automatically puts all keyed state objects of the function into the context of the key of the

可部署在各种集群环境 * 对各种⼤大⼩小的数据规模进⾏行行快速计算 为什什么Flink能做批处理理 Table Stream Bounded Data Unbounded Data SQL Runtime SQL ⾼高吞吐 低延时 Hive vs. Spark vs. Flink Batch Hive/Hadoop Spark Flink 模型 MR MR(Memory/Disk)

shedding techniques operate in a dynamic fashion: the system detects an overload situation during runtime and selectively drops tuples according to a QoS specification. • Similar to congestion control

application throughout the development phase. By doing so, you can ask the right questions about the runtime behaviour of your application, and learn much more about Flink’s internals early on. Last but not