org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink. ˓→createSinkFunctionTransformation(CommonExecSink.java:331) at org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink at org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink. ˓→createSinkTransformation(CommonExecSink.java:146) at org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecSink. ˓→translateToPlanInternal(StreamExecSink.java:140) at org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase. ˓→translateToPlan(ExecNodeBase.java:134) This is an issue around Java 17. It still

org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink. ˓→createSinkFunctionTransformation(CommonExecSink.java:331) at org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink at org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecSink. ˓→createSinkTransformation(CommonExecSink.java:146) at org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecSink. ˓→translateToPlanInternal(StreamExecSink.java:140) at org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase. ˓→translateToPlan(ExecNodeBase.java:134) This is an issue around Java 17. It still

monitors input rates or other system metrics and can access information about the running query plan • It detects overload and decides what actions to take in order to maintain acceptable latency • detect overload quickly to avoid latency increase • monitor input rates • Where in the query plan? • dropping at the sources vs. dropping at bottleneck operators • How much load to shed? • enough Where in the query plan to drop tuples, which tuples, and how many • The question of where is equivalent to placing special drop operators in the best positions in the query plan • Drop operators

of streaming operator execution • state, parallelism, selectivity • Dataflow optimizations • plan translation alternatives • Runtime optimizations • load management, scheduling, state management optimization 11 Parsed program representation Optimizer statistics input plan A plan B output Lowest-cost plan ??? Vasiliki Kalavri | Boston University 2020 12 • What does efficient mean in queries run continuously • streams are unbounded • In traditional ad-hoc database queries, the query plan is generated on- the-fly. Different plans can be used for two consecutive executions of the same

as a static table. • Spark automatically converts this batch-like query to a streaming execution plan. ▶ 2. Specify triggers to control when to update the results. • Each time a trigger fires, Spark as a static table. • Spark automatically converts this batch-like query to a streaming execution plan. ▶ 2. Specify triggers to control when to update the results. • Each time a trigger fires, Spark as a static table. • Spark automatically converts this batch-like query to a streaming execution plan. ▶ 2. Specify triggers to control when to update the results. • Each time a trigger fires, Spark

increased load • scale in to save resources • Fix bugs or change business logic • Optimize execution plan • Change operator placement • skew and straggler mitigation • Migrate to a different cluster

Twitter source Extract hashtags Count topics Trends sink w1 w2 w3 w4 w5 w6 w7 w8 Logic Query Plan Deployment 39 Vasiliki Kalavri | Boston University 2020 source sink input port output port dataflow