1) (and,1) (let,1) (live,2) ## Distributed architecture TaskManager Flink program web dashboard TaskManager client JobManager TaskManager ## DataStream API Basics ## Example: Sensor Readings case might need to adjust: jobmanager.heap.size: JVM heap size for the JobManager (coordinator). taskmanager.heap.size: JVM heap size for the TaskManagers (workers). paralleliasm.default: Default parallelism env.setParallelism() in your application. taskmanager.numberOfTaskSlots: The number of parallel operator or user function instances that a single TaskManager can run. This value is typically proportional

containerized environment you should additionally monitor the overall memory consumption of the Job- and TaskManager containers to ensure they don’t exceed their resource limits. This is particularly important, when JVM.Memory.NonHeap.Committed|job-/taskmanager|The amount of non-heap memory guaranteed to be available to the JVM (in bytes).| |Status.JVM.Memory.Heap.Used|job-/taskmanager|The amount of heap memory currently JVM.Memory.Heap.Committed|job-/taskmanager|The amount of heap memory guaranteed to be available to the JVM (in bytes).| |Status.JVM.Memory.Direct.MemoryUsed|job-/taskmanager|The amount of memory used by

\ -Dkubernetes.cluster-id= \ -Dtaskmanager.memory.process.size=4096m \ -Dkubernetes.taskmanager.cpu=2 \ -Dtaskmanager.numberOfTaskSlots=4 \ (continues on next page) (continued from flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) at org.apache.flink.runtime.taskmanager.Task.doRun(Task doRun(Task.java:766) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver at java.net

\ -Dkubernetes.cluster-id= \ -Dtaskmanager.memory.process.size=4096m \ -Dkubernetes.taskmanager.cpu=2 \ -Dtaskmanager.numberOfTaskSlots=4 \ (continues on next page) (continued from flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:958) at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:927) at org.apache.flink.runtime.taskmanager.Task.doRun(Task doRun(Task.java:766) at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: com.mysql.cj.jdbc.Driver at java.net

RocksDB ## Which backend to choose? ## MemoryStateBackend • Stores state as regular objects on TaskManager’s heap • Low read/write latencies • OutOfMemoryError if large grows too large, GC pauses • Checkpoints FsStateBackend • Stores state on TaskManager’s heap but checkpoints it to a remote file system • In-memory speed for local accesses and fault tolerance • Limited to TaskManager’s memory and might suffer from

in Flink  • A TaskManager can execute several tasks at the same time. - It is statically configured with a certain number The TaskManagers ship data from sending tasks to receiving tasks. - The network component of a TaskManager collects records in buffers before they are shipped, i.e., records are not shipped one by one but in the same process, the sender task serializes the outgoing records into a byte buffer. • A TaskManager needs one dedicated network buffer for each receiving task that any of its tasks need to send data

example: flink.parallellism.default 2 flink.taskmanager.memory.process.size 5g The below options in kyuubi-defaults.conf will set parallelism.default: 2 and taskmanager.memory.process.size: 5g into flink configurations for each SQL engine. For example: jdbc:hive2://localhost:10009/default;#paralleliasm.default=2;taskmanager.memory.process.size=5g ## V ia SET Statements Please refer to the Flink official online documentation a typical Flink application. Thus, its logs mainly contain the logs of a Flink JobManager and TaskManager. Meanwhile, it also includes how all the services of an engine start/stop, how does it response

example: flink.parallelism.default 2 flink.taskmanager.memory.process.size 5g The below options in kyuubi-defaults.conf will set parallelism.default: 2 and taskmanager.memory.process.size: 5g into flink configurations for each SQL engine. For example: jdbc:hive2://localhost:10009/default;#paralleliasm.default=2;taskmanager.memory.process.size=5g ## V ia SET Statements Please refer to the Flink official online documentation a typical Flink application. Thus, its logs mainly contain the logs of a Flink JobManager and TaskManager. Meanwhile, it also includes how all the services of an engine start/stop, how does it response

example: flink.parallelizem.default 2 flink.taskmanager.memory.process.size 5g The below options in kyuubi-defaults.conf will set parallelism.default: 2 and taskmanager.memory.process.size: 5g into flink configurations engine. For example: jdbc:hive2://localhost:10009/default;#flink.parallelism.default=2;flink.taskmanager.memory.process.size=5g ## V ia SET Statements Please refer to the Flink official online documentation a typical Flink application. Thus, its logs mainly contain the logs of a Flink JobManager and TaskManager. Meanwhile, it also includes how all the services of an engine start/stop, how does it response

example: flink.parallelism.default 2 flink.taskmanager.memory.process.size 5g The below options in kyuubi-defaults.conf will set parallelism.default: 2 and taskmanager.memory.process.size: 5g into flink configurations for each SQL engine. For example: jdbc:hive2://localhost:10009/default;#paralleliasm.default=2;taskmanager.memory.process.size=5g ## V ia SET Statements Please refer to the Flink official online documentation a typical Flink application. Thus, its logs mainly contain the logs of a Flink JobManager and TaskManager. Meanwhile, it also includes how all the services of an engine start/stop, how does it response