DynamicTableFactory’ in the classpath . . . . . . . 26 1.3.4.2 O2: ClassNotFoundException: com.mysql.cj.jdbc.Driver . . . . . . . . . . . . . . 29 1.3.4.3 O3: NoSuchMethodError: org.apache.flink.table.factories management page of official PyFlink documentation. 1.3.4.2 O2: ClassNotFoundException: com.mysql.cj.jdbc.Driver py4j.protocol.Py4JJavaError: An error occurred while calling o13.execute. : org.apache.flink.runtime IOException: unable to open JDBC writer at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.open(JdbcOutputFormat. ˓→java:145) at org.apache.flink.connector.jdbc.internal.GenericJdbcSinkFunction

DynamicTableFactory’ in the classpath . . . . . . . 26 1.3.4.2 O2: ClassNotFoundException: com.mysql.cj.jdbc.Driver . . . . . . . . . . . . . . 29 1.3.4.3 O3: NoSuchMethodError: org.apache.flink.table.factories management page of official PyFlink documentation. 1.3.4.2 O2: ClassNotFoundException: com.mysql.cj.jdbc.Driver py4j.protocol.Py4JJavaError: An error occurred while calling o13.execute. : org.apache.flink.runtime IOException: unable to open JDBC writer at org.apache.flink.connector.jdbc.internal.JdbcOutputFormat.open(JdbcOutputFormat. ˓→java:145) at org.apache.flink.connector.jdbc.internal.GenericJdbcSinkFunction

is executed in the driver process. 31 / 79 Output Operations (2/4) ▶ What’s wrong with this code? ▶ This requires the connection object to be serialized and sent from the driver to the worker. dstream dstream.foreachRDD { rdd => val connection = createNewConnection() // executed at the driver rdd.foreach { record => connection.send(record) // executed at the worker } } 32 / 79 Output Operations object to be serialized and sent from the driver to the worker. dstream.foreachRDD { rdd => val connection = createNewConnection() // executed at the driver rdd.foreach { record => connection.send(record)

• True streaming at its core • Streaming & Batch API Historic data Kafka, RabbitMQ, ... HDFS, JDBC, ... Event logs ETL, Graphs,
 Machine Learning
 Relational, … Low latency,
 windowing, aggregations

Long a unique id for each taxi driverId Long a unique id for each driver isStart Boolean TRUE for ride start events, FALSE for ride end events startTime Long a unique id for each taxi driverId Long a unique id for each driver startTime DateTime the start time of a ride paymentType String CSH or CRD

static content. There is a JIRA Ticket6 to limit the size to 250 megabyte by default • The biggest driver of Direct memory is by far the number of Flink’s network buffers, which can be configured7. •