sources. 13 / 79 Input Operations - Basic Sources ▶ Socket connection • Creates a DStream from text data received over a TCP socket connection. ssc.socketTextStream("localhost", 9999) ▶ File stream ory) 14 / 79 Input Operations - Basic Sources ▶ Socket connection • Creates a DStream from text data received over a TCP socket connection. ssc.socketTextStream("localhost", 9999) ▶ File stream private def receive() { ... socket = new Socket(host, port) val reader = ... // read from the socket connection val userInput = reader.readLine() while(! isStopped && userInput ! = null) { store(userInput)

context of streaming? • queries run continuously • streams are unbounded • In traditional ad-hoc database queries, the query plan is generated on- the-fly. Different plans can be used for two consecutive example: URL access frequency 26 map() reduce() GET /dumprequest HTTP/1.1 Host: rve.org.uk Connection: keep-alive Accept: text/html,application/ xhtml+xml,application/ xml;q=0.9,*/*;q=0.8 User-Agent: en;q=0.8 Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3 GET /dumprequest HTTP/1.1 Host: rve.org.uk Connection: keep-alive Accept: text/html,application/ xhtml+xml,application/ xml;q=0.9,*/*;q=0.8 User-Agent:

userSentPayment 4 Connecting producers to consumers • Indirectly • Producer writes to a file or database • Consumer periodically polls and retrieves new data • polling overhead, latency? • Consumer consumers. • It receives messages from the producers and pushes them to the consumers. • A TCP connection is a simple messaging system which connects one sender with one recipient. • A general messaging Databases • DBs keep data until explicitly deleted while MBs delete messages once consumed. • Use a database for long-term data storage! • MBs assume a small working set. If consumers are slow, throughput

api.ValidationException: Unable to create a source for reading␣ ˓→table 'default_catalog.default_database.sourceKafka'. Table options are: 'connector'='kafka' 'format'='json' 'properties.bootstrap.servers'='192 flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider. ˓→loadDriver(SimpleJdbcConnectionProvider.java:90) at org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider ˓→getLoadedDriver(SimpleJdbcConnectionProvider.java:100) at org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider. (continues on next page) 1.3. Frequently Asked Questions (FAQ)

api.ValidationException: Unable to create a source for reading␣ ˓→table 'default_catalog.default_database.sourceKafka'. Table options are: 'connector'='kafka' 'format'='json' 'properties.bootstrap.servers'='192 flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider. ˓→loadDriver(SimpleJdbcConnectionProvider.java:90) at org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider ˓→getLoadedDriver(SimpleJdbcConnectionProvider.java:100) at org.apache.flink.connector.jdbc.internal.connection.SimpleJdbcConnectionProvider. (continues on next page) 1.3. Frequently Asked Questions (FAQ)

traditional data processing applications, we know the entire dataset in advance, e.g. tables stored in a database. A data stream is a data set that is produced incrementally over time, rather than being available not know when the stream ends. 3 Vasiliki Kalavri | Boston University 2020 DW DBMS SDW DSMS Database Management System • ad-hoc queries, data manipulation tasks • insertions, updates, deletions Single-node execution Synopses and sketches Approximate results In-order data processing Stream Database Systems 2000 1992 2013 MapReduce 2004 Tapestry NiagaraCQ Aurora TelegraphCQ STREAM Naiad

Kalavri | Boston University 2020 ESL: Expressive Stream Language • Ad-hoc SQL queries • Updates on database tables • Continuous queries on data streams • New streams (derived) are defined as virtual views of the 10th international conference on Database Theory (ICDT’05). • Yan-Nei Law, Haixun Wang, and Carlo Zaniolo. Query languages and data models for database sequences and data streams. In Proceedings

it is on the TCP channel. • If there is no buffer on the consumer side, reading from the TCP connection is interrupted. • The producer uses a threshold to control how much data is in-flight. • through an ATM network, each pair of endpoints first needs to establish a virtual circuit (VC) or connection. • CFC uses a credit system to signal the availability of buffer space from receivers to senders pairs of communicating tasks only • it does not interfere with other tasks sharing the same TCP connection. • CFC maximizes network utilization and prevents faults caused by high congestion. • In the

management • checkpointing state to remote and persistent storage, e.g. a distributed filesystem or a database system • Available state backends in Flink: • In-memory • File system • RocksDB State backends

persistent message queue, before it is processed by Apache Flink, which then writes the results to a database or calls a downstream system. In such a pipeline, latency can be introduced at each stage and for