came back online? • How long do we have to wait before we decide that we have seen all events? ## Watermarks ## Stream progress  confident that no more delayed events will arrive. • Watermarks provide a logical clock which informs the system about the current event time. Watermarks (in Flink) flow along dataflow edges. They are special upstream stages • minimum of output watermarks of all upstream tasks • The output watermark captures the progress of the stage itself • minimum of input watermarks and event-times of non-late data ##

t.getExecutionEnvironment(); env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime); ## Watermarks 23 19 22 24 21 14 17 13 12 15 9 11 7 2 4 → • Data may arrive out of order • Sorting data is expensive and may not always be required • Watermark is a good heuristic to bound out of orderness ## Watermarks DataStream stream = ... DataStream withTimestampsAndWatermarks = stream.as

#34;), "1 hour", "5 minutes")) ▶ Spark streaming uses watermarks to measure progress in event time. ▶ Watermarks flow as part of the data stream and carry a timestamp t. ▶ A $ W(t) $

windows and transformations might not be suitable: • they provide access to record timestamps and watermarks • they can register timers that trigger at a specific time in the future ProcessFunction, KeyedProcessFunction

from upstream and processes them in parallel with the primary but it doesn't output results • Watermarks are used to identify duplicate output tuples and trim the secondary's output queue • Negligible

aggregations, distinct... • State is commonly partitioned by key - State can be cleared based on watermarks or punctuations • window fires, post becomes inactive ## Example: Apache Flink DataStream API

aggregations, distinct... - State is commonly partitioned by key - State can be cleared based on watermarks or punctuations • window fires, post becomes inactive ## Operator selectivity • The number of

• How can we perform iterative computation in a streaming dataflow engine? How can we propagate watermarks? • Do we need to run the computation from scratch for every new edge? - Can we use graph synopses

document.pdf") # save the document with a new filename ## Note: Taking it further Adding watermarks is essentially as simple as adding an image at the base of each page. You should ensure that the background or foreground image for the page, like a copyright or a watermark. Please remember, that watermarks require a transparent image if put in foreground. 4. The image may be inserted uncompressed, e separate output page (see posterize.py). • include PDF-based vector images like company logos, watermarks, etc., see $ svg-logo.py $ , which puts an SVG-based logo on each page (requires additional packages

examples 4-up.py and posterize.py). Or use it to include PDF-based vector images (company logos, watermarks, etc.). ## Note Unfortunately, garbage collection currently does not detect multiple copies of