vkalavri@bu.edu Spring 2020 1/21: Introduction Vasiliki Kalavri | Boston University 2020 Course Information • Instructor: Vasiliki Kalavri • Office: MCS 206 • Contact: vkalavri@bu.edu • Course Time At the end of the course, you will hopefully: • know when to use stream processing vs other technology • be able to comprehensively compare features and processing guarantees of streaming systems Analysis of real-time vehicle locations to improve traffic conditions • Provide real-time scheduling information for public transport • Optimize transport network flow and recommend alternative routes Example:

Post-failure output is identical to no-failure • Rollback recovery (at-least-once) • It avoids information loss • The output may contain duplicates • A backup needs to rebuild state of the failed node Post-failure output is identical to no-failure • Rollback recovery (at-least-once) • It avoids information loss • The output may contain duplicates • A backup needs to rebuild state of the failed node recovery (at-most-once) • It drops data during failure • The backup starts from most recent information 8 Vasiliki Kalavri | Boston University 2020 Recovery semantics Given a dataflow Q, let Oe be

LONG() It should be noted that Types.BIG_INT() represents type information for the Java BigInteger, while Types.LONG() represents type information for long integer. There are several users are using Types.BIG_INT()

LONG() It should be noted that Types.BIG_INT() represents type information for the Java BigInteger, while Types.LONG() represents type information for long integer. There are several users are using Types.BIG_INT()

watermark in each passing record, e.g. if the stream contains special records that encode watermark information. val env = StreamExecutionEnvironment.getExecutionEnvironment env.setStreamTimeCharacteristi

publish publish notify() subscribe() unsubscribe() subscribe notify unsubscribe advertise(): information reg. future events Publish/Subscribe Systems 17 Pub/Sub levels of de-coupling • Space: interacting

holistic aggregates • Compute on most recent events only • when providing real-time traffic information, you probably don't care about an accident that happened 2 hours ago • Recent might mean different

• The load shedder continuously monitors input rates or other system metrics and can access information about the running query plan • It detects overload and decides what actions to take in order

optimization • Statis Viglas and Jeffrey Naughton. Rate-based Query Optimization for Streaming Information Sources. SIGMOD 2002. Further reading

University 2020 Lecture references • Gianpaolo Cugola and Alessandro Margara. Processing flows of information: From data stream to complex event processing. ACM Comput. Surv. 44, 3, Article 15 (June 2012)