Real-Time Unified Data Layers: A New Era for Scalable Analytics, Search, and AI # Table of Contents 1\. Introduction 2. The Interconnection of Analytics, Search, and AI 3. What is a Real-Time Unified Data ## For example: - FMCG & Manufacturing need real-time production dashboards and historical analytics shared across plants & lines to improve Overall Equipment Effectiveness (OEE). • Energy companies platforms require real-time monitoring and analytics to personalize experiences and ensure performance. # 2. The Interconnection of Analytics, Search, and AI Analytics, search, and AI are deeply interconnected

# CS 591 K1: Data Stream Processing and Analytics Spring 2020 4/16: Skew mitigation Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Key partitioning ![Image](/uploads/documents/5/5/9/6/5596d571f02610c

# CS 591 K1: Data Stream Processing and Analytics Spring 2020 2/25: State Management Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## State in dataflow computations Any non-trivial streaming computation

## CS 591 K1: Data Stream Processing and Analytics Spring 2020 ## 4 /14: Stream processing optimizations Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Topics covered in this lecture • Costs of streaming

## CS 591 K1: Data Stream Processing and Analytics Spring 2020 2/11: Windows and Triggers Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Window operators • Practical way to perform operations on unbounded

## CS 591 K1: Data Stream Processing and Analytics Spring 2020 1/21: Introduction Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Course Information • Instructor: Vasiliki Kalavri • Office: MCS 206 • analysis • Visualization and aggregation • impressions, clicks, transactions, likes, comments • Analytics on user activity • Filtering, aggregation, joins with static data (e.g. user profile data) ## [Image](/uploads/documents/2/8/8/5/2885b429b889e942d6d827594bab5473/p33_1.jpg) actions, alerts continuous analytics ## Optional reading - The 8 Requirements of Real-Time Stream Processing http://cs.brown.edu/~ugur/8rulesSigRec

## CS 591 K1: Data Stream Processing and Analytics Spring 2020 2/06: Notions of time and progress Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Mobile game application • input stream: user activity

## CS 591 K1: Data Stream Processing and Analytics Spring 2020 ## 1 /23: Stream Processing Fundamentals Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## What is a stream? - In traditional data processing materialized view updates • pre-aggregated, pre-processed streams and historical data DW DBMS analytics SDW DSMS ## Database Management System streaming data • ad-hoc queries, data manipulation tasks [Image](/uploads/documents/b/6/e/7/b6e7f87c9aa281c017a086c7749f72cd/p8_1.jpg) actions, alerts continuous analytics ## Basic Stream Models A stream can be viewed as a massive, dynamic, one-dimensional vector A[1

## CS 591 K1: Data Stream Processing and Analytics Spring 2020 ## 4 /23: Cardinality and frequency estimation Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Counting distinct elements ## How can we count

## CS 591 K1: Data Stream Processing and Analytics Spring 2020 4/28: Graph Streaming Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Modeling the world as a graph ![Image](/uploads/documents/d/c/6/7/d computation from scratch for every new edge? - Can we use graph synopses and summaries and compute graph analytics in one-pass? ## Connectivity & Bipartite property ## Streaming Connected Components • State: