Scalable Stream Processing - Spark Streaming and Flink Amir H. Payberah payberah@kth.se 05/10/2018 The Course Web Page https://id2221kth.github.io 1 / 79 Where Are We? 2 / 79 Stream Processing Systems

Implementation of Quantifiable Stack/Queue Live Demonstration Design and Implementation of Highly Scalable Quantifiable Data Structures in C++ 2MotivationMotivation Quantifiability Vector Space Entropy Measurement yields 64! = 1089 FYI: number of atoms in the universe is 1082 Design and Implementation of Highly Scalable Quantifiable Data Structures in C++ 4Motivation Quantifiability Vector Space Entropy Measurement response, the method call takes effect in real-time order. Design and Implementation of Highly Scalable Quantifiable Data Structures in C++ 5Motivation Quantifiability Vector Space Entropy Measurement

Real-Time Unified Data Layers: A New Era for Scalable Analytics, Search, and AI v 1.1Table of Contents Introduction 1. The Interconnection of Analytics, Search, and AI 2. What is a Real-Time Unified analytics, AI-driven decision-making, and intelligent search. However, many organizations lack a scalable infrastructure, leading to slow processing, high costs, and missed opportunities. Security, Compliance evolving demands of modern industries by consolidating diverse data sources into a single, real-time, scalable system. Built with industry use cases in mind, it enables seamless access, processing, and analysis—ensuring

Reclamation Beyond Concurrency TS2 – Maged Michael Why use hazard pointers? • Fast (low nanoseconds) and scalable protection. • Supports arbitrary protection duration (e.g., the user-defined fn is allowed to block Extra synchronization Removed private cache from Folly Using sharded domain lists instead Slow Not Scalable Is One Call to Global Cleanup Always Enough? No Hazard Pointer Synchronous Reclamation Beyond Concurrency of threads. Is there a good solution using the default domain? Is there any good solution? Fast Scalable Robust Synchronous Reclamation Hazard Pointer Synchronous Reclamation Beyond Concurrency TS2 – Maged

functions, and then manage how such defects can be detected and mitigated, in a portable and scalable fashion, during program evaluation. – Joshua Berne54 Copyright (c) Timur Doumler | functions, and then manage how such defects can be detected and mitigated, in a portable and scalable fashion, during program evaluation. – Joshua Berne55 Copyright (c) Timur Doumler | functions, and then manage how such defects can be detected and mitigated, in a portable and scalable fashion, during program evaluation. – Joshua Berne56 Copyright (c) Timur Doumler |

to read, understand, and write • Practical safety guarantees and testing mechanism • Fast and scalable development • Current hardware architectures, OS platforms, and environments as they evolve CppCon to read, understand, and write • Practical safety guarantees and testing mechanism • Fast and scalable development • Current hardware architectures, OS platforms, and environments as they evolve CppCon to read, understand, and write • Practical safety guarantees and testing mechanism • Fast and scalable development • Current hardware architectures, OS platforms, and environments as they evolve CppCon

cells that are used as building 6 Zoph, Barret, et al. "Learning transferable architectures for scalable image recognition." Proceedings of the IEEE conference on computer vision and pattern recognition higher number of normal and reduction cell blocks. Source: Learning transferable architectures for scalable image recognition. NASNet cells are composed of blocks. A single block corresponds to two hidden primitive operations 3x3 conv and 2x2 maxpool. Source: Learning transferable architectures for scalable image recognition. The output of a cell is the concatenated output of all the component blocks

OS/Architectures • Cross compile on many opera*ng systems • Access to scalable cloud compu*ng environments • MGO driver for Go provides excellent MongoDB with the Go language and template framework, we have a scalable, redundant and feature rich solu*on. • Go, MGO, MongoDB and Beego process datasets. • These technologies can be used to create highly scalable systems. CONCLUSION MongoDB and Go are technologies that work

Difficult Easy Communication efficiency Low High Technology stack selection Restricted Flexible Scalable Restricted Flexible Reusability Low High Difficulty in breaking down business complexity Difficult rebuilding model Automatic sharding Automatically generate microservice code Distributed Scalable Horizontaly scalable View Control Model XX UIs Service orchestration Microservice -based Cloud-based

stores Programming Models dapr: Distributed Application Runtime Building blocks for building scalable distributed apps Open Application Model (OAM) Platform agnostic application model Application Distributed Application Runtime State of Enterprise Developers Being asked to develop resilient, scalable, microservice-based apps Functions and Actors are powerful programming models They write in output bindings to external resources including databases and queues Publish & Subscribe Secure, scalable messaging between services Actors Encapsulate code and data in reusable actor objects as a common