## CS 591 K1: Data Stream Processing and Analytics Spring 2020 ## 4 /02: Elasticity policies and state migration Vasiliki (Vasia) Kalavri vkalavri@bu.edu Streaming applications are long-running • Workload

of the algorithm. ## Execution policies Most algorithms have overloads that accept execution policies. The standard library algorithms support several execution policies, and the library provides corresponding nth/max_element • Much more efficient than sort • Works for any sortable range - Supports execution policies, comparison functions, concepts/requirements • Hides confusing iterator pair Midpoint available

functions of image pulling and pushing. StratoVirt & virtualization: openEuler supports memory elasticity, huge pages, and enhanced I/O subsystems, and leverages multi-channel concurrency to improve I/O mechanism combine to bring a significant improvement in NVDIMM initialization. Optimized scheduling policies of SCHED_IDLE significantly shorten the scheduling delays of prioritized tasks and reduce interference providing flexible resource scaling capabilities for lightweight workloads. ## New Features - Memory elasticity: StratoVirt allocates and reclaims memory on the real-time memory demand of each workload. The

infrastructure complexity. PowerFlex empowers organizations to move faster by delivering flexibility, elasticity, and simplicity with extraordinary predictable performance for mission-critical workloads, while edge devices, private clouds, public clouds, and vSphere and secures them using global security policies. Use Helm or SUSE Rancher App Catalog to deploy and manage applications across any or all these SUSE Rancher lets you automate processes and applies a consistent set of user access and security policies to all your clusters, no matter where they are running. ## Shared tools and services SUSE Rancher

stream processing 6. Ensure availability despite failures 7. Support distribution and automatic elasticity 8. Offer low-latency ## Building a stream processor ![Image](/uploads/documents/b/6/e/7/b6e7 in-memory|partitioned, virtually unlimited, persisted to backends| |Load management|shedding|backpressure, elasticity| |Fault tolerance|limited support, high availability|full support, exactly-once| ## Summary Today

distributed streaming Systems Architecture and design Scheduling and load management Scalability and elasticity Fault-tolerance and guarantees State management Operator semantics Window optimizations Filtering

jpg) ## Cloud Computing Cloud Characteristics Service Goals On-Demand Quick Scaling Rapid elasticity Low Cost Pay per use High Reliable Uncertain downtime High Available ## Fault Oriented Once

Back-pressure  (c) Elasticity ## Selectively drop records: • Temporarily trades-off result accuracy for sustainable performance

Distributed Foreign Data Wrappers • Columnar compressed FDW • SQL on HDFS • High Availability • Elasticity – Break 1-1 mapping of shard to node ![Image](/uploads/documents/5/e/9/9/5e9986560b576f780f906844b3d21eef/p41_1

|服务化能力(Service)|无(单体应用)|部分服务化&缺乏治理(自持技术,初步服务化)|全部服务化&有治理体系(自持技术,具备治理能力)|Mesh化的服务体系(云技术,治理最佳实践)|| |弹性能力(Elasticity)|全人工扩缩容(固定容量)|半闭环(监控+人工扩缩容)|非全云方式闭环(监控+代码伸缩,百节点规模)|基于云全闭环(基于流量等多策略,万节点规模)|| |无服务器化程度(Serverles