competitive landscape 2. Provide stable, reliable, and secure service to the customer b. THE BUSINESS VALUE OF DEVOPS i. Code and change deployments (thirty times more frequent) ii. Code and change deployment significant difference between technology and manufacturing value streams is that our work is invisible. Unlike physical processes, in the technology value stream we cannot easily see where flow is being impeded WASTE IN THE VALUE STREAM 1. In the book Implementing Lean Software Development: From Concept to Cash, Mary and Tom Poppendieck describe waste and hardship in the software development stream as anything

Badger: Fast Key-Value DB in Go Manish R Jain, Dgraph Labs Apr 14,2018 Gopher China, Shanghai Dgraph Labs Fast, Distributed graph database. Sparse data sets. Lots of relationships. https://dgraph https://dgraph.io What is Badger? Badger is an embedded key-value database, written in Go. Licensed under Apache 2.0. Current Status Closing v2.0. Close to 3500 Github starts. 42 contributors. Used 0-stor, Sandglass. Serving 300TB (and growing) at Usenet Express Basic Operations Set a key-value func set() error { fmt.Println("\nRunning SET") return db.Update(func(txn *badger

## CS 591 K1: Data Stream Processing and Analytics Spring 2020 ## 1 /23: Stream Processing Fundamentals Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## What is a stream? - In traditional data processing processing applications, we know the entire dataset in advance, e.g. tables stored in a database. A data stream is a data set that is produced incrementally over time, rather than being available in full before high-volume, real-time data that might be unbounded • we cannot store the entire stream in an accessible way • we have to process stream elements on-the-fly using limited memory ## Properties of data streams

## Scalable Stream Processing - Spark Streaming and Flink Amir H. Payberah payberah@kth.se 05/10/2018 https://id2221kth.github.io ## Data Processing Graph Data Pregel, GraphLab, PowerGraph GraphX GraphX, X-Stream, Chaos Batch Data MapReduce, Dryad FlumeJava, Spark Structured Data Spark SQL Machine Learning Mliib Tensorflow Streaming Data Storm, SEEP, Naiad, Spark Streaming, Flink, Millwheel BigTable, Cassandra ## Distributed Messaging Systems Kafka Resource Management Mesos. YARN ## Stream Processing Systems Design Issues ▶ Continuous vs. micro-batch processing Record-at-a-Time vs. declarative

## 由浅入深学习 map 容器 by 彭于斌 (@archibate) 我负责监督你鞋习！ 本期看点： 用方括号 [ ] 取出 map 元素居然是错误的！ 能不能在遍历的同时删除元素？安全吗？ emplace, emplace hint, try_emplace 的区别?  3. string_view , const char * 的爱恨纠葛 (BV1ja411M7Di) 我们都要认真鞋习哦 4. 万能的 map 容器全家桶及其妙用举例（本期） 5. 函子 functor 与 lambda 表达式知多少 6. 通过实战案例来学习 STL 算法库 7. C++ 标准输入输出流 & 字符串格式化 5e498bf3d00d971de6ee1/p3_1.jpg) ## map 查找元素的两个接口 • map 提供了两个查找元素的接口，一曰 ，二曰 at。 那么他们两个又有什么区别呢？很多新手都分不清他俩，可能只认识☐。 ## 读取 map 元素 • map m; - 读取 map 中指定键值的元素有两种方法。 • val = m["key];

# CS 591 K1: Data Stream Processing and Analytics Spring 2020 ## 1 /28: Stream ingestion and pub/sub systems Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Streaming sources  Where do stream processors read data from? Files, e.g. transaction logs Sockets IoT devices and sensors Databases and KV stores Message queues unpredictable delays • might be producing too fast • stream processor needs to keep up and not shed load • might be producing too slow or become idle • stream processor should be able to make progress • might

Stream Processing with Apache RocketMQ and Apache Flink 王鑫 · The Apache Software Foundation Nov.4, 2018, Shanghai, Apache Flink China Meetup  .setParallelism(2) .process(new ProcessFunction<map, map="">() { @Override public void processElement(Map in, Context ctx, Collector<map> out) throws Exception { // do something

# CS 591 K1: Data Stream Processing and Analytics Spring 2020 4/16: Skew mitigation Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Key partitioning  > δ*N, where N is the number of stream elements • The solution will not contain any item y with frequency: • freq(y) < (δ - ε)*N, for a user-chosen value ε |not \\delta^{\*}N $|| ## Notation (I) Input: a stream of items N: number of items in the stream $ f_{e} $ : true frequency of the item e in the input stream f: estimated frequency of item δ: user-defined

# CS 591 K1: Data Stream Processing and Analytics Spring 2020 2/25: State Management Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## State in dataflow computations Any non-trivial streaming computation d594d85ed844723ae7e0a3cea6fb1/p4_2.jpg) What state types can you think of? • Count, sum, list, map, ... ## State management in Apache Flink All data maintained by a task and used to compute results: to a key defined in the operator's input records - Flink maintains one state instance per key value and partitions all records with the same key to the operator task that maintains the state for this

## CS 591 K1: Data Stream Processing and Analytics Spring 2020 ## 4 /14: Stream processing optimizations Vasiliki (Vasia) Kalavri vkalavri@bu.edu ## Topics covered in this lecture • Costs of streaming 2db614d10387ee7/p3_1.jpg) ## Revisiting the basics A series of transformations on streams in Stream SQL, Scala, Python, Rust, Java...  - Stateful operators maintain state that reflect part of the stream history they have seen • windows, continuous aggregations, distinct... - State is commonly partitioned