MathWorks, Inc. MATLAB与Spark/Hadoop相集成：实现大 数据的处理和价值挖 马文辉 2 内容 ▪ 大数据及其带来的挑战 ▪ MATLAB大数据处理 ➢ tall数组 ➢ 并行与分布式计算 ▪ MATLAB与Spark/Hadoop集成 ➢ MATLAB访问HDFS(Hadoop分布式文件系统） ➢ 在Spark/Hadoop集群上运行MATLAB代码 ▪ 应用演示 数据规模增大、数据复杂度增加，增加处理难度和所需时间； 5 MATLAB的大数据处理 ▪ 编程 ▪ Streaming ▪ Block Processing ▪ Parallel-for loops ▪ GPU Arrays ▪ SPMD and Distributed Arrays ▪ MapReduce ▪ MapReduce (MDCS/PCT) ▪ MATLAB API for Spark API ImageDatastore 6 tall arrays ▪ tall array – 一种新的数据类型，专门用于处理大数据. – 用于处理数据规模超过单个机器或群集的内存承载能力的数据集合 ▪ 使用方式等同于MATLAB 数组(array) – 支持数据类型包括数值型、字符串、时间类型、表等… – 支持众多基本的数学函数、统计函数、索引函数等. – 支持机器学习算法包括分类、聚类和回归 7 tall

regression. These exercises have been extensively tested with Matlab, but they should also work in Octave, which has been called a “free version of Matlab”. If you are using Octave, be sure to install the Image using gradient descent algorithm, based on which, we can predict the height given a new age value. In Matlab/Octave, you can load the training set using the commands x = load ( ’ ex1x . dat ’ ) ; y = load starting gradient descent, we need to add the x0 = 1 intercept term to every example. To do this in Matlab/Octave, the command is m = length (y ) ; % st or e the number of t r a i n i n g examples x = [

the number of colors it contains. To begin, download data6.zip and unpack its contents into your Matlab/Octave working directory. Photo credit: The bird photo used in this exercise belongs to Frank Wouters you will then use the 16 colors to replace the pixels in the large image. 3 K-means in Matlab/Octave In Matlab/Octave, load the small image into your program with the following command: A = double (

someone or something? . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 Why don't you compile Matlab/Python/R/… code to Julia? . . . . . . . . . . . . . . . . . . . . . 411 37.2 Sessions and the REPL . . . 428 38 Noteworthy Differences from other Languages 429 38.1 Noteworthy differences from MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429 38.2 Noteworthy differences from ease and expressiveness for high-level numerical computing, in the same way as languages such as R, MATLAB, and Python, but also supports general program- ming. To achieve this, Julia builds upon the lineage

someone or something? . . . . . . . . . . . . . . . . . . . . . . . . . . . 413 Why don't you compile Matlab/Python/R/… code to Julia? . . . . . . . . . . . . . . . . . . . . . 413 37.2 Sessions and the REPL . . . 430 38 Noteworthy Differences from other Languages 431 38.1 Noteworthy differences from MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 431 38.2 Noteworthy differences from ease and expressiveness for high-level numerical computing, in the same way as languages such as R, MATLAB, and Python, but also supports general program- ming. To achieve this, Julia builds upon the lineage

someone or something? . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 Why don't you compile Matlab/Python/R/… code to Julia? . . . . . . . . . . . . . . . . . . . . . 411 37.2 Sessions and the REPL . . . 428 38 Noteworthy Differences from other Languages 429 38.1 Noteworthy differences from MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429 38.2 Noteworthy differences from ease and expressiveness for high-level numerical computing, in the same way as languages such as R, MATLAB, and Python, but also supports general program- ming. To achieve this, Julia builds upon the lineage

someone or something? . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 Why don't you compile Matlab/Python/R/… code to Julia? . . . . . . . . . . . . . . . . . . . . . 411 37.2 Sessions and the REPL . . . 428 38 Noteworthy Differences from other Languages 429 38.1 Noteworthy differences from MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 429 38.2 Noteworthy differences from ease and expressiveness for high-level numerical computing, in the same way as languages such as R, MATLAB, and Python, but also supports general program- ming. To achieve this, Julia builds upon the lineage

(1995) • Ruby (1995) 脚本语言是集中性大爆发的 大概也就是在 Java 出现的那个年代 数据科学语言发展史 (TOP50) • SQL (1973) • SAS (1976) • MATLAB (1984) • Python (1991) • R (2000) • Julia (2009) • Go+ (2020) 数据科学的发展古老而漫长 但开始进入加速期 语言发展史的启发 • 数据科学是计算机的最初需求，历史悠久但进步缓慢 -因为数据大爆发的时代一直没有到来 02 数据科学的发展 数据科学的原始时期：数学软件时代 • SQL (1973) • SAS (1976) • MATLAB (1984) • Excel (1985) • Limited Domains (有限领域) ，比如 BI (Business Intelligence) • Limited Data (有限数据规模) 时代 • 从前 -Limited Domains (有限领域): 比如 BI (Business Intelligence) -Limited Data (有限数据规模): 比如 Excel、Matlab • 未来 -Full Domains (全领域): 智能应用 (Intelligent Application) • 典型代表：抖音、快手 -Big Data (大规模数据) -Any

. . . 408 38 Noteworthy Differences from other Languages 410 38.1 Noteworthy differences from MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 38.2 Noteworthy differences from ease and expressiveness for high-level numerical computing, in the same way as languages such as R, MATLAB, and Python, but also supports general program- ming. To achieve this, Julia builds upon the lineage following languages, then you should start by reading the section on noteworthy differences from MATLAB, R, Python, C/C++ or Common Lisp. This will help you avoid some common pitfalls since Julia differs

. . . 408 38 Noteworthy Differences from other Languages 410 38.1 Noteworthy differences from MATLAB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 38.2 Noteworthy differences from ease and expressiveness for high-level numerical computing, in the same way as languages such as R, MATLAB, and Python, but also supports general program- ming. To achieve this, Julia builds upon the lineage following languages, then you should start by reading the section on noteworthy differences from MATLAB, R, Python, C/C++ or Common Lisp. This will help you avoid some common pitfalls since Julia differs