2.1 弹性分布数据集（RDD） RDD 是 Spark 的最基本抽象, 是对分布式内存的抽象使用, 实现了以操作本地集合的方式来操作分布式数据集的抽象实现。RDD 是 Spark 最核心的东西, 它表示已被分区, 不可变的并能够被并行操作的数据集合, 不同的数据集格式对应不同的 RDD 实现。RDD 必须是可序列化的。RDD 可以 cache 到内存中, 每次对 RDD 数据集的操作之后的结果 效率提升比较大。 #### 1.2.2 RDD 的转换与操作 对于 RDD 可以有两种计算方式 :转换( 返回值还是一个 RDD )与操作( 返回值不是一个 RDD ) 1. 转换(Transformations) (如：map, filter, groupBy, join 等)，Transformations 操作是 Lazy 的，也就是说从一个 RDD 转换生成另一个 RDD 的操作不是马上执行，Spark Actions 操作会返回结果或把 RDD 数据写到存储系统中。Actions 是触发 Spark 启动计算的动因。 #### 1.2.3 血统 (Lineage) 利用内存加快数据加载，在众多的In-Memory类数据库或Cache类系统中也有实现，Spark的主要区别在于它处理分布式运算环境下的数据容错性（节点实效/数据丢失）问题时采用的方案。为了保证RDD中数据的鲁棒性，RDD数据集通过所谓的血统关系

up the live stream into batches of X seconds. • Treats each batch as RDDs and processes them using RDD operations.  ## Spark up the live stream into batches of X seconds. • Treats each batch as RDDs and processes them using RDD operations. • Discretized Stream Processing (DStream)  ## count • Returns a new DStream of single-element RDDs by counting the number of elements in each RDD of the source DStream

ow gRPC server） • Worker通过读取RDD获取训练样本 • RDD的数据cache到内存或者磁盘供多次迭代训练使用 ## SparkFlow介绍 ## SparkFlow与TensorFlow on Yarn对比： |SparkFlow|TensorFlow on Yarn| |---|---| |通过RDD读取训练样本数据，关心文件存储格式|直接读取HDFS数据，不关心文件存储格式|

Pipeline 中的 mapPartition 在进行下一部分讲解之前，我们先简要介绍一下 Spark 的懒执行机制。Spark 的算子分为 Action 和 Transformation 两大类。RDD 的依赖关系构成了数据处理的有向无环图 DAG。只有当 Action 算子出现时，才会执行 Action 算子与前面一系列 Transformation 算子构成的 DAG。Spark 还会根据 Shuffle 为例，最后 runTask() 方法调用了 rdd.iterator()。 ### ResultTask.scala override def runTask(context: TaskContext): U = { ..... // 源码缩略不进行展示：初始化一些需要的对象 val (rdd, func) = ser.deserialize([RDD[T], (TaskContext, deserializeStartCpuTime } else 0L // 这里的 func() 调用了 rdd.iterator() func(context, rdd.iterator(partition, context)) } 而 RDD 的 iterator 方法的源码如下，其调用逻辑最终都会进入 computeOrRead-Checkpoint 方法，若没有

collect to trigger calculation and collect the entire data set of all tasks(a.k.a. partitions of an RDD), after all partitions data arrived, then the client pulls the result set from the Driver through the using custom RDD resource profile.|1.8.0| |spark.sql.finalWriteStageExecutorCores|fallback spark.exe, corex|Specify the executor core request for final write stage. It would be passed to the RDD resource profile memory|Specify the executor on heap memory request for final write stage. It would be passed to the RDD resource profile.|1.8.0| |spark.sql.finalWriteStageExecutorMemoryOverhead|fallback spark.exe, memoryOverhead|Specify

collect to trigger calculation and collect the entire data set of all tasks(a.k.a. partitions of an RDD), after all partitions data arrived, then the client pulls the result set from the Driver through the using custom RDD resource profile.|1.8.0| |spark.sql.finalWriteStageExecutorCores|fallback spark.exe, corex|Specify the executor core request for final write stage. It would be passed to the RDD resource profile memory|Specify the executor on heap memory request for final write stage. It would be passed to the RDD resource profile.|1.8.0| |spark.sql.finalWriteStageExecutorMemoryOverhead|fallback spark.exe, memoryOverhead|Specify

collect to trigger calculation and collect the entire data set of all tasks(a.k.a. partitions of an RDD), after all partitions data arrived, then the client pulls the result set from the Driver through the using custom RDD resource profile.|1.8.0| |spark.sql.finalWriteStageExecutorCores|fallback spark.exe, corex|Specify the executor core request for final write stage. It would be passed to the RDD resource profile memory|Specify the executor on heap memory request for final write stage. It would be passed to the RDD resource profile.|1.8.0| |spark.sql.finalWriteStageExecutorMemoryOverhead|fallback spark.exe, memoryOverhead|Specify

peline| |吞吐|TB-PB|TB-PB|未经大规模生产验证| |性能|一般(分钟小时级别)|快(秒级)|优秀 x2| |稳定性|好|一般|已在阿里内部验证| |API|差(MR)|最丰富(RDD/DataSet/DataFrame)Python/Scala/R/Java|丰富(TableAPI)Scala/Java| |SQL|HiveSQL|SparkSQL|ANSI SQL| |易用性|一般|易用|一般|

collect to trigger calculation and collect the entire data set of all tasks(a.k.a. partitions of an RDD), after all partitions data arrived, then the client pulls the result set from the Driver through the enabled|false|When true, make final w RDD resource profile.| |spark.sql.finalWriteStageExecutorCores|Fallback spark.executor.cores|Specify the executor core passed to the RDD resou| |spark.sql.finalWriteSta

collect to trigger calculation and collect the entire data set of all tasks(a.k.a. partitions of an RDD), after all partitions data arrived, then the client pulls the result set from the Driver through the enabled|false|When true, make final w RDD resource profile.| |spark.sql.finalWriteStageExecutorCores|Fallback spark.executor.cores|Specify the executor core passed to the RDD resource| |spark.sql.finalWrite