Non-fused operators can run on different threads • The optimizer can interact with the scheduler and fuse operators according to the number of available cores / threads • Fused operators can share the address address space but use separate threads of control • avoid communication cost without losing pipeline parallelism • use a shared buffer for communication • Fused filters / projections at the source

cyclic dataflows and iterations on streams • Operators are data-parallel • distributed workers (threads) execute one parallel instance of one of more operators on disjoint data partitions 36 Vasiliki

spark._ import org.apache.spark.streaming._ // Create a local StreamingContext with two working threads and batch interval of 1 second. val conf = new SparkConf().setMaster("local[2]").setAppName("NetworkWordCount")