training instead of sequential: huge time saver for large trainings Kubeflow • Machine Learning Toolkit for Kubernetes • To make ML workflows on Kubernetes simple, portable, and scalable • Training

Spark on Kubernetes with volcano API-Server Volcano Scheduler Kubernetes master Podgroup Spark driver pod executor pod executor pod executor pod apiVersion：v1 kind: Pod metadata: annotations: scheduling sh/task-spec: spark-driver createTimestamp: “2019-11-27T09:33:19Z” labels: spark-app-selector:spark- 6cc54577d7254b2d84924500375112f7 spark-role: driver name: job-1574739729783-driver namespace: default /api/v1/namespaces/default/pods/job- 1574739729783-driver uid: f26a81f3-10f8-11ea-938f-fa163eddd2ce Spec: containers: … Create podgroup Create driver pod Request create executor Create executor pods

Java File API HDFS Interface S3 Interface REST API POSIX Interface Alluxio是什么 HDFS Driver S3 Driver OSS Driver Web Driver Alibaba Cloud OSS Alluxio 服务器 Alluxio 服务器 大数据查询 大数据ETL 模型训练 Alluxio核心功能一：分布式数据缓存

数据更改观察者，⽇志分配器，⽇志记录和监视适配器，事件发布者等 代理，桥接器和适配器 控制器，管理器，配置器和更新器 ⼀般来说，单个Pod不会运⾏同⼀应⽤的多个实例。 详情请看The Distributed System ToolKit: Patterns for Composite Containers 考虑替代⽅案 为什么不在单个（Docker）容器中运⾏多个程序？ 1. 透明。 使Pod中的容器对基础设施可⻅，以便

/usr/local/lib/pkgconfig ## VM 驱动程序 VirtualBox brew install docker-machine-driver-hyperkit ~/.kube/config ⽂件中查看所有可⽤的环境 启动 # 可添加参数 # --vm-driver=hyperkit minikube start minikube dashboard Minikube Dashboard

KCQL + Data Driver = 泛化调用 Policy cluster.addon.terway.version > 1.0.9.14 Expression Engine KCQL cluster.addon.terway.version Check Expression ${data.expression} > 1.0.9.14 Data Driver 1 Kubernetes

11m calico-typha-84cb54bfd-hh57t 1/1 Running 1 (10m ago) 11m csi-node-driver-fn7zd 2/2 Running 2 (10m ago) 11m # Remove the taints on the docker.io/calico/typha:v3.26.1 docker.io/calico/pod2daemon-flexvol:v3.26.1 docker.io/calico/node-driver-registrar:v3.26.1 docker.io/calico/csi:v3.26.1 docker.io/calico/kube-controllers:v3.26.1 #以下2个要求上传到harbor，必须为在线获取 2 2 2 2 kubernetes.io/os=linux 4h24m calico-system csi-node-driver 2 2 2 2 2 kubernetes.io/os=linux 4h24m # kubectl get deploy

"exec-opts": ["native.cgroupdriver=systemd"], "log-driver": "json-file", "log-opts": { "max-size": "100m" }, "storage-driver": "overlay2", "storage-opts": [ "overlay2.ov

requests! • ��������! • ������! • limits! • ��������! • ������! • �����������! Nvidia GPU Driver! • Nvidia��! • �����K80�M40�etc.! • ������! • ����! • �����������! • ��������! ��NV��HostPath

and management. It provides optional, additive functionality on top of core Kubernetes. Minikube driver for Fusion and Workstation • Minikube is a tool that makes it easy to run Kubernetes locally. Minikube