A Day in the Life of a Data Scientist Conquer Machine Learning Lifecycle on Kubernetes Brian Redmond • Cloud Architect @ Microsoft (18 years) • Azure Global Black Belt Team • Live in Pittsburgh, PA Repeatable/consistent • CI/CD • This has worked well for App Dev. Now time for AI/ML • But, must ensure data scientist are not hindered by structure Why Containers, Kubernetes & Helm? • Container • Contains Scalable • Easy to explore hyper-parameters space • Easy to do distributed training But really, Data Scientists shouldn’t have to care about containers, kubernetes and all that stuff • Pachyderm can

locations with the use of groundbreaking technologies, to enable decision-making at the point of data collection. Fast, insight-driven decision-making in highly dynamic and dangerous conditions is Allen’s innova- tive edge computing solution, SmartEdge, addresses the increasing need to gather data in real time and perform analysis at the point of collection, supplying imme- diate insight which battlefields. The an- alytics enabled and performed by Smart- Edge allow battalions to make real-time, data-driven decisions which dramatically improve operational outcomes and in- crease the probability of

intensive cryptanalytic attacks ● A cryptoperiod is the time during which a key is used to encrypt data Key rotation: cryptoperiod There are lots of factors that influence the choice of cryptoperiod Strength of cryptographic algorithms used ○ Implementation ○ Operating environment ○ Volume of data ○ Re-keying method ○ Number of key copies ○ Personnel turnover ○ Threat model ○ New and disruptive cardholder data against disclosure and misuse. 3.6 Fully document and implement all key-management processes and procedures for cryptographic keys used for encryption of cardholder data, including

scale • Reduce the learning curve for customer and ourselves • Get consistent user experience and data, leverage with PaaS capability • Facilitate our PaaS and micro-service product Kubernetes Capabilities/Advantages agent to collecting log data ElasticSearch ElasticSearch Monitor/Alert Service CronJob Node Pod Node Pod Unified logging、monitoring、alert with PaaS Consistent data Node group of build nodes configuration and history in MySQL • Logging in central logging service - ElasticSearch • Metric data in monitoring system - prometheus • Alertmanager to invoke various alert and related actions docker

FailedPostStartHook, Unhealthy… Trace system Increase of SLO Data Collect Audit log Event The unhealthy node Monitoring Isolation Recover Degrade Data Analysis Failures/Machine Failures/Reason Report User Storage Analysis Platform Trace Report Weakness The trace system Data Collect: Collect Audit log for the whole cluster. Data analysis: Analyze failure reason if pod is failed. Reason analysis: Analyze kubelet metrics daemonset metrics node load slo metrics csi metrics dirty data With huge amount of metrics data collected, statistical methods can be used to check whether the node is healthy

com/developer/article/1627330 Step1: 安装 docker-ce # 安装需要的支撑软件 yum install -y yum-utils device-mapper-persistent-data lvm2 # 添加 yum 源 yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce configmap vi 3-configmap.yaml kind: ConfigMap apiVersion: v1 metadata: name: traefik-config data: traefik.yaml: |- serversTransport: insecureSkipVerify: true api: 的登录密码(登录用户为 admin) kubectl -n rook-ceph get secret rook-ceph-dashboard-password -o jsonpath="{['data']['password']}" | base64 --decode && echo Step4: 配置 storageclass download https://github.c

docker # docker info ★配置docker服务使用systemd去管理（以及信任本地镜像仓库） # vi /etc/docker/daemon.json { "data-root": "/docker_data", "registry-mirrors": [ "h�ps://cof-lee.com:5443" ], "insecure-registries": [ "cof-lee /etc/docker/daemon.json <data-root": "/docker_data", "registry-mirrors": [ "h�ps://cof-lee.com:5443" ], "insecure-registries": [ "cof-lee cm-name2.yaml <data: key1: value1 key2: value2 key3: value3 EOF # kubectl apply -f cm-name2.yaml #创建

node-local-path /data/all-log pod-path /data/all-log kafka-cluster elasticsearch-cluster logstash-cluster kafka-groupid=es topic applog logstash kafka-groupid=file file-storage mkdir -p /data/all- lo pod * app-log log* daemonsets filebeat 容器启动挂载宿主/data/all-log到容器/data/all-log 容器启动时创建namespace和pod-name 创建软连接到logs目录 daemonset在每一个k8s-node节点启动 挂载/data/all-log目录 实时扫描发现新目录日志 logstash通过不同的消费groupid来消费同一

manger manger metadata data data metadata metadata Data and metadata store in different devices File data/metadata Heartbeat Heartbeat metabata Heartbeat metabata data 存储 openebs localPV Why:

��1�DaemonSet or Sidecar • �����DaemonSet • ���PASS���Sidecar DaemonSet Sidecar ��2��������� /app/data/logs Sidecar ������� ������ ����� ����� ��� ���� • �����40����40GB SSD • 5W�����2PB SSD ������� structured Data SQL�NoSQL Log Service / LogShipper Mobile & Web IoT Mobile Logs Web Text & Logs Services & Languages IoT & Devices Camera �� Log Service / LogHub Real-time Data Stream �� Log Service / Dashboard ��� Stream Processing SparkStreaming Function Compute Hadoop HIVE Big Data Analytics Batch Processing Pig PAI MaxCompute EMR Interactive Analytics DLA Log Service /