Introduction to Modern Data Science Sam Kreter Code and Slides github.com/samkreter/KubeconAsia2018 The Process Business Need / Problem Discovery The Process Business Need / Problem Discovery Business Need / Problem Discovery Development Production / Actual User Impact The Data Science Pipeline Pipelines The Data Science Pipeline Principles 1. Autonomy I HAVE A VERY PARTICULAR SET OF SKILLS 1. Single Operations per Container 2. Use Parameterize Data Flow • Data Inputs • Data Outputs Distributing Workloads 2. Reproducibility Data Versioning Reproducibility For Developers Reproducibility

0) consul集群搭建 consul agent -data-dir /tmp/node0 -node=node0 -bind=192.168.64.59 -datacenter=dc1 -ui - lient=192.168.64.59 -server -bootstrap-expect 1 consul agent -data-dir /tmp/node1 -node=node1 -bind=192 -bind=192.168.64.94 -datacenter=dc1 -ui consul agent -data-dir /tmp/node2 -node=node2 -bind=192.168.64.249 -datacenter=dc1 -ui client=192.168.64.249 consul join 192.168.64.59 consul members -rpc-addr=192 249:8400 192.168.64.59 agent -data-dir /tmp/node0 -node=node0 -bind=192.168.64.59 -datacenter=dc1 -ui -server bootstrap-expect 1 consul agent -server -bootstrap-expect 3 -data-dir /tmp/consul -node 192.168

proxy_golang，类似 proxy_wasm ，方便 Nginx、OpenResty 等也能够支持 GoLang 扩展。 MoE 方案介绍 — 功能职责 Envoy MOSN MoE Data Plane HTTP(L7) HTTP Filter GoLang L7 extension Filter HTTP Filter Stream filter GoLang L7 State Upstream&C onnpool Pilot SOFARegistry Consul Nacos Consul Others Control Plane MCP Data flow Control flow xDS Discovery Client Metrics Admin REST API Request Request MOSN 做业务扩展 1 2 4 1 2 Envoy MOSN Data flow Control flow GoLang L7 extension SDK GoLang L7 extension filter 5 Other http filter(via GoLang) CGO 1 2 xDS Control Plane Data Plane MoE 2 4 3 Request

标准化能力-承载无忧-E2E云原生纵深安全保障DevSecOps-1 Applications Data Runtime Middleware OS Virtualization Servers Storage NetWorking PaaS 硬件与虚拟化厂商提供，如果是HCI架构， 作为总体集成方，会降低安全集成成本 可信计算环境：OS安全、TPM加密、TEE可信环境 Read Write DB Server User Space File system Data Router&Cache DB Server User Space File system Data Router&Cache DB Server User Space File system Data Router&Cache RDMA Read Only Read Only R/W Parallel-Raft Protocol&Storage Serverless Data Chunk Data Chunk Data Chunk • 云原生的本质在于为云这种弹性资源下能够为应用提供 稳定的基础架构，所以云原生数据库相对于传统数据库 最大的不同也在这个方面：弹性 • 对于数据存储的高性能、高稳定性、高拓展、资源成本 等等都需要同时满足（和传统CAP相悖） • 接入层

到对 应版本的变更进行回滚。 微服务PAAS-应用架构治理-总论 Applications Data Runtime Middleware OS Virtualization Servers Storage NetWorking 传统IT Applications Data Runtime Middleware OS Virtualization Servers IaaS Applications Data Runtime Middleware OS Virtualization Servers Storage NetWorking PaaS BpmPaas iPaas MFT Paas Baas Container Service RDS Service Cache|MQ Service Big Data Service aPaas 专业PaaS(2B) 通过将服务治理、通信、安全等从微服务框架中以独立进程剥离出来，解决了微服 务治理的碎片化以及基础设施耦合问题。 • 但是Sidecar（数据面）在国际化组织和企业联盟的努力下，也形成了叫做UDPA （Universal Data Plane API）的标准，使得在不同场景下的数据面在API层面达成统 一，比如云端Mesh和边缘Mesh由于API的统一，可以使用统一的控制面(配置下发管 理)来实现统一管理和统一服务化抽象，并向应用角度抽象，降低用户使用成本。

Configurations, Automation, Work Distribution, Policy, Security, Compliance Development & Programming Models Data Placement & Governance Operational & Application Management IBM Cloud Private 3rd Party Private Infrastructure Application Development, Integration, Operations & Management Next Generation Middleware, Data, Integration & Analytics IBM MQ IBM Cloud private Platform Kubernetes-based Platform Multiple driven Network & Storage Microservices z Systems Vulnerability Advisor to prevent risk Middleware, Data, management & Analytics Content Integration Content Solution Architectures & Best Practices Core

DistSQL API P D P D P D PD Cluster Pum p Pum p Pum p Draine r TiDB Binlog DM Worker DM Worker Data Migratio n DM Master Upstream Database Downstrea m Database Backup & Recovery Lightnin g KV consensus algorithm to replicate data and provide fault-tolerance ○ Kill one TiDB instance, if the instance has a leader replica, the QPS may drop because the client can’t write data into the replica now ○ Other

用户案例：荷兰ING银行大数据平台云原生化改造 • Platform Entry-point • Project Management Data Science in a box (Advanced analytics toolbox) • Data Discovery • Metadata engine • SQL+BI toolset • Dashboarding Information Hierarchy Queue • Co-location • Elastic scheduling • TDM • Proportional scheduling • …… Volcano 社区 Data from https://volcano.devstats.cncf.io Volcano communityTop contribution orgination 社区用户示例 加入社区

encoding/json package to determine field names when marshaling and unmarshaling structs Other data formats Used for XML, BSON, SQL ORMS... type Candidate struct { Id string `json:"id,omitempty"` that changes slightly depending on specific fields" In other words If it's easier to decode some data into a map first, then pull it into a struct based on one of its values. Command-line Configuration

Querier …… tagstr no-tag tagint tagint tagint tagint …… tagint tagint 云API、K8s apiserver GUI Data …… ① 采集 ① 采集 ② 存储 ② 存储 ③ 查询 ③ 查询 看云网更清晰 Simplify the growing complexity. Stage-1：采集时的编码 Querier …… tagstr no-tag tagint tagint tagint tagint …… tagint tagint 云API、K8s apiserver GUI Data …… ① 采集 ① 采集 ② 存储 ② 存储 ③ 查询 ③ 查询 看云网更清晰 Simplify the growing complexity. MultistageCodec的性能提升：~10X