high performance cloud native file system (Based on CurveBS / S3 compatible Storage)Operator capability Level OPERATOR CAPABILITY LEVEL CURVE COMMENT BASIC INSTALL Y (by Helm) automated application provisioning tuningCloud native Feature list • Features for Cluster • Features for CurveBS • Features for CurveFS • Features for monitorFeature list for cluster • Install / uninstall/ upgrade and configure CurveBS/CurveFS CurveBS/CurveFS through helm chart • upgrade automation • Supporting Curve Cluster provisioning in helm chart • metadata backup and recovery • MDS / ChunkServer should respect failure domains of

Deploy CurveBS as container service (in Plan) • Config CurveBS by (Cluster and Pool CRDs) in Kubernetes (in Plan) • Support Operator capability level 5 (in Plan) • horizontal / vertical scaling, auto occasionally CAN SYNC WITH REMOTE DISK SERVER Y NI/O Jitter (vs. Ceph) 3 replicas with 9 nodes cluster each node has 20 x SSD, 2xE5-2660 v4 and 256GB mem FAULTS CASE CURVE I/O JITTER CEPH I/O JITTER CurveFS as container service (in Plan) • Config CurveFS by (cluster and storage pools) CRDs in Kubernetes (in Plan) • Support Operator capability level 5 (in Plan) • now support helmCurrent Status

改造 vs 自己实现? redis 中哈希表实现的优点？ 参考 前言 根据之前讨论的结果，元数据节点的架构如下图所示，这里涉及到两部分需要持久化/编码的内容： Raft Log：记录 operator log Raft Snapshot：将内存中的数据结构以特定格式 dump 到文件进行持久化© XXX Page 3 of 12 Raft Log +------+----------- 的方案是不行了. redis 的高可用、高可扩方案？ 主要是 redis cluster + 主从复制 (或者第三方 codis + 哨兵) redis cluster/codis 主要解决扩展性的问题，它会进行分片，每个 redis 实例保存分片的 key 主从复制主要解决高可用，一个分片实例挂 2 个从实例，当主节点挂掉时，cluster/哨兵会自动将从节点升为主节点 redis + muliraft

annual probability of cluster data loss is as follows: 𝑃 = (𝑃! ∗ 𝑃) ∗ 𝑃-) The probability of no data loss in a cluster is as follows: 1 − 𝑃 Assume that in a CurveBS cluster consisting of 1200 minutes, i.e. 0.083 hours. Therefore, the probability of no data loss in the annual of the CurveBS cluster is P = 0.999999781

decreased by 21% and TPS improved by 26%Performance compare with CEPH The test environment ● A cluster consists of six nodes. Each node consists of 20 x SSDS, 256GB memory, and 2 x e5-2660 cpus Performance death 4s Slow response 1s frequentlyData availability analysis Copyset allocation algorithm cluster with 1200 disks (each have MTBF 1.2 million hours) 3 replicas Restore data on a disk within 5 CURVE Community You can take awayKey designs used by CURVE High availability and reliability ● Cluster topology ● CopySet pre-allocation algorithm ● Raft Consistency protocol High performance ● pre-created

partition管理的元数据，data partition管理数据。meta partition管理inode和dentry信息。 创建一个文件系统时，如何初始化meta partition？ master\cluster.go， chubaofs的文件系统使用volume的来表示，在创建一个文件系统的时候，会创建3个meta partition和10个data partition。chubaofs的data p usage来选的，通常选择内存和disk使用率最低的节点。 并去对应的meta node上去创建对应的meta partition。 如何选择partition的host，通过这个函数去选择。 func (c *Cluster) (excludeZone , excludeNodeSets [] , excludeHosts [] , replicaNum , crossZone , specifiedZone partition使用第一个叫做MetaWrapper的结构体组织起来© XXX Page 7 of 19 type MetaWrapper struct { sync.RWMutex cluster string localIP string volname string …… // Partitions and

Server (MDS) l Manages and stores metadata information and persists the data in ETCD l Collect cluster status and schedule. 2. Chunkserverl Responsible for data storage l Multi-replicas consistency 1. Fs-meta Cluster is used to manage the inode and dentry metadata of files. The architecture is like CurveBS, so metadata scalability is very good in this way. 2. Fs-data cluster is used to store

在物理pool上，还创建了一个逻辑pool，逻辑pool使用3个zone，采用 3副本，有100个copyset。 cluster pool1 zone1 zone2 zone3 server1 server2 server3 192.168.0.1:8200 192.168.0.2:8200 192.168.0.3:8200 cluster_map: servers: - name: server1 internalip: OPYSET Copyset的生成策略：Source code : curve/src/mds/copyset/ bool GenCopyset(const ClusterInfo& cluster, int numCopysets, std::vector* out); 例如要在(zone1 zone2 zone3 zone4)中创建8个copyset: result:

from CS1 to CS2的operator给这三个copyset ④ MDS通过RPC在CS2上创建copyset1,2,3这三个copyset ⑤ 假定三个copyset的leader都是CS3，在CS3的下一次心跳的 response中，下发第三步生成的三个operator ⑥ CS3收到change peer from CS1 to CS2的operator，给CS2同步 raft to CS1的operator给 部分copyset，比如copyset1,2,3 ⑤ MDS通过RPC在CS1上创建copyset1,2,3这些copyset ⑥ 假定三个copyset的leader都是CS3，在CS3的下一次心跳的 response中，下发第四步生成的三个operator ⑦ CS3收到change peer from CS2 to CS1的operator，给CS1同步

services can be configured using CRDs in public cloud and on- premises environments • Use the Curve operator to install, upgrade, backup, and expand CurveBS clusters • Use Curve Discover to discover resource services can be configured using CRDs in public cloud and on- premises environments • Use the Curve operator to install, upgrade, backup, and expand CurveFS clusters • Use Curve Discover to discover resource