## Estimation of availability and reliability in CurveBS CurveBS uses the RAFT protocol to maintain consistency of stored data. It generally takes the form of 3 replicas of data. If one replica fails Time of disks is MTBF (Mean Time Before Failure). $$ AFR=\frac{1}{MTBF/\ (24*365)}*100 $$ In CurveBS, the data on one disk is distributed in about 50 copysets, which means when a disk fails, up to 50 P=(P_{1}*P_{2}*P_{3}) $$ The probability of no data loss in a cluster is as follows: Assume that in a CurveBS cluster consisting of 1200 disks, MTBF of each disk is 1.2 million hours, and the data recovery time

## CurveBS I/O processing flow Before introducing IO processing flow, we first describe the overall architecture, data organization and topology structure of CURVE.  CurveBS uses the central node Metadata Server (MDS) to manage virtual disk mapping and data replicas distribution. Decentralization currently leads to excessive software and hardware, the need for decentralization is not particularly strong in most cases. The CurveBS core consists of three parts: ### 1. Metadata Server (MDS) - Manages and stores metadata information

opencurve.io/ ## Agenda • CurveBS Architecture • CurveBS Topology • CurveBS Data Organization • MetaData Server (MDS) • ChunkServer • Client • CurveBS IO processing flow • CurveBS Performance considerations Organization - CurveFS MetaServer • CurveFS Client - CurveFS MKNode Flow • CurveFS Write to S3 Flow ## CurveBS Architecture  copset Chunk\_{5} $|$ Chunk\_{6} $|……| |……|……|……|……| ## CurveBS Topology  ## CurveBS Data Organization VDISK /dev/sda Chunks CopySets

Distributed storage system https://www.opencurve.io/ ## Agenda • What is Curve • Use Cases • CurveBS • Key Features • Comparing to Ceph • CurveFS • Comparing to Ceph • Current Status • Roadmap What is Curve - Curve is an distributed storage system • Components - Curve Block Storage (CurveBS) - CurveBS: a high performance cloud native distributed block storage • Curve File System (CurveFS) system - kubernetes Host / VM ↓ CSI / Storage Class NBD Curve Block API SSD three copies CurveBS HDD / HDD EC Curve Block API APPs ![Image](/uploads/documents/2/7/b/f/27bfb5b5cb83abf4

Storage (CurveBS) - CurveBS: a high performance cloud native distributed block storage • Curve File System (CurveFS) • CurveFS: a high performance cloud native file system (Based on CurveBS / S3 compatible Features for Cluster • Features for CurveBS • Features for CurveFS • Features for monitor ## Feature list for cluster • Install / uninstall/ upgrade and configure CurveBS/CurveFS through helm chart • upgrade for cluster - CurveBS mirroring configured with CRDs • Different Curve clusters may share MDS and ETCD server • Replication between CurveFS configured with CRDs ## Feature list for CurveBS • CSI plugin

## 基于SPDK的CurveBS PFS存储引擎 ## Why ·为了减少使用cpu做内存copy，减少系统调用 发挥某些被操作系统屏蔽的功能，例如nvme write zero ·根据阿里《When Cloud Storage Meets RDMA》的说法 ·在100Gbps网络带宽时，内存带宽成为瓶颈 Intel Memory Latency Checker (MLC)测试得到的CPU内存带宽是61Gbps 为什么用PFS ·对代码比较熟悉 找一个能管理裸盘，具有产品级可靠性的代码挺难的 ·PFS支持类POSIX文件的接口，与使用EXT4的存储引擎代码很像，所以容易移植现有代码到PFS存储引擎 CurveBS对文件系统元数据的操作非常少，对文件系统的要求不高，所以不需要元数据高性能，这方面PFS也合适 ## 对PFS的修改 ## 基于阿里开源的PFS ·不再基于daemon模式，而是直接使用pfs vector的接口主要是为了与brpc的iobuf对接，iobuf由若干地址不连续的block组成，一次IO提交可以提高效率。 ## PFS+SPDK 的部分读写的实现 某些盘只支持4k单位读写，但是CurveBS支持512字节读写 可能存在部分写的并发冲突 ·引入并发的range lock解决冲突 ## PFS+SPDK 的DMA支持 .ssize_t pfs_writev_dma(int fd,

jpg) CBD NBD iSCSI FUSE kernel NFS S3 HDFS Performance First File Storage Block Storage CurveBS NVMe SSD HDD EBS public cloud Performance Capacity Passive/Active Data Lifecycle Management isolation Server Indicates a physical server Chunkserver is a service instance on a physical server CurveBS Cluster  zone

iSCSI软件 • Client端: iscsi initiator，系统自带 • Linux open-iscsi • Windows iSCSI 发起者 • 服务器端 • 必须是CurveBS原生支持的平台，因为需要curve原生接口，目前是Linux ## iSCSI target服务器 • LINUX LILO • 一般用于输出内核本地块设备 • TCMU • 作为LILO支持用户态的接口

b5df49cd2bce19fc844f8319/p3_1.jpg) File Storage Block Storage CurveFS Performance Turbo CurveBS NVMe SSD HDD Ceph RBD/RADOS Performance Capacity Cost Priority doing roadmap Passive/Active

4K大小时延比较(us)  ## CurveBS FIO ## FIO 4K 随机写 UCX 配置双网卡（UCX_MAX_EAGER_RAILS=2, UCX_MAX_RNDV_RAILS=2） 时延单位: us |QD|RDMA-IOPS|clat