Ozone on NVMe Wei-Chiu Chuang (jojochuang) Ritesh Shukla (kerneltime) ## Agenda • Overview of how Ozone and how it scales • Why NVME is important for Ozone for scaling • Benefits of using NVME • Impala Impala performance results from NVME clusters • Write path improvements results from NVME clusters • Summary • Questions ## Ozone Architecture ![Image](/uploads/documents/7/2/6/a/726a4b8f876d2c53cacdbfc4c79c1139/p3_1 limitations, working set can be cached in memory and unused data can be destaged to disk • OM uses NVME to store RocksDBs - Future projects such as Snapshots leverage RocksDB to preserve simplicity of

bottlenecked by IO, meaning CPUs were constantly waiting on HDDs to respond. But when we replace HDDs with NVMe SSDs on a Dual Socket Server, the CPU becomes the bottleneck. In an attempt to minimize the CPU bottleneck Server|5.7.10-3|| |Storage||| |SAS HDD|2x SEAGATE ST600MP0005 15K rpm|| |SATA SSD|2x Samsung 850 PRO|| |NVMe SSD|2x Samsung XS1715|| Quad-socket (28 Core) Configuration

	Database SSD	2x Samsung 850 Pro
SAS SSD	2x Samsung PM1633
NVMe	2x Samsung PM1725

It is generally accepted that OLTP database applications

## 基于SPDK的CurveBS PFS存储引擎 ## Why ·为了减少使用cpu做内存copy，减少系统调用 发挥某些被操作系统屏蔽的功能，例如nvme write zero ·根据阿里《When Cloud Storage Meets RDMA》的说法 ·在100Gbps网络带宽时，内存带宽成为瓶颈 Intel Memory Latency Checker (MLC)测试得到的CPU内存带宽是61Gbps ·读写内存都由网卡进行offline ·应用程序不再通过系统调用在内核和用户态来回切换 ## 磁盘的读写 基于EXT4的存储引擎，依然需要通过系统调用来回切换 ·读写都需要CPU拷贝数据 不能发挥某些NVME的功能，例如write zero ## 为什么用PFS ·对代码比较熟悉 找一个能管理裸盘，具有产品级可靠性的代码挺难的 ·PFS支持类POSIX文件的接口，与使用EXT4的存储引擎代码很像，所以容易移植现有代码到PFS存储引擎 ·直接DMA读写，要求的内存必须是DPDK的hugetlb内存 ·必须符合NVME内存读写地址对齐要求 offset 512对齐 ·为零copy提供接口 ## BRPC IOBuf DMA ·修改BRPC，允许使用dpdk内存作为IOBuf的内存分配器 ·BRPC接收到的数据在IOBuf中，IOBuf直接使用于NVME DMA传输 ·使用IOBuf内存读nvme，避免自己写PRP页面对齐内存分配代码 ## pfs_pwrite_zero

/dev/disk/by-path/pci-0000:87:00.0-nvme-1 - /dev/disk/by-path/pci-0000:88:00.0-nvme-1 optionalPaths: - /dev/disk/by-path/pci-0000:89:00.0-nvme-1 - /dev/disk /dev/disk/by-path/pci-0000:90:00.0-nvme-1 thinPoolConfig: name: thin-pool-1 sizePercent: 90 overprovisionRatio: 10 nodeSelector: nodeSelectorTerms: - matchExpressions: /dev/disk/by-path/pci-0000:87:00.0-nvme-1 - /dev/disk/by-path/pci-0000:88:00.0-nvme-1 optionalPaths: - /dev/disk/by-path/pci-0000:89:00.0-nvme-1 - /dev/disk/

##### 3.3.1. 通过 OpenStack 使用 PCI 透传向 etcd 节点提供存储 要为 etcd 节点提供快速存储以便 etcd 在大规模稳定，使用 PCI 透传将非易失性内存表达(NVMe)设备直接传递给 etcd 节点。要使用 Red Hat OpenStack 11 或更高版本进行设置，请在存在 PCI 设备的 OpenStack 节点上完成以下内容。 1. 确定在 BIOS "product_id":"a820", "device_type":"type-PCI", "name":"nvme" } 6. 在控制器上重启 nova-api 和 nova-scheduler。 7. 在 /etc/nova/nova.conf 中的计算节点上： [pci] passthrough_whitelist "name": "nvme" } 要检索您想要透传的 NVMe 设备的 address、vendor_id 和 product_id 值，请运行： # Ispci -nn | grep devicename 8. 重启计算节点上的 nova-compute。 9. 将您正在运行的 OpenStack 版本配置为使用 NVMe 并启动 etcd 节点。 ####

before you have confirmed that you really want to make the changes. Select disk to partition: /dev/nvme0n1 - 256.1 GB INTEL SSDPEKKF256G8L SCSI1 (0,0,0) (sda) - 15.7 GB SMI USB DISK ## Screenshot Go a/c5fa5933634fc607e7b3dc7b93ac2b4f/p12_2.jpg) Partition disks Selected for partitioning: /dev/nvme0n1 - INTEL SSDPEKKF256G8L: 256.1 GB The disk can be partitioned using one of several different schemes the following devices are changed: /dev/nvme0n1 The following partitions are going to be formatted: partition #1 of /dev/nvme0n1 as ext4 partition #5 of /dev/nvme0n1 as swap Write the changes to disks

disk via RocksDB - No block report load: Background processing is separated from foreground. NVME: OM uses NVME to store RocksDBs • Simplicity: Written with simplicity in mind Example: Snapshots leverage 2_1.jpg) ## HARDWARE TRENDS All NVME clusters are increasingly common Ozone’s metadata is stored on SATA SSD or NVME - Increasing number of customers using all NVME clusters (metadata and data) - High High density nodes with Ozone - High performance workloads – Effectively lower TCO for all NVME clusters. |<nl>| |---| org/apache/.. start_thread ## I MPALA + OZONE Data warehouse is popular

阿里云的环境中创建Polardb的方法 共享盘使用阿里云自带的高性能Nvme盘，注意使用Nvme磁盘对可用区有要求： • 华东1（杭州）可用区I • 华东2（上海）可用区B • 华北2（北京）可用区K • 华南1（深圳）可用区F。 只有某些规格的虚拟机可以挂载Nvme共享盘： • g7se • c7se • r7se 虚拟机要求是按量付费才可以挂载Nvme共享盘 阿里云的VIP功能目前还在内侧阶段，需要申请 需要有共享盘：使用阿里云的高性能Nvme共享盘，Nvme共享盘的详细信息见：https://help.aliyun.com/document_detail/256487.html CLup的高可用需要VIP：阿里云目前的VIP功能处于内侧阶段，可以申请使用 • 操作系统：CentOS7.X 可以满足 - 盘要求路径为/dev/nvmeXnY：阿里云的Nvme共享盘的路径已经是这样的格式，无需特殊处理。

Fusion 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) 86 5.3 The Virtual Media Manager 87 5.4 Special Logic and BusLogic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special:

Fusion 81 5 Virtual storage 83 5.1 Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe 83 5.2 Disk image files (VDI, VMDK, VHD, HDD) 86 5.3 The Virtual Media Manager 87 5.4 Special Logic and BusLogic); see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for details. Whereas providing one of these would be enough for VirtualBox by itself, this present initially. Please see chapter 5.1, Hard disk controllers: IDE, SATA (AHCI), SCSI, SAS, USB MSD, NVMe, page 83 for additional information. VirtualBox also provides a floppy controller, which is special: