Apache Ozone Erasure Coding(EC) The Modern Big Data Object Store with More Than 50% Storage Space Savings Uma Maheswara Rao Gangumalla Sr. Engineering Manager, Cloudera Inc Stephen O’Donnell Sr. Staff Non EC Flow Erasure Coding Requirements ❏ Phase - I ❏ Enable EC at Cluster/Bucket Level ❏ Should be able to Write files in EC format ❏ Should be able to Read the files which were written in EC format 6:3, 10:4 EC Schemes ❏ Should be able to recover the files automatically on failures ❏ Online recovery ❏ Phase - II ❏ Offline recovery ❏ Phase - III ❏ Should provide options to enable EC via Recon

3 ⽬录 I. Ozone 构架 II. Ozone 1.3 新功能 III. 未来展望 4 Ozone 构架 5 Ozone 1.3 新功能 I. 纠删码(Erasure coding) II. 系统均衡器(Container Balancer) III. 性能优化 - ⽂件系统优化(File System Optimization) IV. 性能优化 - 合并Container 100% 3-replica 2 33% EC RS(6,3) 3 67% EC RS(10, 4) 4 71% EC RS(3,2) 2 60% 以计算为代价，满⾜数据可靠性的同时， 降低数据存储成本 数据可靠性 vs. 存储效率 7 Ozone条带纠删码 I. 物理块：每个DN磁盘上的数据块，默认256MB II. 逻辑EC块：满⾜EC策略的⼀个⽤户数据块。例如RS-3-2，⼀个逻辑块3*256MB⼤⼩ 个逻辑块3*256MB⼤⼩ III. 条带：条带的默认粒度1MB，可配置 IV. EC Container Group：给定Container的⼀组满⾜EC策略的副本实例 8 数据写⼊ DN5 C-1 C-2 B-1-p B-2-p DN1 C-1 C-2 B-1-d B-2-d DN2 C-1 C-2 B-1-d B-2-d DN3 C-1 C-2 B-1-d

released in Dec 2021 ● Version 1.3.0 is in-progress ○ Tons of new features and improvements ■ Erasure Coding ■ Container Balancer ■ S3 Multi-Tenancy ■ S3 gRPC improvements ○ 1000+ new commits since streaming • Efficient data path with rack awareness • Zero copy buffers – Simplified IO path for erasure coding • OM - operations per second – Concurrency improvements – Caching background updates – Reducing

threshold 纠删码(HDDS-3816) 数据可靠性 (越⾼越好) 存储效率 (越⾼越好) 1-replica 0 100% 3-replica 2 33% EC RS(6,3) 3 67% EC RS(10, 4) 4 71% EC RS(3,2) 2 60% 以计算为代价，在不降低数据可靠性的同 时，降低数据存储成本 数据可靠性 vs. 存储效率 纠删码策略 • 内建⽀持的策略 C-2 B-1-p B-2-p EC Container Group1 EC Container Group2 客户端 写⼊⽂件 256MB 256MB 256MB 256MB 256MB 256MB 256MB 256MB 256MB 0 data1 data2 data3 parity1 parity2 数据写⼊ • EC Container Group Group：给定Container的⼀组满⾜EC策略的副本实例 • 物理块：每个DN磁盘上的数据块，默认是256MB • 逻辑EC块：属于单个条带，满⾜EC策略的⼀组数据块。例如EC-3-2，⼀个逻辑块 3*256MB⼤⼩ • 条带粒度：条带的粒度默认1MB，可配置 数据读取 DN5 C-2 DN1 C-2 DN2 C-2 DN3 C-2 DN4 C-2 EC Container Group