Apache Ozone Erasure Coding(EC) The Modern Big Data Object Store with More Than 50% Storage Space Savings Uma Maheswara Rao Gangumalla Sr. Engineering Manager, Cloudera Inc Stephen O’Donnell Sr. Staff Non EC Flow Erasure Coding Requirements ❏ Phase - I ❏ Enable EC at Cluster/Bucket Level ❏ Should be able to Write files in EC format ❏ Should be able to Read the files which were written in EC format 6:3, 10:4 EC Schemes ❏ Should be able to recover the files automatically on failures ❏ Online recovery ❏ Phase - II ❏ Offline recovery ❏ Phase - III ❏ Should provide options to enable EC via Recon

Secure access to EC2 (for developers) Agenda – introduction – EC2 in native environment – 3-tier architecture intro – alternative methods of resource connection: – Bastion host – SSM – EC2 Instance Connect DDoS attacks A regular virtual machineIt's a VM. Classic EC2 instance in de‐ fault setting. Resources: EC2SecurityGroup: Type: AWS:: EC2:: SecurityGroup Properties: GroupName: Launch-wizard-13 Launch-wizard-13 GroupDescription: Allow traffic to EC2 SecurityGroupIngress: - CidrIp: 0.0.0.0/0 IpProtocol: -1 SecurityGroupEgress: - CidrIp: 0.0.0.0/0

–前端 –python 人工智能资料下载，可百度访问：尚硅谷官网 1）纠删码操作相关的命令 [atguigu@hadoop102 hadoop-3.1.3]$ hdfs ec Usage: bin/hdfs ec [COMMAND] [-listPolicies] [-addPolicies -policyFile ] [-help ]. 2）查看当前支持的纠删码策略 [atguigu@hadoop102 hadoop-3.1.3] hdfs ec -listPolicies Erasure Coding Policies: ErasureCodingPolicy=[Name=RS-10-4-1024k, Schema=[ECSchema=[Codec=rs, numDataUnits=10 hdfs ec -enablePolicy - 尚硅谷大数据技术之 Hadoop（生产调优手册） ——————————————————————————————————————— 更多 Java –大数据 –前端 –python 人工智能资料下载，可百度访问：尚硅谷官网 policy RS-3-2-1024k Erasure coding policy

3 ⽬录 I. Ozone 构架 II. Ozone 1.3 新功能 III. 未来展望 4 Ozone 构架 5 Ozone 1.3 新功能 I. 纠删码(Erasure coding) II. 系统均衡器(Container Balancer) III. 性能优化 - ⽂件系统优化(File System Optimization) IV. 性能优化 - 合并Container 100% 3-replica 2 33% EC RS(6,3) 3 67% EC RS(10, 4) 4 71% EC RS(3,2) 2 60% 以计算为代价，满⾜数据可靠性的同时， 降低数据存储成本 数据可靠性 vs. 存储效率 7 Ozone条带纠删码 I. 物理块：每个DN磁盘上的数据块，默认256MB II. 逻辑EC块：满⾜EC策略的⼀个⽤户数据块。例如RS-3-2，⼀个逻辑块3*256MB⼤⼩ 个逻辑块3*256MB⼤⼩ III. 条带：条带的默认粒度1MB，可配置 IV. EC Container Group：给定Container的⼀组满⾜EC策略的副本实例 8 数据写⼊ DN5 C-1 C-2 B-1-p B-2-p DN1 C-1 C-2 B-1-d B-2-d DN2 C-1 C-2 B-1-d B-2-d DN3 C-1 C-2 B-1-d

2 What's Coming in Kotlin 1.3 Standard library Tooling Getting Started Basic Syntax Idioms Coding Conventions Basics Basic Types Packages Control Flow: if, when, for, while Returns and Jumps underscores in numeric literals to separate groups of digits: val oneMillion = 1_000_000 val hexBytes = 0xFF_EC_DE_5E val bytes = 0b11010010_01101001_10010100_10010010 Read the KEEP for more details. For properties current coding style for the Kotlin language. Source code organization Naming rules Formatting Documentation comments Avoiding redundant constructs Idiomatic use of language features Coding conventions

1.2 What's New in Kotlin 1.3 Standard library Tooling Getting Started Basic Syntax Idioms Coding Conventions Basics Basic Types Packages Control Flow: if, when, for, while Returns and Jumps in numeric literals to separate groups of digits: val oneMillion = 1_000_000 val hexBytes = 0xFF_EC_DE_5E val bytes = 0b11010010_01101001_10010100_10010010 Read the KEEP for more details. For properties current coding style for the Kotlin language. Source code organization Naming rules Formatting Documentation comments Avoiding redundant constructs Idiomatic use of language features Coding conventions

 纠错码(Erasure Coding)  多个Standby Namenode  Datanode内部balance工具  云计算平台的支持 • YARN • MapReduce HDFS纠错码(Erasure Coding) • 一个简单的例子 1备份: 1,0 需要额外的2位 XOR编码: 1,0 需要额外的1位 HDFS纠错码(Erasure Coding) • Reed-Solomon task能 带来30%的性能提升。 Hadoop 的未来 HDFS的未来 • 对象存储 - HDFS-7240 • 更高性能的Namenode：更高效的内存使用，锁的改进等 • Erasure Coding的完善 YARN的未来 • 更大规模的集群支持 • 更好的资源调度，隔离和多租户 • 支持更多的应用，包括long running的service 谢谢 Q&A

zlib/1.2.11#f1fadf0d3b196dc0332750354ad8ab7b:2a823fda5c9d8b4f682cb27c30caf4124c5726c8 ˓→#48bc7191ec1ee467f1e951033d7d41b2 - Cache Build requirements cmake/3.22.6 ˓→#3e3d8f3a848b2a60afafbe7a0955085 627b54ff3af ˓→#50f91e204d09b64b24b29df3b87a2f3a hello/1.0: Package folder /Users/user/.conan2/p/21ec37b931782de8/p 4.2. Creating packages 71 Conan Documentation, Release 2.0.17 Check how the include from previous page) $ conan list packages hello/0.1#:* Local Cache: hello hello/0.1#d8e4debf31f0b7b5ec7ff910f76f1e2a (2022-12-22 17:38:35 UTC) PID: 522dcea5982a3f8a5b624c16477e47195da2f84f (2022-12-22

zlib/1.2.11#f1fadf0d3b196dc0332750354ad8ab7b:2a823fda5c9d8b4f682cb27c30caf4124c5726c8 ˓→#48bc7191ec1ee467f1e951033d7d41b2 - Cache Build requirements cmake/3.22.6 ˓→#3e3d8f3a848b2a60afafbe7a0955085 627b54ff3af ˓→#50f91e204d09b64b24b29df3b87a2f3a hello/1.0: Package folder /Users/user/.conan2/p/21ec37b931782de8/p 74 Chapter 4. Tutorial Conan Documentation, Release 2.5.0 Check how the include and previous page) $ conan list "hello/0.1#:*" Local Cache hello hello/0.1 revisions d8e4debf31f0b7b5ec7ff910f76f1e2a (2024-05-10 09:13:16 UTC) packages 522dcea5982a3f8a5b624c16477e47195da2f84f info

zlib/1.2.11#f1fadf0d3b196dc0332750354ad8ab7b:2a823fda5c9d8b4f682cb27c30caf4124c5726c8 ˓→#48bc7191ec1ee467f1e951033d7d41b2 - Cache Build requirements cmake/3.22.6 ˓→#3e3d8f3a848b2a60afafbe7a0955085 627b54ff3af ˓→#50f91e204d09b64b24b29df3b87a2f3a hello/1.0: Package folder /Users/user/.conan2/p/21ec37b931782de8/p 74 Chapter 4. Tutorial Conan Documentation, Release 2.1.0 Check how the include and from previous page) $ conan list packages hello/0.1#:* Local Cache: hello hello/0.1#d8e4debf31f0b7b5ec7ff910f76f1e2a (2022-12-22 17:38:35 UTC) PID: 522dcea5982a3f8a5b624c16477e47195da2f84f (2022-12-22