## Apache Ozone Erasure Coding(EC) The Modern Big Data Object Store with More Than 50% Storage Space Savings Uma Maheswara Rao Gangumalla Sr. Engineering Manager, Cloudera Inc Stephen O'Donnell Key |Block|Value| |---|---| |Block 001|{ List of Chunks }| |Block 002|{ List of Chunks }| ## Erasure Coding Requirements ## ☐ Phase - I ☐ Enable EC at Cluster/Bucket Level ☐ Should be able to Write

面对同一个问题，来自不同编程语言的程序员给出了思维方式截然不同的解决方法 一定程度上印证了前面的假说：编程语言影响编程思维 避免Go coding in c way/in java way/in python way… 目标：Go coding in go way 编程语言思维的形成 根本：价值观决定思维和语言结构 核心：思维和语言结构影响语言的应用行为 反馈：语言的应用行为反过来持续影响/优化语言结构 Than One Way To Do It” 显而易见的代码，而不是"聪明"的代码 minimize programmer’s effort Least concern about coding style: Gofmt’s style is no one’s favorite, yet gofmt is everyone’s favorite. "一种"循环：for 常规 return } · 优点： 并发 快 显式timeout/cannel，实现可预知 closing thoughts make programming fun when coding in go way “less is more” Go 2.0? Go 1.9:2017.8 Release! Then Go 1.10 or Go 2.0? 回顾"编程

} }; ## What about runtime polymorphism? • Programmers make the virtual tables • Zoo type-erasure already generates virtual tables • The handles have the accessor to the virtual table *INDEX* as articulate a new concept: A Value Manager ## Clear & Complete Value Management scenario • In Type Erasure: std::any, especially std::function: • The type of the managed value is “forgotten” at compilation manage a value you don’t know its type • There are different types of value managers (for type erasure): • Local: concerns such as buffer size and alignment • Non-local: • Allocation? • How to hide

--debug ## Hadoop 3介绍 • Common • HDFS ■ 纠错码(Erasure Coding) ■ 多个Standby Namenode Datanode内部balance工具 云计算平台的支持 • YARN • MapReduce ## HDFS纠错码(Erasure Coding) ## • 一个简单的例子 |X|Y|X ⊕ Y| |---|---|---| |---|---|---| |0|0|0| |0|1|1| |1|0|1| |1|1|0| 1备份: 1,0 需要额外的2位 XOR编码: 1,0 需要额外的1位 ## HDFS纠错码(Erasure Coding) • Reed-Solomon (RS) 编码 Data $ G^{T} $ ![Image](/uploads/documents/b/9/8/5/b9850874ac9a0437b1b050eae497a4e6/p12_1 能带来30%的性能提升。 ## Hadoop 的未来 ## HDFS的未来 • 对象存储 - HDFS-7240 • 更高性能的Namenode：更高效的内存使用，锁的改进等 • Erasure Coding的完善 ## YARN的未来 • 更大规模的集群支持 • 更好的资源调度，隔离和多租户 • 支持更多的应用，包括long running的service ## 谢谢 Q&A

Emacs mode. As an alternative you can copy the following text to your .emacs file: (load-file (let ((coding-system-for-read 'utf-8)) (shell-command-to-string "agda-mode locate"))) It is also possible that uses --erased-cubical, but names defined using Cubical Agda can only be used if the option --erasure is used. In that case the names are treated as if they had been marked as erased, with an exception visible. ### 3.37 Run-time Irrelevance From version 2.6.1 Agda supports run-time irrelevance (or erasure) annotations. Values marked as erased are not present at run time, and consequently the type checker

Emacs mode. As an alternative you can copy the following text to your .emacs file: (load-file (let ((coding-system-for-read 'utf-8)) (shell-command-to-string "agda-mode locate"))) It is also possible that uses --erased-cubical, but names defined using Cubical Agda can only be used if the option --erasure is used. In that case the names are treated as if they had been marked as erased, with an exception visible. ### 3.37 Run-time Irrelevance From version 2.6.1 Agda supports run-time irrelevance (or erasure) annotations. Values marked as erased are not present at run time, and consequently the type checker

Emacs mode. As an alternative you can copy the following text to your .emacs file: (load-file (let ((coding-system-for-read 'utf-8)) (shell-command-to-string "agda-mode locate"))) It is also possible that uses --erased-cubical, but names defined using Cubical Agda can only be used if the option --erasure is used. In that case the names are treated as if they had been marked as erased, with an exception visible. ### 3.37 Run-time Irrelevance From version 2.6.1 Agda supports run-time irrelevance (or erasure) annotations. Values marked as erased are not present at run time, and consequently the type checker

Emacs mode. As an alternative you can copy the following text to your .emacs file: (load-file (let ((coding-system-for-read 'utf-8)) (shell-command-to-string "agda-mode locate"))) It is also possible that uses --erased-cubical, but names defined using Cubical Agda can only be used if the option --erasure is used. In that case the names are treated as if they had been marked as erased, with an exception visible. ### 3.37 Run-time Irrelevance From version 2.6.1 Agda supports run-time irrelevance (or erasure) annotations. Values marked as erased are not present at run time, and consequently the type checker

又能提供硬盘故障的容错能力。 镜像技术（Mirroring）又称为复制技术（Replication），可提供数据冗余性和高可用性；条带（Striping），可提供并行的数据吞吐能力；纠删码（Erasure Code），把数据切片并增加冗余编码而提供高可用性和高速读写能力。镜像、条带和纠删码是磁盘阵列技术经典的数据分发方式，这3种经典的磁盘技术可通过组合方式提供更加丰富的数据读写性能。 传统的磁 [crush-ruleset-name] [expected-num-objects] ceph osd pool create {poolname} {pg-num} {pgp-num} erasure [erasure-code-profile] [crush-ruleset-name] [expected_num_ 2）配置pool配额。 ceph osd pool set-quota {poolname} 是Ceph官方强烈建议不要在RBD块存储一类对读写操作频繁且性能要求也比较高的场景下使用缓冲池技术，只建议在RGW对象存储之类场景下使用该技术。 ### 11.3 纠删码原理 纠删码（Erasure Coding，EC）是一种编码容错技术，最早是在通信行业解决部分数据在传输中的损耗问题。其基本原理就是把传输的信号分段，加入一定的校验再让各段间发生相互关联，即使在传输过程中丢失部分信号，接收端仍然

released in Dec 2021 • Version 1.3.0 is in-progress ☐ Tons of new features and improvements ■ Erasure Coding Container Balancer ■ S3 Multi-Tenancy ■ S3 gRPC improvements ☐ 1000+ new commits since 1 streaming Efficient data path with rack awareness • Zero copy buffers - Simplified IO path for erasure coding • OM - operations per second - Concurrency improvements - Caching background updates - Reducing