Apache Ozone Erasure Coding(EC) The Modern Big Data Object Store with More Than 50% Storage Space Savings Uma Maheswara Rao Gangumalla Sr. Engineering Manager, Cloudera Inc Stephen O’Donnell Sr. Staff by using OFS protocol Ozone Client Apache Ozone Architecture Quick Overview of Non EC Flow Erasure Coding Requirements ❏ Phase - I ❏ Enable EC at Cluster/Bucket Level ❏ Should be able to Write files

std::spanstream  std::byteswap()  std::to_underlying()  Associative Containers Heterogeneous Erasure  Removed Features  Garbage Collection SupportC++23 Core Language6 Agenda  C++23 Core Language std::spanstream  std::byteswap()  std::to_underlying()  Associative Containers Heterogeneous Erasure  Removed Features  Garbage Collection Support7 Explicit Object Parameters  Instead of implicit std::spanstream  std::byteswap()  std::to_underlying()  Associative Containers Heterogeneous Erasure  Removed Features  Garbage Collection Support13 if consteval  Syntax: if consteval { /* A

that uses --erased-cubical, but names defined using Cubical Agda can only be used if the option --erasure is used. In that case the names are treated as if they had been marked as erased, with an exception be visible. 3.37 Run-time Irrelevance From version 2.6.1 Agda supports run-time irrelevance (or erasure) annotations. Values marked as erased are not present at run time, and consequently the type checker declared erased using the @0 or @erased annotation. (These annotations may only be used if the option --erasure is active.) For example, the following definition of vectors guarantees that the length argument

that uses --erased-cubical, but names defined using Cubical Agda can only be used if the option --erasure is used. In that case the names are treated as if they had been marked as erased, with an exception be visible. 3.37 Run-time Irrelevance From version 2.6.1 Agda supports run-time irrelevance (or erasure) annotations. Values marked as erased are not present at run time, and consequently the type checker declared erased using the @0 or @erased annotation. (These annotations may only be used if the option --erasure is active.) For example, the following definition of vectors guarantees that the length argument

that uses --erased-cubical, but names defined using Cubical Agda can only be used if the option --erasure is used. In that case the names are treated as if they had been marked as erased, with an exception be visible. 3.37 Run-time Irrelevance From version 2.6.1 Agda supports run-time irrelevance (or erasure) annotations. Values marked as erased are not present at run time, and consequently the type checker declared erased using the @0 or @erased annotation. (These annotations may only be used if the option --erasure is active.) For example, the following definition of vectors guarantees that the length argument

that uses --erased-cubical, but names defined using Cubical Agda can only be used if the option --erasure is used. In that case the names are treated as if they had been marked as erased, with an exception be visible. 3.37 Run-time Irrelevance From version 2.6.1 Agda supports run-time irrelevance (or erasure) annotations. Values marked as erased are not present at run time, and consequently the type checker declared erased using the @0 or @erased annotation. (These annotations may only be used if the option --erasure is active.) For example, the following definition of vectors guarantees that the length argument

to inject different functionality • Linking • Inheritance/virtual functions • Templates • Type erasure 29 Bloomberg Dependency Injection BasicsCall any thing satisfying a function signature – via yield; } private : CalculateYield YieldCalculator_; }; 31 Bloomberg Dependency Injection via type erasure Injection Dependency Type Testing Dependency captureTEST(Processor, test_yield) { auto y_calculator processor.process(rdata); EXPECT_EQ(yield.realised, 1); } 32 Bloomberg Dependency Injection via type erasure Injection Dependency Testing VerificationCall any function satisfying a function signature –

contains(id()); } };What about runtime polymorphism? • Programmers make the virtual tables • Zoo type-erasure already generates virtual tables • The handles have the accessor to the virtual table *INDEX* as articulate a new concept: A Value Manager 35Clear & Complete Value Management scenario • In Type Erasure: std::any, especially std::function: • The type of the managed value is “forgotten” at compilation manage a value you don’t know its type • There are different types of value managers (for type erasure): • Local: concerns such as buffer size and alignment • Non-local: • Allocation? • How to

HDFS  纠错码(Erasure Coding)  多个Standby Namenode  Datanode内部balance工具  云计算平台的支持 • YARN • MapReduce HDFS纠错码(Erasure Coding) • 一个简单的例子 1备份: 1,0 需要额外的2位 XOR编码: 1,0 需要额外的1位 HDFS纠错码(Erasure Coding) 集型的task能 带来30%的性能提升。 Hadoop 的未来 HDFS的未来 • 对象存储 - HDFS-7240 • 更高性能的Namenode：更高效的内存使用，锁的改进等 • Erasure Coding的完善 YARN的未来 • 更大规模的集群支持 • 更好的资源调度，隔离和多租户 • 支持更多的应用，包括long running的service 谢谢 Q&A

should be visible. Run-time Irrelevance From version 2.6.1 Agda supports run-time irrelevance (or erasure) annotations. Values marked as erased are not present at run time, and consequently the type checker Marking it erased explictly, however, ensures that it is erased without relying on the analysis. Erasure annotations can also appear in function arguments (both first-order and higher-order). For instance optimization enabled by the erasure annotation is compiling λ {n} → f {suc n} to simply f, removing a terrible space leak from the program. Compare to the result of compiling without erasure: foldl f z xs =