## +24 ## Cross-Platform Floating-Point Determinism Out of the Box ## SHERRY IGNATCHENKO ET AL ## Authors  ## All the 0c9de70c3d925/p2_4.jpg) Guy Davidson gd@6it.dev Advising on C++ standard with regards to floating-point, idea about intrinsics, reviews  Mykhailo Borovyk mbo@6it.dev Implementing fixed-point math with floating-point fallback  Vladyslav

## +24 ## A New Dragon in the Den: Fast Conversion From Floating-Point Numbers ## CASSIO NERI ## 2024 September 15 - 20 WARNING: This PDF poorly reproduces the slides shown at the talk. You are strongly seven-headed beast. It's not rocket science. ## A New Dragon in the Den ## Fast conversion from floating-point numbers ## Cassio Neri (Independent Researcher) CppCon 2024 - Aurora # Statistical mechanics Decimal fixed-point representation ## 1234 ## Decimal floating-point representation ## exponent ### 10 ¹ x 2.34 mantissa ## Decimal floating-point representation exponent ### 10 ¹ x 2.34 mantissa 0

assignment versus mutation 4.3 Stylistic Conventions 5 Integers and Floating-Point Numbers 5.1 Integers 5.2 Floating-Point Numbers 5.3 Arbitrary Precision Arithmetic 5.4 Numeric Literal Coefficients stylistic conventions, see the Style Guide. ## Chapter 5 ## I ntegers and Floating-Point Numbers Integers and floating-point values are the basic building blocks of arithmetic and computation. Built-in representations of integers and floating-point numbers as immediate values in code are known as numeric literals. For example, 1 is an integer literal, while 1.0 is a floating-point literal; their binary in-memory

both be mapped to 0 in the quantized domain. Keeping all that in mind, it is easy to see that floating-point $ x_{min} $ should map to 0, and $ x_{max} $ should map to $ 2^{b}-1 $ . How do we map the precision domain. Visually inspecting figure 2-4, can you work out the formula for mapping a given floating-point value (x) to a quantized value ( $ x_{q} $ ). Assume that you are given values of $ x_{min} [Image](/uploads/documents/5/f/8/a/5f8af6d4905642e38ad2d0bbe54cd375/p6_1.jpg) Figure 2-4: Quantizing floating-point continuous values to discrete unsigned values. The continuous values range from $ x_{min} $

assignment versus mutation 12 4.3 Stylistic Conventions 14 5 Integers and Floating-Point Numbers 15 5.1 Integers 16 5.2 Floating-Point Numbers 20 5.3 Arbitrary Precision Arithmetic 25 5.4 Numeric Literal Coefficients information about stylistic conventions, see the Style Guide. Chapter 5 Integers and Floating-Point Numbers Integers and floating-point values are the basic building blocks of arithmetic and computation. Built-in representations of integers and floating-point numbers as immediate values in code are known as numeric literals. For example, 1 is an integer literal, while 1.0 is a floating-point literal; their binary in-memory

major scenarios. Determinism. Deterministic training is highly beneficial for debugging hardware or software issues. Moreover, when training exhibits anomalies such as loss spikes, determinism enables researchers researchers to more easily pinpoint numerical causes and further refine the model design. Non-determinism in training typically stems from non-deterministic accumulation order, often due to the use of atomic atomicAdd to accumulate gradients for the KV tokens. This introduces non-determinism due to the non-associativity of floating-point addition. To address this problem, we allocate separate accumulation buffers

shapeIndex); [[nodiscard]] const CircleCollider& circle(ShapeIndex shapeIndex) const; ## Floating-point number determinism From Yossi Kreinin - Consistency: how to defeat the purpose of IEEE floating point

such as Rust and Julia have begun to follow the naming scheme described in IEEE Standard for Floating-Point Arithmetic (IEEE 754). This describes single and double precision numbers as Float32 and Float64; introduces a number of other effects a program may have: I/O; Exceptions; Random Numbers; and Non-determinism, giving examples for each, and an extended example combining several effects in one complete program effect described in Effects Summary (page 95). #### 3.3.4 Non-determinism The listing below gives the definition of the non-determinism effect, which allows a program to choose a value non-deterministically

such as Rust and Julia have begun to follow the naming scheme described in IEEE Standard for Floating-Point Arithmetic (IEEE 754). This describes single and double precision numbers as Float32 and Float64; introduces a number of other effects a program may have: I/O; Exceptions; Random Numbers; and Non-determinism, giving examples for each, and an extended example combining several effects in one complete program SYSTEM effect described in Effects Summary (page 132). ## Non-determinism The listing below gives the definition of the non-determinism effect, which allows a program to choose a value non-deterministically

such as Rust and Julia have begun to follow the naming scheme described in IEEE Standard for Floating-Point Arithmetic (IEEE 754). This describes single and double precision numbers as Float32 and Float64; introduces a number of other effects a program may have: I/O; Exceptions; Random Numbers; and Non-determinism, giving examples for each, and an extended example combining several effects in one complete program SYSTEM effect described in Effects Summary (page 136). 4.3.4 Non-determinism The listing below gives the definition of the non-determinism effect, which allows a program to choose a value non-deterministically