serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model tested version ## 1. Install flash attention for ROCm Install ROCm's flash attention (v2.0.4) following the instructions from ROCmSoftwarePlatform/flash attention Note: - If you are using rocm5.7 ROCm flash attention directly. - If you fail to install ROCmSoftwarePlatform/flash-attention, try cloning from the commit 6fd2f8e572805681cd67ef8596c7e2ce521ed3c6. - ROCm's Flash-attention-2 (v2.0.4)

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model BUILD_FA: specifies whether to build CK flash-attention. The default is 1. For Radeon RX 7900 series (gfx1100), this should be set to 0 before flash-attention supports this target. - FX_GFX_ARCHS: specifies flash-attention, for example, gfx90a;gfx942 for MI200 and MI300. The default is gfx90a;gfx942 - FA_BRANCH: specifies the branch used to build the CK flash-attention in ROCm's flash-attention repo.

upgrades in architecture and optimization: (1) a hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency; (2) Manifold- Manifold-Constrained Hyper-Connections 7 2.3 Hybrid Attention with CSA and HCA 9 2.3.1 Compressed Sparse Attention 9 2.3.2 Heavily Compressed Attention 11 2.3.3 Other Details 12 2.3.4 Efficiency Discussion Cost-Effective and Memory-Efficient Implementation of mHC 21 3.5.3 Contextual Parallelism for Long-Context Attention 21 3.5.4 Extended Automatic Differentiation for Flexible Activation Checkpointing 21 3.6 Inference

[Image](/uploads/documents/a/b/7/b/ab7b254a5c187d70765c98d89cffb40d/p6_1.jpg) ### 1 \.Transformer介绍 ## Attention注意力机制 在介绍什么是注意力机制之前，先让大家看一张图片。当大家看到下面图片，会首先看到什么内容？当过载信息映入眼帘时，我们的大脑会把注意力放在主要的信息上，这就是大脑的注意力机制。 ! [Image](/uploads/documents/a/b/7/b/ab7b254a5c187d70765c98d89cffb40d/p7_1.jpg) ### 1 \.Transformer介绍 ## 每个词的Attention计算 ## 每个词的Q会跟整个序列中每一个K计算得分，然后基于得分再分配特征 Q: query，要去查询的 K: key，等着被查的 V: value，实际的特征信息 ![Image Transformer介绍 ## Attention的优点 1. 参数少：相比于 CNN、RNN，其复杂度更小，参数也更少。所以对算力的要求也就更小。 2.速度快：Attention 解决了 RNN 及其变体模型不能并行计算的问题。Attention 机制每一步计算不依赖于上一步的计算结果，因此可以和 CNN 一样并行处理。 3. 效果好：在Attention 机制引入之前，有一个问题大家一直

In the first chapter, we briefly introduced architectures like depthwise separable convolution, attention mechanism and the hashing trick. In this chapter, we will deepdive into their architectures and architectures, and how they help us outperform baseline methods. Another example in this domain is the attention mechanism, which forms the backbone of the state of the art NLP model architectures such as the showing great promise in computer vision applications as well! ## Learn Long-Term Dependencies Using Attention Imagine yourself in your favorite buffet restaurant. A variety of food items from savory to sweet

length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Introduction 4 2 Architecture 6 2.1 Multi-Head Latent Attention: Boosting Inference Efficiency 6 2.1.1 Preliminaries: Standard Multi-Head Attention 6 2.1.2 Low-Rank Key-Value Joint Compression 7 2 ... 29 B.2 Performance Evaluation ..... 30 C Full Formulas of MLA ..... 31 D Ablation of Attention Mechanisms ..... 31 D.1 Ablation of MHA, GQA, and MQA ..... 31 D.2 Comparison Between MLA and

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model BUILD_FA: specifies whether to build CK flash-attention. The default is 1. For Radeon RX 7900 series (gfx1100), this should be set to 0 before flash-attention supports this target. - FX_GFX_ARCHS: specifies flash-attention, for example, gfx90a;gfx942 for MI200 and MI300. The default is gfx90a;gfx942 - FA_BRANCH: specifies the branch used to build the CK flash-attention in ROCm's flash-attention repo.

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model tested version ## 1. Install flash attention for ROCm Install ROCm's flash attention (v2.0.4) following the instructions from ROCmSoftwarePlatform/flash attention ## Note: - If you are using ROCm flash attention directly. - If you fail to install ROCmSoftwarePlatform/flash-attention, try cloning from the commit 6fd2f8e572805681cd67ef8596c7e2ce521ed3c6. - ROCm's Flash-attention-2 (v2.0.4)

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model BUILD_FA: specifies whether to build CK flash-attention. The default is 1. For Radeon RX 7900 series (gfx1100), this should be set to 0 before flash-attention supports this target. - FX_GFX_ARCHS: specifies flash-attention, for example, gfx90a;gfx942 for MI200 and MI300. The default is gfx90a;gfx942 - FA_BRANCH: specifies the branch used to build the CK flash-attention in ROCm's flash-attention repo.

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model BUILD_FA: specifies whether to build CK flash-attention. The default is 1. For Radeon RX 7900 series (gfx1100), this should be set to 0 before flash-attention supports this target. - FX_GFX_ARCHS: specifies flash-attention, for example, gfx90a;gfx942 for MI200 and MI300. The default is gfx90a;gfx942 - FA_BRANCH: specifies the branch used to build the CK flash-attention in ROCm's flash-attention repo.