[Image](/uploads/documents/a/b/7/b/ab7b254a5c187d70765c98d89cffb40d/p6_1.jpg) ### 1 \.Transformer介绍 ## Attention注意力机制 在介绍什么是注意力机制之前，先让大家看一张图片。当大家看到下面图片，会首先看到什么内容？当过载信息映入眼帘时，我们的大脑会把注意力放在主要的信息上，这就是大脑的注意力机制。 ! [Image](/uploads/documents/a/b/7/b/ab7b254a5c187d70765c98d89cffb40d/p7_1.jpg) ### 1 \.Transformer介绍 ## 每个词的Attention计算 ## 每个词的Q会跟整个序列中每一个K计算得分，然后基于得分再分配特征 Q: query，要去查询的 K: key，等着被查的 V: value，实际的特征信息 ![Image Transformer介绍 ## Attention的优点 1. 参数少：相比于 CNN、RNN，其复杂度更小，参数也更少。所以对算力的要求也就更小。 2.速度快：Attention 解决了 RNN 及其变体模型不能并行计算的问题。Attention 机制每一步计算不依赖于上一步的计算结果，因此可以和 CNN 一样并行处理。 3. 效果好：在Attention 机制引入之前，有一个问题大家一直

a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing Introduction 4 2 Architecture 6 2.1 Multi-Head Latent Attention: Boosting Inference Efficiency 6 2.1.1 Preliminaries: Standard Multi-Head Attention 6 2.1.2 Low-Rank Key-Value Joint Compression ... 29 B.2 Performance Evaluation ..... 30 C Full Formulas of MLA ..... 31 D Ablation of Attention Mechanisms ..... 31 D.1 Ablation of MHA, GQA, and MQA ..... 31 D.2 Comparison Between MLA and

upgrades in architecture and optimization: (1) a hybrid attention architecture that combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency; (2) Manifold- Manifold-Constrained Hyper-Connections 7 2.3 Hybrid Attention with CSA and HCA 9 2.3.1 Compressed Sparse Attention 9 2.3.2 Heavily Compressed Attention 11 2.3.3 Other Details 12 2.3.4 Efficiency Discussion Cost-Effective and Memory-Efficient Implementation of mHC 21 3.5.3 Contextual Parallelism for Long-Context Attention 21 3.5.4 Extended Automatic Differentiation for Flexible Activation Checkpointing 21 3.6 Inference

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model tested version ## 1. Install flash attention for ROCm Install ROCm's flash attention (v2.0.4) following the instructions from ROCmSoftwarePlatform/flash attention Note: - If you are using rocm5.7 ROCm flash attention directly. - If you fail to install ROCmSoftwarePlatform/flash-attention, try cloning from the commit 6fd2f8e572805681cd67ef8596c7e2ce521ed3c6. - ROCm's Flash-attention-2 (v2.0.4)

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model BUILD_FA: specifies whether to build CK flash-attention. The default is 1. For Radeon RX 7900 series (gfx1100), this should be set to 0 before flash-attention supports this target. - FX_GFX_ARCHS: specifies flash-attention, for example, gfx90a;gfx942 for MI200 and MI300. The default is gfx90a;gfx942 - FA_BRANCH: specifies the branch used to build the CK flash-attention in ROCm's flash-attention repo.

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model BUILD_FA: specifies whether to build CK flash-attention. The default is 1. For Radeon RX 7900 series (gfx1100), this should be set to 0 before flash-attention supports this target. - FX_GFX_ARCHS: specifies flash-attention, for example, gfx90a;gfx942 for MI200 and MI300. The default is gfx90a;gfx942 - FA_BRANCH: specifies the branch used to build the CK flash-attention in ROCm's flash-attention repo.

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model tested version ## 1. Install flash attention for ROCm Install ROCm's flash attention (v2.0.4) following the instructions from ROCmSoftwarePlatform/flash attention ## Note: - If you are using ROCm flash attention directly. - If you fail to install ROCmSoftwarePlatform/flash-attention, try cloning from the commit 6fd2f8e572805681cd67ef8596c7e2ce521ed3c6. - ROCm's Flash-attention-2 (v2.0.4)

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model BUILD_FA: specifies whether to build CK flash-attention. The default is 1. For Radeon RX 7900 series (gfx1100), this should be set to 0 before flash-attention supports this target. - FX_GFX_ARCHS: specifies flash-attention, for example, gfx90a;gfx942 for MI200 and MI300. The default is gfx90a;gfx942 - FA_BRANCH: specifies the branch used to build the CK flash-attention in ROCm's flash-attention repo.

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model BUILD_FA: specifies whether to build CK flash-attention. The default is 1. For Radeon RX 7900 series (gfx1100), this should be set to 0 before flash-attention supports this target. - FX_GFX_ARCHS: specifies flash-attention, for example, gfx90a;gfx942 for MI200 and MI300. The default is gfx90a;gfx942 - FA_BRANCH: specifies the branch used to build the CK flash-attention in ROCm's flash-attention repo.

serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management of attention key and value memory with PagedAttention - Continuous batching of incoming requests - Fast model BUILD_FA: specifies whether to build CK flash-attention. The default is 1. For Radeon RX 7900 series (gfx1100), this should be set to 0 before flash-attention supports this target. - FX_GFX_ARCHS: specifies flash-attention, for example, gfx90a;gfx942 for MI200 and MI300. The default is gfx90a;gfx942 - FA_BRANCH: specifies the branch used to build the CK flash-attention in ROCm's flash-attention repo.