## +24 ## Boosting Software Efficiency: A Case Study of 100% Performance Improvement in an Embedded C++ System ## GILI KAMMA ## 20 24 September 15 - 20 ☐ The talk today is about software development

## HUAWEI CLOUD Microservice Tool Improves Development Efficiency Department: Application Platform Service Author: Wang Qijun Date: 2019-09-20 ## Contents 1. Tool for Splitting Monolithic Applications ss-level| |Overall availability|Low|High| |Continuous evolution|Difficult|Easy| |Communication efficiency|Low|High| |Technology stack selection|Restricted|Flexible| |Scalable|Restricted|Flexible| |Reusability|Low|High| increases. ## Tool for Splitting Monolithic Applications into Microservices Improves Development Efficiency  ✓ Distributed

## +24 ## Balancing Efficiency and Flexibility: Cost of Abstractions in Embedded Systems MARCELL JUHASZ  zühlke ## whoami

rapid growth. We will establish our motivation behind seeking efficiency in deep learning models. We will also introduce core areas of efficiency techniques (compression techniques, learning techniques, automation Our hope is that even if you just read this chapter, you would be able to appreciate why we need efficiency in deep learning models today, how to think about it in terms of metrics that you care about, and models is rate-limited by their efficiency. While efficiency can be an overloaded term, let us investigate two primary aspects: ## Training Efficiency Training Efficiency involves benchmarking the model

tables 59 Python Module Index 61 Index 63 ## LLM vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management including parallel sampling, beam search, and more - Tensor parallelism support for distributed inference - Streaming outputs - OpenAI-compatible API server - Support NVIDIA GPUs and AMD GPUs - (Experimental) PagedAttention) - vLLM paper (SOSP 2023) - How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al. ## DOCUMENTATION ## 1.1 Installation vLLM is

153 Python Module Index 155 Index 157 ## LLM vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management including parallel sampling, beam search, and more - Tensor parallelism support for distributed inference - Streaming outputs - OpenAI-compatible API server - Support NVIDIA GPUs and AMD GPUs - (Experimental) PagedAttention) - vLLM paper (SOSP 2023) - How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al. - vLLM Meetups. ## DOCUMENTATION ## 1.1 Installation

157 Python Module Index 159 Index 161 ## LLM vLLM is a fast and easy-to-use library for LLM inference and serving. vLLM is fast with: - State-of-the-art serving throughput - Efficient management sampling, beam search, and more - Tensor parallelism and pipeline parallelism support for distributed inference - Streaming outputs - OpenAI-compatible API server - Support NVIDIA GPUs and AMD GPUs - (Experimental) PagedAttention) - vLLM paper (SOSP 2023) - How continuous batching enables 23x throughput in LLM inference while reducing p50 latency by Cade Daniel et al. - vLLM Meetups. ## DOCUMENTATION ## 1.1 Installation

strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE costs and inference efficiency of DeepSeek 67B (Dense) and DeepSeek-V2. ## Contents 1 Introduction 4 2 Architecture 6 2.1 Multi-Head Latent Attention: Boosting Inference Efficiency 6 2.1.1 Preliminaries:

Mixed-Precision Training/Inference PAI (Platform of AI) Alibaba Cloud Intelligence ## Outline • TensorCore AutoCodeGen in TVM • FP16 Mixed-Precision Training on PAI • INT8 Inference on PAI-Blade ## TensorCore PAI-TF  ## I NT8 Inference on PAI-Blade ## PAI-Blade ![Image](/uploads/documents/e/6/f/1/e6f1347cd2b546040bb0c9ad6650060d/p22_2

Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency; (2) Manifold- Constrained Hyper-Connections (mHC) that enhance conventional residual connections; scenarios. In the one-million-token context setting, DeepSeek V4-Pro requires only 27% of single-token inference FLOPs and 10% of KV cache compared with DeepSeek-V3.2. This enables us to routinely support one-million-token Figure 1 | Left: benchmark performance of DeepSeek-V4-Pro-Max and its counterparts. Right: inference FLOPs and KV cache size of DeepSeek-V4 series and DeepSeek-V3.2. Contents 1 Introduction 4 2 Architecture