## Compilation of Quantized Models in TVM Animesh Jain Amazon SageMaker Neo AWS AI ## Quantization Overview • Represent FP32 numbers with a lower-precision INT8 numbers • Integer number stands as ith-tensorrt.pdf ## Quantization in TVM ## • Quantization within TVM - Automatic Quantization • TVM stack ingests a FP32 graph and a small dataset • Finds suitable quantization scale • Produces a quantized QNN dialect  ## Quantization Appraoches in TVM Framework FP32 Graph ![Image](/uploads/documents/a/9/7/6/a97666043c6fb989

compression techniques. By ‘advanced’ we mean that these techniques are slightly more involved than quantization (as discussed in the second chapter). But that doesn’t mean they are harder to learn or implement particular clustering is a generalization of quantization. If you noticed, quantization ensures that any two weights that lie within the same quantization bin, are mapped to the same quantized weight value value. That is an implicit form for weight sharing. However, quantization falls behind in case the data that we are quantizing is not uniformly distributed, i.e. the data is more likely to take values

PagedAttention - Continuous batching of incoming requests - Fast model execution with CUDA/HIP graph - Quantization: GPTQ, AWQ, SqueezeLLM, FP8 KV Cache - Optimized CUDA kernels vLLM is flexible and easy to use model inferencing and serving on AMD GPUs with ROCm. At the moment AWQ quantization is not supported in ROCm, but SqueezeLLM quantization has been ported. Data types currently supported in ROCm are FP16 and MAX_NUM_BATCHED_TOKENS] [--max-num-seqs MAX_NUM_SEQS] [--max-logprobs MAX_LOGPROBS] [--disable-log-stats] [--quantization {awq,gptq,squeezellm,None}] [--enforce-eager] [--max-context-len-to-capture MAX_CONTEXT_LEN_TO_CAPTURE]

we introduce Quantization, a model compression technique that addresses both these issues. We'll start with a gentle introduction to the idea of compression. Details of quantization and its applications after. The quantization section delves into the implementation details using code samples. We finish with a hands-on project that will walk you through the process of applying quantization in practical next section we introduce Quantization, a popular compression technique which is also used in various fields of computer science in addition to deep learning. ## Quantization Before we jump to working

PagedAttention - Continuous batching of incoming requests - Fast model execution with CUDA/HIP graph - Quantization: GPTQ, AWQ, INT4, INT8, and FP8 - Optimized CUDA kernels, including integration with FlashAttention parse_args() main(args) ## 1.10.8 Lora With Quantization Inference Source https://github.com/vllm-project/vllm/blob/main/examples/lora_with_quantization_inference.py. ```python """ This This example shows how to use LoRA with different quantization techniques for offline inference. Requires HuggingFace credentials for access. """ import gc from typing import List, Optional

PagedAttention - Continuous batching of incoming requests - Fast model execution with CUDA/HIP graph - Quantization: GPTQ, AWQ, INT4, INT8, and FP8 - Optimized CUDA kernels, including integration with FlashAttention parse_args() main(args) ## 1.10.8 Lora With Quantization Inference Source https://github.com/vllm-project/vllm/blob/main/examples/lora_with_quantization_inference.py. ```python """ This This example shows how to use LoRA with different quantization techniques for offline inference. Requires HuggingFace credentials for access. """ import gc from typing import List, Optional

PagedAttention - Continuous batching of incoming requests - Fast model execution with CUDA/HIP graph - Quantization: GPTQ, AWQ, INT4, INT8, and FP8 - Optimized CUDA kernels, including integration with FlashAttention parse_args() main(args) ## 1.10.8 Lora With Quantization Inference Source https://github.com/vllm-project/vllm/blob/main/examples/lora_with_quantization_inference.py. ```python """ This This example shows how to use LoRA with different quantization techniques for offline inference. Requires HuggingFace credentials for access. """ import gc from typing import List, Optional

PagedAttention - Continuous batching of incoming requests - Fast model execution with CUDA/HIP graph - Quantization: GPTQ, AWQ, INT4, INT8, and FP8 - Optimized CUDA kernels, including integration with FlashAttention parse_args() main(args) ## 1.10.8 Lora With Quantization Inference Source https://github.com/vllm-project/vllm/blob/main/examples/lora_with_quantization_inference.py. ```python """ This This example shows how to use LoRA with different quantization techniques for offline inference. Requires HuggingFace credentials for access. """ import gc from typing import List, Optional

PagedAttention - Continuous batching of incoming requests - Fast model execution with CUDA/HIP graph - Quantization: GPTQ, AWQ, INT4, INT8, and FP8 - Optimized CUDA kernels, including integration with FlashAttention parse_args() main(args) ## 1.10.8 Lora With Quantization Inference Source https://github.com/vllm-project/vllm/blob/main/examples/lora_with_quantization_inference.py. ```python """ This This example shows how to use LoRA with different quantization techniques for offline inference. Requires HuggingFace credentials for access. """ import gc from typing import List, Optional

PagedAttention - Continuous batching of incoming requests - Fast model execution with CUDA/HIP graph - Quantization: GPTQ, AWQ, INT4, INT8, and FP8 - Optimized CUDA kernels, including integration with FlashAttention parse_args() main(args) ``` ## 1.10.8 Lora With Quantization Inference Source https://github.com/vllm-project/vllm/blob/main/examples/lora_with_quantization_inference.py. ```python """ This This example shows how to use LoRA with different quantization techniques for offline inference. Requires HuggingFace credentials for access. """ import gc ``` (continues on next page)