ate units *， Jamp-/evre/matrix operations exposed in the CUDA WUMAA4 4AP1 FP16 or FP32 FP16 or FP32 Background 和 CUDA C half a[16x8] wmma::fragmentCUDA 本 codegen IR | TensorCore Intrinsics "。Kind of Auto Tensorization 下 CUDA CodeGen *。IR passes to automatically transform sub-tree to TensorCore Intrinsics Pattern Matching

Overview Relay IR Graph Annotation with Your Annotator Graph Partitioning Your Codegen LLVM, CUDA, Metal, VTA Serialized Subgraph Library Relay Runtime (VM, Graph Runtime, Interpreter) Your Overview Relay IR Graph Annotation with Your Annotator Graph Partitioning Your Codegen LLVM, CUDA, Metal, VTA Serialized Subgraph Library Relay Runtime (VM, Graph Runtime, Interpreter) Your Overview Relay IR Graph Annotation with Your Annotator Graph Partitioning Your Codegen LLVM, CUDA, Metal, VTA Serialized Subgraph Library Relay Runtime (VM, Graph Runtime, Interpreter) Your

learning.TVM Stack High-Level Differentiable IR Tensor Expression and Optimization Search Space LLVM, CUDA, Metal VTA Edge FPGA Cloud FPGA ASIC Optimization AutoTVM Device FleetExisting Deep Learning related workloads Credit: Siyuan FengWhere are we goingUnified Runtime For Heterogeneous Devices CUDA Driver NPU Driver Device Drivers External Runtimes NPUModule CUDAModule TFModule tvm::runtime::Module

端机器学习服务的云平台。 这些工具和库专门为加速机器学习模型的训练和推理而设计，通常利 用 GPU 或 TPU 等硬件。这类工具可以显著提高训练和推理的速度， 使得处理大规模数据集和复杂模型变得可行。NVIDIA CUDA 和 Google Cloud TPU 均是此类工具。 这类工具通常由开源社区支持和维护，提供了灵活、可扩展的工具和 库来构建和训练大型机器学习模型，如 TensorFlow 和 PyTorch 昆仑芯 XPU 从AI落地的实际需求出发，按 照复杂前沿的人工智能场景需求开展迭代，致力为开发者提供通用、易用、高性能的算力来源。 DCU 系列产品以 GPGPU 架构为基础，兼容通用的“类 CUDA”环境以及国际主流商业计 算软件和人工智能软件，可广泛应用于大数据处理、人工智能、商业计算等应用领域。 通用 GPU 高端芯片及超级算力系统提供商。拥有云边协同、训推组合的完整通用算力 系统

of opportunity in PyTorch - Graph optimization - Existing fusion infrastructure fairly limited (CUDA-only, injective-only) - Kernel synthesis - Dynamic shapes, stride specialization - Impedance mismatch

Primitives & Optimizations Symbols NNVM & Param Frontends Operators Algorithm &Schedule CUDA TOPI Backends Machine Learning Automated Optimizer Schedule explorer Cost model Mali TOPI ROCM

in TVM Tensor IR .. More targets AutoTVM – Tuning the kernels Optimized Binary Codegen – LLVM, Cuda, C, … Framework Parsers Graph level optimizations Tensor-level optimizations Machine code generation©

computation of shared experts with the expert parallel all-to-all communication. We also customize faster CUDA kernels for communications, routing algorithms, and fused 12 1K 12K 24K 35K 47K 58K 70K 81K 93K