[和| Alibaba AL.Labs 阿里巴巴人工智能实验室 AiILabs & TVM PART 1 : ARM32 CPU CONTENT PART 2 : HIFI4 DSP PART 3 : _ PowervVR GPU [和| Alibaba AL.Labs 阿里巴巴人工智能实验室 ARM 32 CPU Resolution Quantization Orize Orize Kernel ALIOS TVM Alibaba Al.Labs 阿里巴巴人工智能实验室 7= 590一I) rm 一 下Er (mm) =肪+2mxaM0 [5 (全-2)+o| current plan 1 = int16 * int16 erflow-aware int16 = int8 xint8 ent pl 1=int8 int8 * int8 int32 int32 = int16 1 + int16 x int8 Alibaba Al.Labs 阿里巴巴人工智能实验室 CPU : MTK8167S (ARM32 A35 1.5GHz) Model : MobileNetV2_ 1.0_ 224 400 336 350 3丈 300 250

10/22 4/25 800MM Big Six* USA Technology Company CapEx *Apple, NVIDIA, Microsoft, Alphabet, Amazon (AWS only), & Meta Platforms Source: Capital IQ (3/25), Morgan Stanley 2014 2024 CapEx, $B +63% the installed based of smartphones & tablets in 2020. Cloud & data center capex includes Google, Amazon, Microsoft, Meta, Alibaba, Apple, IBM, Oracle, Tencent, & Baidu for ten years ending 2022. ‘Tens the Future of Music’ (5/2/25); Spotify earnings releases; eMarketer, ‘Spotify dominates Apple and Amazon in digital audio’ (4/25) AI-Powered Audio Translation – 5/25, per Spotify Imagine if you’re a creator

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon/Intel Confidentia Presenter: Zhi Chen, Cody Yu Amazon SageMaker Neo, Deep Engine Science Bring Your Own Codegen to TVM TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Considering You... Design and manufacture a deep learning chip which achieves amazing performance on widely-used operators Maximum Suppression (NMS) is too new to be supported by your chip But NMS is supported by TVM!© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Let TVM Be the Compiler of Your Chip

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Deploying GluonCV models using TVM© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Amazon Trademark Deploy GluonCV Models GluonCV Models MXNet Computational Graph Json Acyclic Graph Export As-is Optimize with TVM© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved reserved. Amazon Trademark Deploy GluonCV Models https://arxiv.org/pdf/1907.02154.pdf© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Overall Performance AWS DeepLens

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Presenter: Haichen Shen, Yao Wang Amazon SageMaker Neo, Deep Engine Science Dynamic Model in TVM AWS AI© 2019, Amazon Web Services within a while loop Limitation of TVM/graph runtime ● Cannot compile and run dynamic models© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Support dynamic model in TVM ● Support ○ Graph dispatch for a (sub-)graph In collaboration with Jared Roesch, Zhi Chen, Wei Chen© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. “Any” in Relay typing Any: represent

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Animesh Jain Amazon SageMaker Neo Compilation of Quantized Models in TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates 𝑟𝑒𝑎𝑙_𝑣𝑎𝑙𝑢𝑒 = 𝑠𝑐𝑎𝑙𝑒 ∗ (𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒𝑑_𝑣𝑎𝑙𝑢𝑒 − 𝑧𝑒𝑟𝑜_𝑝𝑜𝑖𝑛𝑡)© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Quantization in TVM • Quantization within ingests a pre-quantized graph in TFLite or MxNet • Use high-level wrapper ops of QNN dialect© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TVM Overview Framework Graph Mxnet

azure.com 671B（全量模型） 需注册微软账户并创建订阅，免费部署，支持参数调节。 亚马逊AWS https://aws.amazon.com/c n/blogs/aws/deepseek-r1- models-now-available-on- aws 671B（全量模型） 需注册AWS账户，填写付款方式，免费部署。 Cerebras https://cerebras.ai 70B

Primitive Tensor operators such as Conv2D eg. cuDNN Offload to heavily optimized DNN operator library FrameworksLimitations of Existing Approach cuDNN Frameworks New operator introduced by SaveToBinary/LoadFromBinary Runtime Module Interface SubclassesUnified Runtime Benefit mod.export_library("mylib.so") Unified library packaging Free API (Py/Java/Go) lib = tvm.module.load("mylib.so") func = lib["npufunction0"] conferenceCommunityOpen Source Community Open source: ~280 contributors from UW, Berkeley, Cornell, UCLA, Amazon, Huawei, NTT, Facebook, Microsoft, Qualcomm, Alibaba, Intel, … Incubated as Apache TVM recently

to TVM o_uTVM: support for microcontrollers in TVM o_ Virtual Machine and dynamic NNs support (w/ AWS folks) o_ Improved NLP support, with focus on transformers QQ octoML Core Infrastructure Refactors currently implemented using copy, 10 Virtual Machine e Many improvements from contributors at UW, AWS, and OctoML. e Initial implementation is quickly moving towards production quality. o _VM compiler Apache(incubating) community members. e ASF Mentors and PMC members who make this awesome project Possiblel ee AWS for hosting the first Bay Area meetup QQ octoML 14 Annual TVM Conference 2019 Organized and participated