23 10/22 4/25 800MM Big Six* USA Technology Company CapEx *Apple, NVIDIA, Microsoft, Alphabet, Amazon (AWS only), & Meta Platforms Source: Capital IQ (3/25), Morgan Stanley 2014 2024 CapEx, $B +63% the installed based of smartphones & tablets in 2020. Cloud & data center capex includes Google, Amazon, Microsoft, Meta, Alibaba, Apple, IBM, Oracle, Tencent, & Baidu for ten years ending 2022. ‘Tens the Future of Music’ (5/2/25); Spotify earnings releases; eMarketer, ‘Spotify dominates Apple and Amazon in digital audio’ (4/25) AI-Powered Audio Translation – 5/25, per Spotify Imagine if you’re a creator

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Presenter: Haichen Shen, Yao Wang Amazon SageMaker Neo, Deep Engine Science Dynamic Model in TVM AWS AI© 2019, Amazon Web Services within a while loop Limitation of TVM/graph runtime ● Cannot compile and run dynamic models© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Support dynamic model in TVM ● Support ○ Graph dispatch for a (sub-)graph In collaboration with Jared Roesch, Zhi Chen, Wei Chen© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. “Any” in Relay typing Any: represent

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon/Intel Confidentia Presenter: Zhi Chen, Cody Yu Amazon SageMaker Neo, Deep Engine Science Bring Your Own Codegen to TVM TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Considering You... Design and manufacture a deep learning chip which achieves amazing performance on widely-used operators Maximum Suppression (NMS) is too new to be supported by your chip But NMS is supported by TVM!© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Let TVM Be the Compiler of Your Chip

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Animesh Jain Amazon SageMaker Neo Compilation of Quantized Models in TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates 𝑟𝑒𝑎𝑙_𝑣𝑎𝑙𝑢𝑒 = 𝑠𝑐𝑎𝑙𝑒 ∗ (𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒𝑑_𝑣𝑎𝑙𝑢𝑒 − 𝑧𝑒𝑟𝑜_𝑝𝑜𝑖𝑛𝑡)© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Quantization in TVM • Quantization within ingests a pre-quantized graph in TFLite or MxNet • Use high-level wrapper ops of QNN dialect© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TVM Overview Framework Graph Mxnet

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Deploying GluonCV models using TVM© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Amazon Trademark Deploy GluonCV Models GluonCV Models MXNet Computational Graph Json Acyclic Graph Export As-is Optimize with TVM© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved reserved. Amazon Trademark Deploy GluonCV Models https://arxiv.org/pdf/1907.02154.pdf© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Overall Performance AWS

available throughout the organization.” They are self-healing ii. Netflix – April 21, 2011; entire Amazon AWS US-EAST availability zone went down; Netflix was unaffected. Since 2008 they were building and defining failure modes and then testing to ensure they operate as expected iv. Great Amazon Reboot of 2014 – 10% of Amazon EC2 servers had to reboot for Xen emergency security patch. At Netflix, zero downtime

this work? (Theory) Flow: The Psychology of Optimal Experience by Mihaly Csikszentmihalyi [3] From Amazon.com Psychologist Mihaly Csikszentmihalyi's famous investigations of "optimal experience" have revealed present#slide=id.i0 [3] Flow: The Psychology of Optimal Experience, Mihaly Csikszentmihalyi, http://www.amazon.com/Flow-Psychology-Experience-Mihaly-Csikszentmihalyi/dp/0061339202/Walkthrough To Do Today Underestimated

been properly coordinated with and agreed to by proper stakeholders. Attributions [1] Amazon, http://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262509/[2] The Three Ways: The Principles

conferenceCommunityOpen Source Community Open source: ~280 contributors from UW, Berkeley, Cornell, UCLA, Amazon, Huawei, NTT, Facebook, Microsoft, Qualcomm, Alibaba, Intel, … Incubated as Apache TVM recently

High Frequency & High Compute Efficiency ˃ Supported on U200 – 3 Instances U250 – 4 Instances Amazon F1 ˃ ~1536 DSPs @ 700MHz Execution Controller Spill / Restore DMA Controller Weights DMA Controller