2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Presenter: Haichen Shen, Yao Wang Amazon SageMaker Neo, Deep Engine Science Dynamic Model in TVM AWS AI© 2019, Amazon Web Services while loop Limitation of TVM/graph runtime ● Cannot compile and run dynamic models© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Support dynamic model in TVM ● Support Any-dim dispatch for a (sub-)graph In collaboration with Jared Roesch, Zhi Chen, Wei Chen© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. “Any” in Relay typing Any: represent an unknown

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon/Intel Confidentia Presenter: Zhi Chen, Cody Yu Amazon SageMaker Neo, Deep Engine Science Bring Your Own Codegen to TVM TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Considering You... Design and manufacture a deep learning chip which achieves amazing performance on widely-used operators Suppression (NMS) is too new to be supported by your chip But NMS is supported by TVM!© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Let TVM Be the Compiler of Your Chip Your chip

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Animesh Jain Amazon SageMaker Neo Compilation of Quantized Models in TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates 𝑟𝑒𝑎𝑙_𝑣𝑎𝑙𝑢𝑒 = 𝑠𝑐𝑎𝑙𝑒 ∗ (𝑞𝑢𝑎𝑛𝑡𝑖𝑧𝑒𝑑_𝑣𝑎𝑙𝑢𝑒 − 𝑧𝑒𝑟𝑜_𝑝𝑜𝑖𝑛𝑡)© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Quantization in TVM • Quantization within TVM pre-quantized graph in TFLite or MxNet • Use high-level wrapper ops of QNN dialect© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. TVM Overview Framework Graph Mxnet TF …. parsers

23 10/22 4/25 800MM Big Six* USA Technology Company CapEx *Apple, NVIDIA, Microsoft, Alphabet, Amazon (AWS only), & Meta Platforms Source: Capital IQ (3/25), Morgan Stanley 2014 2024 CapEx, $B +63% one can make. The magic of watching AI do your work for you feels like the early days of email and web search – technologies that fundamentally changed our world. The better / faster / cheaper impacts the installed based of smartphones & tablets in 2020. Cloud & data center capex includes Google, Amazon, Microsoft, Meta, Alibaba, Apple, IBM, Oracle, Tencent, & Baidu for ten years ending 2022. ‘Tens

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Deploying GluonCV models using TVM© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Amazon Trademark Deploy GluonCV Models GluonCV Models MXNet Computational Graph Json Acyclic Graph Export As-is Optimize with TVM© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved reserved. Amazon Trademark Deploy GluonCV Models https://arxiv.org/pdf/1907.02154.pdf© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Trademark Overall Performance AWS

developer incorporates open source frameworks, uses standardized design patterns, and orchestrates services that are already available.  There are “cookbooks” available with templates for deploying systems adapt the system over time as the business changes versus buying an undefined stream of future services from a vendor who doesn’t know your business and doesn’t have financial incentives to supportyou book is meant to be, really. About the Author Mark Schwartz is an Enterprise Strategist at Amazon Web Services and the author of The Art of Business Value and A Seat at the Table: IT Leadership in the

book is meant to be, really. About the Author Mark Schwartz is an Enterprise Strategist at Amazon Web Services and the author of The Art of Business Value and A Seat at the Table: IT Leadership in the

requires some sort of management.About the Author Mark Schwartz is an Enterprise Strategist at Amazon Web Services and the author of The Art of Business Value and A Seat at the Table: IT Leadership in the

available throughout the organization.” They are self-healing ii. Netflix – April 21, 2011; entire Amazon AWS US-EAST availability zone went down; Netflix was unaffected. Since 2008 they were building and defining failure modes and then testing to ensure they operate as expected iv. Great Amazon Reboot of 2014 – 10% of Amazon EC2 servers had to reboot for Xen emergency security patch. At Netflix, zero downtime responsible for unsupported platforms ii. Systematically review production infrastructure and services for items that are causing disproportionate amounts of failure and unplanned work; plan for elimination

TO START 47 7. Ch. 5 Selecting Which Value Stream to Start With 51 a. GREENFIELD vs BROWNFIELD SERVICES i. DevOps is not just for Greenfield ii. Important Predictor – Is the application architected TECHNICAL DEBT i. Don’t just make the interest payments, pay down the principle ii. If not managed, services become so fragile, delivery grinds to a halt g. INCREASE THE VISIBILITY OF WORK i. Keep the current independently, quickly delivering value ii. Aligned with full life-cycle of the application iii. Used by Amazon and Netflix e. MAKING FUNCTIONAL ORIENTATION WORK i. Requires everyone in the value stream views