INDUSTRIES, INCORPORATED ACCELERATED VISUAL PERCEPTION LIANGFU CHEN 11/16/2019 DEPLOY VTA ON INTEL FPGA©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 2 Moore’s Law is Slowing Down MOTIVATION©2019 Terasic DE10-Nano DEPLOY VTA ON INTEL FPGA©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 5 Software - CMA Contiguous Memory Allocation – Linux Kernel DEPLOY VTA ON INTEL FPGA https://pynq.readthedocs INCORPORATED 6 Software - CMA Contiguous Memory Allocation – Linux Kernel Module DEPLOY VTA ON INTEL FPGA Setup Environment Variables Navigate to 3rdparty/cma and build kernel module Copy kernel module

1/66Bridging the Gap: Writing Portable Programs for CPU and GPU using CUDA Thomas Mejstrik Sebastian Woblistin 2/66Content 1 Motivation Audience etc.. Cuda crash course Quiz time 2 Patterns Oldschool Motivation Patterns The dark path Cuda proposal Thank you Why write programs for CPU and GPU Difference CPU/GPU Algorithms are designed differently Latency/Throughput Memory bandwidth Number of cores Motivation Patterns The dark path Cuda proposal Thank you Why write programs for CPU and GPU Difference CPU/GPU Why it makes sense? Library/Framework developers Embarrassingly parallel algorithms User

http://wongmichael.com/about ● C++11 book in Chinese: https://www.amazon.cn/dp/B00ETOV2OQ We build GPU compilers for some of the most powerful supercomputers in the world 34 Nevin “:-)” Liber nliber@anl Attribution 4.0 International License SYCL Single Source C++ Parallel Programming GPU FPGA DSP Custom Hardware GPU CPU CPU CPU Standard C++ Application Code C++ Libraries ML Frameworks give better performance on complex apps and libs than hand-coding AI/Tensor HW GPU FPGA DSP Custom Hardware GPU CPU CPU CPU AI/Tensor HW Other BackendsSYCL 2020 is here! Open Standard for

performance claims, visit www.intel.com/PerformanceIndex or scan the QR code: © Intel Corporation. Intel, the Intel logo, and other Intel marks are trademarks of Intel Corporation or its subsidiaries about future Intel products. - I work in Intel’s research labs. Work described here will involve experimental prototypes and early research.Problem: writing parallel programs is hard - Multi-GPU, multi-CPU / execution necessary. CPU NIC GPU GPU GPU GPU Xe LinkMulti-GPU Systems - NUMA regions: - 4+ GPUs - 2+ CPUs CPU NIC GPU GPU GPU GPU Xe LinkMulti-GPU Systems - NUMA regions: - 4+ GPUs

Target-independent Relay passes Target-optimized graph Target-dependent Relay passes Intel x86 ARM CPU Nvidia GPU ARM GPU Schedule templates written in TVM Tensor IR .. More targets AutoTVM – Tuning Target-independent Relay passes Target-optimized Int8 Relay Graph Intel x86 schedule ARM CPU schedule Nvidia GPU schedule ARM GPU schedule Relay Int8 Graph Target-dependent Relay layout opt© 2019 its Affiliates. All rights reserved. Outline • QNN Dialect • Design • Operators • Results on Intel Cascade Lake© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Quantized Operators

人 人 e 人 e@ TVM Q@ AliOs Overview TVM @ AliOs ARM CPU TVM @ AliOos Hexagon DSP TVM @ Alios Intel GPU Misc /NiiOS ! 驱动万物智能 PART ONE TVM Q@ AliOs Overview AiOS 1驱动万物智能 AliOs overview 。 AliOs (www AN 2X MobilenetV2 TFLite 1.34X MobilenetV2 QNNPACK AliOs @ Roewe RX5 MAX OpenVINO @ Intel GPU AliDS AR-Nav Product @ SUV Release and adopt TVM (Apollo Lake Gold) Model 1.6X Intel AliOs TVM Arch Model 。 Facelandmark Pedestrian & Vehicle Detection Voice-GUI Gesture Lanenet NLU DMS FacelD Multimodal Interection CPU (ARM、Intel) 1驱动万物智能 Accelerated

transistors incorporated in a chip will approximately double every 24 months." --Gordon Moore, Intel co-founderMoore’s Law (2/2) 29 ● Around 1965 Gordon Moore predicted the number of transistors months." --Gordon Moore, Intel co-founderDennard Scaling (1/3) "The number of transistors incorporated in a chip will approximately double every 24 months." --Gordon Moore, Intel co-founder http://www-cs-faculty transistors incorporated in a chip will approximately double every 24 months." --Gordon Moore, Intel co-founder http://www-cs-faculty.stanford.edu/~eroberts/cs181/projects/2010-11/TechnologicalSing

MiniMax 等。  其次是由 TogetherAI、Groq、Fireworks、Replicate、硅基流动等组成的 GPU 推理集群 服务提供商，它们处理扩展与缩减等技术难题，并在基本计算费用基础上收取额外费用，从 而让应用公司无需承担构建和管理 GPU 推理集群的高昂成本，而是可以直接利用抽象化的 AI 基础设施服务。  第三类是传统的云计算平台，例如亚马逊的 Amazon Vertex AI 等，允许应用开发者轻松部署和使用标准化或定制化的 AI 模型， 并通过 API 接口调用这些模型。  最后一类是本地推理，SGLang、vLLM、TensorRT-LLM 在生产级 GPU 服务负载中表现 出色，受到许多有本地托管模型需求的应用开发者的欢迎，此外，Ollama 和 LM Studio 也 是在个人计算机上运行模型的优选方案。 62 / 111 除模型层面外，应 软件，例如：微控制处理器（MCU）会运行实时操作系统或者直接运行某个特定程序；中央处 理器（CPU）往往会运行 Windows、Linux 等复杂操作系统作为底座支撑整个软件栈；图形 处理器（GPU）一般不加载操作系统而是直接运行图形图像处理程序，神经网络处理器 （NPU）则直接运行深度学习相关程序。 处理器芯片设计是一项很复杂的任务，整个过程犹如一座冰山。冰山水面上是用户或者大 众看到

© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon/Intel Confidentia Presenter: Zhi Chen, Cody Yu Amazon SageMaker Neo, Deep Engine Science Bring Your Own Codegen to TVM Chip© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Example showcase: Intel MKL-DNN (DNNL) library 1. Import packages import numpy as np from tvm import relay 2. Load a pretrained Relay Runtime (VM, Graph Runtime, Interpreter) Your Dispatcher Target Device General Devices (CPU/GPU/FPGA) Mark supported operators or subgraphs 1. Implement an operator-level annotator, OR 2. Implement

iOS, Android, WASM*) Hybrid profiling capabilities (sampling and/or instrumentation) (CPU and GPU instrumentation) Tracing capabilities (values, messages, plots, allocations, …) Hassle-free integration spall https://handmade.network/p/333/spall/ geiger https://github.com/david-grs/geiger Intel IACA https://www.intel.com/content/www/us/en/developer/ articles/tool/architecture-code-analyzer.html“There is experience! Tracy can do it all!Tracy Profiler GUI 13Tracy Profiler GUI 14 Frame Info Menu bar GPU Timeline (per “device”) CPU Timeline (per-thread) Custom Plots & Allocation Trackers15Tracy Client