build_extern(mod, “dnnl”) 4. Run the inference exe = relay.create_executor(“vm”, mod=mod, ctx=tvm.cpu(0)) data = np.random.uniform(size=(1, 3, 224, 224)).astype(“float32”) out = exe.evaluate()(data, **params) Relay Runtime (VM, Graph Runtime, Interpreter) Your Dispatcher Target Device General Devices (CPU/GPU/FPGA) Mark supported operators or subgraphs 1. Implement an operator-level annotator, OR 2. Implement Options Op-level annotation ● Simple and easy to implement 👍 ● One op per subgraph results in overhead 👎 (working on an algorithm to merge annotated ops) Graph-level annotation ● High flexibility

function CPU strategy func GPU strategy func OpStrategy OpStrategy OpStrategy Default implement Specialized implement 1 Specialized implement 2 (e.g., winograd) kernel_size <= 3 b < 8 “cpu” “gpu”© Affiliates. All rights reserved. How to register a strategy? @conv2d_strategy.register("cpu") def conv2d_strategy_cpu(attrs, inputs, out_type, target): strategy = OpStrategy() layout = attrs.data_layout Services, Inc. or its Affiliates. All rights reserved. Why do we need graph dispatcher 1. Minimal overhead: only one dispatching operation is required for each inference. 2. Fit for operator such as conv2d_NCHWc

(baseline), 40us (target) - 85x speedup - Uh ohEnter, TVM and model co-design - PyTorch operator overhead makes interpreter infeasible - Reduce FLOPs with block-sparsified weight matrices - not a new (~10 lines of Relay IR) - A few days of work - TVM sampling model running in 30us on single server CPU core - Beat hand-written, highly optimized baselines (https://github.com/mozilla/LPCNet) by ~40%

import/export, and over 270 bugs fixed. 2.70 -- March 2014: Cycles gets basic volumetric support on the CPU, more improvements to the motion tracker, two new modeling modifiers, some UI consistency improvements GLSL which runs on the GPU for performance but falls back to the CPU for large images which might be slow when loaded with the GPU. Uses CPU for display transform and render images as a 2D texture. Fastest Cycles can use either the CPU or certain GPUs to render images, for more information see the GPU Rendering page. 無 When set to None or when the only option is None: the CPU will be used as the computing

import/export, and over 270 bugs fixed. 2.70 – March 2014: Cycles gets basic volumetric support on the CPU, more improvements to the motion tracker, two new modeling modifiers, some UI consistency improvements GLSL which runs on the GPU for performance but falls back to the CPU for large images which might be slow when loaded with the GPU. Uses CPU for display transform and render images as a 2D texture. Fastest Cycles can use either the CPU or certain GPUs to render images, for more information see the GPU Rendering page. None When set to None or when the only option is None: the CPU will be used as the computing

import/export, and over 270 bugs fixed. 2.70 -- March 2014: Cycles gets basic volumetric support on the CPU, more improvements to the motion tracker, two new modeling modifiers, some UI consistency improvements which runs on the GPU for performance but falls back to the CPU for large images which might be slow when loaded with the GPU. 2D Texture Uses CPU for display transform and render images as a 2D texture Cycles can use either the CPU or certain GPUs to render images, for more information see the GPU Rendering page. None When set to None or when the only option is None: the CPU will be used as the computing

import/export, and over 270 bugs fixed. 2.70 – March 2014: Cycles gets basic volumetric support on the CPU, more improvements to the motion tracker, two new modeling modifiers, some UI consistency improvements which runs on the GPU for performance but falls back to the CPU for large images which might be slow when loaded with the GPU. 2D Texture Uses CPU for display transform and render images as a 2D texture Cycles can use either the CPU or certain GPUs to render images, for more information see the GPU Rendering page. None When set to None or when the only option is None: the CPU will be used as the computing

import/export, and over 270 bugs fixed. 2.70 – March 2014: Cycles gets basic volumetric support on the CPU, more improvements to the motion tracker, two new modeling modifiers, some UI consistency improvements GLSL which runs on the GPU for performance but falls back to the CPU for large images which might be slow when loaded with the GPU. Uses CPU for display transform and render images as a 2D texture. Fastest Cycles can use either the CPU or certain GPUs to render images, for more information see the GPU Rendering page. None When set to None or when the only option is None: the CPU will be used as the computing

import/export, and over 270 bugs fixed. 2.70 -- March 2014: Cycles gets basic volumetric support on the CPU, more improvements to the motion tracker, two new modeling modifiers, some UI consistency improvements GLSL which runs on the GPU for performance but falls back to the CPU for large images which might be slow when loaded with the GPU. Uses CPU for display transform and render images as a 2D texture. Fastest Cycles can use either the CPU or certain GPUs to render images, for more information see the GPU Rendering page. 無 When set to None or when the only option is None: the CPU will be used as the computing

import/export, and over 270 bugs fixed. 2.70 -- March 2014: Cycles gets basic volumetric support on the CPU, more improvements to the motion tracker, two new modeling modifiers, some UI consistency improvements GLSL which runs on the GPU for performance but falls back to the CPU for large images which might be slow when loaded with the GPU. Uses CPU for display transform and render images as a 2D texture. Fastest Cycles can use either the CPU or certain GPUs to render images, for more information see the GPU Rendering page. 無 When set to None or when the only option is None: the CPU will be used as the computing