Confidentia Presenter: Zhi Chen, Cody Yu Amazon SageMaker Neo, Deep Engine Science Bring Your Own Codegen to TVM AWS AI© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Considering testing.mobilenet.get_workload(batch_size=1) 3. Partition and build the network with an external codegen mod = relay.build_extern(mod, “dnnl”) 4. Run the inference exe = relay.create_executor(“vm”, mod=mod reserved. System Overview Relay IR Graph Annotation with Your Annotator Graph Partitioning Your Codegen LLVM, CUDA, Metal, VTA Serialized Subgraph Library Relay Runtime (VM, Graph Runtime, Interpreter)

function to compute the type at runtime ● Virtual machine as a new runtime for Relay ● Dynamic codegen (WIP) ○ Kernel dispatch for a single op ○ Graph dispatch for a (sub-)graph In collaboration with Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dynamic codegen: op dispatch (proposal) ● Goal: support codegen for dynamic shape ● Challenges ○ Single kernel performs poor across different coupled together© 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dynamic codegen: kernel dispatch (proposal) Relay op: conv2d Default function FTVMStrategy A generic function

schedule Se 一人一 了9 。 Normal schedule: the schedule for CUDA 本 codegen IR Passes *。 Need to satisfy TensorCore Intrinsics "。Kind of Auto Tensorization 下 CUDA CodeGen *。IR passes to automatically transform sub-tree to TensorCore Intrinsics Pattern Matching compute_locallo族 了 了 Performance Optimization 计划了全事业部 “Same as non-TensorCore CUDA codegen 。Auto tune tiling sizes 。 Vectorized load/store for higher bandwidth utilization 。Double buffer

as our C locks) to prevent recursion when doing certain operations (incremental package loading, codegen, etc.). The combination of a lock and this flag can be used to make finalizers safe. 2. A second initializes the global jl_root_task struct; and sets jl_current_task to the root task. jl_init_codegen() initializes the LLVM library. jl_init_serializer() initializes 8-bit serialization tags for builtin handled by codegen.cpp. Whenever a Julia function is called for the first time with a given set of argument types, type inference will be run on that function. This information is used by the codegen step

as our C locks) to prevent recursion when doing certain operations (incremental package loading, codegen, etc.). The combination of a lock and this flag can be used to make finalizers safe. 2. A second initializes the global jl_root_task struct; and sets jl_current_task to the root task. jl_init_codegen() initializes the LLVM library. jl_init_serializer() initializes 8-bit serialization tags for builtin handled by codegen.cpp. Whenever a Julia function is called for the first time with a given set of argument types, type inference will be run on that function. This information is used by the codegen step

as our C locks) to prevent recursion when doing certain operations (incremental package loading, codegen, etc.). The combination of a lock and this flag can be used to make finalizers safe. 2. A second initializes the global jl_root_task struct; and sets jl_current_task to the root task. jl_init_codegen() initializes the LLVM library. jl_init_serializer() initializes 8-bit serialization tags for builtin handled by codegen.cpp. Whenever a Julia function is called for the first time with a given set of argument types, type inference will be run on that function. This information is used by the codegen step

as our C locks) to prevent recursion when doing certain operations (incremental package loading, codegen, etc.). The combination of a lock and this flag can be used to make finalizers safe. 2. A second initializes the global jl_root_task struct; and sets jl_current_task to the root task. jl_init_codegen() initializes the LLVM library. jl_init_serializer() initializes 8-bit serialization tags for builtin convenience, as (due to the way OpaqueClosures work) it is not necessarily the rettype used by codegen. • parent The MethodInstance that "owns" this object (if applicable). • edges Forward edges to

as our C locks) to prevent recursion when doing certain operations (incremental package loading, codegen, etc.). The combination of a lock and this flag can be used to make finalizers safe. 2. A second initializes the global jl_root_task struct; and sets jl_current_task to the root task. jl_init_codegen() initializes the LLVM library. jl_init_serializer() initializes 8-bit serialization tags for builtin convenience, as (due to the way OpaqueClosures work) it is not necessarily the rettype used by codegen. • parent The MethodInstance that "owns" this object (if applicable). • edges Forward edges to

as our C locks) to prevent recursion when doing certain operations (incremental package loading, codegen, etc.). The combination of a lock and this flag can be used to make finalizers safe. 2. A second initializes the global jl_root_task struct; and sets jl_current_task to the root task. jl_init_codegen() initializes the LLVM library. jl_init_serializer() initializes 8-bit serialization tags for builtin convenience, as (due to the way OpaqueClosures work) it is not necessarily the rettype used by codegen. • parent The MethodInstance that "owns" this object (if applicable). • edges Forward edges to

as our C locks) to prevent recursion when doing certain operations (incremental package loading, codegen, etc.). The combination of a lock and this flag can be used to make finalizers safe. 2. A second initializes the global jl_root_task struct; and sets jl_current_task to the root task. jl_init_codegen() initializes the LLVM library. jl_init_serializer() initializes 8-bit serialization tags for builtin convenience, as (due to the way OpaqueClosures work) it is not necessarily the rettype used by codegen. • parent The MethodInstance that "owns" this object (if applicable). • edges Forward edges to