FrameworksLimitations of Existing Approach cuDNN Frameworks New operator introduced by operator fusion optimization potential benefit: 1.5x speedup Engineering intensiveMachine Learning based Program models in product Competitive on benchmarking type model Quickly enables other optimizations: fusion, layout, parallelization Portable performance across devicesWhy Automation is the Future 1 1 1

today in FBGEMMPyTorch and TVM - Lots of opportunity in PyTorch - Graph optimization - Existing fusion infrastructure fairly limited (CUDA-only, injective-only) - Kernel synthesis - Dynamic shapes

especially for multi-branch networks (i.e. YOLOv3, SSD)© Copyright 2018 Xilinx TVM Graph Partitioning/Fusion >> 8 Subgraph 1 Parallel Subgraphs Post-Processing Pre-Processing CPU FPGA CPU CPU FPGA© Copyright

Completely add new ops from scratch • New Relay passes and TVM schedules required • AlterOpLayout, Graph Fusion etc require work/operator • No reuse of existing Relay and TVM infrastructure. Option 2 – Lower

array.] This loop fusion is not a compiler optimization that may or may not occur, it is a syntactic guarantee whenever nested f.(args...) calls are encountered. Technically, the fusion stops as soon as each vectorized operation. 35.17 Fewer dots: Unfuse certain intermediate broadcasts The dot loop fusion mentioned above enables concise and idiomatic code to express highly performant operations. However

array.] This loop fusion is not a compiler optimization that may or may not occur, it is a syntactic guarantee whenever nested f.(args...) calls are encountered. Technically, the fusion stops as soon as each vectorized operation. 35.17 Fewer dots: Unfuse certain intermediate broadcasts The dot loop fusion mentioned above enables concise and idiomatic code to express highly performant operations. However

array.] This loop fusion is not a compiler optimization that may or may not occur, it is a syntactic guarantee whenever nested f.(args...) calls are encountered. Technically, the fusion stops as soon as each vectorized operation. 35.17 Fewer dots: Unfuse certain intermediate broadcasts The dot loop fusion mentioned above enables concise and idiomatic code to express highly performant operations. However

array.] This loop fusion is not a compiler optimization that may or may not occur, it is a syntactic guarantee whenever nested f.(args...) calls are encountered. Technically, the fusion stops as soon as CHAPTER 35. PERFORMANCE TIPS 477 Fewer dots: Unfuse certain intermediate broadcasts The dot loop fusion mentioned above enables concise and idiomatic code to express highly performant operations. However

array.] This loop fusion is not a compiler optimization that may or may not occur, it is a syntactic guarantee whenever nested f.(args...) calls are encountered. Technically, the fusion stops as soon as CHAPTER 35. PERFORMANCE TIPS 478 Fewer dots: Unfuse certain intermediate broadcasts The dot loop fusion mentioned above enables concise and idiomatic code to express highly performant operations. However

array.] This loop fusion is not a compiler optimization that may or may not occur, it is a syntactic guarantee whenever nested f.(args...) calls are encountered. Technically, the fusion stops as soon as CHAPTER 35. PERFORMANCE TIPS 477 Fewer dots: Unfuse certain intermediate broadcasts The dot loop fusion mentioned above enables concise and idiomatic code to express highly performant operations. However