Deep Learning Landscape Frameworks and Inference engines DL Compilers Kenrel Libraries Hardware CuDNN NNPack MKL-DNN Hand optimized Open source, automated end-to- end optimization framework for Primitive Tensor operators such as Conv2D eg. cuDNN Offload to heavily optimized DNN operator library FrameworksLimitations of Existing Approach cuDNN Frameworks New operator introduced by is the Future 1 1 1 1 0.76 0.83 1.16 1.44 Large MatMul BatchConv Small MatMul BatchMatMul CuDNN w/ TensorCores tvm w/ TensorCores 1.4x better on emerging workloads Transformer related workloads