(TensorFlow版) 张校捷 2019/9/21 ## 目录 >> 低精度的概念和意义 TensorFlow的FP16模型 >> TensorRT的FP16/Int8模型 总结   FP16: E8M7 (TPU, tf.bfloat16) FP16: E5M10 (GPU, tf.float16) Int8 ## 低精度浮点数的优点 ### 1. 节约内存/显存的使用（FP16为原来的1/2，int8为原来的1/4） 2. 特殊的硬件专门用于低精度浮点数的计算加速（TensorCore） FP16 storage/input Full precision 使用低精度的意义 ## TensorCores适用条件 1. 卷积：K（输入通道），C（输出通道） 2. 通用矩阵乘法（GEMM）：MxK，KxN，（M，N，K） FP16: 大小为8x Int8: 大小为16x 如果FP32要使用，可以设置（内部转为FP16）： TF_ENABLE_CUBLAS_TENSOR_OP_MATH_FP32=1 TF_ENABLE_CUDNN_TENSOR_OP_MATH_FP32=1

TVM@ARM CPU • Support TFLite (Open Source and Upstream Master) • Optimize on INT8 & FP32 ## AliOS TVM @ ARM CPU INT8 Convolution • NHWC layout • im2col + pack • Tensorize GEMM  ## AliOS TVM @ ARM CPU INT8 Depthwise Convolution instruction if your ARM does not have dot 3. compute_at is very important ## AliOS TVM @ ARM CPU INT8 TVM / QNNPACK Speed Up @ Mobilenet V2 @ rasp 3b+ AARCH64 ![Image](/uploads/documents/9/0/e/a/9

Intelligence ## Outline • TensorCore AutoCodeGen in TVM • FP16 Mixed-Precision Training on PAI • INT8 Inference on PAI-Blade ## TensorCore ## AutoCodeGen ## Background ## • TensorCore • A revolutionary

Array(Array(Int8)), _teams.players.blk, Array(Array(Int8)), _teams.players.drb, Array(Array(Int8)), _teams.players.fg, Array(Array(Int8)), _teams.players.fg3, Array(Array(Int8)), _teams _teams.players.fg3a, Array(Array(Int8)), _teams.players.fg_pct, Array(Array(String)), _teams.players.fga, Array(Array(Int8)), _teams.players.ft, Array(Array(Int8)), _teams.players.ft_pct, _teams.players.fta, Array(Array(Int8)), _teams.players.mp, Array(Array(String)), _teams.players.orb, Array(Array(Int8)), _teams.players.pf, Array(Array(Int8)), _teams.players.player, Array(Array(String))

primitive numeric types: Integer types: Type Signed? Number of bits Smallest value Largest value Int8 √ 8 -2^7 2^7-1 UInt8 8 0 2^8-1 Int16 √ 16 -2^15 2^15-1 UInt16 16 0 2^16-1 Int32 √ 32 -2^31 2147483647) julia> for T in [Int8,Int16,Int32,Int64,Int128,UInt8,UInt16,UInt32,UInt64,UInt128] println("$(lpad(T,7)): [$(typemin(T)),$(typemax(T))]") end Int8: [-128,127] Int16: forms. julia> Int8(127) 127 julia> Int8(128) ERROR: InexactError: trunc(Int8, 128) Stacktrace: [...] julia> Int8(127.0) 127 julia> Int8(3.14) ERROR: InexactError: Int8(3.14) Stacktrace:

primitive numeric types: Integer types: Type Signed? Number of bits Smallest value Largest value Int8 √ 8 -2^7 2^7-1 UInt8 8 0 2^8-1 Int16 √ 16 -2^15 2^15-1 UInt16 16 0 2^16-1 Int32 √ 32 -2^31 2147483647) julia> for T in [Int8,Int16,Int32,Int64,Int128,UInt8,UInt16,UInt32,UInt64,UInt128] println("$(lpad(T,7)): [$(typemin(T)),$(typemax(T))]") end Int8: [-128,127] Int16: forms. julia> Int8(127) 127 julia> Int8(128) ERROR: InexactError: trunc(Int8, 128) Stacktrace: [...] julia> Int8(127.0) 127 julia> Int8(3.14) ERROR: InexactError: Int8(3.14) Stacktrace:

Integer types: |Type|Signed?|Number of bits|Smallest value|Largest value| |---|---|---|---|---| |Int8||8|$ -2^{7} $|$ 2^{7} - 1 $| |UInt8||8|0|$ 2^{8} - 1 $| |Int16||16|$ -2^{15} $|$ 2^{15} - 1 $| |UInt16||16|0|$ 2147483647) julia> for T in [Int8, Int16, Int32, Int64, Int128, Int8, Int16, Int32, Int64, Int128] println("$(lpad(T,7)): [$(typemin(T)),$(typemax(T))]") end Int8: [-128,127] Int16: [-32768,32767] forms. julia> Int8(127) 127 julia> Int8(128) ERROR: InexactError: trunc(Int8, 128) Stacktrace: [...] julia> Int8(127.0) 127 julia> Int8(3.14) ERROR: InexactError: Int8(3.14) Stacktrace:

Integer types: |Type|Signed?|Number of bits|Smallest value|Largest value| |---|---|---|---|---| |Int8||8|$ -2^{7} $|$ 2^{7} - 1 $| |UInt8||8|0|$ 2^{8} - 1 $| |Int16||16|$ -2^{15} $|$ 2^{15} - 1 $| |UInt16||16|0|$ 2147483647) julia> for T in [Int8, Int16, Int32, Int64, Int128, Int8, Int16, Int32, Int64, Int128] println("$(lpad(T,7)): [$(typemin(T)),$(typemax(T))]") end Int8: [-128,127] Int16: [-32768,32767] forms. julia> Int8(127) 127 julia> Int8(128) ERROR: InexactError: trunc(Int8, 128) Stacktrace: [...] julia> Int8(127.0) 127 julia> Int8(3.14) ERROR: InexactError: Int8(3.14) Stacktrace:

Integer types: |Type|Signed?|Number of bits|Smallest value|Largest value| |---|---|---|---|---| |Int8|✓|8|-2^7|2^7 - 1| |UInt8||8|0|2^8 - 1| |Int16|✓|16|-2^15|2^15 - 1| |UInt16||16|0|2^16 - 1| |Int32|✓|32|-2^31|2^31 2147483647) julia> for T in [Int8, Int16, Int32, Int64, Int128, Int8, Int16, Int32, Int64, Int128] println("$(lpad(T,7)): [$(typemin(T)),$(typemax(T))]") end Int8: [-128,127] Int16: [-32768,32767] forms. julia> Int8(127) 127 julia> Int8(128) ERROR: InexactError: trunc(Int8, 128) Stacktrace: [...] julia> Int8(127.0) 127 julia> Int8(3.14) ERROR: InexactError: Int8(3.14) Stacktrace:

Integer types: |Type|Signed?|Number of bits|Smallest value|Largest value| |---|---|---|---|---| |Int8|✓|8|-2^7|2^7 - 1| |UInt8||8|0|2^8 - 1| |Int16|✓|16|-2^15|2^15 - 1| |UInt16||16|0|2^16 - 1| |Int32|✓|32|-2^31|2^31 2147483647) julia> for T in [Int8, Int16, Int32, Int64, Int128, Int8, Int16, Int32, Int64, Int128] println("$(lpad(T,7)): [$(typemin(T)),$(typemax(T))]") end Int8: [-128,127] Int16: [-32768,32767] forms. julia> Int8(127) 127 julia> Int8(128) ERROR: InexactError: trunc(Int8, 128) Stacktrace: [...] julia> Int8(127.0) 127 julia> Int8(3.14) ERROR: InexactError: Int8(3.14) Stacktrace: