TLDR
本节课先回顾了上次课的Linear Quantization, 然后介绍了量化的两个阶段：PTQ和QAT。PTQ指在训练后进行量化，粒度分为per-tensor, per-chanel和per-group（利用多层scale factor）,对于activation来说，如何找到clipping range. 然后介绍了QAT指的是在训练时forward用的是量化后的weights和activation，但是backward时需要保存一个原精度的weights用来计算gradient. 之后介绍了binary和ternary量化的方式，其中量化到1-bit之后可以用XOR计算来进一度加速。最后介绍了mixed-precision, 指的是在网络不同层用不同的bit，但是这样对于网络设计的要求更高。
Overview
02:06Lecture Plan
03:22 Review of last lecture
K-Means Quantization, Linear Quantization
06:43 Post-Training Quantization
including Granularity, Dynamic Range Clipping and Rounding
26:07Dynamic Range Clipping
35:20Quantization-Aware Training
53:08Binary/Ternary Quantization
01:05:33Mixed-Precision Quantization
01:08:52Summary
Paper List
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference [Jacob et al., CVPR 2018]
VS-Quant: Per-Vector Scaled Quantization for Accurate Low-Precision Neural Network Inference [Steve Dai, et at.]
With Shared Microexponents, A Little Shifting Goes a Long Way [Bita Rouhani et al.]
Post-Training 4-Bit Quantization of Convolution Networks for Rapid-Deployment [Banner et al., NeurIPS 2019]
8-bit Inference with TensorRT [Szymon Migacz, 2017]
Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training [Sakr et al., ICML 2022]
XNOR-Net: ImageNet Classification using Binary Convolutional Neural Networks [Rastegari et al., ECCV 2016]
Trained Ternary Quantization [Zhu et al., ICLR 2017]
HAQ: Hardware-Aware Automated Quantization with Mixed Precision [Wang et al., CVPR 2019]
Notes
Trades accuracy (fine-grained) for hardware acceleration (coarse-grained)
Tensor Quantization
Paper: Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference [Jacob et al., CVPR 2018]
一个tensor内，32个chanel，weight的range可能会很不同，导致accuracy不高
q-max is determined by 2-bit
Per-Tensor Quantization loss is 2.28
Per-Channel Quantization
re-scale for each row
loss is 2.08 < 2.28
works super well for vision models
One more scaling factor
两层scale，Sq - integer scale factor
Gamma - fp scale factor
Paper: VS-Quant: Per-Vector Scaled Quantization for Accurate Low-Precision Neural Network Inference [Steve Dai, et at.]
Multi-level scaling scheme
Paper: With Shared Microexponents, A Little Shifting Goes a Long Way [Bita Rouhani et al.]
S1M2 means 1 sign bit + 2 mantissa bit
active research topic
26:07Clipping with Dynamic Range
How to find range for activations?
During training, EMA exponential moving average, smoothed across training steps
calibration batch, separate from train and test dataset
Gaussian Distribution
Paper: Post-Training 4-Bit Quantization of Convolution Networks for Rapid-Deployment [Banner et al., NeurIPS 2019]
KL Divergence to find best location to clip
Goal is to minimize loss of information
Paper: 8-bit Inference with TensorRT [Szymon Migacz, 2017]
Clip example
MSE using newton-raphson method
Paper: Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training [Sakr et al., ICML 2022]
35:20Quantization-Aware Training
一般用fp pretrain模型的weights再进行quantization和fine-tune，而不是train from scratch
Computation Graph
fp weights are kept in the back
quantized weights are used for inference
forward: int computation
backward: fp computation
How to backprop?
STE pass the gradients through the quantization as if identity function
Accuracy is improved from PTQ to QAT
53:08Binary/Ternary Quantization
change weight to binary (2-bit)
How to Binarize weights?
Deterministic by sign
Stochastic by computing probs (complex compute)
Accuracy Drop 21.2% after Binarization
can use a scaling factor
loss improved from 9.28 to 9.24
Accuracy Drop 0.2%
Paper XNOR-Net: ImageNet Classification using Binary Convolutional Neural Networks [Rastegari et al., ECCV 2016]
what if weights and activations are both binarized?
relate to XNOR computation
popcount: count number of 1
hardware efficient
use xnor and popcount to replace +-
BWN: binarized weights + scale factor
BNN: binarized without scale factor
XNOR-Net: binarized for both activation and weights
Ternary Weight Networks (TWN)
2-bit: -1, 1, 0
TWN accuracy better than BWN
TTQ: Trained Ternary Quantization [Zhu et al., ICLR 2017]
trained parameters to represent positive and negative scales
01:05:33Mixed-Precision Quantization
8-bit everywhere
bit varies for layers
huge design space
Solution: deign automation
HAQ: Hardware-Aware Automated Quantization with Mixed Precision [Wang et al., CVPR 2019]
Better than Uniform
Edge and Cloud
MX Strategy Difference
