Chapter 2 - Compression Techniques “I have made this longer than usual because I have not had time to make it shorter.” Blaise Pascal In the last chapter, we discussed a few ideas to improve the deep deep learning efficiency. Now, we will elaborate on one of those ideas, the compression techniques. Compression techniques aim to reduce the model footprint (size, latency, memory etc.). We can reduce the chapter, we introduce Quantization, a model compression technique that addresses both these issues. We’ll start with a gentle introduction to the idea of compression. Details of quantization and its applications

Advanced Compression Techniques “The problem is that we attempt to solve the simplest questions cleverly, thereby rendering them unusually complex. One should seek the simple solution.” — Anton Pavlovich Pavlovich Chekhov In this chapter, we will discuss two advanced compression techniques. By ‘advanced’ we mean that these techniques are slightly more involved than quantization (as discussed in the second of our models. Did we get you excited yet? Let’s learn about these techniques together! Model Compression Using Sparsity Sparsity or Pruning refers to the technique of removing (pruning) weights during

an 80% accuracy with the same number of training steps and labels. Thus, delivering a 2x model compression. Again, this is a hypothetical scenario which illustrates how learning techniques are leveraged samples. Similar to mixup, the parameter is sampled from a probability distribution and is used as the ratio of the area of the cut-mixed section ( ) to the area of the sample image ( ). The positional offsets

训练模型 由于VGG‐11比AlexNet计算量更大，因此我们构建了一个通道数较少的网络，足够用于训练Fashion‐MNIST数 据集。 ratio = 4 small_conv_arch = [(pair[0], pair[1] // ratio) for pair in conv_arch] net = vgg(small_conv_arch) 除了使用略高的学习率外，模型训练过程与 通过均匀采样获得的连续值。 shape_aug = torchvision.transforms.RandomResizedCrop( (200, 200), scale=(0.1, 1), ratio=(0.5, 2)) apply(img, shape_aug) 13.1. 图像增广 551 改变颜色 另一种增广方法是改变颜色。我们可以改变图像颜色的四个方面：亮度、对比度、饱和度和色调。在下面的 整区域边界从而更准确地预测目标的真实边界框（ground‐truth bounding box）。不同的模型使用的区域采样 方法可能不同。这里我们介绍其中的一种方法：以每个像素为中心，生成多个缩放比和宽高比（aspect ratio） 不同的边界框。这些边界框被称为锚框（anchor box）我们将在 13.7节中设计一个基于锚框的目标检测模型。 首先，让我们修改输出精度，以获得更简洁的输出。 %matplotlib inline

right: RMSE (cm) of keyframes, the starting ratio (i.e. dividing the initialization frame index by the total frame number), and the tracking success ratio after initialization. Group A: simple translation

out的输出向量?，这个过程可以看成是特征降维的 过程，把原始的高维输入向量?变换到低维的变量?。特征降维(Dimensionality Reduction)在 机器学习中有广泛的应用，比如文件压缩(Compression)、数据预处理(Preprocessing)等。最 常见的降维算法有主成分分析法(Principal components analysis，简称 PCA)，通过对协方差 矩阵进行特征分解而得到数据的主要成分，但是 重要性采样 ratio = (pi_a / tf.gather(old_action_log_prob, index, axis=0 )) surr1 = ratio * advantage surr2 = tf.clip_by_value(ratio, 1 - epsilon, 1 + epsilon)

= PCA(n_components=3) 训练模型 pca.fit(X) 投影后各个特征维度的方差比例(这里是三个主成分) print(pca.explained_variance_ratio_) 投影后的特征维度的方差 print(pca.explained_variance_) 20 2.Scikit-learn主要用法 无监督学习算法-聚类 DBSCAN 层次聚类

Moving on, we define a few configurations for the controller to decide the exploitation/exploration ratio, the hidden size of the recurrent cell, the number of training episodes for the controller and the

efficiency in deep learning models. We will also introduce core areas of efficiency techniques (compression techniques, learning techniques, automation, efficient models & layers, infrastructure). Our hope leeway in model quality, we can trade off some of it for a smaller footprint by using lossy model compression techniques7. For example, when compressing a model naively we might reduce the model size, RAM core areas, with infrastructure and hardware forming the foundation (see Figure 1-7). 7 Lossy compression techniques allow you to compress data very well, but you lose some information too when you try

chapter 2. We could also incorporate compression techniques such as sparsity, k-means clustering, etc. which will be discussed in the later chapters. 2. Even after compression, the vocabulary itself is large: embedding model’s quality and footprint metrics as discussed. We can combine other ideas from compression techniques and learning techniques on top of efficient architectures. As an example, we can train embeddings based model using data augmentation to achieve higher performance and subsequently apply compression or distillation to further reduce its footprint. With this chapter, we hope to have set the stage