memorize something for the long term, you need to improve recall by repetition (i.e., increase the weight of that connection). Can we do the same with neural networks? Can we optimally prune the network latency gains with a minimal performance tradeoff. Next, the chapter goes over weight sharing using clustering. Weight sharing, and in particular clustering is a generalization of quantization. If you that lie within the same quantization bin, are mapped to the same quantized weight value. That is an implicit form for weight sharing. However, quantization falls behind in case the data that we are quantizing

architectures. A classical example is Quantization (see Figure 1-8), which tries to compress the weight matrix of a layer, by reducing its precision (eg., from 32-bit floating point values to 8-bit unsigned unsigned / signed integers). Quantization can generally be applied to any network which has a weight matrix. It can often help reduce the model size 2 - 8x, while also speeding up the inference latency. of continuous high-precision values to discrete fixed-point integer values. Another example is Pruning (see Figure 1-9), where weights that are not important for the network’s quality are removed / pruned

attention domain. The attention (Luong) mechanism learns three weight matrices namely WQ (query weight), WK (key weight) and WV (value weight) which are used to compute the query, key and value matrices number of elements in each sequence and d represents the number of dimensions of each element. The weight matrices WQ, WK, and WV are identically shaped as (d, dk). The query, key and the value matrices learned. In the next chapter we will explore some more advanced model compression techniques like pruning.

，这 样就可能对训练样本学习的“太好”了，把训练样本的一些特点当做所有数据 都具有的一般性质，从而导致过拟合。 剪枝的基本策略有“预剪枝”（prepruning）和“后剪枝”（post-pruning） 通过剪枝处理去掉一些分支来降低过拟合的风险。 20 C4.5的剪枝 预剪枝（prepruning） 预剪枝不仅可以降低过拟合的风险而且还可以 减少训练时间，但另一方面它是基于“贪心”

是添加到模型的层的列表。 4.2.3 Sequential 模型方法 4.2.3.1 compile compile(self, optimizer, loss, metrics=None, sample_weight_mode=None, weighted_metrics=None, target_tensors=None) 用于配置训练模型。 参数 • optimizer: 字符串（优化器名）或者优化器对象。详见 • sample_weight_mode: 如果你需要执行按时间步采样权重（2D 权重） ，请将其设置为 temporal。默认为 None，为采样权重（1D）。如果模型有多个输出，则可以通过传递 mode 的字典或列表，以在每个输出上使用不同的 sample_weight_mode。 • weighted_metrics: 在训练和测试期间，由 sample_weight 或 class_weight class_weight 评估和加权的度 量标准列表。 • target_tensors: 默认情况下，Keras 将为模型的目标创建一个占位符，在训练过程中将使用 目标数据。相反，如果你想使用自己的目标张量（反过来说，Keras 在训练期间不会载入 这些目标张量的外部 Numpy 数据），您可以通过 target_tensors 参数指定它们。它应该 是单个张量（对于单输出 Sequential 模型）。

NeuralNetwork 内部定义函数： def weight_init ( s e l f ) : #遍 历 网 络 的 每 一 层 fo r m in s e l f . modules () : #如 果 该 层 是 线 性 连 接 层 i f i s i n s t a n c e (m, nn . Linear ) : print (m. weight . shape ) print (m (m. bias . shape ) #权 重 分 布 符 合 正 态 分 布 m. weight . data . normal_ ( 0 . 0 , 1) #偏 置 归 0 m. bias . data . zero_ () Chapter 3. 更完善的神经网络 17 注意 bias 是权重，因为当前层的 bias 会连接下一层的每个神经元，所以 bias 的 shape 是下 model . weight_init () 我们开始训练，发现第一个 epoch 训练的结果正确率就达到了 78%，而最终训练结果能达到 百分之 81%。说明合理地初始化权重具有很重要的意义。 如果 weight 全都初始化成同一个值，例如： m. weight . data . f i l l _ (0.05) 训练效果就会变得特别差。 还有另一种初始化方式： m. weight . data=torch

Operation. BC is the most expensive operation latency-wise. In typical neural-networks: C is the weight matrix. D is often a one-dimension vector, hence the addition is cheap both from the latency point a matrix) using quantization. Weight Quantization We just learnt that the weights in deep learning models are stored in an N-dimensional matrix (tensor), and the weight matrix W is most expensive in terms terms of storage. Can we efficiently represent this weight matrix W to reduce the model size? We already have worked with 1-D vectors and 2-D vectors in previous exercises. We can extend that learning

Overview of Deep learning Keras 12 Here,  Multiple input along with weight represents dendrites.  Sum of input along with activation function represents neurons. Sum actually initializer to set the weight for each input and finally activators to transform the output to make it non-linear. In between, constraints restricts and specify the range in which the weight of input data to specified. Initializers In Machine Learning, weight will be assigned to all input data. Initializers module provides different functions to set these initial weight. Some of the Keras Initializer function

线性假设是指目标（房屋价格）可以表示为特征（面积和房龄）的加权和，如下面的式子： price = warea · area + wage · age + b. (3.1.1) (3.1.1)中的warea和wage 称为权重（weight），权重决定了每个特征对我们预测值的影响。b称为偏置（bias）、 偏移量（offset）或截距（intercept）。偏置是指当所有特征都取值为0时，预测值应该为多少。即使现实中不 会有任 Linear时指定输入和输出尺寸一样，现在我们能直接访问参数以设定它们的初始值。我 们通过net[0]选择网络中的第一个图层，然后使用weight.data和bias.data方法访问参数。我们还可以使用 替换方法normal_和fill_来重写参数值。 net[0].weight.data.normal_(0, 0.01) net[0].bias.data.fill_(0) tensor([0.]) 练获得的模型参数。要访问参数，我们首先从net访 问所需的层，然后读取该层的权重和偏置。正如在从零开始实现中一样，我们估计得到的参数与生成数据的 真实参数非常接近。 w = net[0].weight.data print('w的估计误差：', true_w - w.reshape(true_w.shape)) b = net[0].bias.data print('b的估计误差：',

个节点的网络层，并通过全连接层的 kernel 成员名查 看其权值矩阵?： In [45]: # 定义全连接层的输出节点为 3, 输入节点为 4 fc = nn.Linear(4, 3) fc.weight # 查看权值矩阵 W Out[45]: Parameter containing: tensor([[-0.1410, 0.1454, -0.3955, -0.1701], out.shape # 输出大小 Out[48]: torch.Size([4, 16, 30, 30]) 其中卷积核张量?也是 4 维张量，可以通过 weight 成员变量访问： In [49]: layer.weight.shape # 访问卷积核权值张量 Out[49]: torch.Size([16, 3, 3, 3]) 4.6 索引与切片 通过索引与切片操 ]的连接汇集为变量?，即： ? = ?1 ?1 + ?2 ?2 + ⋯ + ?? ?? + ? 其中?称为感知机的偏置(Bias)，一维向量? = [?1, ?2, … , ??]称为感知机的权值(Weight)，? 称为感知机的净活性值(Net Activation)。 ?1 ?2 ? ?? ? ?1 ?2 ? ?? 输入? 输出 图 6.1