LOSS及其梯度 主讲人：龙良曲 Typical Loss ▪ Mean Squared Error ▪ Cross Entropy Loss ▪ binary ▪ multi-class ▪ +softmax ▪ Leave it to Logistic Regression Part MSE ▪ lo?? = σ[? − (?? + ?)]2 ▪ ?2 − ???? = ∇? = 2 σ ? − ?? ? ∗ ∇??(?) ∇? autograd.grad loss.backward Gradient API ▪ torch.autograd.grad(loss, [w1, w2,…]) ▪ [w1 grad, w2 grad…] ▪ loss.backward() ▪ w1.grad ▪ w2.grad Softmax ▪ soft version

............................................................................................ 60 Loss ................................................................................................. to learn by training and finally do to prediction. This step requires us to choose loss function and Optimizer. loss function and Optimizer are used in learning phase to find the error (deviation from convolution layer, pooling layer, etc., Keras model and layer access Keras modules for activation function, loss function, regularization function, etc., Using Keras model, Keras Layer, and Keras modules, any ANN

layer at the bottom (right after the input layer). We compile the model with a sparse cross entropy loss function (discussed in chapter 2) and the adam optimizer. from tensorflow.keras import applications activation='softmax') ]) adam = optimizers.Adam(learning_rate=LEARNING_RATE) model.compile( optimizer=adam, loss='sparse_categorical_crossentropy', metrics=['accuracy']) return model model = create_model() model 968ms/step - loss: 4.5983 - accuracy: 0.3833 - val_loss: 1.8394 - val_accuracy: 0.6833 Epoch 2/100 43/43 [==============================] - 21s 493ms/step - loss: 0.1100 - accuracy: 0.9784 - val_loss: 2.3430

(OBD) paper approximate the saliency score using a second-derivative of the weights , where is the loss function, and is the candidate parameter for removal. Why do we want to compute the second-derivative derivative helps us compute the instantaneous slope of the loss function with respect to . gives us the rate at which the curvature of the loss function is changing with respect to and . The second derivative derivative gives us a clearer insight into how important might be to minimize the loss. Since computing pairwise second-derivatives for all and might be very expensive (even with just weights, this quickly

virtual system disks. As a result, this should not normally be done, since it can potentially cause data loss or an inconsistent state of the guest system on disk. 42 1 First Steps As an exception, if your created since the snapshot and all other file changes will be lost. In order to prevent such data loss while still making use of the snapshot feature, it is possible to add a second hard drive in write-through a USB device, it will be disconnected from the host without a proper shutdown. This may cause data loss. Note: Oracle Solaris hosts have a few known limitations regarding USB support. See Known Limitations

either humans who will consume the information will not notice the loss of some information, or do not necessarily care about the loss in quality. Figure 2-2: On the left is a high quality image of a cat can be discarded, based on the tolerance for loss in quality. The JPEG and MP3 formats are able to achieve a 10-11x compression without any perceptible loss in quality. However, further compression might arrival? If so, what would be the ideal tradeoff on how much compression we want v/s how much quality loss can we tolerate? Let us slowly build up to that by exploring how quantization can help us. A Generic

remain identical across the different features. 9 Implementation Detail: Using the cross entropy loss when $$N$$ is large can be computationally expensive due to the N-way softmax calculation. In the controls the number of unique words for which we learn embeddings. A small value for would result in loss of information because most of the words would get mapped to the OOV token. However, if is too large bow_model_w2v = get_bow_model(get_pretrained_embedding_layer(trainable=True)) bow_model_w2v.compile( loss='sparse_categorical_crossentropy', optimizer='adam', metrics=["accuracy"] ) bow_model_w2v.summary()

then we can simply add a few additional layers (known as the prediction head), use the appropriate loss function, and train the model with the labeled data for the task at hand. We can keep the original 6-3: Pre-training and Fine-tuning steps for BERT. Source: Develin et al. For BERT, the pre-training loss is the mean of the losses for the above two tasks. Similar to NLU, the pretext tasks in vision have Model(bert_inputs, output) bert_classifier.compile( optimizer=tf.keras.optimizers.Adam(learning_rate), loss='sparse_categorical_crossentropy', metrics=['accuracy'] ) if not keep_tfhub_weights: shuffle_

样本集合? = {(?(1), ?(1)),(?(2),?(2)),… , (?(?), ?(?))}，然后找出一条“最好”的直线，使得它尽可能地 让所有采样点到该直线的误差(Error，或损失 Loss)之和最小。 ? = 1. ? + . (?(1), ?(1)) (?(2), ?(2)) 图 2.4 带观测误差的估计模型 也就是说，由于观测误差?的存在，当采集了多个数据点 np.array(points), lr) loss = mse(b, w, points) # 计算当前的均方差，用于监控训练进度 if step%50 == 0: # 打印误差和实时的 w,b 值 print(f"iteration:{step}, loss:{loss}, w:{w}, b:{b}") return [b, w] 次，返回最优 w*,b*和训练 Loss 的下降过程 [b, w]= gradient_descent(data, initial_b, initial_w, lr, num_iterations) loss = mse(b, w, data) # 计算最优数值解 w,b 上的均方差 print(f'Final loss:{loss}, w:{w}, b:{b}')

吧。有的人可能会疑惑输出为什么不用在 forward 里面定义 Softmax 或者 Cross-Entropy，这是因为这些东西是在 NeuralNetwork 之外定义： #损 失 函 数 为 交 叉 熵 loss_function = nn . CrossEntropyLoss () # 学 习 率 learning_rate = 1e−3 # 优 化 器 为 随 机 梯 度 下 降 optimizer train_dataloader , model , loss_function , optimizer ) test_loop ( test_dataloader , model , loss_function ) print ( ”Done ! ” ) 然后就是训练和测试的程序，训练一轮的程序如下： def train_loop ( dataloader , model , loss_function , optimizer pred = model (X) l o s s = loss_function ( pred , y) # Backpropagation optimizer . zero_grad () #梯 度 归0w l o s s . backward () optimizer . step () i f batch % 100 == 0: loss , current = l o s s . item