mse_loss(torch.ones(1), x*w) In [25]: torch.autograd.grad(mse, [w]) Out[25]: (tensor([2.]),) ### loss.backward ## ☀️ ☁️ ☀️ ☁️ In [15]: x=torch.ones(1) In [17]: w=torch.full([1],2) In [19]: mse=F.mse_loss(torch x*w) In [27]: mse.backward() In [28]: w.grad Out[28]: tensor([2.]) ## Gradient API ####### - torch.autograd.grad(loss, [w1, w2, ...]) - [w1 grad, w2 grad...] ### loss.backward() w1.grad w2.grad dim=0) In [35]: p.backward() RuntimeError: Trying to backward through the graph a second time, but the buffers have already been freed. Specify retain_graph=True when calling backward the first time.

需前向计算，再后向计算微分 1. struct Backward { 2. value : Double // 当前节点计算值 3. backward : (Double) -> Unit // 对当前子表达式微分并累加 4. } 5. 6. fn Backward::var(value : Double, diff : Ref[Double]) -> Backward { 7. // value, backward: fn { d => diff.val = diff.val + d } } 9. } 10. 11. fn Backward::constant(d : Double) -> Backward { 12. { value: d, backward: fn { _ => () } } 13. } 14. 15. fn Backward::backward(b Backward::backward(b : Backward, d : Double) { (b.backward)(d) } 16. 17. fn Backward::value(backward : Backward) -> Double { backward.value } ## 后向微分 - 需前向计算，再后向计算微分 $$ \circ f=g+h,\frac{\partial f}{\partial

requires_grad=True) z = torch.rand((N, D), requires_grad=True) a = x * y b = a + z c = torch.sum(b) c.backward() print(c.grad_fn) print(x.data) print(x.grad) Package • Don’t need to worry about partial differentiation, chain rule etc.. • backward() does that • loss.backward() • Gradients are accumulated for each step by default: • Need to zero out gradients requires_grad=True) z = torch.rand((N, D), requires_grad=True) a = x * y b = a + z c = torch.sum(b) c.backward() ## Autograd (continued) ## • Manual Weight Update - example a = torch.randn(1, requires_grad=True

bias=True) (fc3): Linear(in_features=84, out_features=10, bias=True) ) 在模型中必须要定义 forward 函数，backward 函数（用来计算梯度）会被 autograd 自动可以在 forward 函数中使用任何针对 Tensor 的操作。 net.parameters() 返回可被学习的参数（权重）列表和值 params 0.0184, -0.0300]], grad_fn=) 将所有参数的梯度缓存清零，然后进行随机梯度的反向传播： net.zero_grad() out.backward(torch.randn(1, 10)) ## note torch.nn 只支持小批量输入。整个 torch.nn 包都只支持小批量样本，而不支持单个样本。 例如，nn.Conv2d 接受一个4维的张量，每一维分别是 宽）。如果你有单个样本，只需使用 input.unsqueeze(0) 来添加其它的维数 在继续之前，我们回顾一下到目前为止用到的类。 ## 回顾: - torch.Tensor：一个用过自动调用backward()实现支持自动梯度计算的多维数组，并且保存关于个向量的梯度 w.r.t. - nn.Module：神经网络模块。封装参数、移动到 GPU 上运行、导出、加载等。 - nn.Paramet

utograd引擎会按照逆序，通过Function的backward依次计算梯度。 ### 2. Autograd自动求导 ## backward函数 backward函数是反向传播的入口点，在需要被求导的节点上调用 backward函数会计算梯度值到相应的节点上。 backward函数是反向求导数，使用链式法则求导。 backward需要一个重要的参数grad_tensor，对非标量节点求导 ensors的shape必须和y的相同。 但如果节点只含有一个标量值，这个参数就可以省略（例如最普遍的loss.backward()与loss.backward(torch.tensor(1))等价） ### 2. Autograd自动求导 ## grad属性 backward函数本身没有返回值，它计算出来的梯度存放在叶子节点的grad属性中。PyTorch文档中提到，如果grad属性不为空 ，新计算出来的梯度值会直接加到旧值上面。为什么不直接覆盖旧的结果呢？ 这是因为有些Tensor可能有多个输出，那么就需要调用多个backward。叠加的处理方式使得backward不需要考虑之前有没有被计算过导数，只需要加上去就行了，这使得设计变得更简单。 因此我们用户在反向传播之前，常常需要用zero_grad函数对导数手动清零，确保计算出来的是正确的结果。 ### 2. Autograd自动求导

record: record Enumeration A : Set where constructor enumeration field start : A forward : A → A backward : A → A This gives an interface that allows us to move along the elements of a data type A. For e))) Or we can go back 2 positions starting from a given a: backward-2 : {A : Set} → Enumeration A → A → A backward-2 e a = backward (backward a) where open Enumeration e Now, we want to use these methods = 0 ; forward = suc ; backward = pred } where pred : Nat → Nat pred zero = zero pred (suc x) = x test₁ : 3rd enum-Nat 3 test₁ = refl test₂ : backward-2 enum-Nat 5 3 test₂ = refl

record: record Enumeration A : Set where constructor enumeration field start : A forward : A → A backward : A → A This gives an interface that allows us to move along the elements of a data type A. For e))) Or we can go back 2 positions starting from a given a: backward-2 : {A : Set} → Enumeration A → A → A backward-2 e a = backward (backward a) where open Enumeration e Now, we want to use these methods = 0 ; forward = suc ; backward = pred } where pred : Nat → Nat pred zero = zero pred (suc x) = x test₁ : 3rd enum-Nat ≡ 3 test₁ = refl test₂ : backward-2 enum-Nat 5 ≡ 3 test₂ = refl

level, component]) Wrapper that sends a log message through the Spider's logger, kept for backward compatibility. For more information see Logging from Spiders. ## closed (reason) Called when the attribute) ## body_as_unicode() The same as text, but available as a method. This method is kept for backward compatibility; please prefer response.text. ## HtmlResponse objects ####### class scrapy.http the default reactor defined by Twisted for the current platform will be used. This is to maintain backward compatibility and avoid possible problems caused by using a non-default reactor. For additional

level, component]) Wrapper that sends a log message through the Spider's logger, kept for backward compatibility. For more information see Logging from Spiders. closed(reason) Called when the spider the default reactor defined by Twisted for the current platform will be used. This is to maintain backward compatibility and avoid possible problems caused by using a non-default reactor. For additional support for wildcard matching • doesn’t use the length based rule It is faster than Protego and backward-compatible with versions of Scrapy before 1.8.0. In order to use this parser, set: • ROBOTSTXT_PARSER

log(message[, level, component]) Wrapper that sends a log message through the Spider’s logger, kept for backward compatibility. For more information see Logging from Spiders. ## closed(reason) Called when the the default reactor defined by Twisted for the current platform will be used. This is to maintain backward compatibility and avoid possible problems caused by using a non-default reactor. For additional support for wildcard matching • doesn’t use the length based rule It is faster than Protego and backward-compatible with versions of Scrapy before 1.8.0. In order to use this parser, set: • ROBOTSTXT_PARSER