than vanilla distillation. We will now go over stochastic depth, a technique which can be useful if you are training very deep networks. Stochastic Depth Deep networks with hundreds of layers such as block, the output of the previous layer ( ) skips the layers represented by the function . The stochastic depth idea takes this one step further by probabilistically dropping a residual block with a probability final probability ( ). Under these conditions, the expected network depth during training reduces to . By expected network depth we informally mean the number of blocks that are enabled in expectation

layers.SeparableConv2D(filters, kernel_size, strides=(1, 1), padding='valid', data_format=None, depth_multiplier=1, activation=None, use_bias=True, depthwise_initializer='glorot_uniform', pointwise bias_constraint=None) 深度方向的可分离 2D 卷积。 可分离的卷积的操作包括，首先执行深度方向的空间卷积（分别作用于每个输入通道），紧 接一个将所得输出通道混合在一起的逐点卷积。depth_multiplier 参数控制深度步骤中每个输 入通道生成多少个输出通道。 直观地说，可分离的卷积可以理解为一种将卷积核分解成两个较小的卷积核的方法，或者 作为 Inception 块的一个极端版本。 json 中找到的 image_data_format 值。如果你从未设 置它，将使用”channels_last”。 • depth_multiplier: 每个输入通道的深度方向卷积输出通道的数量。深度方向卷积输出通道 的总数将等于 filterss_in * depth_multiplier。 • activation: 要使用的激活函数 (详见 activations)。如果你不指定，则不使用激活函数

difficult ??? Vasiliki Kalavri | Boston University 2020 10 Stochastic averaging ??? Vasiliki Kalavri | Boston University 2020 10 Stochastic averaging Use one hash function to simulate many by splitting of the M-bit hash value to select a sub-stream and the next M-p bits to compute the rank(.): Stochastic averaging Use one hash function to simulate many by splitting the hash value into two parts of the M-bit hash value to select a sub-stream and the next M-p bits to compute the rank(.): Stochastic averaging Use one hash function to simulate many by splitting the hash value into two parts

Early Stop,Dropout 主讲人：龙良曲 Tricks ▪ Early Stopping ▪ Dropout ▪ Stochastic Gradient Descent Early Stopping ▪ Regularization How-To ▪ Validation set to select parameters ▪ Monitor validation performance Batch- Norm Stochastic Gradient Descent ▪ Stochastic ▪ not random! ▪ Deterministic Gradient Descent https://towardsdatascience.com/difference-between-batch-gradient-descent-and- stochastic-gradient- https://towardsdatascience.com/difference-between-batch-gradient-descent-and- stochastic-gradient-descent-1187f1291aa1 ?? ??? Stochastic Gradient Descent ▪ Not single usually ▪ batch = 16, 32, 64, 128… Why

Supervised Learning: Regression and Classification 2 Linear Regression 3 Gradient Descent Algorithm 4 Stochastic Gradient Descent 5 Revisiting Least Square 6 A Probabilistic Interpretation to Linear Regression 0.6 , = 0.06 , = 0.07 , = 0.071 Feng Li (SDU) Linear Regression September 13, 2023 20 / 31 Stochastic Gradient Descent (SGD) What if the training set is huge? In the above batch gradient descent iteration A considerable computation cost is induced! Stochastic gradient descent (SGD), also known as incremental gradient descent, is a stochastic approximation of the gradient descent optimiza- tion

the GD algorithm. We illustrate the convergence processes under di↵erent step sizes in Fig. 3. 3 Stochastic Gradient Descent According to Eq. 5, it is observed that we have to visit all training data in convergence of GD algorithm under di↵erent step sizes. Stochastic Gradient Descent (SGD), also known as incremental gradient descent, is a stochastic approximation of the gradient descent optimization method x(i) � y(i))x(i) (6) and the update rule is ✓j ✓j � ↵(✓T x(i) � y(i))x(i) j (7) Algorithm 2: Stochastic Gradient Descent for Linear Regression 1: Given a starting point ✓ 2 dom J 2: repeat 3: Randomly

梯度下降 ? 学习率 步长 11 梯度下降的三种形式 批量梯度下降（Batch Gradient Descent,BGD） 梯度下降的每一步中，都用到了所有的训练样本 随机梯度下降（Stochastic Gradient Descent,SGD） 梯度下降的每一步中，用到一个样本，在每一次计算之后 便更新参数 ，而不需要首先将所有的训练集求和 小批量梯度下降（Mini-Batch Gradient 1 ? ෍ ?=1 ? ℎ ?(?) − ?(?) ⋅ ?? (?) (同步更新?? ，(j=0,1,...,n )) 梯度 学习率 13梯度下降的三种形式 随机梯度下降（Stochastic Gradient Descent） ? = ? − ? ⋅ ??(?) ?? = ? ??? 1 2 ℎ ? ? − ? ? 2 = 2 ⋅ 1 2 ℎ ? ? − ? ? ⋅ = ℎ ? ? − ? ? ⋅ ? ??? ൱ ෍ ?=0 ? (?? ?? ? − ? ? = ℎ ? ? − ? ? ?? ? 14梯度下降的三种形式 随机梯度下降（Stochastic Gradient Descent） 梯度下降的每一步中，用到一个样本，在每一次计算之后便更新参数，而不 需要首先将所有的训练集求和 参数更新 ??: = ?? − ? ℎ ?(?) −

梯度下降 ? 学习率 步长 13 梯度下降的三种形式 批量梯度下降（Batch Gradient Descent,BGD） 梯度下降的每一步中，都用到了所有的训练样本 随机梯度下降（Stochastic Gradient Descent,SGD） 梯度下降的每一步中，用到一个样本，在每一次计算之后 便更新参数 ，而不需要首先将所有的训练集求和 小批量梯度下降（Mini-Batch Gradient 1 ? ෍ ?=1 ? ℎ ?(?) − ?(?) ⋅ ?? (?) (同步更新?? ，(j=0,1,...,n )) 梯度 学习率 15梯度下降的三种形式 随机梯度下降（Stochastic Gradient Descent） ? = ? − ? ⋅ ??(?) ?? = ? ??? 1 2 ℎ ? ? − ? ? 2 = 2 ⋅ 1 2 ℎ ? ? − ? ? ⋅ ℎ ? ? − ? ? ⋅ ? ??? (෍ ?=0 ? ( ???? (?) − ?(?))) = ℎ ? ? − ? ? ?? ? 16梯度下降的三种形式 随机梯度下降（Stochastic Gradient Descent） 梯度下降的每一步中，用到一个样本，在每一次计算之后便更新参数，而不 需要首先将所有的训练集求和 参数更新 ??: = ?? − ? ℎ ?(?) −

Iterations 0 1 2 3 4 0 81, 1 27, 3 9, 9 6, 27 5, 81 3 Jamieson, Kevin, and Ameet Talwalkar. "Non-stochastic best arm identification and hyperparameter optimization." Artificial intelligence and statistics performance on the image and language benchmark datasets. Moreover, their NAS model could generate variable depth child networks. Figure 7-4 shows a sketch of their search procedure. It involves a controller which

α∂L(θ) ∂θj , ∀j where α is so-called learning rate • Variations • Gradient ascent algorithm • Stochastic gradient descent/ascent • mini-batch gradient descent/ascent 9 / 19 Back-Propagation: Warm Up ∂L/∂w[l] jk = a[l−1] k δ[l] j and ∂L/∂b[l] j = δ[l] j • BP algorithm is usually combined with stochastic gradient descent algorithm or mini-batch gradient descent algorithm 18 / 19 Thanks! Q & A 19