is the gradient of L and can be defined as $$ \nabla_{\theta}L=\frac{1}{m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})x^{(i)} $$ One approach to minimize the above objective function is gradient descent |L^{+}(\theta)-L(\theta)|\leq\epsilon $$ Try to resolve the logistic regression problem using gradient descent method with the initialization $ \theta = 0 $ , and answer the following questions: 1. Assume $ \epsilon = 10^{-6} $ . How many iterations are required to achieve convergence? Note that gradient descent method has a very slow convergence rate and may take a long while to achieve the minimum. 2

## PyTorch ## Early Stop,Dropout 主讲人：龙良曲 ## Tricks Early Stopping Dropout ■ Stochastic Gradient Descent ## Early Stopping ■ Regularization =\frac{ \underbrace{}_{m} \underbrace{\frac{m}{\sum_{i=1}^{m}} (\hat{y}^{i} - y^{i})} x_{j}^{i} $$ ## Gradient Descent ③ Stochastic G.D. for i in range(M): $$ \theta_{j} := \theta_{j} - \alpha \cdot \frac{\overline{(only

data to the hyperplane is denoted by $ |\theta^{T} x^{(i)} - y^{(i)}| $ . ## 2 Gradient Descent Gradient Descent (GD) method is a first-order iterative optimization algorithm for finding the minimum J(\theta) $ decreases fastest if one goes from $ \theta $ in the direction of the negative gradient of J at $ \theta $ . Let $$ \nabla J(\theta)=[\frac{\partial J}{\partial\theta_{0}},\frac{\partial \frac{\partial J}{\partial\theta_{1}},\cdots,\frac{\partial J}{\partial\theta_{n}}]^{T} $$ denote the gradient of $ J(\theta) $ . In each iteration, we update $ \theta $ according to the following rule:

8687b8d2ce249d05/p2_3.jpg) 3 Gradient Descent Algorithm  4 Stochastic Gradient Descent =\sum_{i=1}^{n}f_{i}^{\prime}(x)\cdot u_{i} $$ ### Gradient (Contd.) ## Proof Letting $ g(h) = f(x + hu) $ , we have $$ g^{\prime}(0)=\lim_{h\to0}\f f_{i}'(x) u_{i} $ , by substituting which into (1), we complete the proof. ## Definition Gradient: The gradient of f is a vector function $ \nabla f : R^{n} \rightarrow R^{n} $ defined by $$ \nabla

J(\theta)=\frac{1}{2m}\sum_{i=1}^{m}(h_{\theta}(x^{(i)})-y^{(i)})^{2} $$ One of the optimization approach is gradient descent algorithm. The algorithm is performed iteratively, and in each iteration, we update parameter $ \alpha $ is so-called “learning rate” based on which we can tune the convergence of the gradient descent. ## 3 2D Linear Regression We start a very simple case where n = 1. Download data1.zip, and are m = 50 training examples, and you will use them to develop a linear regression model using gradient descent algorithm, based on which, we can predict the height given a new age value. In Matlab/Octave

批量梯度下降（Batch Gradient Descent, BGD） 梯度下降的每一步中，都用到了所有的训练样本 随机梯度下降（Stochastic Gradient Descent, SGD） 梯度下降的每一步中，用到一个样本，在每一次计算之后便更新参数，而不需要首先将所有的训练集求和 小批量梯度下降（Mini-Batch Gradient Descent, MBGD） 梯度下降的每一步中，用到了一定批量的训练样本 梯度下降的三种形式 批量梯度下降 (Batch Gradient Descent) 梯度下降的每一步中，都用到了所有的训练样本 学习率 参数更新 梯度 (同步更新 $ w_{j} $ ， $ (j=0,1,\ldots,n) $ ) ## 梯度下降的三种形式 ## 随机梯度下降 (Stochastic Gradient Descent) $$ \begin{aligned} 推导 ig(x^{(i)}\big)-y^{(i)}\big)x_{j}^{(i)}\end{aligned} $$ ## 梯度下降的三种形式 ## 随机梯度下降（Stochastic Gradient Descent） 梯度下降的每一步中，用到一个样本，在每一次计算之后便更新参数，而不需要首先将所有的训练集求和 ## 参数更新 $$ w_{j}\text{:=}w_{j}-\alpha\b

1.jpg)  ## Gradient Descent (GD) Algorithm • If the multi-variable cost (or loss) function $ \mathcal{L}(\theta) $ is from $ \theta $ in the direction of the negative gradient of L at $ \theta $ • Find a local minimum of a differentiable function using gradient descent $$ \theta_{j}\leftarrow\theta_{j}-\alpha\frac \alpha $ is so-called learning rate • Variations • Gradient ascent algorithm • Stochastic gradient descent/ascent • mini-batch gradient descent/ascent ## Back-Propagation: Warm Up • $ w_{jk}^{[l]}

批量梯度下降（Batch Gradient Descent, BGD） 梯度下降的每一步中，都用到了所有的训练样本 随机梯度下降（Stochastic Gradient Descent, SGD） 梯度下降的每一步中，用到一个样本，在每一次计算之后便更新参数，而不需要首先将所有的训练集求和 小批量梯度下降（Mini-Batch Gradient Descent, MBGD） 梯度下降的每一步中，用到了一定批量的训练样本 梯度下降的每一步中，用到了一定批量的训练样本 ## 梯度下降的三种形式 批量梯度下降 (Batch Gradient Descent) 梯度下降的每一步中，都用到了所有的训练样本 学习率 参数更新  梯度 (同步更新 $ w_{j} $ $ ， $ (j=0,1,\ldots,n) $ ) ## 梯度下降的三种形式 ## 随机梯度下降 (Stochastic Gradient Descent) $$ \begin{aligned} 推导 w&=w-\alpha\cdot\frac{\partial J(w)}{\partial w}\quad h(x)=w^{\mathrm{T}}X=w_{0}x_{0}+w_{1}x

parameters as well as the magnitude of $ \lambda $ ### Regularized Linear Regression (Contd.) ## • Gradient descent • Repeat { $$ \theta_{0}:=\theta_{0}-\alpha\frac{1}{m}\sum_{i=1}^{m}\big(h_{\theta}\big( bda}{2m}\sum_{j=1}^{n}\theta_{j}^{2} $$ ### Regularized Logistic Regression (Contd.) ## • Gradient descent: ## Repeat $$ \bullet\theta_{0}:=\theta_{0}-\alpha\frac{1}{m}\sum_{i=1}^{m}\big(h_{\thet \log[1+\exp(-y^{(i)}\theta^{T}x^{(i)})] $$ • No close-form solution exists, but we can do gradient descent on $ \theta $ ## Logistic Regression: MAP Solution • Again, assume θ follows a Gaussian distribution

|Adam|tensorflow/python/training/adam.py| |Ftrl|tensorflow/python/training/ftrl.py| |Gradient Descent|tensorflow/python/training/gradient\_descent.py| |Momentum|tensorflow/python/training/momentum.py| |Proximal Adagr Adagrad|tensorflow/python/training/proximal\_adagrad.py| |Proximal Gradient Descent|tensorflow/python/training/proximal\_gradient\_descent.py| |Rmsprop|tensorflow/python/training/rmsprop.py| |Synchronize R