Lecture 4: Regularization and Bayesian Statistics Feng Li Shandong University fli@sdu.edu.cn September 20, 2023 Feng Li (SDU) Regularization and Bayesian Statistics September 20, 2023 1 / 25 Lecture Lecture 4: Regularization and Bayesian Statistics 1 Overfitting Problem 2 Regularized Linear Regression 3 Regularized Logistic Regression 4 MLE and MAP Feng Li (SDU) Regularization and Bayesian Statistics Overfitting Problem y = θ0 + θ1x y = θ0 + θ1x + θ2x2 y = θ0 + θ1x + · · · + θ5x5 Feng Li (SDU) Regularization and Bayesian Statistics September 20, 2023 3 / 25 Overfitting Problem (Contd.) Underfitting

Regularization 主讲人：龙良曲 Occam's Razor ▪ More things should not be used than are necessary. Reduce Overfitting ▪ More data ▪ Constraint model complexity ▪ shallow ▪ regularization ▪ Dropout ▪ Data ▪ Early Stopping Regularization Enforce Weights close to 0 Weight Decay Intuition How ▪ L1-regularization ▪ L2-regularization lambda L2-regularization L1-regularization 下一课时 动量与学习率衰 减 Thank

layer, etc., Keras model and layer access Keras modules for activation function, loss function, regularization function, etc., Using Keras model, Keras Layer, and Keras modules, any ANN algorithm (CNN, RNN penalties on the layer parameter during optimization. Keras regularization module provides below functions to set penalties on the layer. Regularization applies per-layer basis only. Keras 35 L1 Regularizer It provides L1 based regularization. from keras.models import Sequential from keras.layers import Activation, Dense from keras import regularizers my_regularizer = regularizers

to how label smoothing can help us avoid overfitting. Label Smoothing Label smoothing is a regularization method that helps reduce the overfitting we might see with our models where the model predicts way too noisy for the model to learn anything. You should treat label smoothing as yet another regularization technique. In fact this paper17 goes into details about when label smoothing helps. The original optimizer should prefer a flatter minima over a steeper minima. This idea is intuitively analogous to regularization where we prefer to find solutions with model parameters having smaller absolute values due to

smaller footprints. In the first chapter, we briefly introduced learning techniques such as regularization, dropout, data augmentation, and distillation to improve quality. These techniques can boost namely data augmentation and distillation, to discuss in this chapter. This is because, firstly, regularization and dropout are fairly straight-forward to enable in any modern deep learning framework. Secondly sections are sampled from a probability distribution as follows: 10 Yun, Sangdoo, et al. "Cutmix: Regularization strategy to train strong classifiers with localizable features." Proceedings of the IEEE/CVF

algorithms in machine learning. The topics include linear regression, logistic re- gression, regularization, Gaussian discriminant analysis, Naive Bayes, EM algorithm, SVM, K-means, factor analysis, PCA we have very well, but do poorly on new data (poor generalization ability). Cross-validation, regularization, Reducing dimensionality is another possibility. It is apparent that things become simpler

The key to prevent Overfitting Sample more data? Limited Data ▪ Small network capacity ▪ Regularization ▪ Data argumentation Recap Data argumentation ▪ Flip ▪ Rotate ▪ Random Move & Crop ▪ GAN

主讲人：龙良曲 Tricks ▪ Early Stopping ▪ Dropout ▪ Stochastic Gradient Descent Early Stopping ▪ Regularization How-To ▪ Validation set to select parameters ▪ Monitor validation performance ▪ Stop at the

Least Cost among plans How to compute Wasserstein Distance 1-Lipschitz function WGAN Sort of Regularization WGAN-Gradient Penalty More stable Training Progress Indicator Thank You.

到更多更有效的特征，减小噪声的影响。 2.降维 即丢弃一些不能帮助我们正确预测的特征。可以是手工选择保留哪些特征，或者使用一 些模型选择的算法来帮忙（例如PCA）。 3.正则化 正则化(regularization)的技术，保留所有的特征，但是减少参数的大小（magnitude） ，它可以改善或者减少过拟合问题。 4.集成学习方法 集成学习是把多个模型集成在一起，来降低单一模型的过拟合风险。