Lecture 4: Regularization and Bayesian Statistics Feng Li Shandong University fli@sdu.edu.cn September 20, 2023 Feng Li (SDU) Regularization and Bayesian Statistics September 20, 2023 1 / 25 Lecture Regularization and Bayesian Statistics 1 Overfitting Problem 2 Regularized Linear Regression 3 Regularized Logistic Regression 4 MLE and MAP Feng Li (SDU) Regularization and Bayesian Statistics September y = θ0 + θ1x y = θ0 + θ1x + θ2x2 y = θ0 + θ1x + · · · + θ5x5 Feng Li (SDU) Regularization and Bayesian Statistics September 20, 2023 3 / 25 Overfitting Problem (Contd.) Underfitting, or high bias,

smartly allocate resources to promising ranges of hyper-parameters like Bayesian Optimization (Figure 1-12 illustrates Bayesian Optimization). These algorithms construct ‘trials’ of hyper-parameters, where What varies across them is how future trials are constructed based on past results. Figure 1-12: Bayesian Optimization over two dimensions x1 and x2. Red contour lines denote a high loss value, and blue to the algorithm. Each cross is a trial (pair of x1 and x2 values) that the algorithm evaluated. Bayesian Optimization picks future trials in regions that were more favorable. Source. As an extension to

could sample more in the favorable regions? The next search strategy does exactly that! Bayesian Optimization Bayesian Optimization Search (BOS) is a sequential model based search technique where the search called Configuration Evaluation. Let's discuss it in detail in the next section. Figure 7-3: (a) Bayesian Optimization Search on a two dimensional search space. The red areas correspond to lower validation

how to reduce the large number of variables to a small number. Averaging over complexity is the Bayesian approach. Use as complex a model might be needed, but don’t choose a single parameter values. Instead with terms of arbitrarily high order. How can it be good to use a model that we know is false? The Bayesian answer: It is not good. We should abandon the idea of using the best parameters and instead average

is true, and P(A) and P(B) are probability of observing A and B, respectively. We now introduce Bayesian inference by taking image recognition as an example. Our aim is to identify if there is a cat in calculate P(X = x), since both of 1 them share the same denominator P(X = x). Therefore, to perform Bayesian interference, the parameters we have to compute are only P(X = x | Y = y) and P(Y = y). Recalling

capabilities for the upper layer, including classification and clustering for model identification and Bayesian optimization for parameter search. A-Tune software architecture A-Tune client (atune-adm) A-Tune 3) MPI/CPI Data sampling System parameter configuration Classification Al engine Clustering Bayesian optimization Server Efficient Concurrency and Ultimate Performance 040 openEuler OS Technical

scientific computing package -- notably smoothing, optimization and machine learning. ● PyMC is for your Bayesian/MCMC/hierarchical modeling needs. ● PyMix for mixture models ● If speed becomes a problem, consider

o��r���F �����T���������������I��� � +�Dong Chen, Xudong Cao, Liwei Wang, Fang Wen, Jian Sun. Bayesian face revisited: a joint formulation. 2012, european conference on computer vision. MSRA “Feature

nouns  Titles and Names  Titles: capitalize meaning-carrying words  Names: capitalize, e.g., Bayesian, Euclidean  References like Figure 1, Table 2, Section 3, Chapter 4, Equation 5 are names as well

WordCount – TeraSort – Enhanced DFSIO – Nutch Indexing – Page Rank Machine Learning – Bayesian Classification – K-Means Clustering Analytical Query HiBench 1.0 paper (“The HiBench Suite: