Lecture Notes on Linear Regression Feng Li fli@sdu.edu.cn Shandong University, China 1 Linear Regression Problem In regression problem, we aim at predicting a continuous target value given an input n-dimensional feature vector is denoted by x 2 Rn, while y 2 R is the output variable. In linear regression models, the hypothesis function is defined by h✓(x) = ✓nxn + ✓n�1xn�1 + · · · + ✓1x1 + ✓0 Geometrically i=1 ⇣ h✓(x(i)) � y(i)⌘2 Our linear regression problem can be formulated as min ✓ J(✓) = 1 2 m X i=1 ⇣ ✓T x(i) � y(i)⌘2 1 Figure 1: 3D linear regression. Specifically, we aim at minimizing J(✓)

Experiment 1: Linear Regression August 27, 2018 1 Description This first exercise will give you practice with linear regression. These exercises have been extensively tested with Matlab, but they should option in the installer, and available for Linux from Octave-Forge ). 2 Linear Regression Recall that the linear regression model is hθ(x) = θT x = n � j=0 θjxj, (1) where θ is the parameter which we intercept item x0 = 1. Therefore, the resulting feature vector is (n + 1)-dimensional. 1 3 2D Linear Regression We start a very simple case where n = 1. Download data1.zip, and extract the files (ex1x.dat and

Lecture 2: Linear Regression Feng Li Shandong University fli@sdu.edu.cn September 13, 2023 Feng Li (SDU) Linear Regression September 13, 2023 1 / 31 Lecture 2: Linear Regression 1 Supervised Learning: Learning: Regression and Classification 2 Linear Regression 3 Gradient Descent Algorithm 4 Stochastic Gradient Descent 5 Revisiting Least Square 6 A Probabilistic Interpretation to Linear Regression Feng Feng Li (SDU) Linear Regression September 13, 2023 2 / 31 Supervised Learning Regression: Predict a continuous value Classification: Predict a discrete value, the class Living area (feet2) Price (1000$s)

Lecture 3: Logistic Regression Feng Li Shandong University fli@sdu.edu.cn September 20, 2023 Feng Li (SDU) Logistic Regression September 20, 2023 1 / 29 Lecture 3: Logistic Regression 1 Classification Classification 2 Logistic Regression 3 Newton’s Method 4 Multiclass Classification Feng Li (SDU) Logistic Regression September 20, 2023 2 / 29 Classification Classification problems Email: Spam / Not Spam? Class” (e.g., benign tumor) 1 : “Positive Class” (e.g., malignant tumor) Feng Li (SDU) Logistic Regression September 20, 2023 3 / 29 Warm-Up What if applying linear regress to classification? Tumor Size

1 Aurora InnovationContinuous Regression Testing for Safer and Faster Refactoring Pejman Ghorbanzade Aurora Innovation3 Aurora Innovation Engineers spend 17 hours per week maintaining software. *Stripe is continuous regression testing How does regression testing work in practice How to build a regression testing system Going beyond �nding behavioral regressions How to use regression testing effectively "Write tests. Not too many. Mostly integration." - Guillermo Rauch17 Aurora Innovation Continuous regression testing Continuously verifying that the software works as well as before, during the development

Experiment 2: Logistic Regression and Newton’s Method August 29, 2018 1 Description In this exercise, you will use Newton’s Method to implement logistic regression on a classification problem. 2 Data 1 score 40 45 50 55 60 65 70 75 80 85 90 Exam 2 score 4 Logistic Regression Recall that in logistic regression, the hypothesis function is hθ(x) = g(θT x) = 1 1 + e−θT x = P(y = 1 | x; θ) than (or equal to) some threshold ϵ, i.e. |L+(θ) − L(θ)| ≤ ϵ (7) Try to resolve the logistic regression problem using gradient de- scent method with the initialization θ = 0, and answer the following

Logistic Regression 主讲人：龙良曲 Recap ▪ for continuous: ? = ?? + ? ▪ for probability output: ? = ? ?? + ? ▪ ?: ??????? ?? ???????? Binary Classification ▪ interpret network as ?: ? → ? ? ?; ? ▪ output output ∈ 0, 1 ▪ which is exactly what logistic function comes in! Goal v.s. Approach ▪ For regression: ▪ Goal: ???? = ? ▪ Approach: minimize ????(????, ?) ▪ For classification: ▪ Goal: maximize benchmark since the number of correct is not continuous Q2. why call logistic regression ▪ use sigmoid ▪ Controversial! ▪ MSE => regression ▪ Cross Entropy => classification 0.7 0.3 0.7 MSE CEL Binary Classification

problems which do not occur with other, similar servers. 8. Is the problem a regression? Knowing that an issue is a regression usually makes it signifi- cantly easier to find the solution. In this case, conditions (bug #13487) • Host services: fixed a crash during VM shutdown under rare conditions (4.3.32 regression; bug #14841) • ExtPack: black-list Extension Packs older than 4.3.30 due to incompatible changes hosts: several El-Capitan fixes • X11 Additions: fixed wrong DPI value with certain guests (4.3.28 regression; bug #14151) • Solaris Additions: added quiesce support to co-operate with Solaris’ fast-reboot

problems which do not occur with other, similar servers. 8. Is the problem a regression? Knowing that an issue is a regression usually makes it signifi- cantly easier to find the solution. In this case, to propagate any DNS name server / domain / search string information to the NAT network (4.3.24 regression; bugs #13915, #13918) • NAT Network: don’t delay the shutdown of VBoxSVC on Windows hosts • Mouse the mouse could not be moved under rare conditions if no Guest Additions are installed (4.3.24 regression; bug #13935) • Storage: if the guest ejects a virtual CD/DVD medium, make the change permanent

problems which do not occur with other, similar servers. 8. Is the problem a regression? Knowing that an issue is a regression usually makes it signifi- cantly easier to find the solution. In this case, X11 hosts. • GUI: fix occasional loss of focus in full-screen mode on X11 host systems (4.3.16 regression) • GUI: Mac OS X: wizards should have Cancel button (bug #12541) • GUI: added a global option circum- stances (bug #13190) • ACPI: fixed occassional Guru Meditations in ACPI timer code (4.3.18 regression; bug #13521) • EFI: improved performance of IDE disk access • EFI: fixed a bug in the EFI video