Lecture 4: Regularization and Bayesian Statistics Feng Li Shandong University fli@sdu.edu.cn September 20, 2023 Feng Li (SDU) Regularization and Bayesian Statistics September 20, 2023 1 / 25 Lecture Regularization and Bayesian Statistics 1 Overfitting Problem 2 Regularized Linear Regression 3 Regularized Logistic Regression 4 MLE and MAP Feng Li (SDU) Regularization and Bayesian Statistics September 20, 2023 θ1x y = θ0 + θ1x + θ2x2 y = θ0 + θ1x + · · · + θ5x5 Feng Li (SDU) Regularization and Bayesian Statistics September 20, 2023 3 / 25 Overfitting Problem (Contd.) Underfitting, or high bias, is when the

180 About Database Statistics in Greenplum Database 180 System Statistics 181 Table Size 181 The pg_statistic System Table and pg_stats View 181 Sampling 185 Updating Statistics 185 Analyzing Partitioned Partitioned Tables 185 Configuring Statistics 186 Statistics Target 186 Automatic Statistics Collection 187 Installing and Upgrading Greenplum 189 Estimating Storage Capacity 190 Calculating Usable Query 573 Collecting Root Partition Statistics 573 Running ANALYZE 574 GPORCA and Leaf Partition Statistics 574 Deactivating Automatic Root Partition Statistics Collection 575 Considerations when Using

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · 95 2.3.8 Statistics · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · analyze_concurrency-new-in-v840">Concurrent automatic statistics �→ collection 43 You can set the concurrency within a single automatic statistics �→ collection task using the system variable concurrency of �→ scanning tasks based on node scale and hardware specifications. This �→ improves statistics collection efficiency by fully utilizing system �→ resources, reduces manual tuning, and ensures

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · 90 2.3.8 Statistics · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · analyze_concurrency-new-in-v840">Concurrent automatic statistics �→ collection 38 You can set the concurrency within a single automatic statistics �→ collection task using the system variable concurrency of �→ scanning tasks based on node scale and hardware specifications. This �→ improves statistics collection efficiency by fully utilizing system �→ resources, reduces manual tuning, and ensures

· · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · 68 2.3.8 Statistics · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · · Ignoring unnecessary columns when �→ collecting statistics Under the premise of ensuring that the optimizer can obtain the �→ necessary information, TiDB speeds up statistics collection, improves �→ the timeliness of statistics, and thus ensures that the optimal 37 �→ execution plan is selected

329 About Database Statistics in Greenplum Database 329 System Statistics 330 Table Size 330 The pg_statistic System Table and pg_stats View 330 Sampling 334 Updating Statistics 334 Analyzing Partitioned Partitioned Tables 334 Configuring Statistics 335 Statistics Target 335 Automatic Statistics Collection 336 VMware Greenplum on vSphere 338 Overview 338 Getting Started 339 VMware Greenplum on vSphere Query 778 Collecting Root Partition Statistics 778 Running ANALYZE 778 GPORCA and Leaf Partition Statistics 779 Deactivating Automatic Root Partition Statistics Collection 779 Considerations when Using

297 About Database Statistics in Greenplum Database 297 System Statistics 298 Table Size 298 The pg_statistic System Table and pg_stats View 298 Sampling 302 Updating Statistics 302 Analyzing Partitioned Partitioned Tables 302 Configuring Statistics 303 Statistics Target 303 Automatic Statistics Collection 304 VMware Tanzu Greenplum on vSphere 306 Overview 306 Getting Started 307 VMware Tanzu Greenplum Query 769 Collecting Root Partition Statistics 770 Running ANALYZE 770 GPORCA and Leaf Partition Statistics 771 Deactivating Automatic Root Partition Statistics Collection 771 Considerations when Using

315 About Database Statistics in Greenplum Database 315 System Statistics 316 Table Size 316 The pg_statistic System Table and pg_stats View 316 Sampling 320 Updating Statistics 320 Analyzing Partitioned Partitioned Tables 320 Configuring Statistics 321 Statistics Target 321 Automatic Statistics Collection 322 VMware Greenplum on vSphere 324 Overview 324 Getting Started 325 VMware Greenplum on vSphere Query 749 Collecting Root Partition Statistics 749 Running ANALYZE 749 GPORCA and Leaf Partition Statistics 750 Deactivating Automatic Root Partition Statistics Collection 750 Considerations when Using

286 About Database Statistics in Greenplum Database 286 System Statistics 287 Table Size 287 The pg_statistic System Table and pg_stats View 287 Sampling 291 Updating Statistics 291 Analyzing Partitioned 292 VMware Tanzu Greenplum 6 Documentation VMware, Inc 12 Configuring Statistics 292 Statistics Target 292 Automatic Statistics Collection 293 VMware Tanzu Greenplum on vSphere 295 Supported Platforms Query 742 Collecting Root Partition Statistics 742 Running ANALYZE 743 GPORCA and Leaf Partition Statistics 744 Deactivating Automatic Root Partition Statistics Collection 744 Considerations when Using

256 About Database Statistics in Greenplum Database 256 System Statistics 257 Table Size 257 The pg_statistic System Table and pg_stats View 257 Sampling 260 Updating Statistics 260 Analyzing Partitioned Partitioned Tables 261 Configuring Statistics 261 Statistics Target 261 Automatic Statistics Collection 262 VMware Tanzu Greenplum on vSphere 264 Supported Platforms 264 Overview 265 Getting Started a Query 640 Collecting Root Partition Statistics 640 Running ANALYZE 640 GPORCA and Leaf Partition Statistics 641 Disabling Automatic Root Partition Statistics Collection 641 Considerations when Using