September 6, 2023 50 / 57 The Curse of Dimensionality Handling complexity Involve many variables, how can we handle this complexity without get- ting into trouble. Optimization and Integration Usually Overview September 6, 2023 51 / 57 How to Handle Complexity Properly dealing with complexity is a crucial issue for machine learning. Limiting complexity is one approach Use a model that is complex enough if can find out how to reduce the large number of variables to a small number. Averaging over complexity is the Bayesian approach. Use as complex a model might be needed, but don’t choose a single parameter

to the linear computation complexity of RNNs. However, attention is still faster in wall clock time because it processes entire sequences together. The quadratic complexity of attention is addressed through transformer variants with efficient self-attention mechanisms. These ideas tackle the quadratic complexity at various levels. The simplest idea is to chunk the input sequence of length n into blocks of the input sequence, thus compressing it in the process. Sparse attention attempts to reduce the complexity by picking a subset of parameters for attention computation. This subset can also be learned during

has been a race to create deeper networks with an ever larger number of parameters and increased complexity. In Computer Vision, several model architectures such as VGGNet, Inception, ResNet etc. (refer classification. Each new breakthrough in neural networks has led to an increase in the network complexity, number of parameters, the amount of training resources required to train the network, prediction

techniques using the metrics relevant to our use case. In some cases, these techniques can help reduce complexity and improve generalization. Let us consider an arbitrary neural network layer. We can abstract B, C, and D are all matrices. The number of MACs in a model is another metric to measure its complexity. And since they are such a fundamental block of models, they have been optimized both in hardware

things should not be used than are necessary. Reduce Overfitting ▪ More data ▪ Constraint model complexity ▪ shallow ▪ regularization ▪ Dropout ▪ Data argumentation ▪ Early Stopping Regularization

linearly or exponentially to reach a value of 1.0 in the final epoch. It controls when and how much complexity you want to introduce in the training. Figure 6-12 shows multiple examples of pacing functions