Chapter 4 - Efficient Architectures “Any sufficiently advanced technology is indistinguishable from magic.” — Arthur C. Clarke, “Hazards of Prophecy: The Failure of Imagination” (1962) “Any technology gain orders of magnitude in terms of footprint or quality, we should consider employing suitable efficient architectures. The progress of deep learning is characterized by the phases of architectural breakthroughs deployment challenges. What good is a model that cannot be deployed in practical applications! Efficient Architectures aim to improve model deployability by proposing novel ways to reduce model footprint

or exceeded the contemporary state of the art models. These child networks were smaller and more efficient than the human designed models. However, the key contribution of NASNet was the focus on predicting

Chapter 1 - Introduction to Efficient Deep Learning Welcome to the book! This chapter is a preview of what to expect in the book. We start off by providing an overview of the state of deep learning, its introduce core areas of efficiency techniques (compression techniques, learning techniques, automation, efficient models & layers, infrastructure). Our hope is that even if you just read this chapter, you would hands dirty with practical projects. With that being said, let’s start off on our journey to more efficient deep learning models. Introduction to Deep Learning Machine learning is being used in countless

recall or other performance metrics). We designate a new model training setup to be more sample efficient, if it achieves similar or better performance with fewer data samples when compared to the baseline same accuracy by seeing a smaller number of samples, that process would be sample efficient. Similarly, a sample efficient model training process requires fewer samples to achieve the same performance, which adopt this hypothetical sample efficient model training. Figure 3-1: The above plot demonstrates sample efficiency between two model training setups. The sample efficient model achieves about the same

of parameters, but this could hurt the quality. Compression techniques are used to achieve an efficient representation of one or more layers in a neural network with a possible quality trade off. The constrained environment like a mobile device. To summarize, compression techniques help to achieve an efficient representation of a layer or a collection of layers, such that it meets the desired tradeoff goals chosen to work with Tensorflow 2.0 (TF) because it has exhaustive support for building and deploying efficient models on devices ranging from TPUs to edge devices at the time of writing. However, we encourage

Leveraging the power of C++ for efficient machine learning on embedded devices Adrian Stanciu adrian.stanciu.pub@gmail.com CppCon, 2023 1 / 50About me ◮ I am a software engineer from Romania ◮ I have data to make predictions 11 / 50Neural network (NN) 13 / 50Convolutional neural network (CNN) ◮ Efficient in image classification ◮ A convolutional layer can apply filters to detect: ◮ Edges ◮ Shapes ◮

kernels represents a channel. As you might notice, with such structured sparsity we can obtain efficient implementations which can drop unnecessary computation. In the case of this convolutional layer loss value. Han et al. in their seminal paper titled "Learning both Weights and Connections for Efficient Neural Networks8" proposed a three step approach for pruning. The three steps are: Train Connectivity preprint arXiv:1803.03635 (2018). 8 Han, Song, et al. "Learning both weights and connections for efficient neural network." Advances in neural information processing systems 28 (2015). 7 Dettmers, Tim,

quality you can achieve while retaining the same labeling costs i.e., training data-efficient (specifically, label efficient) models. We will describe the general principles of Self-Supervised learning which second limitation, training large models from scratch for every slightly different task is not efficient either. In many cases we might be limited by our training compute budget, so this approach is a which ensure that the model learns general representations of the inputs. Pre-training is data-efficient since we end up saving on the number of labels required to achieve the desired model quality on

DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI research@deepseek.com Abstract We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while