Chapter 4 - Efficient Architectures “Any sufficiently advanced technology is indistinguishable from magic.” — Arthur C. Clarke, “Hazards of Prophecy: The Failure of Imagination” (1962) “Any technology gain orders of magnitude in terms of footprint or quality, we should consider employing suitable efficient architectures. The progress of deep learning is characterized by the phases of architectural breakthroughs biggest architectural breakthroughs in the field of neural networks. It introduced the idea of stacking layers to learn complex relationships. Convolutional Neural Nets (CNNs) were another important breakthrough

techniques in the last few chapters to improve efficiency and boost the quality of deep learning models. These techniques are just a small subset of the available techniques. It is often tedious to decide momentum are geared towards model convergence. However, they all work in conjunction to produce better models faster. Let's say that we are optimizing the validation loss, , for a given dataset on a model target classes. import random import tensorflow as tf import numpy as np from tensorflow.keras import layers, losses, optimizers X = tf.random.uniform((20, 5)) Y = tf.squeeze( tf.one_hot(tf.random.uniform((20

Chapter 1 - Introduction to Efficient Deep Learning Welcome to the book! This chapter is a preview of what to expect in the book. We start off by providing an overview of the state of deep learning, its efficiency in deep learning models. We will also introduce core areas of efficiency techniques (compression techniques, learning techniques, automation, efficient models & layers, infrastructure). Our hope you just read this chapter, you would be able to appreciate why we need efficiency in deep learning models today, how to think about it in terms of metrics that you care about, and finally the tools at your

chapter, our focus will be on the techniques that enable us to achieve our quality goals. High quality models have an additional benefit in footprint constrained environments like mobile and edge devices where with samples and labels, distillation transfers knowledge from a large model or ensemble of models to smaller models. The obvious question at this point is: why are we talking about them in the same breadth recall or other performance metrics). We designate a new model training setup to be more sample efficient, if it achieves similar or better performance with fewer data samples when compared to the baseline

parameters. However, this approach has two drawbacks. First, it is hard to determine the parameters or layers that can be removed without significantly impacting the performance. It requires many trials and lead to degradation in quality. In our case, we are concerned about compressing the deep learning models. What do we really mean by compressing though? As mentioned in chapter 1, we can break down the metrics footprint. In the case of deep learning models, the model quality is often correlated with the number of layers, and the number of parameters (assuming that the models are well-tuned). If we naively reduce

with an eye towards conceptual understanding as well as practically using them in your deep learning models. We start with sparsity. If your goal was to optimize your brain for storage, you can often trim ensures the decoded value deviates less from the original value and can help improve the quality of our models. Did we get you excited yet? Let’s learn about these techniques together! Model Compression Using of removing (pruning) weights during the model training to achieve smaller models. Such models are called sparse or pruned models. The simplest form of pruning is to zero out a certain, say p, percentage

learning which has been instrumental in the success of natural language models like BERT. Self-Supervised learning helps models to quickly achieve impressive quality with a small number of labels. As you can achieve while retaining the same labeling costs i.e., training data-efficient (specifically, label efficient) models. We will describe the general principles of Self-Supervised learning which are tasks requires new models to be trained from scratch. For models that share the same domain, it is likely that the first few layers learn similar features. Hence training new models from scratch for these

它允许构建任意的神经网络图。 Sequential 顺序模型如下所示： from keras.models import Sequential model = Sequential() 可以简单地使用 .add() 来堆叠模型： KERAS: 基于 PYTHON 的深度学习库 2 from keras.layers import Dense model.add(Dense(units=64, activation='relu' 顺序模型是多个网络层的线性堆叠。 你可以通过将层的列表传递给 Sequential 的构造函数，来创建一个 Sequential 模型： from keras.models import Sequential from keras.layers import Dense, Activation model = Sequential([ Dense(32, input_shape=(784,)), 的字符级文本生成 … 等等。 3.1.5.1 基于多层感知器 (MLP) 的 softmax 多分类： import keras from keras.models import Sequential from keras.layers import Dense, Dropout, Activation from keras.optimizers import SGD # 生成虚拟数据

Keras. This tutorial walks through the installation of Keras, basics of deep learning, Keras models, Keras layers, Keras modules and finally conclude with some real-time applications. Audience This ................................................................................ 24 7. Keras ― Layers ................................................................................................ .......................................................................... 26 Basic Concept of Layers ................................................................................................

Conv3-64 Max-Pool Max-Pool FC-512 Output ConvNet Configuration Stacked layers Previous input x F(x) y=F(x) Stacked layers Previous input x F(x) y=F(x)+x x identity + 15 2.深度残差网络 ResNets使用了许多same卷积 Learning) 是把已学训练好的模型参数用作新训练模型的起 始参数。迁移学习是深度学习中非常重要和常用的⼀个策略。 4.卷积神经网络使用技巧 29 迁移学习步骤 1.使用预训练的模型 net = models.resnet18(pretrained=True) 2.冻结模型权重 for param in net.parameters(): #遍历每个模型参数 param.requires_grad DenseNet：Densely Connected Convolutional Networks (Gao Huang et al., 2017) • MobileNet ： MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications (Andrew G. Howard et al., 2017) •