# Chapter 4 - Efficient Architectures “Any sufficiently advanced technology is indistinguishable from magic.” — Arthur C. Clarke, “Hazards of Prophecy: The Failure of Imagination” (1962) “Any technology in terms of footprint or quality, we should consider employing suitable efficient architectures. The progress of deep learning is characterized by the phases of architectural breakthroughs to improve (CNNs) were another important breakthrough that enabled learning spatial features in the input. Recurrent Neural Nets (RNNs) facilitated learning from the sequences and temporal data. These breakthroughs

# Chapter 3 - Learning Techniques "The more that you read, the more things you will know. The more that you learn, the more places you'll go." — Dr. Seuss Model quality is an important benchmark benchmark to evaluate the performance of a deep learning model. A language translation application that uses a low quality model would struggle with consumer adoption because it wouldn't serve its flexibility to trade off some quality for smaller footprints. In the first chapter, we briefly introduced learning techniques such as regularization, dropout, data augmentation, and distillation to improve quality

variety of techniques in the last few chapters to improve efficiency and boost the quality of deep learning models. These techniques are just a small subset of the available techniques. It is often tedious these four options to make an informed decision. Blessed with a large research community, the deep learning field is growing at a rapid pace. Over the past few years, we have seen newer architectures, techniques performance benchmarks higher. Figure 7-1 shows some of the choices we face when working on a deep learning problem in the vision domain for instance. Some of these choices are boolean, others have discrete

Introduction to Efficient Deep Learning Welcome to the book! This chapter is a preview of what to expect in the book. We start off by providing an overview of the state of deep learning, its applications behind seeking efficiency in deep learning models. We will also introduce core areas of efficiency techniques (compression techniques, learning techniques, automation, efficient models & layers, infrastructure) even if you just read this chapter, you would be able to appreciate why we need efficiency in deep learning models today, how to think about it in terms of metrics that you care about, and finally the tools

make it shorter.” Blaise Pascal In the last chapter, we discussed a few ideas to improve the deep learning efficiency. Now, we will elaborate on one of those ideas, the compression techniques. Compression gentle introduction to the idea of compression. Details of quantization and its applications in deep learning follow right after. The quantization section delves into the implementation details using code compression might lead to degradation in quality. In our case, we are concerned about compressing the deep learning models. What do we really mean by compressing though? As mentioned in chapter 1, we can break

Advanced Learning Techniques “Tell me and I forget, teach me and I may remember, involve me and I learn.” – Benjamin Franklin This chapter is a continuation of Chapter 3, where we introduced learning techniques techniques. To recap, learning techniques can help us meet our model quality goals. Techniques like distillation and data augmentation improve the model quality, without increasing the footprint of the model this chapter by presenting self-supervised learning which has been instrumental in the success of natural language models like BERT. Self-Supervised learning helps models to quickly achieve impressive

them, with an eye towards conceptual understanding as well as practically using them in your deep learning models. We start with sparsity. If your goal was to optimize your brain for storage, you can different possible methods of picking the connections and nodes to prune, and how to prune a given deep learning model to achieve storage and latency gains with a minimal performance tradeoff. Next, the chapter effectiveness. Later on in this chapter, we have a project that relies on it for sparsifying a deep learning model. The authors of the Optimal Brain Damage (OBD) paper approximate the saliency score using

a ■ Web：爬虫 • 2016的目标：Web爬虫 + 深度学习 + 自然语言处理 = ? ## 今年最激动人心的事件？ ## Mastering the Game of Go with Deep Neural Networks and Tree Search David Silver $ ^{1*} $ , Aja Huang $ ^{1*} $ , Chris J. Maddison policy networks to select moves. These deep neural networks are trained by a novel combination of supervised learning from human expert games, and reinforcement learning from games of self-play. Without any 分类 - 半监督学习 - 介于监督学习和无监督学习之间，算法: Graph Inference 或者 Laplacian SVM - 强化学习- 通过观察来学习做成如何的动作, 算法：Q-Learning以及时间差学习 机器学习－方法及流程 • 输入特征选择 - 基于什么进行预测 • 目标－预测什么 • 预测功能 - 回归、聚类、降维... $ X_{n} \rightarrow F(x_{n})

+23 ## Leveraging the Power of C++ for Efficient Machine Learning on Embedded Devices ADRIAN STANCIU # Leveraging the power of C++ for efficient machine learning on embedded devices Adrian Stanciu adrian Motivation ▶ Image classification ▶ Hand gesture recognition Summary ▶ Q&A Motivation ## Machine Learning (ML) ▶ Subfield of Artificial Intelligence (AI) Enables computers to learn from data and then consumption ▶ May have real-time performance constraints ## Machine learning on embedded devices Alternative to cloud-based machine learning Advantages: ▶ Real-time processing ▶ Low latency ▶ Reduced bandwidth

Machine Learning Lecture 10: Neural Networks and Deep Learning Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 2018 ## Deep Feedforward usually a highly non-linear function • Feedforward networks are of extreme importance to machine learning practitioners - The conventional neural networks (CNN) used for object recognition from photos h^{[2]} $ Output y ### Neural Feedforward Networks (Contd.) • We approximate $ f^{*}(x) $ by learning $ f(x) $ from the given training data • In the output layer, $ f(x) \approx y $ for each