Chapter 3 - Learning Techniques “The more that you read, the more things you will know. The more that you learn, the more places you'll go.” ― Dr. Seuss Model quality is an important benchmark to evaluate evaluate the performance of a deep learning model. A language translation application that uses a low quality model would struggle with consumer adoption because it wouldn’t serve its intended purpose translation accuracy would garner better consumer support. In this chapter, our focus will be on the techniques that enable us to achieve our quality goals. High quality models have an additional benefit in

Advanced Learning Techniques “Tell me and I forget, teach me and I may remember, involve me and I learn.” – Benjamin Franklin This chapter is a continuation of Chapter 3, where we introduced learning techniques techniques. To recap, learning techniques can help us meet our model quality goals. Techniques like distillation and data augmentation improve the model quality, without increasing the footprint of the model this chapter by presenting self-supervised learning which has been instrumental in the success of natural language models like BERT. Self-Supervised learning helps models to quickly achieve impressive

Compression Techniques “I have made this longer than usual because I have not had time to make it shorter.” Blaise Pascal In the last chapter, we discussed a few ideas to improve the deep learning efficiency efficiency. Now, we will elaborate on one of those ideas, the compression techniques. Compression techniques aim to reduce the model footprint (size, latency, memory etc.). We can reduce the model footprint gentle introduction to the idea of compression. Details of quantization and its applications in deep learning follow right after. The quantization section delves into the implementation details using code samples

Advanced Compression Techniques “The problem is that we attempt to solve the simplest questions cleverly, thereby rendering them unusually complex. One should seek the simple solution.” — Anton Pavlovich Pavlovich Chekhov In this chapter, we will discuss two advanced compression techniques. By ‘advanced’ we mean that these techniques are slightly more involved than quantization (as discussed in the second them, with an eye towards conceptual understanding as well as practically using them in your deep learning models. We start with sparsity. If your goal was to optimize your brain for storage, you can often

Machine Learning Lecture 10: Neural Networks and Deep Learning Feng Li fli@sdu.edu.cn https://funglee.github.io School of Computer Science and Technology Shandong University Fall 2018 Deep Feedforward usually a highly non-linear function • Feedforward networks are of extreme importance to machine learning practioners • The conventional neural networks (CNN) used for object recognition from photos are units), and output layer 7 / 19 Neural Feedforward Networks (Contd.) • We approximate f ∗(x) by learning f(x) from the given training data • In the output layer, f(x) ≈ y for each training data, but the

Machine Learning Pytorch Tutorial TA : 曾元（Yuan Tseng） 2022.02.18 Outline ● Background: Prerequisites & What is Pytorch? ● Training & Testing Neural Networks in Pytorch ● Dataset & Dataloader ● Tensors link3 2. Deep Learning Basics ■ Prof. Lee’s 1st & 2nd lecture videos from last year ■ ref: link1, link2 Some knowledge of NumPy will also be useful! What is PyTorch? ● An machine learning framework in computing with more cores for arithmetic calculations ○ See What is a GPU and do you need one in deep learning? Tensors – Gradient Calculation >>> x = torch.tensor([[1., 0.], [-1., 1.]], requires_grad=True)

a variety of techniques in the last few chapters to improve efficiency and boost the quality of deep learning models. These techniques are just a small subset of the available techniques. It is often tedious Blessed with a large research community, the deep learning field is growing at a rapid pace. Over the past few years, we have seen newer architectures, techniques and training procedures pushing the performance performance benchmarks higher. Figure 7-1 shows some of the choices we face when working on a deep learning problem in the vision domain for instance. Some of these choices are boolean, others have discrete parameters

Introduction to Efficient Deep Learning Welcome to the book! This chapter is a preview of what to expect in the book. We start off by providing an overview of the state of deep learning, its applications, and motivation behind seeking efficiency in deep learning models. We will also introduce core areas of efficiency techniques (compression techniques, learning techniques, automation, efficient models & layers, even if you just read this chapter, you would be able to appreciate why we need efficiency in deep learning models today, how to think about it in terms of metrics that you care about, and finally the tools

Schmidt in ANALOG magazine (1991) So far, we have discussed generic techniques which are agnostic to the model architecture. These techniques can be applied in NLP, vision, speech or other domains. However or quality, we should consider employing suitable efficient architectures. The progress of deep learning is characterized by the phases of architectural breakthroughs to improve on previous results and (CNNs) were another important breakthrough that enabled learning spatial features in the input. Recurrent Neural Nets (RNNs) facilitated learning from the sequences and temporal data. These breakthroughs

A Day in the Life of a Data Scientist Conquer Machine Learning Lifecycle on Kubernetes Brian Redmond • Cloud Architect @ Microsoft (18 years) • Azure Global Black Belt Team • Live in Pittsburgh, PA • Intro to Kubeflow, Helm, Argo • Demos • Image classification with Inception v3 and transfer learning • Automate repeatable ML experiments with containers • Deploy ML components to Kubernetes with Kubeflow Parallel training instead of sequential: huge time saver for large trainings Kubeflow • Machine Learning Toolkit for Kubernetes • To make ML workflows on Kubernetes simple, portable, and scalable •