Chapter 7 - Automation "There's a lot of automation that can happen that isn't a replacement of humans but of mind-numbing behavior." - Stewart Butterfield, Founder (Slack) We have talked about a variety and resource allocation changes across multiple brackets in a Hyperband. Source: Hyperband In chapter 3, we trained a model to classify flowers in the oxford_flowers102 dataset. In the next section, with a twist! Project: Oxford Flower Classification With Hyperparameter Tuning Recall that in chapter 3, we trained a ResNet based model to classify oxford_flowers102 flowers dataset. We used two hyperparameters:

Chapter 1 - Introduction to Efficient Deep Learning Welcome to the book! This chapter is a preview of what to expect in the book. We start off by providing an overview of the state of deep learning, its automation, efficient models & layers, infrastructure). Our hope is that even if you just read this chapter, you would be able to appreciate why we need efficiency in deep learning models today, how to think of the book’s main points, but you will lose the finer details. We cover these in more detail in Chapter 2. (Figure 1-7: A mental model of Efficient Deep Learning, which comprises the core areas and relevant

Chapter 4 - Efficient Architectures “Any sufficiently advanced technology is indistinguishable from magic.” — Arthur C. Clarke, “Hazards of Prophecy: The Failure of Imagination” (1962) “Any technology giant counterparts. In the first chapter, we briefly introduced architectures like depthwise separable convolution, attention mechanism and the hashing trick. In this chapter, we will deepdive into their architectures popular tools for dimensionality reduction. We will explain these techniques in further detail in chapter 6. A Petting Zoo for Kids Let’s go back to our example of cute and dangerous animals, and represent

Chapter 3 - Learning Techniques “The more that you read, the more things you will know. The more that you learn, the more places you'll go.” ― Dr. Seuss Model quality is an important benchmark to evaluate quality model with a reasonable translation accuracy would garner better consumer support. In this chapter, our focus will be on the techniques that enable us to achieve our quality goals. High quality models where they provide the flexibility to trade off some quality for smaller footprints. In the first chapter, we briefly introduced learning techniques such as regularization, dropout, data augmentation, and

Chapter 2 - Compression Techniques “I have made this longer than usual because I have not had time to make it shorter.” Blaise Pascal In the last chapter, we discussed a few ideas to improve the deep doesn’t generalize well because the model designs are subjective to the specific problem. In this chapter, we introduce Quantization, a model compression technique that addresses both these issues. We’ll compressing the deep learning models. What do we really mean by compressing though? As mentioned in chapter 1, we can break down the metrics we care about into two categories: footprint metrics such as model

In this chapter, we will discuss two advanced compression techniques. By ‘advanced’ we mean that these techniques are slightly more involved than quantization (as discussed in the second chapter). But that network connections, remove extraneous nodes, etc. while retaining the model’s performance? In this chapter we introduce the intuition behind sparsity, different possible methods of picking the connections learning model to achieve storage and latency gains with a minimal performance tradeoff. Next, the chapter goes over weight sharing using clustering. Weight sharing, and in particular clustering is a generalization

Chapter 3. Rendering Text The pixels library can do slightly more than just putPixel: it also offers a minimal drawText proc, to put letters and words on the screen. Its declaration looks like this: text we want to render. It is of type string. We will look at strings in more depth in the next chapter, for now it is enough to know that string is a builtin type that is roughly a sequence of char, position (10, i*10) where i is in one of the numbers in the 1..3 range, for each loop iteration. 18 Chapter 4. Sequences We have said that a string is a sequence of characters. Nim also supports sequences

Chapter 6 - Advanced Learning Techniques “Tell me and I forget, teach me and I may remember, involve me and I learn.” – Benjamin Franklin This chapter is a continuation of Chapter 3, where we introduced can be traded off for a smaller footprint as desired. Continuing with the theme of chapter 3, we will start this chapter by presenting self-supervised learning which has been instrumental in the success models to quickly achieve impressive quality with a small number of labels. As we described in chapter 3’s ‘Learning Techniques and Efficiency’ section, labeling of training data is an expensive undertaking

above outline do not belong in the RTL, but should be included in the packages, or in the FCL. 139 Chapter 1 Reference for unit ’BaseUnix’ 1.1 Used units Table 1.1: Used units by unit ’BaseUnix’ Name Flag for various *at calls to indicate current working directory AT_NO_AUTOMOUNT = $800 140 CHAPTER 1. REFERENCE FOR UNIT ’BASEUNIX’ AT_RECURSIVE = $8000 AT_REMOVEDIR = $200 Unlink at: specify rmdir System error: Cannot assign requested address ESysEADV = 68 System error: Advertise error 141 CHAPTER 1. REFERENCE FOR UNIT ’BASEUNIX’ ESysEAFNOSUPPORT = 97 System error: Address family not supported

are other units that have counterparts in Delphi, but most of them are unique to Free Pascal. 88 Chapter 1 Reference for unit ’ascii85’ 1.1 Used units Table 1.1: Used units by unit ’ascii85’ Name Page ascOneEncodedChar,ascTwoEncodedChars, ascThreeEncodedChars,ascFourEncodedChars, ascNoEncodedChar,ascPrefix) 89 CHAPTER 1. REFERENCE FOR UNIT ’ASCII85’ Table 1.2: Enumeration values for type TASCII85State Value Explanation instance, and sets aStream as the source stream. See also: TASCII85DecoderStream.Destroy (91) 90 CHAPTER 1. REFERENCE FOR UNIT ’ASCII85’ 1.4.5 TASCII85DecoderStream.Decode Synopsis: Decode source byte