the pace of change catalyzed by the internet was unprecedented. Consider now that AI user and usage trending is ramping materially faster…and the machines can outpace us. The pace and scope of change technology evolution is indeed unprecedented, as supported by the data. This document is filled with user, usage and revenue charts that go up-and-to-the-right… often supported by spending charts that also go up-and-to-the • AI User + Usage + CapEx Growth = Unprecedented • AI Model Compute Costs High / Rising + Inference Costs Per Token Falling = Performance Converging + Developer Usage Rising • AI Usage + Cost + Loss

part of the systeml e Haichen and | will discuss more details at TVMConf. Oo oo QQ octoML 11 VM Memory Planning e Recently shipped a first version fn enain(0) -> Tensor[tk,)，f32] { ofdynamicmemory Planmng Let t2 3 memory planning,， storage Let s = alLLoc_storage(40，64，f32) ; Tet outl = attoc_tensor(s，(19,)，f32); coalescing, memory re-use for invoke_ l，t2)，(outl,))3 Out1l loops, and offloading dynamic } allocation to devices. QQ octoML VM Memory Abstractions Old New t1: Tensor t1: Tensor

the KV joint compression in MLA reduces the KV cache. Moreover, in order to reduce the activation memory during training, we also perform 7 low-rank compression for the queries, even if it cannot reduce relatively few activated parameters, and a portion of the operators are recomputed to save acti- vation memory, it can be trained without the necessity of tensor parallelism, thereby decreasing the communication demands on the training framework. It requires careful engineering optimization to manage the GPU memory and RAM pressure, and meanwhile maintain a fast training speed. For this goal, we implement the following

VTA ON INTEL FPGA©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 5 Software - CMA Contiguous Memory Allocation – Linux Kernel DEPLOY VTA ON INTEL FPGA https://pynq.readthedocs.io/en/v2.0/pynq_package/pynq 08.02_pr.tar.gz©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 6 Software - CMA Contiguous Memory Allocation – Linux Kernel Module DEPLOY VTA ON INTEL FPGA Setup Environment Variables Navigate INTERNATIONAL INDUSTRIES, INCORPORATED 7 Software - Driver Cyclone V & Arria V SoC HPS Physical Memory Map DEPLOY VTA ON INTEL FPGA©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 8 Hardware Configure

TensorCore Intrinsics 。Authored by @Hzfengsy 。 Intrinsics: tvm_load_matrix_sync tvm_mma_sync … “New Memory Scopes: wmma.matrix_a/b, accumulator 。Tensorization on warp level schedule Motivation load/store for higher bandwidth utilization 。Double buffer to hide memory load latency 。 storage align to reduce bank conflicts of shared memory 。 Virtual threads for data reuse (on going) Performance on V100

FABRIC IMG RD SCHEDULER WEIGHTS RD SCHEDULER PE Array PE PE PE PE DISPATCHER ... EXTERNAL MEMORY INSTR FETCHER DECODER REG MAP WB WR SCHEDULER CTRL SIGNALS MISC CALC AVG POOL MAX POOL aster/examples/deployment_modes/mp_classify.py) Streamlined multi-process pipeline using shared memory Usually need >4 Pre-Process cores running to keep up with FPGA ˃ TVM pipeline needed. CPU/FPGA

type 02 The model returns a response without any tool calls (e.g., a direct user message) Example usage: Python 1 Agents.run(agent, [UserMessage( )]) "What's the capital of the USA?" This concept of a while

Specialized Accelerators Tensor Compute Primitives Unified Buffer Acc FIFO Explicitly Managed Memory Subsystem TPUsTensorization Challenge Compute primitives scalar vector tensor Challenge: Build

>$1%Agent?@ABCDE • 1⃣ ‚•«Front-end¬5A+%ã:}GÏï÷øÑÒ> • *˜5StreamlitcFlaskcGradiocNode.jscNEXT.js> • 2⃣ ™•«Memory¬5š›•‘ AI *+Gœ•“”,Ì()@Z°ž–+Ê> • *˜5ZepcMemgcCognéecLetta> • 3⃣ Ÿ «Authentication¬5š›%ã¡¢£ C¤=¥+>