Richard Hirsh; John McCallum; OpenAI Details on Page 138 0 Years 72 Years Electric Power Computer Memory AI Inference AI Monetization Threats = Rising Competition + Open-Source Momentum + China’s Rise Assistant – 6/18-2/25, per Bank of America Erica acts as both a personal concierge and mission control for our clients. Our data science team has made more than 50,000 updates to Erica’s performance relative to prior analytical techniques with the remainder relative to a random baseline or holdout control.’ We indicate 2020 as the start year for JP Morgan’s AI Modernization (2020 Letter to Shareholders:

cost. As we employ expert parallelism during training, we also devise supplementary mechanisms to control communication overheads and ensure load balance. By combining these two techniques, DeepSeek-V2 features the KV joint compression in MLA reduces the KV cache. Moreover, in order to reduce the activation memory during training, we also perform 7 low-rank compression for the queries, even if it cannot reduce relatively few activated parameters, and a portion of the operators are recomputed to save acti- vation memory, it can be trained without the necessity of tensor parallelism, thereby decreasing the communication

a code change, 
 or generating a report. Applications that integrate LLMs but don’t use them to control workflow execution—think simple chatbots, single-turn LLMs, or sentiment classifiers—are not agents proactively correct its actions if needed. In case 
 of failure, it can halt execution and transfer control back to the user. 02 It has access to various tools to interact with external systems—both to gather orchestrate a network of specialized agents seamlessly through tool calls. Instead of losing context or control, the manager intelligently delegates tasks to the right agent at the right time, effortlessly synthesizing

part of the systeml e Haichen and | will discuss more details at TVMConf. Oo oo QQ octoML 11 VM Memory Planning e Recently shipped a first version fn enain(0) -> Tensor[tk,)，f32] { ofdynamicmemory Planmng Let t2 3 memory planning,， storage Let s = alLLoc_storage(40，64，f32) ; Tet outl = attoc_tensor(s，(19,)，f32); coalescing, memory re-use for invoke_ l，t2)，(outl,))3 Out1l loops, and offloading dynamic } allocation to devices. QQ octoML VM Memory Abstractions Old New t1: Tensor t1: Tensor

Design with simplicity 55 Be specific about the output 56 Use Instructions over Constraints 56 Control the max token length 58 Use variables in prompts 58 Experiment with input formats and writing styles need to figure out the model configuration. Most LLMs come with various configuration options that control the LLM’s output. Effective prompt engineering requires setting these configurations optimally for higher, all tokens become equally likely to be the next predicted token. The Gemini temperature control can be understood in a similar way to the softmax function used in machine learning. A low temperature

VTA ON INTEL FPGA©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 5 Software - CMA Contiguous Memory Allocation – Linux Kernel DEPLOY VTA ON INTEL FPGA https://pynq.readthedocs.io/en/v2.0/pynq_package/pynq 08.02_pr.tar.gz©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 6 Software - CMA Contiguous Memory Allocation – Linux Kernel Module DEPLOY VTA ON INTEL FPGA Setup Environment Variables Navigate INTERNATIONAL INDUSTRIES, INCORPORATED 7 Software - Driver Cyclone V & Arria V SoC HPS Physical Memory Map DEPLOY VTA ON INTEL FPGA©2019 HARMAN INTERNATIONAL INDUSTRIES, INCORPORATED 8 Hardware Configure

TensorCore Intrinsics 。Authored by @Hzfengsy 。 Intrinsics: tvm_load_matrix_sync tvm_mma_sync … “New Memory Scopes: wmma.matrix_a/b, accumulator 。Tensorization on warp level schedule Motivation load/store for higher bandwidth utilization 。Double buffer to hide memory load latency 。 storage align to reduce bank conflicts of shared memory 。 Virtual threads for data reuse (on going) Performance on V100

FABRIC IMG RD SCHEDULER WEIGHTS RD SCHEDULER PE Array PE PE PE PE DISPATCHER ... EXTERNAL MEMORY INSTR FETCHER DECODER REG MAP WB WR SCHEDULER CTRL SIGNALS MISC CALC AVG POOL MAX POOL aster/examples/deployment_modes/mp_classify.py) Streamlined multi-process pipeline using shared memory Usually need >4 Pre-Process cores running to keep up with FPGA ˃ TVM pipeline needed. CPU/FPGA

dynamism ● Control flow (if, loop, etc) ● Dynamic shapes ○ Dynamic inputs: batch size, image size, sequence length, etc. ○ Output shape of some ops are data dependent: arange, nms, etc. ○ Control flow:

Specialized Accelerators Tensor Compute Primitives Unified Buffer Acc FIFO Explicitly Managed Memory Subsystem TPUsTensorization Challenge Compute primitives scalar vector tensor Challenge: Build